Oct 30 01:14:31.169726 ip-10-0-138-223 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 01:14:31.169738 ip-10-0-138-223 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 01:14:31.169745 ip-10-0-138-223 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 01:14:31.169996 ip-10-0-138-223 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 01:14:41.418861 ip-10-0-138-223 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 01:14:41.418877 ip-10-0-138-223 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 463af845f3b94975a32aae96d317f323 -- Oct 30 01:16:39.312275 ip-10-0-138-223 systemd[1]: Starting Kubernetes Kubelet... Oct 30 01:16:39.727579 ip-10-0-138-223 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 01:16:39.727579 ip-10-0-138-223 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 01:16:39.727579 ip-10-0-138-223 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 01:16:39.727579 ip-10-0-138-223 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 01:16:39.732204 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732109 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:16:39.732204 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732203 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:16:39.732204 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732206 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:16:39.732318 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732211 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:16:39.732318 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732214 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:16:39.732318 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732216 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:16:39.732318 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732220 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:16:39.732318 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732223 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:16:39.732318 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732226 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:16:39.732318 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732228 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:16:39.732318 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732236 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:16:39.732318 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732239 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:16:39.732318 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732241 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:16:39.732318 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732244 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:16:39.732318 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732247 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:16:39.732318 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732249 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:16:39.732318 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732252 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:16:39.732318 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732255 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:16:39.732318 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732257 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:16:39.732318 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732260 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:16:39.732318 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732263 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:16:39.732318 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732265 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:16:39.732790 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732268 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:16:39.732790 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732272 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:16:39.732790 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732275 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:16:39.732790 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732277 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:16:39.732790 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732280 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:16:39.732790 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732283 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:16:39.732790 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732286 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:16:39.732790 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732288 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:16:39.732790 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732291 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:16:39.732790 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732294 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:16:39.732790 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732296 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:16:39.732790 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732299 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:16:39.732790 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732302 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:16:39.732790 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732304 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:16:39.732790 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732308 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:16:39.732790 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732311 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:16:39.732790 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732313 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:16:39.732790 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732316 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:16:39.732790 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732318 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:16:39.733247 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732321 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:16:39.733247 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732324 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:16:39.733247 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732328 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:16:39.733247 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732331 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:16:39.733247 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732334 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:16:39.733247 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732336 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:16:39.733247 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732339 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:16:39.733247 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732341 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:16:39.733247 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732344 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:16:39.733247 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732346 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:16:39.733247 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732349 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:16:39.733247 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732351 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:16:39.733247 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732354 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:16:39.733247 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732356 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:16:39.733247 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732359 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:16:39.733247 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732361 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:16:39.733247 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732364 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:16:39.733247 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732367 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:16:39.733247 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732369 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:16:39.733247 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732372 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:16:39.733743 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732374 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:16:39.733743 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732377 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:16:39.733743 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732381 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:16:39.733743 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732386 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:16:39.733743 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732389 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:16:39.733743 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732392 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:16:39.733743 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732396 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:16:39.733743 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732399 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:16:39.733743 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732402 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:16:39.733743 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732405 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:16:39.733743 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732408 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:16:39.733743 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732410 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:16:39.733743 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732413 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:16:39.733743 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732416 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:16:39.733743 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732419 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:16:39.733743 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732422 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:16:39.733743 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732425 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:16:39.733743 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732428 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:16:39.733743 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732430 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:16:39.734218 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732433 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:16:39.734218 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732435 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:16:39.734218 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732438 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:16:39.734218 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732441 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:16:39.734218 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732443 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:16:39.734218 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732446 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:16:39.734218 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732448 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:16:39.734218 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732451 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:16:39.734218 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732453 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:16:39.734218 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732456 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:16:39.734218 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732458 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:16:39.734218 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732461 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:16:39.734218 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732464 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:16:39.734218 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732467 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:16:39.734218 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732469 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:16:39.734218 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732472 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:16:39.734218 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732474 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:16:39.734218 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732477 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:16:39.734218 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732482 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:16:39.734654 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732487 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:16:39.734654 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732491 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:16:39.734654 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732494 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:16:39.734654 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732497 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:16:39.734654 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732499 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:16:39.734654 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732502 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:16:39.734654 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.732505 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:16:39.734654 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733340 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:16:39.734654 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733347 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:16:39.734654 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733351 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:16:39.734654 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733354 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:16:39.734654 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733357 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:16:39.734654 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733360 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:16:39.734654 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733363 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:16:39.734654 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733365 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:16:39.734654 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733369 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:16:39.734654 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733372 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:16:39.734654 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733375 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:16:39.734654 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733377 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:16:39.734654 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733380 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:16:39.735158 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733383 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:16:39.735158 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733386 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:16:39.735158 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733389 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:16:39.735158 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733391 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:16:39.735158 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733394 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:16:39.735158 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733396 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:16:39.735158 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733399 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:16:39.735158 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733402 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:16:39.735158 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733404 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:16:39.735158 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733407 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:16:39.735158 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733409 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:16:39.735158 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733412 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:16:39.735158 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733415 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:16:39.735158 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733418 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:16:39.735158 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733421 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:16:39.735158 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733424 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:16:39.735158 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733426 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:16:39.735158 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733429 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:16:39.735158 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733431 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:16:39.735158 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733434 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:16:39.735631 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733436 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:16:39.735631 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733439 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:16:39.735631 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733441 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:16:39.735631 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733445 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:16:39.735631 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733447 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:16:39.735631 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733450 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:16:39.735631 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733452 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:16:39.735631 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733455 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:16:39.735631 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733457 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:16:39.735631 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733460 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:16:39.735631 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733463 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:16:39.735631 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733465 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:16:39.735631 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733468 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:16:39.735631 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733471 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:16:39.735631 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733473 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:16:39.735631 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733476 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:16:39.735631 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733479 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:16:39.735631 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733482 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:16:39.735631 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733484 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:16:39.736101 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733487 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:16:39.736101 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733489 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:16:39.736101 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733492 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:16:39.736101 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733495 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:16:39.736101 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733497 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:16:39.736101 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733500 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:16:39.736101 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733502 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:16:39.736101 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733505 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:16:39.736101 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733508 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:16:39.736101 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733511 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:16:39.736101 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733513 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:16:39.736101 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733516 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:16:39.736101 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733518 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:16:39.736101 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733521 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:16:39.736101 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733523 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:16:39.736101 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733526 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:16:39.736101 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733530 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:16:39.736101 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733534 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:16:39.736101 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733537 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:16:39.736558 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733540 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:16:39.736558 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733543 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:16:39.736558 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733545 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:16:39.736558 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733548 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:16:39.736558 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733550 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:16:39.736558 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733553 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:16:39.736558 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733555 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:16:39.736558 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733558 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:16:39.736558 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733560 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:16:39.736558 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733563 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:16:39.736558 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733565 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:16:39.736558 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733568 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:16:39.736558 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733571 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:16:39.736558 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733573 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:16:39.736558 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733576 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:16:39.736558 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733581 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:16:39.736558 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733585 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:16:39.736558 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733588 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:16:39.736558 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733591 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:16:39.737030 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733593 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:16:39.737030 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733595 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:16:39.737030 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733598 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:16:39.737030 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733601 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:16:39.737030 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733603 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:16:39.737030 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733606 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:16:39.737030 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733609 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:16:39.737030 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733612 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:16:39.737030 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733614 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:16:39.737030 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733617 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:16:39.737030 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733619 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:16:39.737030 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733622 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:16:39.737030 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733624 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:16:39.737030 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733627 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:16:39.737030 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733630 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:16:39.737030 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.733634 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:16:39.737030 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734145 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:16:39.737030 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734151 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:16:39.737030 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734154 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:16:39.737030 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734157 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:16:39.737522 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734160 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:16:39.737522 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734162 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:16:39.737522 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734165 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:16:39.737522 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734168 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:16:39.737522 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734171 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:16:39.737522 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734175 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:16:39.737522 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734178 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:16:39.737522 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734180 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:16:39.737522 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734183 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:16:39.737522 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734186 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:16:39.737522 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734189 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:16:39.737522 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734192 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:16:39.737522 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734194 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:16:39.737522 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734198 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:16:39.737522 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734203 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:16:39.737522 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734206 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:16:39.737522 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734209 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:16:39.737522 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734212 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:16:39.737522 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734214 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:16:39.738180 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734217 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:16:39.738180 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734220 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:16:39.738180 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734222 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:16:39.738180 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734225 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:16:39.738180 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734228 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:16:39.738180 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734238 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:16:39.738180 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734241 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:16:39.738180 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734244 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:16:39.738180 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734247 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:16:39.738180 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734249 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:16:39.738180 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734252 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:16:39.738180 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734255 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:16:39.738180 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734257 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:16:39.738180 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734260 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:16:39.738180 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734263 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:16:39.738180 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734265 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:16:39.738180 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734268 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:16:39.738180 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734271 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:16:39.738180 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734274 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:16:39.738180 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734277 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:16:39.738805 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734279 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:16:39.738805 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734282 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:16:39.738805 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734285 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:16:39.738805 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734288 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:16:39.738805 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734291 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:16:39.738805 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734293 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:16:39.738805 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734296 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:16:39.738805 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734299 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:16:39.738805 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734301 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:16:39.738805 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734304 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:16:39.738805 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734307 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:16:39.738805 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734309 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:16:39.738805 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734312 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:16:39.738805 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734315 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:16:39.738805 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734318 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:16:39.738805 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734321 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:16:39.738805 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734323 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:16:39.738805 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734326 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:16:39.738805 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734329 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:16:39.738805 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734332 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:16:39.739417 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734335 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:16:39.739417 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734338 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:16:39.739417 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734341 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:16:39.739417 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734343 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:16:39.739417 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734346 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:16:39.739417 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734348 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:16:39.739417 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734354 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:16:39.739417 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734357 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:16:39.739417 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734359 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:16:39.739417 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734362 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:16:39.739417 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734364 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:16:39.739417 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734367 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:16:39.739417 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734370 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:16:39.739417 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734372 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:16:39.739417 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734375 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:16:39.739417 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734377 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:16:39.739417 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734380 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:16:39.739417 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734382 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:16:39.739417 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734385 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:16:39.739417 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734387 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:16:39.739893 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734390 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:16:39.739893 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734392 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:16:39.739893 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734395 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:16:39.739893 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734398 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:16:39.739893 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734400 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:16:39.739893 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734403 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:16:39.739893 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734405 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:16:39.739893 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734408 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:16:39.739893 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734410 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:16:39.739893 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734413 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:16:39.739893 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734416 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:16:39.739893 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734419 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:16:39.739893 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734422 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:16:39.739893 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734425 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:16:39.739893 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734427 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:16:39.739893 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734430 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:16:39.739893 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734433 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:16:39.739893 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734436 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:16:39.739893 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734442 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:16:39.740358 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734445 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:16:39.740358 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734448 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:16:39.740358 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734451 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:16:39.740358 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.734454 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:16:39.740358 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735437 2583 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 01:16:39.740358 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735454 2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 01:16:39.740358 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735463 2583 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 01:16:39.740358 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735468 2583 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 01:16:39.740358 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735473 2583 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 01:16:39.740358 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735476 2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 01:16:39.740358 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735481 2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 01:16:39.740358 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735485 2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 01:16:39.740358 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735489 2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 01:16:39.740358 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735492 2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 01:16:39.740358 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735495 2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 01:16:39.740358 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735499 2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 01:16:39.740358 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735503 2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 01:16:39.740358 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735506 2583 flags.go:64] FLAG: --cgroup-root="" Oct 30 01:16:39.740358 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735508 2583 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 01:16:39.740358 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735512 2583 flags.go:64] FLAG: --client-ca-file="" Oct 30 01:16:39.740358 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735515 2583 flags.go:64] FLAG: --cloud-provider="external" Oct 30 01:16:39.740358 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735518 2583 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 01:16:39.740358 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735521 2583 flags.go:64] FLAG: --cluster-domain="" Oct 30 01:16:39.740914 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735524 2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 01:16:39.740914 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735528 2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 01:16:39.740914 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735533 2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 01:16:39.740914 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735537 2583 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 01:16:39.740914 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735542 2583 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 01:16:39.740914 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735546 2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 01:16:39.740914 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735549 2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 01:16:39.740914 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735553 2583 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 01:16:39.740914 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735557 2583 flags.go:64] FLAG: --contention-profiling="false" Oct 30 01:16:39.740914 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735560 2583 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 01:16:39.740914 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735563 2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 01:16:39.740914 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735566 2583 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 01:16:39.740914 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735569 2583 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 01:16:39.740914 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735574 2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 01:16:39.740914 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735577 2583 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 01:16:39.740914 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735580 2583 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 01:16:39.740914 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735583 2583 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 01:16:39.740914 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735586 2583 flags.go:64] FLAG: --enable-server="true" Oct 30 01:16:39.740914 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735589 2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 01:16:39.740914 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735594 2583 flags.go:64] FLAG: --event-burst="100" Oct 30 01:16:39.740914 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735598 2583 flags.go:64] FLAG: --event-qps="50" Oct 30 01:16:39.740914 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735601 2583 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 01:16:39.740914 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735604 2583 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 01:16:39.740914 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735607 2583 flags.go:64] FLAG: --eviction-hard="" Oct 30 01:16:39.742141 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735612 2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 01:16:39.742141 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735615 2583 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 01:16:39.742141 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735618 2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 01:16:39.742141 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735621 2583 flags.go:64] FLAG: --eviction-soft="" Oct 30 01:16:39.742141 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735624 2583 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 01:16:39.742141 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735627 2583 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 01:16:39.742141 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735630 2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 01:16:39.742141 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735633 2583 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 01:16:39.742141 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735636 2583 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 01:16:39.742141 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735639 2583 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 01:16:39.742141 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735644 2583 flags.go:64] FLAG: --feature-gates="" Oct 30 01:16:39.742141 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735649 2583 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 01:16:39.742141 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735652 2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 01:16:39.742141 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735655 2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 01:16:39.742141 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735658 2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 01:16:39.742141 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735661 2583 flags.go:64] FLAG: --healthz-port="10248" Oct 30 01:16:39.742141 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735665 2583 flags.go:64] FLAG: --help="false" Oct 30 01:16:39.742141 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735668 2583 flags.go:64] FLAG: --hostname-override="ip-10-0-138-223.ec2.internal" Oct 30 01:16:39.742141 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735671 2583 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 01:16:39.742141 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735674 2583 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 01:16:39.742141 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735677 2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 01:16:39.742141 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735681 2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 01:16:39.742141 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735684 2583 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 01:16:39.742689 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735687 2583 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 01:16:39.742689 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735690 2583 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 01:16:39.742689 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735694 2583 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 01:16:39.742689 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735696 2583 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 01:16:39.742689 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735699 2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 01:16:39.742689 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735703 2583 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 01:16:39.742689 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735706 2583 flags.go:64] FLAG: --kube-reserved="" Oct 30 01:16:39.742689 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735708 2583 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 01:16:39.742689 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735711 2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 01:16:39.742689 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735714 2583 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 01:16:39.742689 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735717 2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 01:16:39.742689 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735721 2583 flags.go:64] FLAG: --lock-file="" Oct 30 01:16:39.742689 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735724 2583 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 01:16:39.742689 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735727 2583 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 01:16:39.742689 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735730 2583 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 01:16:39.742689 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735735 2583 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 01:16:39.742689 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735738 2583 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 01:16:39.742689 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735741 2583 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 01:16:39.742689 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735744 2583 flags.go:64] FLAG: --logging-format="text" Oct 30 01:16:39.742689 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735748 2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 01:16:39.742689 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735751 2583 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 01:16:39.742689 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735755 2583 flags.go:64] FLAG: --manifest-url="" Oct 30 01:16:39.742689 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735758 2583 flags.go:64] FLAG: --manifest-url-header="" Oct 30 01:16:39.742689 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735762 2583 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 01:16:39.742689 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735766 2583 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 01:16:39.743282 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735770 2583 flags.go:64] FLAG: --max-pods="110" Oct 30 01:16:39.743282 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735773 2583 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 01:16:39.743282 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735776 2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 01:16:39.743282 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735779 2583 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 01:16:39.743282 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735782 2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 01:16:39.743282 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735785 2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 01:16:39.743282 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735788 2583 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 01:16:39.743282 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735791 2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 01:16:39.743282 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735799 2583 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 01:16:39.743282 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735802 2583 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 01:16:39.743282 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735805 2583 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 01:16:39.743282 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735808 2583 flags.go:64] FLAG: --pod-cidr="" Oct 30 01:16:39.743282 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735811 2583 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 01:16:39.743282 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735814 2583 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 01:16:39.743282 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735817 2583 flags.go:64] FLAG: --pods-per-core="0" Oct 30 01:16:39.743282 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735820 2583 flags.go:64] FLAG: --port="10250" Oct 30 01:16:39.743282 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735824 2583 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 01:16:39.743282 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735827 2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f0e1a6b194e4cd23" Oct 30 01:16:39.743282 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735830 2583 flags.go:64] FLAG: --qos-reserved="" Oct 30 01:16:39.743282 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735833 2583 flags.go:64] FLAG: --read-only-port="10255" Oct 30 01:16:39.743282 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735836 2583 flags.go:64] FLAG: --register-node="true" Oct 30 01:16:39.743282 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735839 2583 flags.go:64] FLAG: --register-with-taints="" Oct 30 01:16:39.743282 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735842 2583 flags.go:64] FLAG: --registry-burst="10" Oct 30 01:16:39.743282 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735845 2583 flags.go:64] FLAG: --registry-qps="5" Oct 30 01:16:39.743282 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735848 2583 flags.go:64] FLAG: --reserved-cpus="" Oct 30 01:16:39.743282 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735851 2583 flags.go:64] FLAG: --reserved-memory="" Oct 30 01:16:39.743899 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735855 2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 01:16:39.743899 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735859 2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 01:16:39.743899 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735862 2583 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 01:16:39.743899 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735865 2583 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 01:16:39.743899 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735868 2583 flags.go:64] FLAG: --runonce="false" Oct 30 01:16:39.743899 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735871 2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 01:16:39.743899 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735875 2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 01:16:39.743899 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735878 2583 flags.go:64] FLAG: --seccomp-default="false" Oct 30 01:16:39.743899 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735881 2583 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 01:16:39.743899 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735884 2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 01:16:39.743899 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735887 2583 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 01:16:39.743899 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735890 2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 01:16:39.743899 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735893 2583 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 01:16:39.743899 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735896 2583 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 01:16:39.743899 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735899 2583 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 01:16:39.743899 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735902 2583 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 01:16:39.743899 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735905 2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 01:16:39.743899 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735908 2583 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 01:16:39.743899 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735911 2583 flags.go:64] FLAG: --system-cgroups="" Oct 30 01:16:39.743899 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735914 2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 01:16:39.743899 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735919 2583 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 01:16:39.743899 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735922 2583 flags.go:64] FLAG: --tls-cert-file="" Oct 30 01:16:39.743899 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735925 2583 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 01:16:39.743899 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735929 2583 flags.go:64] FLAG: --tls-min-version="" Oct 30 01:16:39.743899 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735931 2583 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 01:16:39.744548 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735947 2583 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 01:16:39.744548 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735950 2583 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 01:16:39.744548 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735953 2583 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 01:16:39.744548 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735956 2583 flags.go:64] FLAG: --v="2" Oct 30 01:16:39.744548 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735965 2583 flags.go:64] FLAG: --version="false" Oct 30 01:16:39.744548 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735970 2583 flags.go:64] FLAG: --vmodule="" Oct 30 01:16:39.744548 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735974 2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 01:16:39.744548 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.735977 2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 01:16:39.744548 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736138 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:16:39.744548 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736143 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:16:39.744548 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736146 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:16:39.744548 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736149 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:16:39.744548 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736152 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:16:39.744548 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736156 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:16:39.744548 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736159 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:16:39.744548 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736161 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:16:39.744548 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736164 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:16:39.744548 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736167 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:16:39.744548 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736169 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:16:39.744548 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736172 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:16:39.744548 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736175 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:16:39.745092 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736177 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:16:39.745092 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736180 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:16:39.745092 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736183 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:16:39.745092 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736186 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:16:39.745092 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736188 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:16:39.745092 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736191 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:16:39.745092 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736194 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:16:39.745092 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736197 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:16:39.745092 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736199 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:16:39.745092 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736202 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:16:39.745092 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736205 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:16:39.745092 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736207 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:16:39.745092 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736210 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:16:39.745092 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736213 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:16:39.745092 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736216 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:16:39.745092 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736218 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:16:39.745092 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736221 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:16:39.745092 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736224 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:16:39.745092 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736226 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:16:39.745092 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736229 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:16:39.745555 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736231 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:16:39.745555 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736234 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:16:39.745555 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736237 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:16:39.745555 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736240 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:16:39.745555 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736244 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:16:39.745555 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736247 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:16:39.745555 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736249 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:16:39.745555 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736252 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:16:39.745555 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736254 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:16:39.745555 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736257 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:16:39.745555 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736260 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:16:39.745555 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736262 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:16:39.745555 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736265 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:16:39.745555 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736268 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:16:39.745555 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736270 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:16:39.745555 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736273 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:16:39.745555 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736275 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:16:39.745555 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736278 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:16:39.745555 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736281 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:16:39.746041 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736283 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:16:39.746041 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736286 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:16:39.746041 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736290 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:16:39.746041 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736292 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:16:39.746041 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736295 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:16:39.746041 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736298 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:16:39.746041 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736300 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:16:39.746041 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736303 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:16:39.746041 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736306 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:16:39.746041 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736308 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:16:39.746041 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736311 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:16:39.746041 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736313 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:16:39.746041 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736316 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:16:39.746041 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736318 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:16:39.746041 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736321 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:16:39.746041 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736323 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:16:39.746041 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736326 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:16:39.746041 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736331 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:16:39.746041 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736334 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:16:39.746041 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736337 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:16:39.746514 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736339 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:16:39.746514 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736342 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:16:39.746514 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736344 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:16:39.746514 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736347 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:16:39.746514 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736350 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:16:39.746514 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736353 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:16:39.746514 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736355 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:16:39.746514 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736358 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:16:39.746514 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736361 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:16:39.746514 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736363 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:16:39.746514 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736366 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:16:39.746514 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736369 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:16:39.746514 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736372 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:16:39.746514 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736376 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:16:39.746514 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736382 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:16:39.746514 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736386 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:16:39.746514 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736389 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:16:39.746514 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736392 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:16:39.746514 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736396 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:16:39.746978 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736399 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:16:39.746978 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736403 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:16:39.746978 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736406 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:16:39.746978 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736408 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:16:39.746978 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736411 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:16:39.746978 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736414 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:16:39.746978 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736416 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:16:39.746978 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736419 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:16:39.746978 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736422 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:16:39.746978 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736424 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:16:39.746978 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736429 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:16:39.746978 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736432 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:16:39.746978 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736435 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:16:39.746978 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736437 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:16:39.746978 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736440 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:16:39.746978 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.736465 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 01:16:39.747353 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736501 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:16:39.747353 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736506 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:16:39.747353 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736509 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:16:39.747353 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736511 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:16:39.747353 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736515 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:16:39.747353 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736518 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:16:39.747353 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736521 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:16:39.747353 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736523 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:16:39.747353 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736526 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:16:39.747353 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736528 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:16:39.747353 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736531 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:16:39.747353 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736535 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:16:39.747353 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736537 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:16:39.747353 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736540 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:16:39.747353 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736542 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:16:39.747353 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736545 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:16:39.747353 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736548 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:16:39.747353 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736550 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:16:39.747353 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736553 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:16:39.747353 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736556 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:16:39.747819 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736559 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:16:39.747819 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736561 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:16:39.747819 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736564 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:16:39.747819 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736567 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:16:39.747819 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736570 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:16:39.747819 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736572 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:16:39.747819 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736575 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:16:39.747819 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736579 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:16:39.747819 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736582 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:16:39.747819 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736585 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:16:39.747819 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736587 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:16:39.747819 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736590 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:16:39.747819 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736593 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:16:39.747819 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736596 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:16:39.747819 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736598 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:16:39.747819 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736601 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:16:39.747819 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736604 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:16:39.747819 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736606 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:16:39.747819 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736609 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:16:39.748288 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736612 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:16:39.748288 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736614 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:16:39.748288 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736617 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:16:39.748288 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736619 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:16:39.748288 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736623 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:16:39.748288 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736625 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:16:39.748288 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736628 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:16:39.748288 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736631 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:16:39.748288 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736633 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:16:39.748288 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736636 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:16:39.748288 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736638 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:16:39.748288 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736641 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:16:39.748288 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736644 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:16:39.748288 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736646 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:16:39.748288 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736649 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:16:39.748288 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736651 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:16:39.748288 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736654 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:16:39.748288 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736657 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:16:39.748288 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736660 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:16:39.748288 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736662 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:16:39.748763 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736666 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:16:39.748763 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736668 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:16:39.748763 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736671 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:16:39.748763 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736674 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:16:39.748763 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736677 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:16:39.748763 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736679 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:16:39.748763 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736681 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:16:39.748763 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736684 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:16:39.748763 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736687 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:16:39.748763 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736690 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:16:39.748763 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736692 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:16:39.748763 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736695 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:16:39.748763 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736697 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:16:39.748763 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736700 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:16:39.748763 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736703 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:16:39.748763 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736705 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:16:39.748763 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736711 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:16:39.748763 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736715 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:16:39.748763 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736718 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:16:39.749333 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736721 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:16:39.749333 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736725 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:16:39.749333 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736728 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:16:39.749333 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736731 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:16:39.749333 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736734 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:16:39.749333 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736737 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:16:39.749333 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736739 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:16:39.749333 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736742 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:16:39.749333 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736745 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:16:39.749333 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736747 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:16:39.749333 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736750 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:16:39.749333 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736753 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:16:39.749333 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736755 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:16:39.749333 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736759 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:16:39.749333 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736762 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:16:39.749333 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736764 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:16:39.749333 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736767 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:16:39.749333 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736770 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:16:39.749333 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736772 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:16:39.749779 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736776 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:16:39.749779 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736778 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:16:39.749779 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736781 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:16:39.749779 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736784 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:16:39.749779 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736786 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:16:39.749779 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736789 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:16:39.749779 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736791 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:16:39.749779 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736794 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:16:39.749779 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.736796 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:16:39.749779 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.745296 2583 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 01:16:39.749779 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.745317 2583 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 01:16:39.749779 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.745335 2583 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 01:16:39.749779 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.745342 2583 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 01:16:39.749779 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745402 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:16:39.749779 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745408 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:16:39.749779 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745411 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:16:39.749779 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745414 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:16:39.749779 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745417 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:16:39.749779 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745419 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:16:39.749779 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745422 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:16:39.750315 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745426 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:16:39.750315 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745428 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:16:39.750315 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745431 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:16:39.750315 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745434 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:16:39.750315 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745436 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:16:39.750315 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745439 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:16:39.750315 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745443 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:16:39.750315 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745448 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:16:39.750315 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745453 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:16:39.750315 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745457 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:16:39.750315 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745460 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:16:39.750315 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745463 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:16:39.750315 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745466 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:16:39.750315 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745469 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:16:39.750315 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745472 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:16:39.750315 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745474 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:16:39.750315 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745477 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:16:39.750315 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745480 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:16:39.750315 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745483 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:16:39.750773 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745485 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:16:39.750773 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745488 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:16:39.750773 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745491 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:16:39.750773 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745494 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:16:39.750773 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745497 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:16:39.750773 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745500 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:16:39.750773 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745503 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:16:39.750773 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745505 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:16:39.750773 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745508 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:16:39.750773 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745511 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:16:39.750773 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745513 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:16:39.750773 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745516 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:16:39.750773 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745519 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:16:39.750773 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745522 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:16:39.750773 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745524 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:16:39.750773 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745527 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:16:39.750773 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745529 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:16:39.750773 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745532 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:16:39.750773 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745535 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:16:39.751257 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745539 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:16:39.751257 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745543 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:16:39.751257 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745547 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:16:39.751257 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745550 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:16:39.751257 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745553 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:16:39.751257 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745556 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:16:39.751257 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745560 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:16:39.751257 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745563 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:16:39.751257 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745566 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:16:39.751257 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745569 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:16:39.751257 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745572 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:16:39.751257 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745575 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:16:39.751257 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745578 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:16:39.751257 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745581 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:16:39.751257 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745584 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:16:39.751257 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745587 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:16:39.751257 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745589 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:16:39.751257 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745592 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:16:39.751257 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745595 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:16:39.751257 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745598 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:16:39.751727 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745601 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:16:39.751727 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745604 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:16:39.751727 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745606 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:16:39.751727 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745609 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:16:39.751727 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745611 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:16:39.751727 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745614 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:16:39.751727 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745617 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:16:39.751727 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745619 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:16:39.751727 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745622 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:16:39.751727 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745625 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:16:39.751727 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745627 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:16:39.751727 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745630 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:16:39.751727 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745632 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:16:39.751727 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745635 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:16:39.751727 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745639 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:16:39.751727 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745642 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:16:39.751727 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745644 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:16:39.751727 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745647 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:16:39.751727 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745650 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:16:39.752194 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745652 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:16:39.752194 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745655 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:16:39.752194 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745658 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:16:39.752194 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745661 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:16:39.752194 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745663 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:16:39.752194 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745666 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:16:39.752194 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745668 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:16:39.752194 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745671 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:16:39.752194 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745674 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:16:39.752194 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745677 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:16:39.752194 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745679 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:16:39.752194 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745682 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:16:39.752194 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745685 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:16:39.752194 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745687 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:16:39.752194 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745690 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:16:39.752194 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745692 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:16:39.752194 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745695 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:16:39.752194 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745698 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:16:39.752194 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745700 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:16:39.752635 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745703 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:16:39.752635 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745705 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:16:39.752635 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745708 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:16:39.752635 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.745736 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 01:16:39.752635 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745880 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:16:39.752635 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745885 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:16:39.752635 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745888 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:16:39.752635 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745891 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:16:39.752635 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745894 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:16:39.752635 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745903 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:16:39.752635 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745906 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:16:39.752635 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745909 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:16:39.752635 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745912 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:16:39.752635 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745925 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:16:39.752635 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745928 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:16:39.752635 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745931 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:16:39.753146 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745948 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:16:39.753146 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745953 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:16:39.753146 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745956 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:16:39.753146 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745959 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:16:39.753146 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745962 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:16:39.753146 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745964 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:16:39.753146 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745967 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:16:39.753146 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745970 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:16:39.753146 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745973 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:16:39.753146 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745975 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:16:39.753146 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745978 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:16:39.753146 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745981 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:16:39.753146 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745983 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:16:39.753146 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745986 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:16:39.753146 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745989 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:16:39.753146 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745991 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:16:39.753146 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745994 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:16:39.753146 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.745996 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:16:39.753146 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746000 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:16:39.753146 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746004 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:16:39.753634 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746007 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:16:39.753634 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746010 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:16:39.753634 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746012 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:16:39.753634 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746015 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:16:39.753634 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746018 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:16:39.753634 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746020 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:16:39.753634 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746024 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:16:39.753634 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746027 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:16:39.753634 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746030 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:16:39.753634 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746032 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:16:39.753634 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746035 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:16:39.753634 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746037 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:16:39.753634 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746040 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:16:39.753634 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746043 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:16:39.753634 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746046 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:16:39.753634 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746049 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:16:39.753634 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746052 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:16:39.753634 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746054 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:16:39.753634 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746057 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:16:39.754129 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746060 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:16:39.754129 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746063 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:16:39.754129 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746065 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:16:39.754129 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746068 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:16:39.754129 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746070 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:16:39.754129 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746074 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:16:39.754129 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746076 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:16:39.754129 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746079 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:16:39.754129 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746081 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:16:39.754129 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746084 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:16:39.754129 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746086 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:16:39.754129 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746089 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:16:39.754129 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746092 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:16:39.754129 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746095 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:16:39.754129 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746098 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:16:39.754129 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746100 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:16:39.754129 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746103 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:16:39.754129 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746105 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:16:39.754129 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746108 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:16:39.754129 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746110 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:16:39.754592 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746113 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:16:39.754592 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746116 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:16:39.754592 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746118 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:16:39.754592 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746121 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:16:39.754592 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746124 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:16:39.754592 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746127 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:16:39.754592 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746129 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:16:39.754592 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746132 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:16:39.754592 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746134 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:16:39.754592 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746137 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:16:39.754592 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746140 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:16:39.754592 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746143 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:16:39.754592 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746147 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:16:39.754592 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746150 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:16:39.754592 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746154 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:16:39.754592 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746157 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:16:39.754592 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746160 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:16:39.754592 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746163 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:16:39.754592 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746166 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:16:39.755055 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746168 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:16:39.755055 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746171 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:16:39.755055 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746174 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:16:39.755055 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746176 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:16:39.755055 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746179 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:16:39.755055 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746181 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:16:39.755055 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746184 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:16:39.755055 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746187 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:16:39.755055 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746189 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:16:39.755055 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746192 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:16:39.755055 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746194 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:16:39.755055 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746197 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:16:39.755055 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746199 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:16:39.755055 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746202 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:16:39.755055 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746205 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:16:39.755055 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746208 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:16:39.755055 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.746235 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 01:16:39.755446 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746271 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:16:39.755446 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746275 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:16:39.755446 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746279 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:16:39.755446 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746282 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:16:39.755446 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746285 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:16:39.755446 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746288 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:16:39.755446 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746291 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:16:39.755446 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746295 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:16:39.755446 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746299 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:16:39.755446 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746301 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:16:39.755446 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746304 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:16:39.755446 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746307 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:16:39.755446 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746310 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:16:39.755446 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746312 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:16:39.755446 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746315 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:16:39.755446 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746318 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:16:39.755446 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746321 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:16:39.755446 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746323 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:16:39.755446 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746326 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:16:39.755920 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746328 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:16:39.755920 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746331 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:16:39.755920 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746334 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:16:39.755920 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746336 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:16:39.755920 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746339 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:16:39.755920 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746341 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:16:39.755920 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746344 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:16:39.755920 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746346 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:16:39.755920 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746349 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:16:39.755920 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746354 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:16:39.755920 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746357 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:16:39.755920 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746360 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:16:39.755920 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746363 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:16:39.755920 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746366 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:16:39.755920 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746368 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:16:39.755920 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746371 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:16:39.755920 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746374 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:16:39.755920 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746376 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:16:39.755920 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746379 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:16:39.755920 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746381 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:16:39.756537 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746384 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:16:39.756537 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746386 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:16:39.756537 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746389 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:16:39.756537 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746391 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:16:39.756537 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746394 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:16:39.756537 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746397 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:16:39.756537 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746399 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:16:39.756537 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746401 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:16:39.756537 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746404 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:16:39.756537 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746407 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:16:39.756537 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746409 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:16:39.756537 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746412 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:16:39.756537 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746415 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:16:39.756537 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746418 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:16:39.756537 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746420 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:16:39.756537 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746423 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:16:39.756537 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746425 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:16:39.756537 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746428 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:16:39.756537 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746431 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:16:39.756537 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746434 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:16:39.757061 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746437 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:16:39.757061 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746442 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:16:39.757061 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746446 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:16:39.757061 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746450 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:16:39.757061 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746454 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:16:39.757061 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746457 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:16:39.757061 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746460 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:16:39.757061 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746463 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:16:39.757061 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746465 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:16:39.757061 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746468 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:16:39.757061 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746470 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:16:39.757061 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746473 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:16:39.757061 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746476 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:16:39.757061 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746478 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:16:39.757061 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746481 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:16:39.757061 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746483 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:16:39.757061 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746486 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:16:39.757061 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746489 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:16:39.757061 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746491 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:16:39.757493 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746494 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:16:39.757493 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746496 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:16:39.757493 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746499 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:16:39.757493 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746502 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:16:39.757493 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746504 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:16:39.757493 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746507 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:16:39.757493 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746510 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:16:39.757493 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746512 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:16:39.757493 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746515 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:16:39.757493 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746518 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:16:39.757493 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746520 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:16:39.757493 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746523 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:16:39.757493 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746526 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:16:39.757493 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746529 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:16:39.757493 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746532 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:16:39.757493 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746535 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:16:39.757493 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746538 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:16:39.757493 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746541 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:16:39.757493 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746544 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:16:39.757493 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746547 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:16:39.758218 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746549 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:16:39.758218 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746553 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:16:39.758218 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746556 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:16:39.758218 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746559 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:16:39.758218 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746561 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:16:39.758218 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746564 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:16:39.758218 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746566 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:16:39.758218 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:39.746569 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:16:39.758218 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.746637 2583 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 01:16:39.758218 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.748720 2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 01:16:39.758218 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.751688 2583 server.go:1014] "Starting client certificate rotation" Oct 30 01:16:39.758218 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.751781 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 01:16:39.758218 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.752506 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 01:16:39.758218 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.754636 2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 01:16:39.760303 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.760286 2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 01:16:39.770203 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.770182 2583 log.go:25] "Validated CRI v1 runtime API" Oct 30 01:16:39.773883 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.773863 2583 log.go:25] "Validated CRI v1 image API" Oct 30 01:16:39.774744 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.774719 2583 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 01:16:39.780590 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.780511 2583 fs.go:135] Filesystem UUIDs: map[18798d15-a7a2-4adb-b7e4-7f663ea9309b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e41383ef-8ce0-477f-8523-ffd98ae4edd0:/dev/nvme0n1p3] Oct 30 01:16:39.780692 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.780588 2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 01:16:39.786414 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.786285 2583 manager.go:217] Machine: {Timestamp:2026-10-30 01:16:39.785029719 +0000 UTC m=+0.352131998 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097936 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec281e2b4acea78f00f4ee495cc050e5 SystemUUID:ec281e2b-4ace-a78f-00f4-ee495cc050e5 BootID:463af845-f3b9-4975-a32a-ae96d317f323 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ee:f6:e3:43 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ee:f6:e3:43 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4a:ea:a6:10:04:04 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 01:16:39.786919 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.786904 2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 01:16:39.787056 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.787039 2583 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 01:16:39.790056 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.790026 2583 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 01:16:39.790193 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.790057 2583 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-223.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 01:16:39.790249 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.790196 2583 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 01:16:39.790249 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.790204 2583 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 01:16:39.790249 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.790220 2583 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 01:16:39.790249 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.790235 2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 01:16:39.791036 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.791026 2583 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 01:16:39.793688 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.793675 2583 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 01:16:39.793807 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.793797 2583 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 01:16:39.794545 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.794536 2583 kubelet.go:493] "Attempting to sync node with API server" Oct 30 01:16:39.794580 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.794549 2583 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 01:16:39.794580 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.794560 2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 01:16:39.794580 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.794569 2583 kubelet.go:405] "Adding apiserver pod source" Oct 30 01:16:39.794580 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.794579 2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 01:16:39.794909 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.794898 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 01:16:39.797364 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.797343 2583 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 01:16:39.802310 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.802295 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 01:16:39.803215 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.803198 2583 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 01:16:39.803258 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.803232 2583 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 01:16:39.803400 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.803388 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 01:16:39.803431 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.803404 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 01:16:39.803431 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.803410 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 01:16:39.803431 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.803416 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 01:16:39.803431 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.803422 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 01:16:39.803431 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.803429 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 01:16:39.804073 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.804064 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 01:16:39.804102 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.804074 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 01:16:39.804102 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.804081 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 01:16:39.804102 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.804087 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 01:16:39.804102 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.804097 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 01:16:39.804225 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.804115 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 01:16:39.804845 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.804832 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 01:16:39.804881 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.804847 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 01:16:39.808746 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.808731 2583 server.go:1263] "Started kubelet" Oct 30 01:16:39.808835 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.808801 2583 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 01:16:39.808916 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.808872 2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 01:16:39.809340 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.809316 2583 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 01:16:39.809746 ip-10-0-138-223 systemd[1]: Started Kubernetes Kubelet. Oct 30 01:16:39.810504 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.810486 2583 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 01:16:39.811201 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.811177 2583 server.go:317] "Adding debug handlers to kubelet server" Oct 30 01:16:39.816413 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:39.816384 2583 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 01:16:39.816992 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.816973 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 01:16:39.817856 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.817835 2583 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 01:16:39.818080 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:39.818058 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-223.ec2.internal\" not found" Oct 30 01:16:39.818572 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.818550 2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 01:16:39.818749 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.818733 2583 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 01:16:39.818823 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.818751 2583 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 01:16:39.819141 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.819121 2583 reconstruct.go:97] "Volume reconstruction finished" Oct 30 01:16:39.819141 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.819141 2583 reconciler.go:29] "Reconciler: start to sync state" Oct 30 01:16:39.819268 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.819178 2583 factory.go:55] Registering systemd factory Oct 30 01:16:39.819268 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.819195 2583 factory.go:223] Registration of the systemd container factory successfully Oct 30 01:16:39.819559 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.819544 2583 factory.go:153] Registering CRI-O factory Oct 30 01:16:39.819625 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.819560 2583 factory.go:223] Registration of the crio container factory successfully Oct 30 01:16:39.819625 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.819616 2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 01:16:39.819710 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.819636 2583 factory.go:103] Registering Raw factory Oct 30 01:16:39.819710 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.819659 2583 manager.go:1199] Started watching for new ooms in manager Oct 30 01:16:39.820551 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.820531 2583 manager.go:319] Starting recovery of all containers Oct 30 01:16:39.831550 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.831531 2583 manager.go:324] Recovery completed Oct 30 01:16:39.832122 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.832103 2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 01:16:39.832204 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.832184 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w99fn" Oct 30 01:16:39.834703 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.834680 2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-223.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 01:16:39.834799 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:39.834689 2583 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-223.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 01:16:39.835442 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.835420 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w99fn" Oct 30 01:16:39.835700 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:39.834682 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-223.ec2.internal.18e329c45322668f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-223.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-223.ec2.internal,},FirstTimestamp:2026-10-30 01:16:39.808681615 +0000 UTC m=+0.375783894,LastTimestamp:2026-10-30 01:16:39.808681615 +0000 UTC m=+0.375783894,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-223.ec2.internal,}" Oct 30 01:16:39.835992 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.835974 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:16:39.838269 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.838251 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:16:39.838341 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.838282 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:16:39.838341 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.838292 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:16:39.838968 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.838953 2583 cpu_manager.go:226] "Starting" policy="none" Oct 30 01:16:39.838968 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.838966 2583 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 01:16:39.839084 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.838966 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:16:39.839084 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.838985 2583 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 01:16:39.839652 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.839635 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:16:39.839723 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.839666 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:16:39.839723 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.839677 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:16:39.839989 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.839978 2583 policy_none.go:50] "Start" Oct 30 01:16:39.840025 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.839993 2583 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 01:16:39.840025 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.840005 2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 01:16:39.841012 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.840996 2583 policy_none.go:44] "Start" Oct 30 01:16:39.849295 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:39.849264 2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-223.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 01:16:39.864500 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.864376 2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:16:39.881637 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.881601 2583 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 01:16:39.885818 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.885802 2583 manager.go:341] "Starting Device Plugin manager" Oct 30 01:16:39.885902 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:39.885832 2583 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 01:16:39.885902 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.885841 2583 server.go:87] "Starting device plugin registration server" Oct 30 01:16:39.886153 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.886141 2583 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 01:16:39.886211 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.886155 2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 01:16:39.886268 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.886252 2583 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 01:16:39.886603 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.886335 2583 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 01:16:39.886603 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.886345 2583 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 01:16:39.886862 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:39.886843 2583 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 01:16:39.886921 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:39.886893 2583 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-223.ec2.internal\" not found" Oct 30 01:16:39.893849 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.893831 2583 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:16:39.945579 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.945547 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 01:16:39.946683 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.946667 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 01:16:39.946752 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.946689 2583 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 01:16:39.946752 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.946726 2583 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 01:16:39.946833 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:39.946763 2583 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 01:16:39.966649 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.966616 2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:16:39.986810 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.986725 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:16:39.987740 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.987725 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:16:39.987832 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.987754 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:16:39.987832 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.987767 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:16:39.987832 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:39.987794 2583 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-223.ec2.internal" Oct 30 01:16:40.016488 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:40.016452 2583 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-223.ec2.internal" Oct 30 01:16:40.016488 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:40.016483 2583 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-223.ec2.internal\": node \"ip-10-0-138-223.ec2.internal\" not found" Oct 30 01:16:40.047387 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:40.047351 2583 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal"] Oct 30 01:16:40.047536 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:40.047421 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:16:40.050262 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:40.050240 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:16:40.050326 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:40.050280 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:16:40.050326 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:40.050295 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:16:40.051767 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:40.051754 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:16:40.051927 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:40.051911 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal" Oct 30 01:16:40.051985 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:40.051955 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:16:40.052587 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:40.052568 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:16:40.052689 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:40.052642 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:16:40.052689 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:40.052568 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:16:40.052689 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:40.052661 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:16:40.052689 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:40.052681 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:16:40.052877 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:40.052698 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:16:40.054020 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:40.054005 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal" Oct 30 01:16:40.054108 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:40.054037 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:16:40.054923 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:40.054903 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:16:40.055036 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:40.054932 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:16:40.055036 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:40.054960 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:16:40.070635 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:40.070602 2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-223.ec2.internal\" not found" node="ip-10-0-138-223.ec2.internal" Oct 30 01:16:40.075661 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:40.075641 2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-223.ec2.internal\" not found" node="ip-10-0-138-223.ec2.internal" Oct 30 01:16:40.094712 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:40.094690 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-223.ec2.internal\" not found" Oct 30 01:16:40.119949 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:40.119908 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/129d2d6479043c5ffd993e1a7481e29a-config\") pod \"kube-apiserver-proxy-ip-10-0-138-223.ec2.internal\" (UID: \"129d2d6479043c5ffd993e1a7481e29a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal" Oct 30 01:16:40.120021 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:40.119962 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b222e074fa1fff6b3c88cd830a3f1d7d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal\" (UID: \"b222e074fa1fff6b3c88cd830a3f1d7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal" Oct 30 01:16:40.120055 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:40.120020 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b222e074fa1fff6b3c88cd830a3f1d7d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal\" (UID: \"b222e074fa1fff6b3c88cd830a3f1d7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal" Oct 30 01:16:40.195358 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:40.195295 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-223.ec2.internal\" not found" Oct 30 01:16:40.220945 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:40.220914 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b222e074fa1fff6b3c88cd830a3f1d7d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal\" (UID: \"b222e074fa1fff6b3c88cd830a3f1d7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal" Oct 30 01:16:40.221025 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:40.220965 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b222e074fa1fff6b3c88cd830a3f1d7d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal\" (UID: \"b222e074fa1fff6b3c88cd830a3f1d7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal" Oct 30 01:16:40.221025 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:40.220987 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/129d2d6479043c5ffd993e1a7481e29a-config\") pod \"kube-apiserver-proxy-ip-10-0-138-223.ec2.internal\" (UID: \"129d2d6479043c5ffd993e1a7481e29a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal" Oct 30 01:16:40.221090 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:40.221032 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/129d2d6479043c5ffd993e1a7481e29a-config\") pod \"kube-apiserver-proxy-ip-10-0-138-223.ec2.internal\" (UID: \"129d2d6479043c5ffd993e1a7481e29a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal" Oct 30 01:16:40.221090 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:40.221035 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b222e074fa1fff6b3c88cd830a3f1d7d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal\" (UID: \"b222e074fa1fff6b3c88cd830a3f1d7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal" Oct 30 01:16:40.221090 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:40.221032 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b222e074fa1fff6b3c88cd830a3f1d7d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal\" (UID: \"b222e074fa1fff6b3c88cd830a3f1d7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal" Oct 30 01:16:40.295732 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:40.295669 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-223.ec2.internal\" not found" Oct 30 01:16:40.374382 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:40.374354 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal" Oct 30 01:16:40.379096 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:40.378884 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal" Oct 30 01:16:40.396717 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:40.396687 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-223.ec2.internal\" not found" Oct 30 01:16:42.031957 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:42.031928 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-223.ec2.internal\" not found" Oct 30 01:16:42.132469 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:42.132392 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-223.ec2.internal\" not found" Oct 30 01:16:42.232928 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:42.232893 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-223.ec2.internal\" not found" Oct 30 01:16:42.286414 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:42.286376 2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 01:16:42.286904 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:42.286576 2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 01:16:42.286904 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:42.286574 2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 01:16:42.286904 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:42.286600 2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 01:16:42.333973 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:42.333934 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-223.ec2.internal\" not found" Oct 30 01:16:42.374367 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:42.374323 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 01:11:41 +0000 UTC" deadline="2028-07-08 00:42:49.477485802 +0000 UTC" Oct 30 01:16:42.374367 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:42.374360 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14807h26m7.103130125s" Oct 30 01:16:42.434637 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:42.434609 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-223.ec2.internal\" not found" Oct 30 01:16:42.479789 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:42.479757 2583 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:16:42.496279 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:42.496241 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod129d2d6479043c5ffd993e1a7481e29a.slice/crio-16b8d3afdfe0b8d1782acbd5b377b2fa9b2f56255bc2a6aba928ce62ab88f227 WatchSource:0}: Error finding container 16b8d3afdfe0b8d1782acbd5b377b2fa9b2f56255bc2a6aba928ce62ab88f227: Status 404 returned error can't find the container with id 16b8d3afdfe0b8d1782acbd5b377b2fa9b2f56255bc2a6aba928ce62ab88f227 Oct 30 01:16:42.496510 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:42.496488 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb222e074fa1fff6b3c88cd830a3f1d7d.slice/crio-61c3f93c78077742aa562727b783c60bb0f3a3341a17838f2669945a71f08ca1 WatchSource:0}: Error finding container 61c3f93c78077742aa562727b783c60bb0f3a3341a17838f2669945a71f08ca1: Status 404 returned error can't find the container with id 61c3f93c78077742aa562727b783c60bb0f3a3341a17838f2669945a71f08ca1 Oct 30 01:16:42.502852 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:42.502825 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:16:42.553304 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:42.553267 2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal" Oct 30 01:16:42.574918 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:42.574893 2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 01:16:42.575031 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:42.574998 2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal" Oct 30 01:16:42.595272 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:42.595250 2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 01:16:43.161618 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.161582 2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:16:43.330499 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.330460 2583 apiserver.go:52] "Watching apiserver" Oct 30 01:16:43.352614 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.352574 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 01:16:43.374352 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.374312 2583 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 01:16:43.374658 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.374631 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-dmnzr","openshift-network-operator/iptables-alerter-xv9wf","openshift-cluster-node-tuning-operator/tuned-cmcx6","openshift-image-registry/node-ca-54hlq","openshift-multus/multus-m8h9g","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal","openshift-multus/network-metrics-daemon-64xhb","openshift-ovn-kubernetes/ovnkube-node-6zn5n","kube-system/global-pull-secret-syncer-j7l6k","kube-system/konnectivity-agent-67v6w","kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal","openshift-dns/node-resolver-4dl5p","openshift-multus/multus-additional-cni-plugins-5wdsg"] Oct 30 01:16:43.376452 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.376428 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:16:43.376571 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:43.376513 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7l6k" podUID="88f5d337-fa16-4430-a82e-5f433354c164" Oct 30 01:16:43.378024 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.378003 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4dl5p" Oct 30 01:16:43.379407 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.379387 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.381807 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.381785 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-54hlq" Oct 30 01:16:43.382953 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.382931 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.384473 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.384451 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:16:43.386163 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.385870 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.387429 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.387150 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:16:43.389443 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.389142 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xv9wf" Oct 30 01:16:43.390934 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.390914 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:16:43.391031 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:43.390997 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmnzr" podUID="1c5a490f-f8db-4afb-9a1f-b1accc9eadea" Oct 30 01:16:43.392220 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.392200 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-67v6w" Oct 30 01:16:43.392934 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.392917 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pcscp\"" Oct 30 01:16:43.393029 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.392989 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-srg4h\"" Oct 30 01:16:43.393029 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.393012 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 01:16:43.393139 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.393058 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 01:16:43.393687 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.393662 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:16:43.393827 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.393716 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 01:16:43.394345 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.394325 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:16:43.394437 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:43.394403 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64xhb" podUID="f4b29936-72de-4ddc-b7f1-a964a614ff61" Oct 30 01:16:43.397251 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.397234 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 01:16:43.397540 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.397519 2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 01:16:43.397634 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.397595 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-nkfsc" Oct 30 01:16:43.404312 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.404244 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nkfsc" Oct 30 01:16:43.433820 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.433793 2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:16:43.454397 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.454369 2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 01:16:43.466818 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.466784 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-node-log\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.466976 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.466829 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-cnibin\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:16:43.466976 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.466899 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/88f5d337-fa16-4430-a82e-5f433354c164-dbus\") pod \"global-pull-secret-syncer-j7l6k\" (UID: \"88f5d337-fa16-4430-a82e-5f433354c164\") " pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:16:43.466976 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.466927 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqpz6\" (UniqueName: \"kubernetes.io/projected/f4b29936-72de-4ddc-b7f1-a964a614ff61-kube-api-access-hqpz6\") pod \"network-metrics-daemon-64xhb\" (UID: \"f4b29936-72de-4ddc-b7f1-a964a614ff61\") " pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:16:43.466976 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.466944 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-kubernetes\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.466976 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.466966 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-cnibin\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.467224 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.466992 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-host-run-multus-certs\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.467224 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.467035 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-systemd-units\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.467224 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.467064 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-modprobe-d\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.467224 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.467080 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda047-30e9-496c-b854-af4043a83656-device-dir\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:16:43.467224 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.467122 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-slash\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.467224 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.467153 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-run-netns\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.467224 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.467185 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx5fw\" (UniqueName: \"kubernetes.io/projected/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-kube-api-access-sx5fw\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:16:43.467224 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.467217 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-sysctl-conf\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.467569 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.467265 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-run\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.467569 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.467307 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-log-socket\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.467569 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.467324 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:16:43.467569 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.467340 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czkw9\" (UniqueName: \"kubernetes.io/projected/1c5a490f-f8db-4afb-9a1f-b1accc9eadea-kube-api-access-czkw9\") pod \"network-check-target-dmnzr\" (UID: \"1c5a490f-f8db-4afb-9a1f-b1accc9eadea\") " pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:16:43.467569 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.467354 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-lib-modules\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.467569 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.467414 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-cni-netd\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.467569 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.467475 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/03772f27-6bc6-4048-a515-d8d4e9abf431-ovn-node-metrics-cert\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.467569 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.467530 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46pfs\" (UniqueName: \"kubernetes.io/projected/03772f27-6bc6-4048-a515-d8d4e9abf431-kube-api-access-46pfs\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.467922 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.467573 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-cni-binary-copy\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:16:43.467922 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.467608 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e7922f23-5ba7-4931-8d19-598e2c4fe041-serviceca\") pod \"node-ca-54hlq\" (UID: \"e7922f23-5ba7-4931-8d19-598e2c4fe041\") " pod="openshift-image-registry/node-ca-54hlq" Oct 30 01:16:43.467922 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.467652 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/262c01e0-8469-4fbe-aaa4-8a8c50551fce-konnectivity-ca\") pod \"konnectivity-agent-67v6w\" (UID: \"262c01e0-8469-4fbe-aaa4-8a8c50551fce\") " pod="kube-system/konnectivity-agent-67v6w" Oct 30 01:16:43.467922 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.467691 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6babafa-f85a-4a1b-9fe9-8bac54198754-tmp-dir\") pod \"node-resolver-4dl5p\" (UID: \"a6babafa-f85a-4a1b-9fe9-8bac54198754\") " pod="openshift-dns/node-resolver-4dl5p" Oct 30 01:16:43.467922 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.467731 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-systemd\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.467922 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.467785 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-host\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.467922 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.467822 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-hostroot\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.467922 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.467870 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-run-ovn\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.467922 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.467892 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:16:43.468266 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.467933 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7922f23-5ba7-4931-8d19-598e2c4fe041-host\") pod \"node-ca-54hlq\" (UID: \"e7922f23-5ba7-4931-8d19-598e2c4fe041\") " pod="openshift-image-registry/node-ca-54hlq" Oct 30 01:16:43.468266 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.467954 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-multus-socket-dir-parent\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.468266 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.467969 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a6babafa-f85a-4a1b-9fe9-8bac54198754-hosts-file\") pod \"node-resolver-4dl5p\" (UID: \"a6babafa-f85a-4a1b-9fe9-8bac54198754\") " pod="openshift-dns/node-resolver-4dl5p" Oct 30 01:16:43.468266 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.467984 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-sysconfig\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.468266 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468026 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-tuned\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.468266 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468048 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-host-var-lib-kubelet\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.468266 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468080 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc85r\" (UniqueName: \"kubernetes.io/projected/4b97aea6-bab9-498d-bce7-95e2a47c8c05-kube-api-access-gc85r\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.468266 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468096 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-kubelet\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.468266 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468112 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-run-openvswitch\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.468266 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468125 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-sys\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.468266 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468140 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-run-systemd\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.468266 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468198 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.468266 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468214 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-sysctl-d\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.468266 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468227 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04631a42-b08e-4aff-9fc4-78dd7392d59d-tmp\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.468266 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468242 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-system-cni-dir\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.468266 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468256 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda047-30e9-496c-b854-af4043a83656-kubelet-dir\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:16:43.468800 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468282 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9ceda047-30e9-496c-b854-af4043a83656-sys-fs\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:16:43.468800 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468299 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-var-lib-openvswitch\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.468800 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468315 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-cni-bin\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.468800 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468338 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/03772f27-6bc6-4048-a515-d8d4e9abf431-ovnkube-script-lib\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.468800 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468359 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b00a159-66f4-4b9d-bcf1-560094dd69bb-host-slash\") pod \"iptables-alerter-xv9wf\" (UID: \"8b00a159-66f4-4b9d-bcf1-560094dd69bb\") " pod="openshift-network-operator/iptables-alerter-xv9wf" Oct 30 01:16:43.468800 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468394 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/88f5d337-fa16-4430-a82e-5f433354c164-kubelet-config\") pod \"global-pull-secret-syncer-j7l6k\" (UID: \"88f5d337-fa16-4430-a82e-5f433354c164\") " pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:16:43.468800 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468413 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b29936-72de-4ddc-b7f1-a964a614ff61-metrics-certs\") pod \"network-metrics-daemon-64xhb\" (UID: \"f4b29936-72de-4ddc-b7f1-a964a614ff61\") " pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:16:43.468800 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468427 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-host-run-netns\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.468800 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468448 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-host-var-lib-cni-multus\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.468800 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468463 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-run-ovn-kubernetes\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.468800 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468528 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/03772f27-6bc6-4048-a515-d8d4e9abf431-ovnkube-config\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.468800 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468554 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:16:43.468800 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468570 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/88f5d337-fa16-4430-a82e-5f433354c164-original-pull-secret\") pod \"global-pull-secret-syncer-j7l6k\" (UID: \"88f5d337-fa16-4430-a82e-5f433354c164\") " pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:16:43.468800 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468592 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-var-lib-kubelet\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.468800 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468607 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6wcg\" (UniqueName: \"kubernetes.io/projected/e7922f23-5ba7-4931-8d19-598e2c4fe041-kube-api-access-z6wcg\") pod \"node-ca-54hlq\" (UID: \"e7922f23-5ba7-4931-8d19-598e2c4fe041\") " pod="openshift-image-registry/node-ca-54hlq" Oct 30 01:16:43.468800 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468622 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-multus-cni-dir\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.469297 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468636 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-host-var-lib-cni-bin\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.469297 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468650 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/262c01e0-8469-4fbe-aaa4-8a8c50551fce-agent-certs\") pod \"konnectivity-agent-67v6w\" (UID: \"262c01e0-8469-4fbe-aaa4-8a8c50551fce\") " pod="kube-system/konnectivity-agent-67v6w" Oct 30 01:16:43.469297 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468664 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-multus-conf-dir\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.469297 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468678 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4b97aea6-bab9-498d-bce7-95e2a47c8c05-multus-daemon-config\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.469297 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468693 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-etc-kubernetes\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.469297 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468710 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda047-30e9-496c-b854-af4043a83656-socket-dir\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:16:43.469297 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468761 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda047-30e9-496c-b854-af4043a83656-registration-dir\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:16:43.469297 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468787 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqww5\" (UniqueName: \"kubernetes.io/projected/9ceda047-30e9-496c-b854-af4043a83656-kube-api-access-tqww5\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:16:43.469297 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468816 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-system-cni-dir\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:16:43.469297 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468832 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8b00a159-66f4-4b9d-bcf1-560094dd69bb-iptables-alerter-script\") pod \"iptables-alerter-xv9wf\" (UID: \"8b00a159-66f4-4b9d-bcf1-560094dd69bb\") " pod="openshift-network-operator/iptables-alerter-xv9wf" Oct 30 01:16:43.469297 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468848 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jnnc\" (UniqueName: \"kubernetes.io/projected/04631a42-b08e-4aff-9fc4-78dd7392d59d-kube-api-access-9jnnc\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.469297 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468863 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-host-run-k8s-cni-cncf-io\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.469297 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468878 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9ceda047-30e9-496c-b854-af4043a83656-etc-selinux\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:16:43.469297 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468892 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-etc-openvswitch\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.469297 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468912 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/03772f27-6bc6-4048-a515-d8d4e9abf431-env-overrides\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.469297 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468935 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm4x7\" (UniqueName: \"kubernetes.io/projected/8b00a159-66f4-4b9d-bcf1-560094dd69bb-kube-api-access-fm4x7\") pod \"iptables-alerter-xv9wf\" (UID: \"8b00a159-66f4-4b9d-bcf1-560094dd69bb\") " pod="openshift-network-operator/iptables-alerter-xv9wf" Oct 30 01:16:43.469893 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468952 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm5kk\" (UniqueName: \"kubernetes.io/projected/a6babafa-f85a-4a1b-9fe9-8bac54198754-kube-api-access-lm5kk\") pod \"node-resolver-4dl5p\" (UID: \"a6babafa-f85a-4a1b-9fe9-8bac54198754\") " pod="openshift-dns/node-resolver-4dl5p" Oct 30 01:16:43.469893 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.468967 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b97aea6-bab9-498d-bce7-95e2a47c8c05-cni-binary-copy\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.487519 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.487434 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal" event={"ID":"129d2d6479043c5ffd993e1a7481e29a","Type":"ContainerStarted","Data":"16b8d3afdfe0b8d1782acbd5b377b2fa9b2f56255bc2a6aba928ce62ab88f227"} Oct 30 01:16:43.489081 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.489049 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal" event={"ID":"b222e074fa1fff6b3c88cd830a3f1d7d","Type":"ContainerStarted","Data":"61c3f93c78077742aa562727b783c60bb0f3a3341a17838f2669945a71f08ca1"} Oct 30 01:16:43.494716 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.494667 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 01:16:43.494873 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.494731 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 01:16:43.494873 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.494767 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 01:16:43.494873 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.494803 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 01:16:43.495030 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.494891 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 01:16:43.495030 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.494956 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 01:16:43.495162 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.495141 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 01:16:43.495313 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.495206 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 01:16:43.495313 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.495269 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 01:16:43.495452 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.495363 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 01:16:43.495536 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.495521 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j8bsj\"" Oct 30 01:16:43.495622 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.495576 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 01:16:43.495622 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.495593 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hb7tb\"" Oct 30 01:16:43.495622 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.495611 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 01:16:43.495622 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.495615 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 01:16:43.495862 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.495640 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 01:16:43.495862 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.495655 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fsgft\"" Oct 30 01:16:43.495862 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.495690 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 01:16:43.495862 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.495692 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 01:16:43.495862 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.495711 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 01:16:43.495862 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.495829 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:16:43.505562 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.505536 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 01:16:43.506617 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.506583 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 01:16:43.506617 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.506582 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p2b59\"" Oct 30 01:16:43.506832 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.506659 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-p4hjp\"" Oct 30 01:16:43.506832 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.506679 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7dsj7\"" Oct 30 01:16:43.507543 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.507469 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 01:16:43.507543 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.507507 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-58lhf\"" Oct 30 01:16:43.507703 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.507677 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 01:16:43.544025 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.543988 2583 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:16:43.569310 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.569272 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-lib-modules\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.569485 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.569320 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-cni-netd\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.569485 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.569347 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/03772f27-6bc6-4048-a515-d8d4e9abf431-ovn-node-metrics-cert\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.569485 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.569374 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46pfs\" (UniqueName: \"kubernetes.io/projected/03772f27-6bc6-4048-a515-d8d4e9abf431-kube-api-access-46pfs\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.569485 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.569397 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-cni-binary-copy\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:16:43.569485 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.569413 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e7922f23-5ba7-4931-8d19-598e2c4fe041-serviceca\") pod \"node-ca-54hlq\" (UID: \"e7922f23-5ba7-4931-8d19-598e2c4fe041\") " pod="openshift-image-registry/node-ca-54hlq" Oct 30 01:16:43.569485 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.569429 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/262c01e0-8469-4fbe-aaa4-8a8c50551fce-konnectivity-ca\") pod \"konnectivity-agent-67v6w\" (UID: \"262c01e0-8469-4fbe-aaa4-8a8c50551fce\") " pod="kube-system/konnectivity-agent-67v6w" Oct 30 01:16:43.569485 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.569446 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6babafa-f85a-4a1b-9fe9-8bac54198754-tmp-dir\") pod \"node-resolver-4dl5p\" (UID: \"a6babafa-f85a-4a1b-9fe9-8bac54198754\") " pod="openshift-dns/node-resolver-4dl5p" Oct 30 01:16:43.569485 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.569467 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-systemd\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.569485 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.569485 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-host\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.569845 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.569498 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-hostroot\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.569845 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.569513 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-run-ovn\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.569845 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.569536 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:16:43.569845 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.569559 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7922f23-5ba7-4931-8d19-598e2c4fe041-host\") pod \"node-ca-54hlq\" (UID: \"e7922f23-5ba7-4931-8d19-598e2c4fe041\") " pod="openshift-image-registry/node-ca-54hlq" Oct 30 01:16:43.569845 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.569580 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-multus-socket-dir-parent\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.569845 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.569601 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a6babafa-f85a-4a1b-9fe9-8bac54198754-hosts-file\") pod \"node-resolver-4dl5p\" (UID: \"a6babafa-f85a-4a1b-9fe9-8bac54198754\") " pod="openshift-dns/node-resolver-4dl5p" Oct 30 01:16:43.569845 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.569625 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-sysconfig\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.569845 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.569642 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-tuned\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.569845 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.569664 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-host-var-lib-kubelet\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.569845 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.569689 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc85r\" (UniqueName: \"kubernetes.io/projected/4b97aea6-bab9-498d-bce7-95e2a47c8c05-kube-api-access-gc85r\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.569845 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.569708 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-kubelet\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.569845 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.569723 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-run-openvswitch\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.569845 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.569736 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-sys\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.569845 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.569769 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-run-systemd\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.569845 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.569795 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.569845 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.569812 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-sysctl-d\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.569845 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.569830 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04631a42-b08e-4aff-9fc4-78dd7392d59d-tmp\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.570405 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.569868 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-system-cni-dir\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.570405 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.569883 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda047-30e9-496c-b854-af4043a83656-kubelet-dir\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:16:43.570405 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.569898 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9ceda047-30e9-496c-b854-af4043a83656-sys-fs\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:16:43.570405 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.569914 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-var-lib-openvswitch\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.570405 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.569991 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-var-lib-openvswitch\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.570405 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.570120 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-lib-modules\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.570405 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.570165 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-cni-netd\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.570651 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.570502 2583 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 01:16:43.570833 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.570810 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-cni-bin\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.570897 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.570848 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/03772f27-6bc6-4048-a515-d8d4e9abf431-ovnkube-script-lib\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.570897 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.570881 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-hostroot\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.570961 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.570949 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-host\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.571013 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.571003 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-sysconfig\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.571066 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.571049 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-run-ovn\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.571228 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.571208 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:16:43.571284 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.571268 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7922f23-5ba7-4931-8d19-598e2c4fe041-host\") pod \"node-ca-54hlq\" (UID: \"e7922f23-5ba7-4931-8d19-598e2c4fe041\") " pod="openshift-image-registry/node-ca-54hlq" Oct 30 01:16:43.571401 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.571381 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-multus-socket-dir-parent\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.571530 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.571430 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a6babafa-f85a-4a1b-9fe9-8bac54198754-hosts-file\") pod \"node-resolver-4dl5p\" (UID: \"a6babafa-f85a-4a1b-9fe9-8bac54198754\") " pod="openshift-dns/node-resolver-4dl5p" Oct 30 01:16:43.571530 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.571481 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/03772f27-6bc6-4048-a515-d8d4e9abf431-ovnkube-script-lib\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.571647 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.571577 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-cni-bin\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.571647 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.571623 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-cni-binary-copy\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:16:43.571743 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.571694 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-systemd\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.571743 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.571730 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b00a159-66f4-4b9d-bcf1-560094dd69bb-host-slash\") pod \"iptables-alerter-xv9wf\" (UID: \"8b00a159-66f4-4b9d-bcf1-560094dd69bb\") " pod="openshift-network-operator/iptables-alerter-xv9wf" Oct 30 01:16:43.571743 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.571733 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6babafa-f85a-4a1b-9fe9-8bac54198754-tmp-dir\") pod \"node-resolver-4dl5p\" (UID: \"a6babafa-f85a-4a1b-9fe9-8bac54198754\") " pod="openshift-dns/node-resolver-4dl5p" Oct 30 01:16:43.571914 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.571784 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/88f5d337-fa16-4430-a82e-5f433354c164-kubelet-config\") pod \"global-pull-secret-syncer-j7l6k\" (UID: \"88f5d337-fa16-4430-a82e-5f433354c164\") " pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:16:43.571914 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.571813 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b29936-72de-4ddc-b7f1-a964a614ff61-metrics-certs\") pod \"network-metrics-daemon-64xhb\" (UID: \"f4b29936-72de-4ddc-b7f1-a964a614ff61\") " pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:16:43.571914 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.571839 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-host-run-netns\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.571914 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.571867 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-host-var-lib-cni-multus\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.571914 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.571891 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-run-ovn-kubernetes\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.572158 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.571917 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/03772f27-6bc6-4048-a515-d8d4e9abf431-ovnkube-config\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.572158 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.571941 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:16:43.572158 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.571963 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e7922f23-5ba7-4931-8d19-598e2c4fe041-serviceca\") pod \"node-ca-54hlq\" (UID: \"e7922f23-5ba7-4931-8d19-598e2c4fe041\") " pod="openshift-image-registry/node-ca-54hlq" Oct 30 01:16:43.572158 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.572006 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b00a159-66f4-4b9d-bcf1-560094dd69bb-host-slash\") pod \"iptables-alerter-xv9wf\" (UID: \"8b00a159-66f4-4b9d-bcf1-560094dd69bb\") " pod="openshift-network-operator/iptables-alerter-xv9wf" Oct 30 01:16:43.572158 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:43.572053 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:16:43.572158 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.572057 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/88f5d337-fa16-4430-a82e-5f433354c164-kubelet-config\") pod \"global-pull-secret-syncer-j7l6k\" (UID: \"88f5d337-fa16-4430-a82e-5f433354c164\") " pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:16:43.572158 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:43.572102 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88f5d337-fa16-4430-a82e-5f433354c164-original-pull-secret podName:88f5d337-fa16-4430-a82e-5f433354c164 nodeName:}" failed. No retries permitted until 2026-10-30 01:16:44.072085984 +0000 UTC m=+3.104376983 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/88f5d337-fa16-4430-a82e-5f433354c164-original-pull-secret") pod "global-pull-secret-syncer-j7l6k" (UID: "88f5d337-fa16-4430-a82e-5f433354c164") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:16:43.572158 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.571966 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/88f5d337-fa16-4430-a82e-5f433354c164-original-pull-secret\") pod \"global-pull-secret-syncer-j7l6k\" (UID: \"88f5d337-fa16-4430-a82e-5f433354c164\") " pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:16:43.572158 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.572130 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-var-lib-kubelet\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.572158 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:43.572133 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:16:43.572158 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.572148 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6wcg\" (UniqueName: \"kubernetes.io/projected/e7922f23-5ba7-4931-8d19-598e2c4fe041-kube-api-access-z6wcg\") pod \"node-ca-54hlq\" (UID: \"e7922f23-5ba7-4931-8d19-598e2c4fe041\") " pod="openshift-image-registry/node-ca-54hlq" Oct 30 01:16:43.572158 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.572164 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-multus-cni-dir\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.572828 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.572182 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-host-var-lib-cni-bin\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.572828 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:43.572199 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4b29936-72de-4ddc-b7f1-a964a614ff61-metrics-certs podName:f4b29936-72de-4ddc-b7f1-a964a614ff61 nodeName:}" failed. No retries permitted until 2026-10-30 01:16:44.072192665 +0000 UTC m=+3.104483624 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f4b29936-72de-4ddc-b7f1-a964a614ff61-metrics-certs") pod "network-metrics-daemon-64xhb" (UID: "f4b29936-72de-4ddc-b7f1-a964a614ff61") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:16:43.572828 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.572214 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-host-run-netns\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.572828 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.572243 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/262c01e0-8469-4fbe-aaa4-8a8c50551fce-agent-certs\") pod \"konnectivity-agent-67v6w\" (UID: \"262c01e0-8469-4fbe-aaa4-8a8c50551fce\") " pod="kube-system/konnectivity-agent-67v6w" Oct 30 01:16:43.572828 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.572271 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-multus-conf-dir\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.572828 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.572278 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/262c01e0-8469-4fbe-aaa4-8a8c50551fce-konnectivity-ca\") pod \"konnectivity-agent-67v6w\" (UID: \"262c01e0-8469-4fbe-aaa4-8a8c50551fce\") " pod="kube-system/konnectivity-agent-67v6w" Oct 30 01:16:43.572828 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.572299 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4b97aea6-bab9-498d-bce7-95e2a47c8c05-multus-daemon-config\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.572828 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.572327 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-etc-kubernetes\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.572828 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.572357 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda047-30e9-496c-b854-af4043a83656-socket-dir\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:16:43.572828 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.572387 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda047-30e9-496c-b854-af4043a83656-registration-dir\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:16:43.572828 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.572416 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqww5\" (UniqueName: \"kubernetes.io/projected/9ceda047-30e9-496c-b854-af4043a83656-kube-api-access-tqww5\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:16:43.572828 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.572444 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-system-cni-dir\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:16:43.572828 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.572474 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8b00a159-66f4-4b9d-bcf1-560094dd69bb-iptables-alerter-script\") pod \"iptables-alerter-xv9wf\" (UID: \"8b00a159-66f4-4b9d-bcf1-560094dd69bb\") " pod="openshift-network-operator/iptables-alerter-xv9wf" Oct 30 01:16:43.572828 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.572531 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jnnc\" (UniqueName: \"kubernetes.io/projected/04631a42-b08e-4aff-9fc4-78dd7392d59d-kube-api-access-9jnnc\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.572828 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.572565 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-host-run-k8s-cni-cncf-io\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.572828 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.572594 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9ceda047-30e9-496c-b854-af4043a83656-etc-selinux\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:16:43.573650 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.572601 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-multus-cni-dir\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.573650 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.572621 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-etc-openvswitch\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.573650 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.572627 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-host-var-lib-cni-bin\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.573650 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.572647 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/03772f27-6bc6-4048-a515-d8d4e9abf431-env-overrides\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.573650 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.572660 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-host-var-lib-cni-multus\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.573650 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.572675 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm4x7\" (UniqueName: \"kubernetes.io/projected/8b00a159-66f4-4b9d-bcf1-560094dd69bb-kube-api-access-fm4x7\") pod \"iptables-alerter-xv9wf\" (UID: \"8b00a159-66f4-4b9d-bcf1-560094dd69bb\") " pod="openshift-network-operator/iptables-alerter-xv9wf" Oct 30 01:16:43.573650 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.572704 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm5kk\" (UniqueName: \"kubernetes.io/projected/a6babafa-f85a-4a1b-9fe9-8bac54198754-kube-api-access-lm5kk\") pod \"node-resolver-4dl5p\" (UID: \"a6babafa-f85a-4a1b-9fe9-8bac54198754\") " pod="openshift-dns/node-resolver-4dl5p" Oct 30 01:16:43.573650 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.572719 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/03772f27-6bc6-4048-a515-d8d4e9abf431-ovnkube-config\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.573650 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.572741 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b97aea6-bab9-498d-bce7-95e2a47c8c05-cni-binary-copy\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.573650 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.572806 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-node-log\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.573650 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.572843 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-cnibin\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:16:43.573650 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.572894 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-cnibin\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:16:43.573650 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.572948 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-run-ovn-kubernetes\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.573650 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.572986 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-node-log\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.573650 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.573329 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4b97aea6-bab9-498d-bce7-95e2a47c8c05-multus-daemon-config\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.573650 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.573375 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/88f5d337-fa16-4430-a82e-5f433354c164-dbus\") pod \"global-pull-secret-syncer-j7l6k\" (UID: \"88f5d337-fa16-4430-a82e-5f433354c164\") " pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:16:43.573650 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.573376 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b97aea6-bab9-498d-bce7-95e2a47c8c05-cni-binary-copy\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.573650 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.573405 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqpz6\" (UniqueName: \"kubernetes.io/projected/f4b29936-72de-4ddc-b7f1-a964a614ff61-kube-api-access-hqpz6\") pod \"network-metrics-daemon-64xhb\" (UID: \"f4b29936-72de-4ddc-b7f1-a964a614ff61\") " pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:16:43.574499 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.573437 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-kubernetes\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.574499 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.573445 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:16:43.574499 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.573463 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-cnibin\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.574499 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.573489 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-host-run-multus-certs\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.574499 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.573500 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-multus-conf-dir\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.574499 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.573514 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-systemd-units\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.574499 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.573574 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-etc-kubernetes\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.574499 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.573805 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-host-run-k8s-cni-cncf-io\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.574499 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.573905 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9ceda047-30e9-496c-b854-af4043a83656-etc-selinux\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:16:43.574499 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.573943 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-system-cni-dir\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:16:43.574499 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.574070 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda047-30e9-496c-b854-af4043a83656-socket-dir\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:16:43.574499 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.574135 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda047-30e9-496c-b854-af4043a83656-registration-dir\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:16:43.574499 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.572391 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-var-lib-kubelet\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.574499 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.574196 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/88f5d337-fa16-4430-a82e-5f433354c164-dbus\") pod \"global-pull-secret-syncer-j7l6k\" (UID: \"88f5d337-fa16-4430-a82e-5f433354c164\") " pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:16:43.574499 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.574467 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-kubernetes\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.574499 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.574487 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8b00a159-66f4-4b9d-bcf1-560094dd69bb-iptables-alerter-script\") pod \"iptables-alerter-xv9wf\" (UID: \"8b00a159-66f4-4b9d-bcf1-560094dd69bb\") " pod="openshift-network-operator/iptables-alerter-xv9wf" Oct 30 01:16:43.575264 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.574521 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-cnibin\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.575264 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.574552 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-etc-openvswitch\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.575264 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.574560 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-host-run-multus-certs\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.575264 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.574589 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/03772f27-6bc6-4048-a515-d8d4e9abf431-ovn-node-metrics-cert\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.575264 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.574596 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-modprobe-d\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.575264 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.574624 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda047-30e9-496c-b854-af4043a83656-device-dir\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:16:43.575264 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.574648 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-slash\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.575264 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.574673 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-run-netns\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.575264 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.574700 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx5fw\" (UniqueName: \"kubernetes.io/projected/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-kube-api-access-sx5fw\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:16:43.575264 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.574723 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-sysctl-conf\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.575264 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.574764 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-run\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.575264 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.574793 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-log-socket\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.575264 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.574817 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:16:43.575264 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.574843 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czkw9\" (UniqueName: \"kubernetes.io/projected/1c5a490f-f8db-4afb-9a1f-b1accc9eadea-kube-api-access-czkw9\") pod \"network-check-target-dmnzr\" (UID: \"1c5a490f-f8db-4afb-9a1f-b1accc9eadea\") " pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:16:43.575264 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.574930 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/03772f27-6bc6-4048-a515-d8d4e9abf431-env-overrides\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.575264 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.574984 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-systemd-units\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.575264 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.575038 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-run-netns\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.576061 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.575116 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-modprobe-d\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.576061 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.575191 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda047-30e9-496c-b854-af4043a83656-device-dir\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:16:43.576061 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.575232 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-slash\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.576061 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.575259 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-sysctl-conf\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.576061 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.575298 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-run\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.576061 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.575323 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-log-socket\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.576061 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.575477 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-run-systemd\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.576061 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.575536 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-host-var-lib-kubelet\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.576061 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.575686 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:16:43.576061 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.575738 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-kubelet\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.576061 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.575763 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-run-openvswitch\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.576061 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.575824 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-sys\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.576061 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.576035 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.576061 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.576034 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/262c01e0-8469-4fbe-aaa4-8a8c50551fce-agent-certs\") pod \"konnectivity-agent-67v6w\" (UID: \"262c01e0-8469-4fbe-aaa4-8a8c50551fce\") " pod="kube-system/konnectivity-agent-67v6w" Oct 30 01:16:43.576715 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.576109 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda047-30e9-496c-b854-af4043a83656-kubelet-dir\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:16:43.576715 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.576138 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-sysctl-d\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.576715 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.576166 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-system-cni-dir\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.576715 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.576201 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9ceda047-30e9-496c-b854-af4043a83656-sys-fs\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:16:43.576951 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.576895 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-tuned\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.578224 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.578202 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04631a42-b08e-4aff-9fc4-78dd7392d59d-tmp\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.591310 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.591276 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46pfs\" (UniqueName: \"kubernetes.io/projected/03772f27-6bc6-4048-a515-d8d4e9abf431-kube-api-access-46pfs\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.592035 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:43.591952 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:16:43.592035 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:43.592035 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:16:43.592212 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:43.592049 2583 projected.go:196] Error preparing data for projected volume kube-api-access-czkw9 for pod openshift-network-diagnostics/network-check-target-dmnzr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:16:43.592212 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:43.592115 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c5a490f-f8db-4afb-9a1f-b1accc9eadea-kube-api-access-czkw9 podName:1c5a490f-f8db-4afb-9a1f-b1accc9eadea nodeName:}" failed. No retries permitted until 2026-10-30 01:16:44.092097267 +0000 UTC m=+3.124388240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-czkw9" (UniqueName: "kubernetes.io/projected/1c5a490f-f8db-4afb-9a1f-b1accc9eadea-kube-api-access-czkw9") pod "network-check-target-dmnzr" (UID: "1c5a490f-f8db-4afb-9a1f-b1accc9eadea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:16:43.593134 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.593110 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6wcg\" (UniqueName: \"kubernetes.io/projected/e7922f23-5ba7-4931-8d19-598e2c4fe041-kube-api-access-z6wcg\") pod \"node-ca-54hlq\" (UID: \"e7922f23-5ba7-4931-8d19-598e2c4fe041\") " pod="openshift-image-registry/node-ca-54hlq" Oct 30 01:16:43.593811 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.593692 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqww5\" (UniqueName: \"kubernetes.io/projected/9ceda047-30e9-496c-b854-af4043a83656-kube-api-access-tqww5\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:16:43.594595 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.594388 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jnnc\" (UniqueName: \"kubernetes.io/projected/04631a42-b08e-4aff-9fc4-78dd7392d59d-kube-api-access-9jnnc\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.595216 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.595195 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqpz6\" (UniqueName: \"kubernetes.io/projected/f4b29936-72de-4ddc-b7f1-a964a614ff61-kube-api-access-hqpz6\") pod \"network-metrics-daemon-64xhb\" (UID: \"f4b29936-72de-4ddc-b7f1-a964a614ff61\") " pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:16:43.598694 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.596235 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm4x7\" (UniqueName: \"kubernetes.io/projected/8b00a159-66f4-4b9d-bcf1-560094dd69bb-kube-api-access-fm4x7\") pod \"iptables-alerter-xv9wf\" (UID: \"8b00a159-66f4-4b9d-bcf1-560094dd69bb\") " pod="openshift-network-operator/iptables-alerter-xv9wf" Oct 30 01:16:43.598694 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.597992 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm5kk\" (UniqueName: \"kubernetes.io/projected/a6babafa-f85a-4a1b-9fe9-8bac54198754-kube-api-access-lm5kk\") pod \"node-resolver-4dl5p\" (UID: \"a6babafa-f85a-4a1b-9fe9-8bac54198754\") " pod="openshift-dns/node-resolver-4dl5p" Oct 30 01:16:43.598694 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.598336 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc85r\" (UniqueName: \"kubernetes.io/projected/4b97aea6-bab9-498d-bce7-95e2a47c8c05-kube-api-access-gc85r\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.599405 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.599382 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx5fw\" (UniqueName: \"kubernetes.io/projected/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-kube-api-access-sx5fw\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:16:43.691403 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.691334 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4dl5p" Oct 30 01:16:43.702410 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:43.702374 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6babafa_f85a_4a1b_9fe9_8bac54198754.slice/crio-f19bfb393c5bc8f29de77632302d48be4377d9623cb8c2a1d62d72b200c4e4e5 WatchSource:0}: Error finding container f19bfb393c5bc8f29de77632302d48be4377d9623cb8c2a1d62d72b200c4e4e5: Status 404 returned error can't find the container with id f19bfb393c5bc8f29de77632302d48be4377d9623cb8c2a1d62d72b200c4e4e5 Oct 30 01:16:43.705056 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.704873 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:16:43.713225 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:43.713193 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04631a42_b08e_4aff_9fc4_78dd7392d59d.slice/crio-090266d1192e81b9761a472f7b7b5867fe0cb83774b48f82646863417326a201 WatchSource:0}: Error finding container 090266d1192e81b9761a472f7b7b5867fe0cb83774b48f82646863417326a201: Status 404 returned error can't find the container with id 090266d1192e81b9761a472f7b7b5867fe0cb83774b48f82646863417326a201 Oct 30 01:16:43.714286 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.714261 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-54hlq" Oct 30 01:16:43.721741 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.721721 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-m8h9g" Oct 30 01:16:43.724480 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:43.724446 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7922f23_5ba7_4931_8d19_598e2c4fe041.slice/crio-a1d852b890b43a7fcf9884a14d90e565e73bf0e32d149b3ef12324d8e15319b8 WatchSource:0}: Error finding container a1d852b890b43a7fcf9884a14d90e565e73bf0e32d149b3ef12324d8e15319b8: Status 404 returned error can't find the container with id a1d852b890b43a7fcf9884a14d90e565e73bf0e32d149b3ef12324d8e15319b8 Oct 30 01:16:43.731102 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.731080 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:16:43.731386 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:43.731354 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b97aea6_bab9_498d_bce7_95e2a47c8c05.slice/crio-2e7b0dcd5c52d7ed1b1ab65a474ce3b34b5a528207d6a62d0901cc211d6c1daf WatchSource:0}: Error finding container 2e7b0dcd5c52d7ed1b1ab65a474ce3b34b5a528207d6a62d0901cc211d6c1daf: Status 404 returned error can't find the container with id 2e7b0dcd5c52d7ed1b1ab65a474ce3b34b5a528207d6a62d0901cc211d6c1daf Oct 30 01:16:43.741234 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:43.741175 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ceda047_30e9_496c_b854_af4043a83656.slice/crio-a89f78f9ff48ec09e65d336ba89ab39eba3f6e47e0e47b47d563d8370fb90a98 WatchSource:0}: Error finding container a89f78f9ff48ec09e65d336ba89ab39eba3f6e47e0e47b47d563d8370fb90a98: Status 404 returned error can't find the container with id a89f78f9ff48ec09e65d336ba89ab39eba3f6e47e0e47b47d563d8370fb90a98 Oct 30 01:16:43.748272 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.748241 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:16:43.755724 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.755698 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:16:43.757436 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.757364 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xv9wf" Oct 30 01:16:43.763316 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:43.763287 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03772f27_6bc6_4048_a515_d8d4e9abf431.slice/crio-6f4aa6724eabbbabd00cb2f9343dc873cf56f83d7f5a7b511923590f68455e50 WatchSource:0}: Error finding container 6f4aa6724eabbbabd00cb2f9343dc873cf56f83d7f5a7b511923590f68455e50: Status 404 returned error can't find the container with id 6f4aa6724eabbbabd00cb2f9343dc873cf56f83d7f5a7b511923590f68455e50 Oct 30 01:16:43.764018 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:43.764000 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-67v6w" Oct 30 01:16:43.771277 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:43.771250 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b00a159_66f4_4b9d_bcf1_560094dd69bb.slice/crio-b2039c18dcbb8325a89eeee1c2021801fdc09d8b18b5cb888aaec7e98a9ee3d7 WatchSource:0}: Error finding container b2039c18dcbb8325a89eeee1c2021801fdc09d8b18b5cb888aaec7e98a9ee3d7: Status 404 returned error can't find the container with id b2039c18dcbb8325a89eeee1c2021801fdc09d8b18b5cb888aaec7e98a9ee3d7 Oct 30 01:16:43.772274 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:43.772245 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba9644c4_ec1e_4bb8_ae9a_d47acaa0dbb2.slice/crio-27a1569120759fae8a09aba518f9bcfd8599fa09d5a4a3d906ef8ac73a041c05 WatchSource:0}: Error finding container 27a1569120759fae8a09aba518f9bcfd8599fa09d5a4a3d906ef8ac73a041c05: Status 404 returned error can't find the container with id 27a1569120759fae8a09aba518f9bcfd8599fa09d5a4a3d906ef8ac73a041c05 Oct 30 01:16:43.785059 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:16:43.785024 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod262c01e0_8469_4fbe_aaa4_8a8c50551fce.slice/crio-2a28007d9fd8ecf4d20c4694935fff740ce2fa87946bb4f514fab6cbcef3b920 WatchSource:0}: Error finding container 2a28007d9fd8ecf4d20c4694935fff740ce2fa87946bb4f514fab6cbcef3b920: Status 404 returned error can't find the container with id 2a28007d9fd8ecf4d20c4694935fff740ce2fa87946bb4f514fab6cbcef3b920 Oct 30 01:16:44.078602 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:44.078578 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b29936-72de-4ddc-b7f1-a964a614ff61-metrics-certs\") pod \"network-metrics-daemon-64xhb\" (UID: \"f4b29936-72de-4ddc-b7f1-a964a614ff61\") " pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:16:44.078714 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:44.078611 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/88f5d337-fa16-4430-a82e-5f433354c164-original-pull-secret\") pod \"global-pull-secret-syncer-j7l6k\" (UID: \"88f5d337-fa16-4430-a82e-5f433354c164\") " pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:16:44.078797 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:44.078721 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:16:44.078848 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:44.078737 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:16:44.078899 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:44.078876 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4b29936-72de-4ddc-b7f1-a964a614ff61-metrics-certs podName:f4b29936-72de-4ddc-b7f1-a964a614ff61 nodeName:}" failed. No retries permitted until 2026-10-30 01:16:45.078852573 +0000 UTC m=+4.111143549 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f4b29936-72de-4ddc-b7f1-a964a614ff61-metrics-certs") pod "network-metrics-daemon-64xhb" (UID: "f4b29936-72de-4ddc-b7f1-a964a614ff61") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:16:44.078951 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:44.078899 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88f5d337-fa16-4430-a82e-5f433354c164-original-pull-secret podName:88f5d337-fa16-4430-a82e-5f433354c164 nodeName:}" failed. No retries permitted until 2026-10-30 01:16:45.078888127 +0000 UTC m=+4.111179092 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/88f5d337-fa16-4430-a82e-5f433354c164-original-pull-secret") pod "global-pull-secret-syncer-j7l6k" (UID: "88f5d337-fa16-4430-a82e-5f433354c164") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:16:44.179303 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:44.179075 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czkw9\" (UniqueName: \"kubernetes.io/projected/1c5a490f-f8db-4afb-9a1f-b1accc9eadea-kube-api-access-czkw9\") pod \"network-check-target-dmnzr\" (UID: \"1c5a490f-f8db-4afb-9a1f-b1accc9eadea\") " pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:16:44.179512 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:44.179494 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:16:44.179608 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:44.179519 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:16:44.179608 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:44.179532 2583 projected.go:196] Error preparing data for projected volume kube-api-access-czkw9 for pod openshift-network-diagnostics/network-check-target-dmnzr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:16:44.179608 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:44.179585 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c5a490f-f8db-4afb-9a1f-b1accc9eadea-kube-api-access-czkw9 podName:1c5a490f-f8db-4afb-9a1f-b1accc9eadea nodeName:}" failed. No retries permitted until 2026-10-30 01:16:45.179568495 +0000 UTC m=+4.211859458 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-czkw9" (UniqueName: "kubernetes.io/projected/1c5a490f-f8db-4afb-9a1f-b1accc9eadea-kube-api-access-czkw9") pod "network-check-target-dmnzr" (UID: "1c5a490f-f8db-4afb-9a1f-b1accc9eadea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:16:44.406159 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:44.406048 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 01:11:43 +0000 UTC" deadline="2028-07-13 09:50:04.91497395 +0000 UTC" Oct 30 01:16:44.406159 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:44.406108 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14936h33m20.508869851s" Oct 30 01:16:44.505568 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:44.505524 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xv9wf" event={"ID":"8b00a159-66f4-4b9d-bcf1-560094dd69bb","Type":"ContainerStarted","Data":"b2039c18dcbb8325a89eeee1c2021801fdc09d8b18b5cb888aaec7e98a9ee3d7"} Oct 30 01:16:44.516252 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:44.516157 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" event={"ID":"9ceda047-30e9-496c-b854-af4043a83656","Type":"ContainerStarted","Data":"a89f78f9ff48ec09e65d336ba89ab39eba3f6e47e0e47b47d563d8370fb90a98"} Oct 30 01:16:44.523522 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:44.523428 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-m8h9g" event={"ID":"4b97aea6-bab9-498d-bce7-95e2a47c8c05","Type":"ContainerStarted","Data":"2e7b0dcd5c52d7ed1b1ab65a474ce3b34b5a528207d6a62d0901cc211d6c1daf"} Oct 30 01:16:44.531507 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:44.531409 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4dl5p" event={"ID":"a6babafa-f85a-4a1b-9fe9-8bac54198754","Type":"ContainerStarted","Data":"f19bfb393c5bc8f29de77632302d48be4377d9623cb8c2a1d62d72b200c4e4e5"} Oct 30 01:16:44.535858 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:44.535100 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal" event={"ID":"129d2d6479043c5ffd993e1a7481e29a","Type":"ContainerStarted","Data":"6ff6924d1949f8a7539a748cd3e6453202f29f76629a80f14200e88a7ff0b70f"} Oct 30 01:16:44.542663 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:44.542543 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" event={"ID":"03772f27-6bc6-4048-a515-d8d4e9abf431","Type":"ContainerStarted","Data":"6f4aa6724eabbbabd00cb2f9343dc873cf56f83d7f5a7b511923590f68455e50"} Oct 30 01:16:44.558162 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:44.558103 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-54hlq" event={"ID":"e7922f23-5ba7-4931-8d19-598e2c4fe041","Type":"ContainerStarted","Data":"a1d852b890b43a7fcf9884a14d90e565e73bf0e32d149b3ef12324d8e15319b8"} Oct 30 01:16:44.565845 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:44.565760 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" event={"ID":"04631a42-b08e-4aff-9fc4-78dd7392d59d","Type":"ContainerStarted","Data":"090266d1192e81b9761a472f7b7b5867fe0cb83774b48f82646863417326a201"} Oct 30 01:16:44.572438 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:44.572408 2583 generic.go:356] "Generic (PLEG): container finished" podID="b222e074fa1fff6b3c88cd830a3f1d7d" containerID="a21211a8ecfda058796f06998ebf0b00316e2b4441b7fd62b0d237432acd4d62" exitCode=0 Oct 30 01:16:44.572702 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:44.572624 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal" event={"ID":"b222e074fa1fff6b3c88cd830a3f1d7d","Type":"ContainerDied","Data":"a21211a8ecfda058796f06998ebf0b00316e2b4441b7fd62b0d237432acd4d62"} Oct 30 01:16:44.578280 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:44.578231 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal" podStartSLOduration=2.578215503 podStartE2EDuration="2.578215503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:44.577366013 +0000 UTC m=+3.609656995" watchObservedRunningTime="2026-10-30 01:16:44.578215503 +0000 UTC m=+3.610506485" Oct 30 01:16:44.585554 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:44.585523 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-67v6w" event={"ID":"262c01e0-8469-4fbe-aaa4-8a8c50551fce","Type":"ContainerStarted","Data":"2a28007d9fd8ecf4d20c4694935fff740ce2fa87946bb4f514fab6cbcef3b920"} Oct 30 01:16:44.602838 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:44.602720 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5wdsg" event={"ID":"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2","Type":"ContainerStarted","Data":"27a1569120759fae8a09aba518f9bcfd8599fa09d5a4a3d906ef8ac73a041c05"} Oct 30 01:16:45.087566 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:45.087471 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b29936-72de-4ddc-b7f1-a964a614ff61-metrics-certs\") pod \"network-metrics-daemon-64xhb\" (UID: \"f4b29936-72de-4ddc-b7f1-a964a614ff61\") " pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:16:45.087566 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:45.087526 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/88f5d337-fa16-4430-a82e-5f433354c164-original-pull-secret\") pod \"global-pull-secret-syncer-j7l6k\" (UID: \"88f5d337-fa16-4430-a82e-5f433354c164\") " pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:16:45.087839 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:45.087686 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:16:45.087839 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:45.087737 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88f5d337-fa16-4430-a82e-5f433354c164-original-pull-secret podName:88f5d337-fa16-4430-a82e-5f433354c164 nodeName:}" failed. No retries permitted until 2026-10-30 01:16:47.087723913 +0000 UTC m=+6.120014873 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/88f5d337-fa16-4430-a82e-5f433354c164-original-pull-secret") pod "global-pull-secret-syncer-j7l6k" (UID: "88f5d337-fa16-4430-a82e-5f433354c164") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:16:45.088085 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:45.088070 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:16:45.088146 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:45.088110 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4b29936-72de-4ddc-b7f1-a964a614ff61-metrics-certs podName:f4b29936-72de-4ddc-b7f1-a964a614ff61 nodeName:}" failed. No retries permitted until 2026-10-30 01:16:47.088100107 +0000 UTC m=+6.120391065 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f4b29936-72de-4ddc-b7f1-a964a614ff61-metrics-certs") pod "network-metrics-daemon-64xhb" (UID: "f4b29936-72de-4ddc-b7f1-a964a614ff61") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:16:45.188521 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:45.188482 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czkw9\" (UniqueName: \"kubernetes.io/projected/1c5a490f-f8db-4afb-9a1f-b1accc9eadea-kube-api-access-czkw9\") pod \"network-check-target-dmnzr\" (UID: \"1c5a490f-f8db-4afb-9a1f-b1accc9eadea\") " pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:16:45.188730 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:45.188656 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:16:45.188730 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:45.188697 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:16:45.188730 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:45.188710 2583 projected.go:196] Error preparing data for projected volume kube-api-access-czkw9 for pod openshift-network-diagnostics/network-check-target-dmnzr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:16:45.188921 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:45.188794 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c5a490f-f8db-4afb-9a1f-b1accc9eadea-kube-api-access-czkw9 podName:1c5a490f-f8db-4afb-9a1f-b1accc9eadea nodeName:}" failed. No retries permitted until 2026-10-30 01:16:47.188777197 +0000 UTC m=+6.221068161 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-czkw9" (UniqueName: "kubernetes.io/projected/1c5a490f-f8db-4afb-9a1f-b1accc9eadea-kube-api-access-czkw9") pod "network-check-target-dmnzr" (UID: "1c5a490f-f8db-4afb-9a1f-b1accc9eadea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:16:45.407315 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:45.407201 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 01:11:43 +0000 UTC" deadline="2028-07-02 17:58:53.832740705 +0000 UTC" Oct 30 01:16:45.407315 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:45.407270 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14680h42m8.425475882s" Oct 30 01:16:45.485534 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:45.484614 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:16:45.485534 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:45.484786 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7l6k" podUID="88f5d337-fa16-4430-a82e-5f433354c164" Oct 30 01:16:45.485534 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:45.485195 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:16:45.485534 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:45.485302 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmnzr" podUID="1c5a490f-f8db-4afb-9a1f-b1accc9eadea" Oct 30 01:16:45.485534 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:45.485386 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:16:45.485534 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:45.485472 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64xhb" podUID="f4b29936-72de-4ddc-b7f1-a964a614ff61" Oct 30 01:16:45.631725 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:45.631655 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal" event={"ID":"b222e074fa1fff6b3c88cd830a3f1d7d","Type":"ContainerStarted","Data":"a8bd72ff55c6dc61f27c0ed60e4e5b8cc138b011e4ee066c7d411b98098fbb4d"} Oct 30 01:16:45.670077 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:45.669102 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal" podStartSLOduration=3.669083575 podStartE2EDuration="3.669083575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:45.668870632 +0000 UTC m=+4.701161617" watchObservedRunningTime="2026-10-30 01:16:45.669083575 +0000 UTC m=+4.701374557" Oct 30 01:16:46.639140 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:46.639097 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5wdsg" event={"ID":"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2","Type":"ContainerStarted","Data":"ae1e4e6158ca6d91aef2ee5636da23e0f68e574f7bc2ddc22994e5efe2f25ee9"} Oct 30 01:16:47.102659 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:47.102613 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b29936-72de-4ddc-b7f1-a964a614ff61-metrics-certs\") pod \"network-metrics-daemon-64xhb\" (UID: \"f4b29936-72de-4ddc-b7f1-a964a614ff61\") " pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:16:47.102909 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:47.102679 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/88f5d337-fa16-4430-a82e-5f433354c164-original-pull-secret\") pod \"global-pull-secret-syncer-j7l6k\" (UID: \"88f5d337-fa16-4430-a82e-5f433354c164\") " pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:16:47.102909 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:47.102878 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:16:47.103047 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:47.102950 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88f5d337-fa16-4430-a82e-5f433354c164-original-pull-secret podName:88f5d337-fa16-4430-a82e-5f433354c164 nodeName:}" failed. No retries permitted until 2026-10-30 01:16:51.102929126 +0000 UTC m=+10.135220089 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/88f5d337-fa16-4430-a82e-5f433354c164-original-pull-secret") pod "global-pull-secret-syncer-j7l6k" (UID: "88f5d337-fa16-4430-a82e-5f433354c164") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:16:47.103358 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:47.103336 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:16:47.103440 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:47.103394 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4b29936-72de-4ddc-b7f1-a964a614ff61-metrics-certs podName:f4b29936-72de-4ddc-b7f1-a964a614ff61 nodeName:}" failed. No retries permitted until 2026-10-30 01:16:51.103381431 +0000 UTC m=+10.135672390 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f4b29936-72de-4ddc-b7f1-a964a614ff61-metrics-certs") pod "network-metrics-daemon-64xhb" (UID: "f4b29936-72de-4ddc-b7f1-a964a614ff61") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:16:47.203867 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:47.203830 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czkw9\" (UniqueName: \"kubernetes.io/projected/1c5a490f-f8db-4afb-9a1f-b1accc9eadea-kube-api-access-czkw9\") pod \"network-check-target-dmnzr\" (UID: \"1c5a490f-f8db-4afb-9a1f-b1accc9eadea\") " pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:16:47.204043 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:47.204014 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:16:47.204043 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:47.204033 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:16:47.204162 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:47.204049 2583 projected.go:196] Error preparing data for projected volume kube-api-access-czkw9 for pod openshift-network-diagnostics/network-check-target-dmnzr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:16:47.204162 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:47.204106 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c5a490f-f8db-4afb-9a1f-b1accc9eadea-kube-api-access-czkw9 podName:1c5a490f-f8db-4afb-9a1f-b1accc9eadea nodeName:}" failed. No retries permitted until 2026-10-30 01:16:51.20409253 +0000 UTC m=+10.236383492 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-czkw9" (UniqueName: "kubernetes.io/projected/1c5a490f-f8db-4afb-9a1f-b1accc9eadea-kube-api-access-czkw9") pod "network-check-target-dmnzr" (UID: "1c5a490f-f8db-4afb-9a1f-b1accc9eadea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:16:47.489028 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:47.488997 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:16:47.489396 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:47.489141 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64xhb" podUID="f4b29936-72de-4ddc-b7f1-a964a614ff61" Oct 30 01:16:47.489505 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:47.489483 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:16:47.489603 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:47.489582 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7l6k" podUID="88f5d337-fa16-4430-a82e-5f433354c164" Oct 30 01:16:47.489658 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:47.489629 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:16:47.489707 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:47.489689 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmnzr" podUID="1c5a490f-f8db-4afb-9a1f-b1accc9eadea" Oct 30 01:16:47.650137 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:47.650056 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-67v6w" event={"ID":"262c01e0-8469-4fbe-aaa4-8a8c50551fce","Type":"ContainerStarted","Data":"1a39c8866ec51a34a59fb487e5d6de6744b6994d85a3ee884a7b0f5a36fdbf9d"} Oct 30 01:16:47.652565 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:47.652533 2583 generic.go:356] "Generic (PLEG): container finished" podID="ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2" containerID="ae1e4e6158ca6d91aef2ee5636da23e0f68e574f7bc2ddc22994e5efe2f25ee9" exitCode=0 Oct 30 01:16:47.652684 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:47.652585 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5wdsg" event={"ID":"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2","Type":"ContainerDied","Data":"ae1e4e6158ca6d91aef2ee5636da23e0f68e574f7bc2ddc22994e5efe2f25ee9"} Oct 30 01:16:47.741623 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:47.741539 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-67v6w" podStartSLOduration=3.261629026 podStartE2EDuration="6.74151987s" totalImagesPullingTime="3.479890844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:47.691449297 +0000 UTC m=+6.723740273" watchObservedRunningTime="2026-10-30 01:16:47.74151987 +0000 UTC m=+6.773810852" Oct 30 01:16:48.199909 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:48.199867 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-67v6w" Oct 30 01:16:49.068940 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:49.068861 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-67v6w" Oct 30 01:16:49.069414 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:49.069307 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-67v6w" Oct 30 01:16:49.485334 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:49.484893 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:16:49.485334 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:49.485007 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7l6k" podUID="88f5d337-fa16-4430-a82e-5f433354c164" Oct 30 01:16:49.485334 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:49.485085 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:16:49.485334 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:49.485167 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmnzr" podUID="1c5a490f-f8db-4afb-9a1f-b1accc9eadea" Oct 30 01:16:49.485334 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:49.485192 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:16:49.485334 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:49.485264 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64xhb" podUID="f4b29936-72de-4ddc-b7f1-a964a614ff61" Oct 30 01:16:49.666455 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:49.666351 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" event={"ID":"9ceda047-30e9-496c-b854-af4043a83656","Type":"ContainerStarted","Data":"757f7866bb2bb357f6aa38a6041254243986a52069cacd0c2413e01852f3302f"} Oct 30 01:16:49.670538 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:49.670506 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-54hlq" event={"ID":"e7922f23-5ba7-4931-8d19-598e2c4fe041","Type":"ContainerStarted","Data":"7157ba897d0522bccbbb8966a950ebc33f6f50a4fac013b19a0f3ceaffab7148"} Oct 30 01:16:49.671206 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:49.671161 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-67v6w" Oct 30 01:16:49.760498 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:49.759887 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-54hlq" podStartSLOduration=3.689323907 podStartE2EDuration="8.759865124s" totalImagesPullingTime="5.070541217s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:49.715089946 +0000 UTC m=+8.747380929" watchObservedRunningTime="2026-10-30 01:16:49.759865124 +0000 UTC m=+8.792156114" Oct 30 01:16:51.137693 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:51.137650 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b29936-72de-4ddc-b7f1-a964a614ff61-metrics-certs\") pod \"network-metrics-daemon-64xhb\" (UID: \"f4b29936-72de-4ddc-b7f1-a964a614ff61\") " pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:16:51.138188 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:51.137710 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/88f5d337-fa16-4430-a82e-5f433354c164-original-pull-secret\") pod \"global-pull-secret-syncer-j7l6k\" (UID: \"88f5d337-fa16-4430-a82e-5f433354c164\") " pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:16:51.138188 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:51.137887 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:16:51.138188 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:51.137947 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88f5d337-fa16-4430-a82e-5f433354c164-original-pull-secret podName:88f5d337-fa16-4430-a82e-5f433354c164 nodeName:}" failed. No retries permitted until 2026-10-30 01:16:59.137930004 +0000 UTC m=+18.170220967 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/88f5d337-fa16-4430-a82e-5f433354c164-original-pull-secret") pod "global-pull-secret-syncer-j7l6k" (UID: "88f5d337-fa16-4430-a82e-5f433354c164") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:16:51.138344 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:51.138317 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:16:51.138391 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:51.138360 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4b29936-72de-4ddc-b7f1-a964a614ff61-metrics-certs podName:f4b29936-72de-4ddc-b7f1-a964a614ff61 nodeName:}" failed. No retries permitted until 2026-10-30 01:16:59.138346872 +0000 UTC m=+18.170637847 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f4b29936-72de-4ddc-b7f1-a964a614ff61-metrics-certs") pod "network-metrics-daemon-64xhb" (UID: "f4b29936-72de-4ddc-b7f1-a964a614ff61") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:16:51.239002 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:51.238968 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czkw9\" (UniqueName: \"kubernetes.io/projected/1c5a490f-f8db-4afb-9a1f-b1accc9eadea-kube-api-access-czkw9\") pod \"network-check-target-dmnzr\" (UID: \"1c5a490f-f8db-4afb-9a1f-b1accc9eadea\") " pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:16:51.239157 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:51.239138 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:16:51.239232 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:51.239164 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:16:51.239232 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:51.239178 2583 projected.go:196] Error preparing data for projected volume kube-api-access-czkw9 for pod openshift-network-diagnostics/network-check-target-dmnzr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:16:51.239296 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:51.239237 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c5a490f-f8db-4afb-9a1f-b1accc9eadea-kube-api-access-czkw9 podName:1c5a490f-f8db-4afb-9a1f-b1accc9eadea nodeName:}" failed. No retries permitted until 2026-10-30 01:16:59.239218324 +0000 UTC m=+18.271509289 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-czkw9" (UniqueName: "kubernetes.io/projected/1c5a490f-f8db-4afb-9a1f-b1accc9eadea-kube-api-access-czkw9") pod "network-check-target-dmnzr" (UID: "1c5a490f-f8db-4afb-9a1f-b1accc9eadea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:16:51.267871 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:51.267837 2583 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 01:16:51.425743 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:51.425542 2583 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T01:16:51.267854461Z","UUID":"5997e81d-1127-4787-8fc0-a2c552b907c4","Handler":null,"Name":"","Endpoint":""} Oct 30 01:16:51.427407 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:51.427384 2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 01:16:51.427526 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:51.427431 2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 01:16:51.482547 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:51.482513 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:16:51.482701 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:51.482666 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64xhb" podUID="f4b29936-72de-4ddc-b7f1-a964a614ff61" Oct 30 01:16:51.482789 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:51.482772 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:16:51.482893 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:51.482858 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7l6k" podUID="88f5d337-fa16-4430-a82e-5f433354c164" Oct 30 01:16:51.482953 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:51.482921 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:16:51.483001 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:51.482989 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmnzr" podUID="1c5a490f-f8db-4afb-9a1f-b1accc9eadea" Oct 30 01:16:51.682738 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:51.682655 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" event={"ID":"9ceda047-30e9-496c-b854-af4043a83656","Type":"ContainerStarted","Data":"48ad875f22933c1c301982d8659746b4f4d522e614ca397dc569e7c61cb570b0"} Oct 30 01:16:51.685684 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:51.685638 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-m8h9g" event={"ID":"4b97aea6-bab9-498d-bce7-95e2a47c8c05","Type":"ContainerStarted","Data":"d93a0f3ee6e436c257ec325229cae841cc1fbacb4373f8f1a71e998baf1465f9"} Oct 30 01:16:51.729993 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:51.729925 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-m8h9g" podStartSLOduration=2.993589506 podStartE2EDuration="10.729910256s" totalImagesPullingTime="7.73632075s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:51.729250926 +0000 UTC m=+10.761541909" watchObservedRunningTime="2026-10-30 01:16:51.729910256 +0000 UTC m=+10.762201238" Oct 30 01:16:52.691706 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:52.691458 2583 generic.go:356] "Generic (PLEG): container finished" podID="ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2" containerID="42f27733be2f82c99d03d96478b5b25cdb5855f889c8c1211bb23a75d34a55ee" exitCode=0 Oct 30 01:16:52.692194 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:52.691568 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5wdsg" event={"ID":"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2","Type":"ContainerDied","Data":"42f27733be2f82c99d03d96478b5b25cdb5855f889c8c1211bb23a75d34a55ee"} Oct 30 01:16:52.695117 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:52.695088 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" event={"ID":"9ceda047-30e9-496c-b854-af4043a83656","Type":"ContainerStarted","Data":"b1175a444421365dcd3872a7377d1265d2b32f2e00534f581522c1d744768d58"} Oct 30 01:16:52.697698 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:52.697674 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4dl5p" event={"ID":"a6babafa-f85a-4a1b-9fe9-8bac54198754","Type":"ContainerStarted","Data":"586c4975254d6638126badea67f49f902548bdebfe1f5389fda85a2da74d7e64"} Oct 30 01:16:52.781263 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:52.781170 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-4dl5p" podStartSLOduration=3.85447074 podStartE2EDuration="11.781153089s" totalImagesPullingTime="7.926682349s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:52.780597687 +0000 UTC m=+11.812888668" watchObservedRunningTime="2026-10-30 01:16:52.781153089 +0000 UTC m=+11.813444071" Oct 30 01:16:52.821334 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:52.821276 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" podStartSLOduration=3.5421410460000002 podStartE2EDuration="11.821259207s" totalImagesPullingTime="8.279118161s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:16:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:52.820566177 +0000 UTC m=+11.852857162" watchObservedRunningTime="2026-10-30 01:16:52.821259207 +0000 UTC m=+11.853550189" Oct 30 01:16:53.485148 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:53.485116 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:16:53.485148 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:53.485149 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:16:53.485337 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:53.485156 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:16:53.485337 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:53.485229 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmnzr" podUID="1c5a490f-f8db-4afb-9a1f-b1accc9eadea" Oct 30 01:16:53.485337 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:53.485285 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7l6k" podUID="88f5d337-fa16-4430-a82e-5f433354c164" Oct 30 01:16:53.485465 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:53.485365 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64xhb" podUID="f4b29936-72de-4ddc-b7f1-a964a614ff61" Oct 30 01:16:53.701463 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:53.701415 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xv9wf" event={"ID":"8b00a159-66f4-4b9d-bcf1-560094dd69bb","Type":"ContainerStarted","Data":"d685c0eeeaded82176edcc13080fb93d39b50448038ddf4c76af504f9d0e3f28"} Oct 30 01:16:53.738983 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:53.738890 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xv9wf" podStartSLOduration=4.988774448 podStartE2EDuration="12.73887093s" totalImagesPullingTime="7.750096482s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:53.738576714 +0000 UTC m=+12.770867691" watchObservedRunningTime="2026-10-30 01:16:53.73887093 +0000 UTC m=+12.771161914" Oct 30 01:16:54.705950 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:54.705914 2583 generic.go:356] "Generic (PLEG): container finished" podID="ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2" containerID="6e9f4186c23bd5392346aa525b7164dc3dee4b04bdf92be033d606f7596d0ec2" exitCode=0 Oct 30 01:16:54.706673 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:54.706011 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5wdsg" event={"ID":"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2","Type":"ContainerDied","Data":"6e9f4186c23bd5392346aa525b7164dc3dee4b04bdf92be033d606f7596d0ec2"} Oct 30 01:16:54.707897 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:54.707859 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" event={"ID":"04631a42-b08e-4aff-9fc4-78dd7392d59d","Type":"ContainerStarted","Data":"3bcd3c8370e3c49fe9bbd98fdb719b414514622d8c7d2091fe1cf6e8364a2e45"} Oct 30 01:16:54.822463 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:54.822396 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" podStartSLOduration=3.746569888 podStartE2EDuration="13.822375797s" totalImagesPullingTime="10.075805909s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:54.821453629 +0000 UTC m=+13.853744610" watchObservedRunningTime="2026-10-30 01:16:54.822375797 +0000 UTC m=+13.854666780" Oct 30 01:16:55.483843 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:55.482855 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:16:55.483843 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:55.482897 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:16:55.483843 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:55.482858 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:16:55.483843 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:55.482991 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmnzr" podUID="1c5a490f-f8db-4afb-9a1f-b1accc9eadea" Oct 30 01:16:55.483843 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:55.483091 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64xhb" podUID="f4b29936-72de-4ddc-b7f1-a964a614ff61" Oct 30 01:16:55.483843 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:55.483201 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7l6k" podUID="88f5d337-fa16-4430-a82e-5f433354c164" Oct 30 01:16:55.712790 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:55.712575 2583 generic.go:356] "Generic (PLEG): container finished" podID="ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2" containerID="5cebc9daa6072033ea315d8df77b9800bb019244088ca3cfcf0a1411ea202a12" exitCode=0 Oct 30 01:16:55.713226 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:55.712670 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5wdsg" event={"ID":"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2","Type":"ContainerDied","Data":"5cebc9daa6072033ea315d8df77b9800bb019244088ca3cfcf0a1411ea202a12"} Oct 30 01:16:57.482656 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:57.482629 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:16:57.483090 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:57.482666 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:16:57.483090 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:57.482630 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:16:57.483090 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:57.482777 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7l6k" podUID="88f5d337-fa16-4430-a82e-5f433354c164" Oct 30 01:16:57.483090 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:57.482869 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmnzr" podUID="1c5a490f-f8db-4afb-9a1f-b1accc9eadea" Oct 30 01:16:57.483090 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:57.482950 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64xhb" podUID="f4b29936-72de-4ddc-b7f1-a964a614ff61" Oct 30 01:16:57.719633 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:57.719547 2583 generic.go:356] "Generic (PLEG): container finished" podID="ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2" containerID="264f25e08b0d9ad778d6058b1c16806d36cbf76db794849973247a5cefd19f87" exitCode=0 Oct 30 01:16:57.719633 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:57.719601 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5wdsg" event={"ID":"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2","Type":"ContainerDied","Data":"264f25e08b0d9ad778d6058b1c16806d36cbf76db794849973247a5cefd19f87"} Oct 30 01:16:58.725207 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:58.724961 2583 generic.go:356] "Generic (PLEG): container finished" podID="ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2" containerID="eb3e06c8feb81e01ff8c945c440fc1ea63d771c4f9c11b04b035dfafd5451ff8" exitCode=0 Oct 30 01:16:58.725877 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:58.725047 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5wdsg" event={"ID":"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2","Type":"ContainerDied","Data":"eb3e06c8feb81e01ff8c945c440fc1ea63d771c4f9c11b04b035dfafd5451ff8"} Oct 30 01:16:58.728600 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:58.728570 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" event={"ID":"03772f27-6bc6-4048-a515-d8d4e9abf431","Type":"ContainerStarted","Data":"9ec71a7b7c1e38f66f91c259c26761feaa0310730d1184714e33c0ee64351cc4"} Oct 30 01:16:58.728674 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:58.728657 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" event={"ID":"03772f27-6bc6-4048-a515-d8d4e9abf431","Type":"ContainerStarted","Data":"74fca7942ec8c4e891cca433fb32afc8fd0b3e414255664a17980206b85f2891"} Oct 30 01:16:58.728729 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:58.728712 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" event={"ID":"03772f27-6bc6-4048-a515-d8d4e9abf431","Type":"ContainerStarted","Data":"92c332ec68da1dc25a5624c6e5dae689a0cacace8731e2d9fb33bd5c0941f87e"} Oct 30 01:16:58.728792 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:58.728778 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" event={"ID":"03772f27-6bc6-4048-a515-d8d4e9abf431","Type":"ContainerStarted","Data":"4515f75930d17f704e443c62f85c3d3e05b55d64706e3241cfe63d1b6e856316"} Oct 30 01:16:58.728834 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:58.728822 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" event={"ID":"03772f27-6bc6-4048-a515-d8d4e9abf431","Type":"ContainerStarted","Data":"5fffa1d25095875f3426ae304d89ea5eb878225ca14813c42f0d20e3b390e16d"} Oct 30 01:16:58.728881 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:58.728869 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" event={"ID":"03772f27-6bc6-4048-a515-d8d4e9abf431","Type":"ContainerStarted","Data":"88f53d9a1c4e203122ddf65a8b252c4fbe961829b0d3e303ad889edf2f790ca8"} Oct 30 01:16:59.205023 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:59.204927 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b29936-72de-4ddc-b7f1-a964a614ff61-metrics-certs\") pod \"network-metrics-daemon-64xhb\" (UID: \"f4b29936-72de-4ddc-b7f1-a964a614ff61\") " pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:16:59.205023 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:59.204968 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/88f5d337-fa16-4430-a82e-5f433354c164-original-pull-secret\") pod \"global-pull-secret-syncer-j7l6k\" (UID: \"88f5d337-fa16-4430-a82e-5f433354c164\") " pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:16:59.205183 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:59.205077 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:16:59.205183 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:59.205080 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:16:59.205183 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:59.205131 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88f5d337-fa16-4430-a82e-5f433354c164-original-pull-secret podName:88f5d337-fa16-4430-a82e-5f433354c164 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:15.205116599 +0000 UTC m=+34.237407562 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/88f5d337-fa16-4430-a82e-5f433354c164-original-pull-secret") pod "global-pull-secret-syncer-j7l6k" (UID: "88f5d337-fa16-4430-a82e-5f433354c164") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:16:59.205183 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:59.205146 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4b29936-72de-4ddc-b7f1-a964a614ff61-metrics-certs podName:f4b29936-72de-4ddc-b7f1-a964a614ff61 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:15.205139471 +0000 UTC m=+34.237430430 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f4b29936-72de-4ddc-b7f1-a964a614ff61-metrics-certs") pod "network-metrics-daemon-64xhb" (UID: "f4b29936-72de-4ddc-b7f1-a964a614ff61") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:16:59.305648 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:59.305613 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czkw9\" (UniqueName: \"kubernetes.io/projected/1c5a490f-f8db-4afb-9a1f-b1accc9eadea-kube-api-access-czkw9\") pod \"network-check-target-dmnzr\" (UID: \"1c5a490f-f8db-4afb-9a1f-b1accc9eadea\") " pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:16:59.305833 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:59.305792 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:16:59.305833 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:59.305810 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:16:59.305833 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:59.305819 2583 projected.go:196] Error preparing data for projected volume kube-api-access-czkw9 for pod openshift-network-diagnostics/network-check-target-dmnzr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:16:59.305925 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:59.305866 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c5a490f-f8db-4afb-9a1f-b1accc9eadea-kube-api-access-czkw9 podName:1c5a490f-f8db-4afb-9a1f-b1accc9eadea nodeName:}" failed. No retries permitted until 2026-10-30 01:17:15.305854116 +0000 UTC m=+34.338145088 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-czkw9" (UniqueName: "kubernetes.io/projected/1c5a490f-f8db-4afb-9a1f-b1accc9eadea-kube-api-access-czkw9") pod "network-check-target-dmnzr" (UID: "1c5a490f-f8db-4afb-9a1f-b1accc9eadea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:16:59.481873 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:59.481784 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:16:59.481873 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:59.481806 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:16:59.482047 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:59.481795 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:16:59.482047 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:59.481889 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmnzr" podUID="1c5a490f-f8db-4afb-9a1f-b1accc9eadea" Oct 30 01:16:59.482047 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:59.481987 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64xhb" podUID="f4b29936-72de-4ddc-b7f1-a964a614ff61" Oct 30 01:16:59.482137 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:16:59.482051 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7l6k" podUID="88f5d337-fa16-4430-a82e-5f433354c164" Oct 30 01:16:59.734477 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:59.734395 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5wdsg" event={"ID":"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2","Type":"ContainerStarted","Data":"cc2522964f4bd78018a204916c832c7601ec8096e8087e83f87a01b3ab5aea8e"} Oct 30 01:16:59.779089 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:16:59.779045 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5wdsg" podStartSLOduration=8.327883143 podStartE2EDuration="18.779030671s" totalImagesPullingTime="10.451147528s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:16:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:59.77858629 +0000 UTC m=+18.810877271" watchObservedRunningTime="2026-10-30 01:16:59.779030671 +0000 UTC m=+18.811321651" Oct 30 01:17:00.739914 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:00.739879 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" event={"ID":"03772f27-6bc6-4048-a515-d8d4e9abf431","Type":"ContainerStarted","Data":"46ec246102e590d269e415b12cb103299f2040cb693b6dc5c3fd1c0730b3fe7a"} Oct 30 01:17:01.378357 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:01.378321 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:01.488261 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:01.488230 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:17:01.488406 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:01.488270 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:17:01.488406 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:01.488230 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:17:01.488406 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:01.488352 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7l6k" podUID="88f5d337-fa16-4430-a82e-5f433354c164" Oct 30 01:17:01.488541 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:01.488406 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmnzr" podUID="1c5a490f-f8db-4afb-9a1f-b1accc9eadea" Oct 30 01:17:01.488541 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:01.488472 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64xhb" podUID="f4b29936-72de-4ddc-b7f1-a964a614ff61" Oct 30 01:17:01.803375 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:01.803285 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:02.141158 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:02.141064 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:02.748427 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:02.748388 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" event={"ID":"03772f27-6bc6-4048-a515-d8d4e9abf431","Type":"ContainerStarted","Data":"9d16fec22c91ddc8b18e0606cec6347c366ae691e738d214f4c3e657bf8dd353"} Oct 30 01:17:02.748664 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:02.748645 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:02.748823 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:02.748790 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:02.748915 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:02.748875 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:02.765258 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:02.765224 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:02.767189 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:02.767166 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:02.830052 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:02.829849 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" podStartSLOduration=7.365226404 podStartE2EDuration="21.829836467s" totalImagesPullingTime="14.464610063s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:02.829291745 +0000 UTC m=+21.861582762" watchObservedRunningTime="2026-10-30 01:17:02.829836467 +0000 UTC m=+21.862127448" Oct 30 01:17:03.482025 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:03.481990 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:17:03.482025 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:03.482022 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:17:03.482224 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:03.481992 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:17:03.482224 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:03.482105 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7l6k" podUID="88f5d337-fa16-4430-a82e-5f433354c164" Oct 30 01:17:03.482224 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:03.482152 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmnzr" podUID="1c5a490f-f8db-4afb-9a1f-b1accc9eadea" Oct 30 01:17:03.482224 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:03.482218 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64xhb" podUID="f4b29936-72de-4ddc-b7f1-a964a614ff61" Oct 30 01:17:04.492247 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:04.492212 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:04.681714 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:04.681661 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:04.784357 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:04.783593 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dmnzr"] Oct 30 01:17:04.784357 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:04.783799 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:17:04.784357 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:04.783909 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmnzr" podUID="1c5a490f-f8db-4afb-9a1f-b1accc9eadea" Oct 30 01:17:04.784686 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:04.784381 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-64xhb"] Oct 30 01:17:04.784686 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:04.784490 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:17:04.784686 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:04.784585 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64xhb" podUID="f4b29936-72de-4ddc-b7f1-a964a614ff61" Oct 30 01:17:04.794757 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:04.794719 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j7l6k"] Oct 30 01:17:04.794861 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:04.794849 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:17:04.794946 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:04.794930 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7l6k" podUID="88f5d337-fa16-4430-a82e-5f433354c164" Oct 30 01:17:06.482499 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:06.482462 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:17:06.482969 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:06.482472 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:17:06.482969 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:06.482588 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmnzr" podUID="1c5a490f-f8db-4afb-9a1f-b1accc9eadea" Oct 30 01:17:06.482969 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:06.482640 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7l6k" podUID="88f5d337-fa16-4430-a82e-5f433354c164" Oct 30 01:17:06.482969 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:06.482481 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:17:06.482969 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:06.482776 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64xhb" podUID="f4b29936-72de-4ddc-b7f1-a964a614ff61" Oct 30 01:17:07.398828 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:07.398791 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:07.732449 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:07.732412 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:08.026290 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:08.026198 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:08.482030 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:08.482000 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:17:08.482235 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:08.482000 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:17:08.482235 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:08.482112 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64xhb" podUID="f4b29936-72de-4ddc-b7f1-a964a614ff61" Oct 30 01:17:08.482235 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:08.482005 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:17:08.482235 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:08.482185 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7l6k" podUID="88f5d337-fa16-4430-a82e-5f433354c164" Oct 30 01:17:08.482361 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:08.482248 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmnzr" podUID="1c5a490f-f8db-4afb-9a1f-b1accc9eadea" Oct 30 01:17:08.549411 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:08.549382 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:09.544569 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:09.544479 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:09.933110 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:09.933076 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:10.153712 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:10.153676 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:10.367258 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:10.367164 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:10.482597 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:10.482554 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:17:10.482597 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:10.482577 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:17:10.482838 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:10.482668 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmnzr" podUID="1c5a490f-f8db-4afb-9a1f-b1accc9eadea" Oct 30 01:17:10.482838 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:10.482718 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:17:10.482838 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:10.482823 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64xhb" podUID="f4b29936-72de-4ddc-b7f1-a964a614ff61" Oct 30 01:17:10.482929 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:10.482893 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7l6k" podUID="88f5d337-fa16-4430-a82e-5f433354c164" Oct 30 01:17:10.753865 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:10.753831 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:10.958219 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:10.958185 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:11.398020 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:11.397983 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:11.599347 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:11.599309 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:11.890159 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:11.890124 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:12.095988 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:12.095950 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:12.305087 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:12.305053 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:12.482795 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:12.482729 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:17:12.482795 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:12.482805 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:17:12.482997 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:12.482823 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:17:12.482997 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:12.482934 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7l6k" podUID="88f5d337-fa16-4430-a82e-5f433354c164" Oct 30 01:17:12.483075 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:12.482990 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64xhb" podUID="f4b29936-72de-4ddc-b7f1-a964a614ff61" Oct 30 01:17:12.483075 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:12.483037 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmnzr" podUID="1c5a490f-f8db-4afb-9a1f-b1accc9eadea" Oct 30 01:17:12.634061 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:12.633973 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:12.828864 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:12.828830 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:13.205590 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:13.205556 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:13.293498 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:13.293472 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4dl5p_a6babafa-f85a-4a1b-9fe9-8bac54198754/dns-node-resolver/0.log" Oct 30 01:17:13.536456 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:13.536361 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:13.892032 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:13.891943 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:14.090235 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:14.090203 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:14.290210 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:14.290186 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-54hlq_e7922f23-5ba7-4931-8d19-598e2c4fe041/node-ca/0.log" Oct 30 01:17:14.290690 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:14.290604 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:14.482513 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:14.482481 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:17:14.482665 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:14.482481 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:17:14.482665 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:14.482600 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7l6k" podUID="88f5d337-fa16-4430-a82e-5f433354c164" Oct 30 01:17:14.482730 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:14.482481 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:17:14.482730 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:14.482676 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64xhb" podUID="f4b29936-72de-4ddc-b7f1-a964a614ff61" Oct 30 01:17:14.482828 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:14.482734 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmnzr" podUID="1c5a490f-f8db-4afb-9a1f-b1accc9eadea" Oct 30 01:17:14.689422 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:14.689392 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:14.908474 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:14.908438 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:15.220988 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:15.220951 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b29936-72de-4ddc-b7f1-a964a614ff61-metrics-certs\") pod \"network-metrics-daemon-64xhb\" (UID: \"f4b29936-72de-4ddc-b7f1-a964a614ff61\") " pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:17:15.220988 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:15.220992 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/88f5d337-fa16-4430-a82e-5f433354c164-original-pull-secret\") pod \"global-pull-secret-syncer-j7l6k\" (UID: \"88f5d337-fa16-4430-a82e-5f433354c164\") " pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:17:15.221197 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:15.221108 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:17:15.221197 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:15.221159 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88f5d337-fa16-4430-a82e-5f433354c164-original-pull-secret podName:88f5d337-fa16-4430-a82e-5f433354c164 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:47.221147683 +0000 UTC m=+66.253438642 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/88f5d337-fa16-4430-a82e-5f433354c164-original-pull-secret") pod "global-pull-secret-syncer-j7l6k" (UID: "88f5d337-fa16-4430-a82e-5f433354c164") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:17:15.221197 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:15.221108 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:17:15.221298 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:15.221249 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4b29936-72de-4ddc-b7f1-a964a614ff61-metrics-certs podName:f4b29936-72de-4ddc-b7f1-a964a614ff61 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:47.221233844 +0000 UTC m=+66.253524819 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f4b29936-72de-4ddc-b7f1-a964a614ff61-metrics-certs") pod "network-metrics-daemon-64xhb" (UID: "f4b29936-72de-4ddc-b7f1-a964a614ff61") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:17:15.321889 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:15.321850 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czkw9\" (UniqueName: \"kubernetes.io/projected/1c5a490f-f8db-4afb-9a1f-b1accc9eadea-kube-api-access-czkw9\") pod \"network-check-target-dmnzr\" (UID: \"1c5a490f-f8db-4afb-9a1f-b1accc9eadea\") " pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:17:15.322280 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:15.321976 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:17:15.322280 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:15.321989 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:17:15.322280 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:15.321999 2583 projected.go:196] Error preparing data for projected volume kube-api-access-czkw9 for pod openshift-network-diagnostics/network-check-target-dmnzr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:17:15.322280 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:15.322045 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c5a490f-f8db-4afb-9a1f-b1accc9eadea-kube-api-access-czkw9 podName:1c5a490f-f8db-4afb-9a1f-b1accc9eadea nodeName:}" failed. No retries permitted until 2026-10-30 01:17:47.322032911 +0000 UTC m=+66.354323871 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-czkw9" (UniqueName: "kubernetes.io/projected/1c5a490f-f8db-4afb-9a1f-b1accc9eadea-kube-api-access-czkw9") pod "network-check-target-dmnzr" (UID: "1c5a490f-f8db-4afb-9a1f-b1accc9eadea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:17:15.488642 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:15.488562 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:15.673760 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:15.673704 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:15.882194 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:15.882105 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:16.089631 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:16.089599 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:16.274612 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:16.274579 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:16.459967 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:16.459930 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:16.481807 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:16.481775 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:17:16.481972 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:16.481848 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:17:16.481972 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:16.481938 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:17:16.481972 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:16.481955 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64xhb" podUID="f4b29936-72de-4ddc-b7f1-a964a614ff61" Oct 30 01:17:16.482124 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:16.482014 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7l6k" podUID="88f5d337-fa16-4430-a82e-5f433354c164" Oct 30 01:17:16.482124 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:16.482070 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmnzr" podUID="1c5a490f-f8db-4afb-9a1f-b1accc9eadea" Oct 30 01:17:16.703273 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:16.703224 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:17.514822 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:17.514785 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:17.890542 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:17.890460 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:18.094971 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:18.094937 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:18.288586 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:18.288557 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:18.482814 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:18.482777 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:17:18.482814 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:18.482801 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:17:18.483012 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:18.482917 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64xhb" podUID="f4b29936-72de-4ddc-b7f1-a964a614ff61" Oct 30 01:17:18.483012 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:18.482993 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7l6k" podUID="88f5d337-fa16-4430-a82e-5f433354c164" Oct 30 01:17:18.483078 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:18.483031 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:17:18.483120 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:18.483103 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmnzr" podUID="1c5a490f-f8db-4afb-9a1f-b1accc9eadea" Oct 30 01:17:18.683241 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:18.683204 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:19.494614 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:19.494575 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:19.690024 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:19.689994 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:20.091961 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:20.091926 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:20.292240 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:20.292201 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:20.482124 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:20.482086 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:17:20.482301 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:20.482086 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:17:20.482301 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:20.482207 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7l6k" podUID="88f5d337-fa16-4430-a82e-5f433354c164" Oct 30 01:17:20.482301 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:20.482086 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:17:20.482301 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:20.482269 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmnzr" podUID="1c5a490f-f8db-4afb-9a1f-b1accc9eadea" Oct 30 01:17:20.482428 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:20.482358 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64xhb" podUID="f4b29936-72de-4ddc-b7f1-a964a614ff61" Oct 30 01:17:20.489806 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:20.489772 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:22.482400 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:22.482363 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:17:22.482400 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:22.482391 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:17:22.482954 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:22.482462 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7l6k" podUID="88f5d337-fa16-4430-a82e-5f433354c164" Oct 30 01:17:22.482954 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:22.482540 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:17:22.482954 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:22.482635 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmnzr" podUID="1c5a490f-f8db-4afb-9a1f-b1accc9eadea" Oct 30 01:17:22.482954 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:22.482735 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64xhb" podUID="f4b29936-72de-4ddc-b7f1-a964a614ff61" Oct 30 01:17:24.418698 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:24.418661 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:24.482381 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:24.482347 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:17:24.482530 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:24.482391 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:17:24.482530 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:24.482391 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:17:24.482530 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:24.482473 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64xhb" podUID="f4b29936-72de-4ddc-b7f1-a964a614ff61" Oct 30 01:17:24.482643 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:24.482544 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7l6k" podUID="88f5d337-fa16-4430-a82e-5f433354c164" Oct 30 01:17:24.482643 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:24.482629 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmnzr" podUID="1c5a490f-f8db-4afb-9a1f-b1accc9eadea" Oct 30 01:17:24.628549 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:24.628457 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:24.825353 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:24.825316 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:25.079702 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:25.079662 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:25.607933 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:25.607895 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:25.791741 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:25.791703 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:25.979763 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:25.979714 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:26.169520 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:26.169484 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:26.482108 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:26.482076 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:17:26.482108 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:26.482104 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:17:26.482341 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:26.482113 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:17:26.482341 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:26.482177 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmnzr" podUID="1c5a490f-f8db-4afb-9a1f-b1accc9eadea" Oct 30 01:17:26.482341 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:26.482237 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7l6k" podUID="88f5d337-fa16-4430-a82e-5f433354c164" Oct 30 01:17:26.482341 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:26.482305 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64xhb" podUID="f4b29936-72de-4ddc-b7f1-a964a614ff61" Oct 30 01:17:26.572876 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:26.572846 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:28.482455 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:28.482417 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:17:28.482917 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:28.482514 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:17:28.482917 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:28.482525 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dmnzr" podUID="1c5a490f-f8db-4afb-9a1f-b1accc9eadea" Oct 30 01:17:28.482917 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:28.482544 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:17:28.482917 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:28.482620 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64xhb" podUID="f4b29936-72de-4ddc-b7f1-a964a614ff61" Oct 30 01:17:28.482917 ip-10-0-138-223 kubenswrapper[2583]: E1030 01:17:28.482679 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j7l6k" podUID="88f5d337-fa16-4430-a82e-5f433354c164" Oct 30 01:17:28.592104 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:28.592067 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:28.796773 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:28.796666 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:29.178292 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.178265 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeReady" Oct 30 01:17:29.178458 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.178402 2583 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 01:17:29.233437 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.233406 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-98r9d"] Oct 30 01:17:29.235226 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.235209 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-98r9d" Oct 30 01:17:29.245425 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.245400 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6vtnc"] Oct 30 01:17:29.247009 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.246994 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6vtnc" Oct 30 01:17:29.251091 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.251067 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 01:17:29.251294 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.251278 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k4mfm\"" Oct 30 01:17:29.251363 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.251348 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 01:17:29.258387 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.258366 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-42jd7"] Oct 30 01:17:29.260244 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.260219 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-98r9d"] Oct 30 01:17:29.260360 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.260342 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-42jd7" Oct 30 01:17:29.268184 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.268163 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6vtnc"] Oct 30 01:17:29.292113 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.292077 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-42jd7"] Oct 30 01:17:29.337790 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.337763 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00c444f0-9313-4c0a-b9c8-79a953eaba17-tmp-dir\") pod \"dns-default-98r9d\" (UID: \"00c444f0-9313-4c0a-b9c8-79a953eaba17\") " pod="openshift-dns/dns-default-98r9d" Oct 30 01:17:29.337960 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.337810 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/00c444f0-9313-4c0a-b9c8-79a953eaba17-config-volume\") pod \"dns-default-98r9d\" (UID: \"00c444f0-9313-4c0a-b9c8-79a953eaba17\") " pod="openshift-dns/dns-default-98r9d" Oct 30 01:17:29.337960 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.337850 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00c444f0-9313-4c0a-b9c8-79a953eaba17-metrics-tls\") pod \"dns-default-98r9d\" (UID: \"00c444f0-9313-4c0a-b9c8-79a953eaba17\") " pod="openshift-dns/dns-default-98r9d" Oct 30 01:17:29.337960 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.337933 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5bjt\" (UniqueName: \"kubernetes.io/projected/00c444f0-9313-4c0a-b9c8-79a953eaba17-kube-api-access-j5bjt\") pod \"dns-default-98r9d\" (UID: \"00c444f0-9313-4c0a-b9c8-79a953eaba17\") " pod="openshift-dns/dns-default-98r9d" Oct 30 01:17:29.350346 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.350323 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 01:17:29.350346 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.350324 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 01:17:29.350495 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.350469 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 01:17:29.350495 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.350471 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 01:17:29.350495 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.350488 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 01:17:29.351212 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.351195 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 01:17:29.351319 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.351302 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 01:17:29.351374 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.351308 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ms58v\"" Oct 30 01:17:29.351374 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.351356 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xmk59\"" Oct 30 01:17:29.438978 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.438892 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fad7d0da-535f-447c-89bd-ab36ac1861a8-data-volume\") pod \"insights-runtime-extractor-6vtnc\" (UID: \"fad7d0da-535f-447c-89bd-ab36ac1861a8\") " pod="openshift-insights/insights-runtime-extractor-6vtnc" Oct 30 01:17:29.438978 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.438926 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njqrk\" (UniqueName: \"kubernetes.io/projected/fad7d0da-535f-447c-89bd-ab36ac1861a8-kube-api-access-njqrk\") pod \"insights-runtime-extractor-6vtnc\" (UID: \"fad7d0da-535f-447c-89bd-ab36ac1861a8\") " pod="openshift-insights/insights-runtime-extractor-6vtnc" Oct 30 01:17:29.438978 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.438948 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fad7d0da-535f-447c-89bd-ab36ac1861a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6vtnc\" (UID: \"fad7d0da-535f-447c-89bd-ab36ac1861a8\") " pod="openshift-insights/insights-runtime-extractor-6vtnc" Oct 30 01:17:29.438978 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.438972 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00c444f0-9313-4c0a-b9c8-79a953eaba17-tmp-dir\") pod \"dns-default-98r9d\" (UID: \"00c444f0-9313-4c0a-b9c8-79a953eaba17\") " pod="openshift-dns/dns-default-98r9d" Oct 30 01:17:29.439287 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.439028 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3efd0b05-2dcd-4e6c-a24e-9e955a58fa9b-cert\") pod \"ingress-canary-42jd7\" (UID: \"3efd0b05-2dcd-4e6c-a24e-9e955a58fa9b\") " pod="openshift-ingress-canary/ingress-canary-42jd7" Oct 30 01:17:29.439287 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.439066 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fad7d0da-535f-447c-89bd-ab36ac1861a8-crio-socket\") pod \"insights-runtime-extractor-6vtnc\" (UID: \"fad7d0da-535f-447c-89bd-ab36ac1861a8\") " pod="openshift-insights/insights-runtime-extractor-6vtnc" Oct 30 01:17:29.439287 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.439169 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/00c444f0-9313-4c0a-b9c8-79a953eaba17-config-volume\") pod \"dns-default-98r9d\" (UID: \"00c444f0-9313-4c0a-b9c8-79a953eaba17\") " pod="openshift-dns/dns-default-98r9d" Oct 30 01:17:29.439287 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.439204 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00c444f0-9313-4c0a-b9c8-79a953eaba17-metrics-tls\") pod \"dns-default-98r9d\" (UID: \"00c444f0-9313-4c0a-b9c8-79a953eaba17\") " pod="openshift-dns/dns-default-98r9d" Oct 30 01:17:29.439287 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.439235 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fad7d0da-535f-447c-89bd-ab36ac1861a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6vtnc\" (UID: \"fad7d0da-535f-447c-89bd-ab36ac1861a8\") " pod="openshift-insights/insights-runtime-extractor-6vtnc" Oct 30 01:17:29.439287 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.439275 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00c444f0-9313-4c0a-b9c8-79a953eaba17-tmp-dir\") pod \"dns-default-98r9d\" (UID: \"00c444f0-9313-4c0a-b9c8-79a953eaba17\") " pod="openshift-dns/dns-default-98r9d" Oct 30 01:17:29.439456 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.439292 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5bjt\" (UniqueName: \"kubernetes.io/projected/00c444f0-9313-4c0a-b9c8-79a953eaba17-kube-api-access-j5bjt\") pod \"dns-default-98r9d\" (UID: \"00c444f0-9313-4c0a-b9c8-79a953eaba17\") " pod="openshift-dns/dns-default-98r9d" Oct 30 01:17:29.439456 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.439403 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2d66\" (UniqueName: \"kubernetes.io/projected/3efd0b05-2dcd-4e6c-a24e-9e955a58fa9b-kube-api-access-q2d66\") pod \"ingress-canary-42jd7\" (UID: \"3efd0b05-2dcd-4e6c-a24e-9e955a58fa9b\") " pod="openshift-ingress-canary/ingress-canary-42jd7" Oct 30 01:17:29.439894 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.439876 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/00c444f0-9313-4c0a-b9c8-79a953eaba17-config-volume\") pod \"dns-default-98r9d\" (UID: \"00c444f0-9313-4c0a-b9c8-79a953eaba17\") " pod="openshift-dns/dns-default-98r9d" Oct 30 01:17:29.443142 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.443111 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00c444f0-9313-4c0a-b9c8-79a953eaba17-metrics-tls\") pod \"dns-default-98r9d\" (UID: \"00c444f0-9313-4c0a-b9c8-79a953eaba17\") " pod="openshift-dns/dns-default-98r9d" Oct 30 01:17:29.459218 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.459185 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5bjt\" (UniqueName: \"kubernetes.io/projected/00c444f0-9313-4c0a-b9c8-79a953eaba17-kube-api-access-j5bjt\") pod \"dns-default-98r9d\" (UID: \"00c444f0-9313-4c0a-b9c8-79a953eaba17\") " pod="openshift-dns/dns-default-98r9d" Oct 30 01:17:29.540107 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.540064 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2d66\" (UniqueName: \"kubernetes.io/projected/3efd0b05-2dcd-4e6c-a24e-9e955a58fa9b-kube-api-access-q2d66\") pod \"ingress-canary-42jd7\" (UID: \"3efd0b05-2dcd-4e6c-a24e-9e955a58fa9b\") " pod="openshift-ingress-canary/ingress-canary-42jd7" Oct 30 01:17:29.540558 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.540134 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fad7d0da-535f-447c-89bd-ab36ac1861a8-data-volume\") pod \"insights-runtime-extractor-6vtnc\" (UID: \"fad7d0da-535f-447c-89bd-ab36ac1861a8\") " pod="openshift-insights/insights-runtime-extractor-6vtnc" Oct 30 01:17:29.540558 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.540151 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njqrk\" (UniqueName: \"kubernetes.io/projected/fad7d0da-535f-447c-89bd-ab36ac1861a8-kube-api-access-njqrk\") pod \"insights-runtime-extractor-6vtnc\" (UID: \"fad7d0da-535f-447c-89bd-ab36ac1861a8\") " pod="openshift-insights/insights-runtime-extractor-6vtnc" Oct 30 01:17:29.540558 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.540172 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fad7d0da-535f-447c-89bd-ab36ac1861a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6vtnc\" (UID: \"fad7d0da-535f-447c-89bd-ab36ac1861a8\") " pod="openshift-insights/insights-runtime-extractor-6vtnc" Oct 30 01:17:29.540558 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.540200 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3efd0b05-2dcd-4e6c-a24e-9e955a58fa9b-cert\") pod \"ingress-canary-42jd7\" (UID: \"3efd0b05-2dcd-4e6c-a24e-9e955a58fa9b\") " pod="openshift-ingress-canary/ingress-canary-42jd7" Oct 30 01:17:29.540558 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.540219 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fad7d0da-535f-447c-89bd-ab36ac1861a8-crio-socket\") pod \"insights-runtime-extractor-6vtnc\" (UID: \"fad7d0da-535f-447c-89bd-ab36ac1861a8\") " pod="openshift-insights/insights-runtime-extractor-6vtnc" Oct 30 01:17:29.540558 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.540245 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fad7d0da-535f-447c-89bd-ab36ac1861a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6vtnc\" (UID: \"fad7d0da-535f-447c-89bd-ab36ac1861a8\") " pod="openshift-insights/insights-runtime-extractor-6vtnc" Oct 30 01:17:29.540558 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.540471 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fad7d0da-535f-447c-89bd-ab36ac1861a8-crio-socket\") pod \"insights-runtime-extractor-6vtnc\" (UID: \"fad7d0da-535f-447c-89bd-ab36ac1861a8\") " pod="openshift-insights/insights-runtime-extractor-6vtnc" Oct 30 01:17:29.540558 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.540552 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fad7d0da-535f-447c-89bd-ab36ac1861a8-data-volume\") pod \"insights-runtime-extractor-6vtnc\" (UID: \"fad7d0da-535f-447c-89bd-ab36ac1861a8\") " pod="openshift-insights/insights-runtime-extractor-6vtnc" Oct 30 01:17:29.540843 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.540791 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fad7d0da-535f-447c-89bd-ab36ac1861a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6vtnc\" (UID: \"fad7d0da-535f-447c-89bd-ab36ac1861a8\") " pod="openshift-insights/insights-runtime-extractor-6vtnc" Oct 30 01:17:29.542544 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.542522 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fad7d0da-535f-447c-89bd-ab36ac1861a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6vtnc\" (UID: \"fad7d0da-535f-447c-89bd-ab36ac1861a8\") " pod="openshift-insights/insights-runtime-extractor-6vtnc" Oct 30 01:17:29.542659 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.542641 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3efd0b05-2dcd-4e6c-a24e-9e955a58fa9b-cert\") pod \"ingress-canary-42jd7\" (UID: \"3efd0b05-2dcd-4e6c-a24e-9e955a58fa9b\") " pod="openshift-ingress-canary/ingress-canary-42jd7" Oct 30 01:17:29.546535 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.546512 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-98r9d" Oct 30 01:17:29.558684 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.558655 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njqrk\" (UniqueName: \"kubernetes.io/projected/fad7d0da-535f-447c-89bd-ab36ac1861a8-kube-api-access-njqrk\") pod \"insights-runtime-extractor-6vtnc\" (UID: \"fad7d0da-535f-447c-89bd-ab36ac1861a8\") " pod="openshift-insights/insights-runtime-extractor-6vtnc" Oct 30 01:17:29.559056 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.559039 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2d66\" (UniqueName: \"kubernetes.io/projected/3efd0b05-2dcd-4e6c-a24e-9e955a58fa9b-kube-api-access-q2d66\") pod \"ingress-canary-42jd7\" (UID: \"3efd0b05-2dcd-4e6c-a24e-9e955a58fa9b\") " pod="openshift-ingress-canary/ingress-canary-42jd7" Oct 30 01:17:29.570902 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.570878 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-42jd7" Oct 30 01:17:29.849236 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.849160 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-98r9d"] Oct 30 01:17:29.851692 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:17:29.851663 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c444f0_9313_4c0a_b9c8_79a953eaba17.slice/crio-8ae82ecd7a980f22b2e052f7bae608beaa4523926642ab8ca614d120bc8cb0bb WatchSource:0}: Error finding container 8ae82ecd7a980f22b2e052f7bae608beaa4523926642ab8ca614d120bc8cb0bb: Status 404 returned error can't find the container with id 8ae82ecd7a980f22b2e052f7bae608beaa4523926642ab8ca614d120bc8cb0bb Oct 30 01:17:29.858027 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.858006 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6vtnc" Oct 30 01:17:29.869999 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:29.869976 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-42jd7"] Oct 30 01:17:29.871966 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:17:29.871937 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3efd0b05_2dcd_4e6c_a24e_9e955a58fa9b.slice/crio-76d6428545389bf939771ec6760306f141d7037d715bab0edee0308cd2a9cf45 WatchSource:0}: Error finding container 76d6428545389bf939771ec6760306f141d7037d715bab0edee0308cd2a9cf45: Status 404 returned error can't find the container with id 76d6428545389bf939771ec6760306f141d7037d715bab0edee0308cd2a9cf45 Oct 30 01:17:30.159295 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:30.159273 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6vtnc"] Oct 30 01:17:30.160729 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:17:30.160704 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfad7d0da_535f_447c_89bd_ab36ac1861a8.slice/crio-eb32738b0ac5a00b52a65f77ffc436810dabbd33e62e70360a962b704a1b28ca WatchSource:0}: Error finding container eb32738b0ac5a00b52a65f77ffc436810dabbd33e62e70360a962b704a1b28ca: Status 404 returned error can't find the container with id eb32738b0ac5a00b52a65f77ffc436810dabbd33e62e70360a962b704a1b28ca Oct 30 01:17:30.481965 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:30.481932 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:17:30.482328 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:30.482298 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:17:30.482682 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:30.482663 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:17:30.496775 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:30.496733 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 01:17:30.496932 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:30.496911 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 01:17:30.509192 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:30.509099 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wjscp\"" Oct 30 01:17:30.509192 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:30.509154 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xp5gx\"" Oct 30 01:17:30.509507 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:30.509479 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 01:17:30.520361 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:30.520339 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 01:17:30.520473 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:30.520391 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-n79rv\"" Oct 30 01:17:30.556435 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:30.556402 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:30.815100 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:30.815008 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6vtnc" event={"ID":"fad7d0da-535f-447c-89bd-ab36ac1861a8","Type":"ContainerStarted","Data":"550b22ca82b378e16a00611762ab8155b22f5bfc18b4552b8cfffa2f61fa691f"} Oct 30 01:17:30.815269 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:30.815127 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6vtnc" event={"ID":"fad7d0da-535f-447c-89bd-ab36ac1861a8","Type":"ContainerStarted","Data":"eb32738b0ac5a00b52a65f77ffc436810dabbd33e62e70360a962b704a1b28ca"} Oct 30 01:17:30.817315 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:30.817285 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-98r9d" event={"ID":"00c444f0-9313-4c0a-b9c8-79a953eaba17","Type":"ContainerStarted","Data":"8ae82ecd7a980f22b2e052f7bae608beaa4523926642ab8ca614d120bc8cb0bb"} Oct 30 01:17:30.819280 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:30.819246 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-42jd7" event={"ID":"3efd0b05-2dcd-4e6c-a24e-9e955a58fa9b","Type":"ContainerStarted","Data":"76d6428545389bf939771ec6760306f141d7037d715bab0edee0308cd2a9cf45"} Oct 30 01:17:30.899324 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:30.899289 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:31.206096 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:31.206062 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4dl5p_a6babafa-f85a-4a1b-9fe9-8bac54198754/dns-node-resolver/0.log" Oct 30 01:17:31.826019 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:31.825949 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6vtnc" event={"ID":"fad7d0da-535f-447c-89bd-ab36ac1861a8","Type":"ContainerStarted","Data":"06406317079f0126805684e9ea9fcd014120bc2394d2b8c20e38001b7bcdbe84"} Oct 30 01:17:31.828810 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:31.828779 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-98r9d" event={"ID":"00c444f0-9313-4c0a-b9c8-79a953eaba17","Type":"ContainerStarted","Data":"099fad35c8c2c3e8cad392169f5a8587a45c98def53612074fc5d19e4790182c"} Oct 30 01:17:32.405224 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:32.405190 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-54hlq_e7922f23-5ba7-4931-8d19-598e2c4fe041/node-ca/0.log" Oct 30 01:17:32.832876 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:32.832789 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6vtnc" event={"ID":"fad7d0da-535f-447c-89bd-ab36ac1861a8","Type":"ContainerStarted","Data":"69682e2052f2c51a8c42b1367df63a47d5777c6d5833b390c5f86de7b1dc0867"} Oct 30 01:17:32.834668 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:32.834637 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-98r9d" event={"ID":"00c444f0-9313-4c0a-b9c8-79a953eaba17","Type":"ContainerStarted","Data":"9153796b265522fe943efe97a91ceed15fa7e1903ff7da920f748fdf2259e786"} Oct 30 01:17:32.834818 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:32.834715 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-98r9d" Oct 30 01:17:32.836125 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:32.836104 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-42jd7" event={"ID":"3efd0b05-2dcd-4e6c-a24e-9e955a58fa9b","Type":"ContainerStarted","Data":"b565ff24309ff73b2dcb598a87b4379a99c33372551b643ae210bc69f2a9996a"} Oct 30 01:17:32.874534 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:32.874495 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6vtnc" podStartSLOduration=1.8761688859999999 podStartE2EDuration="3.874483599s" totalImagesPullingTime="1.998314713s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:17:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:32.873641926 +0000 UTC m=+51.905932907" watchObservedRunningTime="2026-10-30 01:17:32.874483599 +0000 UTC m=+51.906774579" Oct 30 01:17:32.893330 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:32.893297 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:32.913186 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:32.913137 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-98r9d" podStartSLOduration=2.154540644 podStartE2EDuration="3.913125057s" totalImagesPullingTime="1.758584413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:32.912925764 +0000 UTC m=+51.945216747" watchObservedRunningTime="2026-10-30 01:17:32.913125057 +0000 UTC m=+51.945416038" Oct 30 01:17:32.951870 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:32.951816 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-42jd7" podStartSLOduration=1.291790723 podStartE2EDuration="3.951807129s" totalImagesPullingTime="2.660016406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:32.951073191 +0000 UTC m=+51.983364172" watchObservedRunningTime="2026-10-30 01:17:32.951807129 +0000 UTC m=+51.984098110" Oct 30 01:17:33.081282 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:33.081239 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:33.205829 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:33.205801 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-42jd7_3efd0b05-2dcd-4e6c-a24e-9e955a58fa9b/serve-healthcheck-canary/0.log" Oct 30 01:17:34.764387 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:34.764357 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:35.686461 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:35.686427 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:36.025138 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:36.025043 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:37.386779 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.386730 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:37.486080 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.486052 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-cnw75"] Oct 30 01:17:37.488015 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.488000 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cnw75" Oct 30 01:17:37.507981 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.507948 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 01:17:37.519877 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.519846 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 01:17:37.520014 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.519929 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 01:17:37.520014 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.519970 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-ntp2h\"" Oct 30 01:17:37.520014 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.520010 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 01:17:37.520167 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.519984 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 01:17:37.520967 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.520947 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-cnw75"] Oct 30 01:17:37.561194 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.561157 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tqcs7"] Oct 30 01:17:37.563412 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.563396 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9"] Oct 30 01:17:37.563546 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.563525 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:37.565257 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.565238 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9" Oct 30 01:17:37.582960 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.582931 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 01:17:37.583203 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.583184 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 01:17:37.598657 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.598623 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-4gtf9\" (UID: \"a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9" Oct 30 01:17:37.598657 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.598660 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-4gtf9\" (UID: \"a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9" Oct 30 01:17:37.598899 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.598682 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f3928614-3b83-4bf6-9e3f-04a71bae52bd-root\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:37.598899 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.598699 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6w4k\" (UniqueName: \"kubernetes.io/projected/f3928614-3b83-4bf6-9e3f-04a71bae52bd-kube-api-access-d6w4k\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:37.598899 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.598819 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwz2j\" (UniqueName: \"kubernetes.io/projected/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab-kube-api-access-wwz2j\") pod \"kube-state-metrics-5dc8d7d89b-4gtf9\" (UID: \"a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9" Oct 30 01:17:37.598899 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.598878 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f3928614-3b83-4bf6-9e3f-04a71bae52bd-node-exporter-tls\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:37.598899 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.598895 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f3928614-3b83-4bf6-9e3f-04a71bae52bd-node-exporter-accelerators-collector-config\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:37.599056 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.598915 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-4gtf9\" (UID: \"a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9" Oct 30 01:17:37.599056 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.598996 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3928614-3b83-4bf6-9e3f-04a71bae52bd-sys\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:37.599056 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.599014 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-4gtf9\" (UID: \"a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9" Oct 30 01:17:37.599056 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.599033 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b2bg\" (UniqueName: \"kubernetes.io/projected/cf531672-6ee1-4188-b38d-2899bf9d6a0f-kube-api-access-6b2bg\") pod \"openshift-state-metrics-7bd7489957-cnw75\" (UID: \"cf531672-6ee1-4188-b38d-2899bf9d6a0f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cnw75" Oct 30 01:17:37.599056 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.599049 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f3928614-3b83-4bf6-9e3f-04a71bae52bd-node-exporter-textfile\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:37.599209 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.599064 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf531672-6ee1-4188-b38d-2899bf9d6a0f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-cnw75\" (UID: \"cf531672-6ee1-4188-b38d-2899bf9d6a0f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cnw75" Oct 30 01:17:37.599209 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.599134 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf531672-6ee1-4188-b38d-2899bf9d6a0f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-cnw75\" (UID: \"cf531672-6ee1-4188-b38d-2899bf9d6a0f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cnw75" Oct 30 01:17:37.599209 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.599167 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf531672-6ee1-4188-b38d-2899bf9d6a0f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-cnw75\" (UID: \"cf531672-6ee1-4188-b38d-2899bf9d6a0f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cnw75" Oct 30 01:17:37.599209 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.599183 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f3928614-3b83-4bf6-9e3f-04a71bae52bd-node-exporter-wtmp\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:37.599323 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.599211 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3928614-3b83-4bf6-9e3f-04a71bae52bd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:37.599323 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.599258 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3928614-3b83-4bf6-9e3f-04a71bae52bd-metrics-client-ca\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:37.599323 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.599294 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-4gtf9\" (UID: \"a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9" Oct 30 01:17:37.603171 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.603146 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9"] Oct 30 01:17:37.609187 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.609161 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 01:17:37.609341 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.609230 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 01:17:37.609341 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.609247 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vthnr\"" Oct 30 01:17:37.609341 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.609273 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 01:17:37.609341 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.609284 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 01:17:37.609341 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.609277 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-xqnkh\"" Oct 30 01:17:37.624060 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.624006 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:37.700460 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.700415 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf531672-6ee1-4188-b38d-2899bf9d6a0f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-cnw75\" (UID: \"cf531672-6ee1-4188-b38d-2899bf9d6a0f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cnw75" Oct 30 01:17:37.700629 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.700466 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f3928614-3b83-4bf6-9e3f-04a71bae52bd-node-exporter-wtmp\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:37.700629 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.700586 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3928614-3b83-4bf6-9e3f-04a71bae52bd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:37.700629 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.700620 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f3928614-3b83-4bf6-9e3f-04a71bae52bd-node-exporter-wtmp\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:37.700832 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.700638 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3928614-3b83-4bf6-9e3f-04a71bae52bd-metrics-client-ca\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:37.700832 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.700663 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-4gtf9\" (UID: \"a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9" Oct 30 01:17:37.700832 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.700720 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-4gtf9\" (UID: \"a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9" Oct 30 01:17:37.700832 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.700774 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-4gtf9\" (UID: \"a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9" Oct 30 01:17:37.700832 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.700802 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f3928614-3b83-4bf6-9e3f-04a71bae52bd-root\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:37.700832 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.700821 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6w4k\" (UniqueName: \"kubernetes.io/projected/f3928614-3b83-4bf6-9e3f-04a71bae52bd-kube-api-access-d6w4k\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:37.701113 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.700852 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwz2j\" (UniqueName: \"kubernetes.io/projected/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab-kube-api-access-wwz2j\") pod \"kube-state-metrics-5dc8d7d89b-4gtf9\" (UID: \"a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9" Oct 30 01:17:37.701113 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.700898 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f3928614-3b83-4bf6-9e3f-04a71bae52bd-root\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:37.701113 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.700907 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f3928614-3b83-4bf6-9e3f-04a71bae52bd-node-exporter-tls\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:37.701113 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.700952 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f3928614-3b83-4bf6-9e3f-04a71bae52bd-node-exporter-accelerators-collector-config\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:37.701113 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.700987 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-4gtf9\" (UID: \"a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9" Oct 30 01:17:37.701113 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.701033 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3928614-3b83-4bf6-9e3f-04a71bae52bd-sys\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:37.701113 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.701073 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-4gtf9\" (UID: \"a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9" Oct 30 01:17:37.701113 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.701109 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b2bg\" (UniqueName: \"kubernetes.io/projected/cf531672-6ee1-4188-b38d-2899bf9d6a0f-kube-api-access-6b2bg\") pod \"openshift-state-metrics-7bd7489957-cnw75\" (UID: \"cf531672-6ee1-4188-b38d-2899bf9d6a0f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cnw75" Oct 30 01:17:37.701487 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.701139 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f3928614-3b83-4bf6-9e3f-04a71bae52bd-node-exporter-textfile\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:37.701487 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.701168 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf531672-6ee1-4188-b38d-2899bf9d6a0f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-cnw75\" (UID: \"cf531672-6ee1-4188-b38d-2899bf9d6a0f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cnw75" Oct 30 01:17:37.701487 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.701201 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf531672-6ee1-4188-b38d-2899bf9d6a0f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-cnw75\" (UID: \"cf531672-6ee1-4188-b38d-2899bf9d6a0f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cnw75" Oct 30 01:17:37.701630 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.701534 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-4gtf9\" (UID: \"a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9" Oct 30 01:17:37.702240 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.702215 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3928614-3b83-4bf6-9e3f-04a71bae52bd-metrics-client-ca\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:37.702384 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.702360 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-4gtf9\" (UID: \"a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9" Oct 30 01:17:37.702452 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.702437 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3928614-3b83-4bf6-9e3f-04a71bae52bd-sys\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:37.702537 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.702262 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-4gtf9\" (UID: \"a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9" Oct 30 01:17:37.702764 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.702723 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f3928614-3b83-4bf6-9e3f-04a71bae52bd-node-exporter-textfile\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:37.702854 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.702789 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f3928614-3b83-4bf6-9e3f-04a71bae52bd-node-exporter-accelerators-collector-config\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:37.703560 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.703535 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf531672-6ee1-4188-b38d-2899bf9d6a0f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-cnw75\" (UID: \"cf531672-6ee1-4188-b38d-2899bf9d6a0f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cnw75" Oct 30 01:17:37.703668 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.703594 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf531672-6ee1-4188-b38d-2899bf9d6a0f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-cnw75\" (UID: \"cf531672-6ee1-4188-b38d-2899bf9d6a0f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cnw75" Oct 30 01:17:37.703766 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.703675 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f3928614-3b83-4bf6-9e3f-04a71bae52bd-node-exporter-tls\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:37.703766 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.703692 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-4gtf9\" (UID: \"a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9" Oct 30 01:17:37.703914 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.703893 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3928614-3b83-4bf6-9e3f-04a71bae52bd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:37.704365 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.704340 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf531672-6ee1-4188-b38d-2899bf9d6a0f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-cnw75\" (UID: \"cf531672-6ee1-4188-b38d-2899bf9d6a0f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cnw75" Oct 30 01:17:37.705166 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.705145 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-4gtf9\" (UID: \"a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9" Oct 30 01:17:37.726074 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.726036 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6w4k\" (UniqueName: \"kubernetes.io/projected/f3928614-3b83-4bf6-9e3f-04a71bae52bd-kube-api-access-d6w4k\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:37.727453 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.727429 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwz2j\" (UniqueName: \"kubernetes.io/projected/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab-kube-api-access-wwz2j\") pod \"kube-state-metrics-5dc8d7d89b-4gtf9\" (UID: \"a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9" Oct 30 01:17:37.727551 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.727466 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b2bg\" (UniqueName: \"kubernetes.io/projected/cf531672-6ee1-4188-b38d-2899bf9d6a0f-kube-api-access-6b2bg\") pod \"openshift-state-metrics-7bd7489957-cnw75\" (UID: \"cf531672-6ee1-4188-b38d-2899bf9d6a0f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cnw75" Oct 30 01:17:37.799837 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.799804 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cnw75" Oct 30 01:17:37.835708 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.835673 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:37.877070 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.877026 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:37.881623 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:37.881596 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9" Oct 30 01:17:37.888271 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:17:37.888242 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3928614_3b83_4bf6_9e3f_04a71bae52bd.slice/crio-758ba053dd70b7aa4ba6e97a8652ecd512db79c8dfeeb0a29de1f494a16fdc23 WatchSource:0}: Error finding container 758ba053dd70b7aa4ba6e97a8652ecd512db79c8dfeeb0a29de1f494a16fdc23: Status 404 returned error can't find the container with id 758ba053dd70b7aa4ba6e97a8652ecd512db79c8dfeeb0a29de1f494a16fdc23 Oct 30 01:17:38.118906 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:38.118880 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-cnw75"] Oct 30 01:17:38.120462 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:17:38.120432 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf531672_6ee1_4188_b38d_2899bf9d6a0f.slice/crio-7ce030618253c5c68cb56a995163f8a4d99ae28ad868ecbc8cc2464bf3595ab6 WatchSource:0}: Error finding container 7ce030618253c5c68cb56a995163f8a4d99ae28ad868ecbc8cc2464bf3595ab6: Status 404 returned error can't find the container with id 7ce030618253c5c68cb56a995163f8a4d99ae28ad868ecbc8cc2464bf3595ab6 Oct 30 01:17:38.137409 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:38.137374 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:38.183896 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:38.182517 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9"] Oct 30 01:17:38.351860 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:38.351821 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:38.757003 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:38.756960 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:38.855845 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:38.855737 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9" event={"ID":"a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab","Type":"ContainerStarted","Data":"3bd77351638fccdbe592f3c0c30162240591236be59aff6b515940657a064a2b"} Oct 30 01:17:38.857763 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:38.857709 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tqcs7" event={"ID":"f3928614-3b83-4bf6-9e3f-04a71bae52bd","Type":"ContainerStarted","Data":"758ba053dd70b7aa4ba6e97a8652ecd512db79c8dfeeb0a29de1f494a16fdc23"} Oct 30 01:17:38.860028 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:38.859995 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cnw75" event={"ID":"cf531672-6ee1-4188-b38d-2899bf9d6a0f","Type":"ContainerStarted","Data":"a973f969dffbac69477dc9404ba912f0d9b9c54929e65d9c1b12cab328260c96"} Oct 30 01:17:38.860160 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:38.860106 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cnw75" event={"ID":"cf531672-6ee1-4188-b38d-2899bf9d6a0f","Type":"ContainerStarted","Data":"64c8bb5ff0f61451f7b2c1f063fbddfbf3ee25e2fa740838691b42a8b5eec6a2"} Oct 30 01:17:38.860160 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:38.860139 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cnw75" event={"ID":"cf531672-6ee1-4188-b38d-2899bf9d6a0f","Type":"ContainerStarted","Data":"7ce030618253c5c68cb56a995163f8a4d99ae28ad868ecbc8cc2464bf3595ab6"} Oct 30 01:17:39.026914 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:39.024919 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:39.234376 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:39.234292 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:39.652553 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:39.652517 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:39.823318 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:39.821939 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-578df9b8f5-9z7nm"] Oct 30 01:17:39.824627 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:39.824598 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:39.851919 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:39.851876 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 01:17:39.851919 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:39.851901 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 01:17:39.851919 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:39.851924 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 01:17:39.852221 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:39.851949 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 01:17:39.852221 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:39.851993 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-2817d7ip80efo\"" Oct 30 01:17:39.852221 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:39.851992 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-hgx58\"" Oct 30 01:17:39.852221 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:39.852031 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 01:17:39.853569 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:39.853545 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-578df9b8f5-9z7nm"] Oct 30 01:17:39.859159 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:39.859126 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:39.866183 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:39.866143 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cnw75" event={"ID":"cf531672-6ee1-4188-b38d-2899bf9d6a0f","Type":"ContainerStarted","Data":"776bd7c5459fd206b01770e862334297e805d8cf37ea05bb440bfe831b73673d"} Oct 30 01:17:39.868655 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:39.868614 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9" event={"ID":"a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab","Type":"ContainerStarted","Data":"25f5d1d1f034029e00128775836e9406563dba1cdfd3733fb904c9963f5885f4"} Oct 30 01:17:39.868881 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:39.868701 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9" event={"ID":"a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab","Type":"ContainerStarted","Data":"6049b3f74424f503c14ffae5dd1d9735e3ab9372691e03a9f07712d21ca874c9"} Oct 30 01:17:39.868881 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:39.868741 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9" event={"ID":"a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab","Type":"ContainerStarted","Data":"e77442fee87368ca0daed21f340560bf0ac91130eef3f1bf02dd4861d1eabe50"} Oct 30 01:17:39.871597 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:39.871553 2583 generic.go:356] "Generic (PLEG): container finished" podID="f3928614-3b83-4bf6-9e3f-04a71bae52bd" containerID="7368301157ab752ac1c80a8f7a2c2d3ed82e4574beb3098fff5de35e30d293a1" exitCode=0 Oct 30 01:17:39.871825 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:39.871606 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tqcs7" event={"ID":"f3928614-3b83-4bf6-9e3f-04a71bae52bd","Type":"ContainerDied","Data":"7368301157ab752ac1c80a8f7a2c2d3ed82e4574beb3098fff5de35e30d293a1"} Oct 30 01:17:39.909897 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:39.909832 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cnw75" podStartSLOduration=1.862264343 podStartE2EDuration="2.909812627s" totalImagesPullingTime="1.047548284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:39.908882113 +0000 UTC m=+58.941173095" watchObservedRunningTime="2026-10-30 01:17:39.909812627 +0000 UTC m=+58.942103610" Oct 30 01:17:39.921319 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:39.921281 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e829899-8a60-4792-b4cc-fabcaff0469f-tmp\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:39.921499 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:39.921332 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9e829899-8a60-4792-b4cc-fabcaff0469f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:39.921775 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:39.921718 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9e829899-8a60-4792-b4cc-fabcaff0469f-secret-thanos-querier-tls\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:39.921849 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:39.921781 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e829899-8a60-4792-b4cc-fabcaff0469f-metrics-client-ca\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:39.921849 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:39.921807 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9e829899-8a60-4792-b4cc-fabcaff0469f-secret-grpc-tls\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:39.921849 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:39.921827 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n85mv\" (UniqueName: \"kubernetes.io/projected/9e829899-8a60-4792-b4cc-fabcaff0469f-kube-api-access-n85mv\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:39.922380 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:39.921886 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9e829899-8a60-4792-b4cc-fabcaff0469f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:39.922380 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:39.921957 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9e829899-8a60-4792-b4cc-fabcaff0469f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:39.922380 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:39.922275 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9e829899-8a60-4792-b4cc-fabcaff0469f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:39.949667 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:39.949381 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9" podStartSLOduration=1.5879691280000001 podStartE2EDuration="2.94935973s" totalImagesPullingTime="1.361390602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:39.94786057 +0000 UTC m=+58.980151552" watchObservedRunningTime="2026-10-30 01:17:39.94935973 +0000 UTC m=+58.981650712" Oct 30 01:17:40.023038 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:40.023001 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9e829899-8a60-4792-b4cc-fabcaff0469f-secret-thanos-querier-tls\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:40.023038 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:40.023048 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e829899-8a60-4792-b4cc-fabcaff0469f-metrics-client-ca\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:40.023298 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:40.023074 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9e829899-8a60-4792-b4cc-fabcaff0469f-secret-grpc-tls\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:40.023298 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:40.023140 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n85mv\" (UniqueName: \"kubernetes.io/projected/9e829899-8a60-4792-b4cc-fabcaff0469f-kube-api-access-n85mv\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:40.023298 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:40.023190 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9e829899-8a60-4792-b4cc-fabcaff0469f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:40.023298 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:40.023247 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9e829899-8a60-4792-b4cc-fabcaff0469f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:40.023298 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:40.023279 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9e829899-8a60-4792-b4cc-fabcaff0469f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:40.023567 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:40.023320 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e829899-8a60-4792-b4cc-fabcaff0469f-tmp\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:40.023567 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:40.023363 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9e829899-8a60-4792-b4cc-fabcaff0469f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:40.023878 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:40.023851 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e829899-8a60-4792-b4cc-fabcaff0469f-metrics-client-ca\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:40.024010 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:40.023918 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e829899-8a60-4792-b4cc-fabcaff0469f-tmp\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:40.026934 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:40.026879 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9e829899-8a60-4792-b4cc-fabcaff0469f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:40.026934 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:40.026908 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9e829899-8a60-4792-b4cc-fabcaff0469f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:40.027089 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:40.026951 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9e829899-8a60-4792-b4cc-fabcaff0469f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:40.027168 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:40.027143 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9e829899-8a60-4792-b4cc-fabcaff0469f-secret-thanos-querier-tls\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:40.027206 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:40.027168 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9e829899-8a60-4792-b4cc-fabcaff0469f-secret-grpc-tls\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:40.028223 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:40.028199 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9e829899-8a60-4792-b4cc-fabcaff0469f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:40.041981 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:40.041913 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n85mv\" (UniqueName: \"kubernetes.io/projected/9e829899-8a60-4792-b4cc-fabcaff0469f-kube-api-access-n85mv\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:40.143391 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:40.143281 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:40.241880 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:40.241840 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:40.423645 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:40.423607 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:40.446069 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:40.446040 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-578df9b8f5-9z7nm"] Oct 30 01:17:40.447661 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:17:40.447638 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e829899_8a60_4792_b4cc_fabcaff0469f.slice/crio-4804e601006c9694ff3e71642609d0091e993141294c870ed271be987c4c434a WatchSource:0}: Error finding container 4804e601006c9694ff3e71642609d0091e993141294c870ed271be987c4c434a: Status 404 returned error can't find the container with id 4804e601006c9694ff3e71642609d0091e993141294c870ed271be987c4c434a Oct 30 01:17:40.870696 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:40.870596 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:40.876988 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:40.876954 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tqcs7" event={"ID":"f3928614-3b83-4bf6-9e3f-04a71bae52bd","Type":"ContainerStarted","Data":"1b605ed4b43fee3599b655e3d5f70c57b542664aca61d174abc1ba7ce0b7d1df"} Oct 30 01:17:40.877175 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:40.877011 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tqcs7" event={"ID":"f3928614-3b83-4bf6-9e3f-04a71bae52bd","Type":"ContainerStarted","Data":"3e03d9e7f35fa98eef7394e0f71da9a25971c90136facd5ca81b436308771800"} Oct 30 01:17:40.878192 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:40.878166 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" event={"ID":"9e829899-8a60-4792-b4cc-fabcaff0469f","Type":"ContainerStarted","Data":"4804e601006c9694ff3e71642609d0091e993141294c870ed271be987c4c434a"} Oct 30 01:17:40.918455 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:40.918392 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tqcs7" podStartSLOduration=2.790659683 podStartE2EDuration="3.918376121s" totalImagesPullingTime="1.048143411s" totalInitContainerRuntime="79.573027ms" isStatefulPod=true podCreationTimestamp="2026-10-30 01:17:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:40.917550444 +0000 UTC m=+59.949841426" watchObservedRunningTime="2026-10-30 01:17:40.918376121 +0000 UTC m=+59.950667140" Oct 30 01:17:41.060137 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:41.060096 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:41.252807 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:41.252765 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:42.322337 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:42.322300 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-f7fkx"] Oct 30 01:17:42.324126 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:42.324105 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-f7fkx" Oct 30 01:17:42.347569 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:42.347537 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-f7fkx"] Oct 30 01:17:42.352576 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:42.352558 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 01:17:42.352637 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:42.352557 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-ng7jx\"" Oct 30 01:17:42.397302 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:42.397161 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:42.444797 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:42.444769 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ljcm\" (UniqueName: \"kubernetes.io/projected/0ae3164f-fec1-4210-863e-70e49e6bd26a-kube-api-access-2ljcm\") pod \"monitoring-plugin-77f4f76594-f7fkx\" (UID: \"0ae3164f-fec1-4210-863e-70e49e6bd26a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-f7fkx" Oct 30 01:17:42.444958 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:42.444810 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0ae3164f-fec1-4210-863e-70e49e6bd26a-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-f7fkx\" (UID: \"0ae3164f-fec1-4210-863e-70e49e6bd26a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-f7fkx" Oct 30 01:17:42.545816 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:42.545775 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ljcm\" (UniqueName: \"kubernetes.io/projected/0ae3164f-fec1-4210-863e-70e49e6bd26a-kube-api-access-2ljcm\") pod \"monitoring-plugin-77f4f76594-f7fkx\" (UID: \"0ae3164f-fec1-4210-863e-70e49e6bd26a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-f7fkx" Oct 30 01:17:42.545966 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:42.545830 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0ae3164f-fec1-4210-863e-70e49e6bd26a-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-f7fkx\" (UID: \"0ae3164f-fec1-4210-863e-70e49e6bd26a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-f7fkx" Oct 30 01:17:42.548284 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:42.548256 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0ae3164f-fec1-4210-863e-70e49e6bd26a-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-f7fkx\" (UID: \"0ae3164f-fec1-4210-863e-70e49e6bd26a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-f7fkx" Oct 30 01:17:42.564548 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:42.564516 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ljcm\" (UniqueName: \"kubernetes.io/projected/0ae3164f-fec1-4210-863e-70e49e6bd26a-kube-api-access-2ljcm\") pod \"monitoring-plugin-77f4f76594-f7fkx\" (UID: \"0ae3164f-fec1-4210-863e-70e49e6bd26a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-f7fkx" Oct 30 01:17:42.580201 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:42.580114 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:42.634831 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:42.634793 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-f7fkx" Oct 30 01:17:42.840994 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:42.840908 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-98r9d" Oct 30 01:17:42.857980 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:42.857936 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-76696fb494-9gc8p"] Oct 30 01:17:42.861424 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:42.861401 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:42.873859 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:42.873831 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-76696fb494-9gc8p"] Oct 30 01:17:42.876371 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:42.876344 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 01:17:42.876371 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:42.876344 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 01:17:42.876371 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:42.876348 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 01:17:42.876577 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:42.876470 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-wz8qb\"" Oct 30 01:17:42.876577 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:42.876504 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 01:17:42.884991 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:42.884959 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 01:17:42.889578 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:42.889544 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" event={"ID":"9e829899-8a60-4792-b4cc-fabcaff0469f","Type":"ContainerStarted","Data":"79b620b1012505e1a0d9c258433093ef8a2041165776b0e152ded7f1d0b3b29f"} Oct 30 01:17:42.889770 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:42.889617 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" event={"ID":"9e829899-8a60-4792-b4cc-fabcaff0469f","Type":"ContainerStarted","Data":"1c298a298b154c5483ab684e8beab389fa5f9858333d48b33339b4d0c66bc129"} Oct 30 01:17:42.889770 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:42.889646 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" event={"ID":"9e829899-8a60-4792-b4cc-fabcaff0469f","Type":"ContainerStarted","Data":"27560ec883bc0ba94a1bf4916b399ae87ce011fc0966f15c4c62a928e67e03d5"} Oct 30 01:17:42.902251 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:42.902215 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 01:17:42.926437 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:42.925554 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-f7fkx"] Oct 30 01:17:42.932435 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:17:42.932398 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ae3164f_fec1_4210_863e_70e49e6bd26a.slice/crio-5fa780ef9495d54c00886fe917a2c6843587f55391ce8b458ae71ac8d377c8e6 WatchSource:0}: Error finding container 5fa780ef9495d54c00886fe917a2c6843587f55391ce8b458ae71ac8d377c8e6: Status 404 returned error can't find the container with id 5fa780ef9495d54c00886fe917a2c6843587f55391ce8b458ae71ac8d377c8e6 Oct 30 01:17:42.949121 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:42.949091 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-secret-telemeter-client\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:42.949267 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:42.949155 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-serving-certs-ca-bundle\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:42.949323 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:42.949262 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-telemeter-client-tls\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:42.949412 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:42.949322 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:42.949412 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:42.949355 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-federate-client-tls\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:42.949412 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:42.949379 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:42.949519 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:42.949494 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-metrics-client-ca\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:42.949611 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:42.949586 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzkj9\" (UniqueName: \"kubernetes.io/projected/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-kube-api-access-zzkj9\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:43.050809 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:43.050742 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-secret-telemeter-client\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:43.051003 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:43.050847 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-serving-certs-ca-bundle\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:43.051003 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:43.050896 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-telemeter-client-tls\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:43.051003 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:43.050926 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:43.051003 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:43.050951 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-federate-client-tls\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:43.051003 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:43.050981 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:43.051246 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:43.051010 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-metrics-client-ca\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:43.051246 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:43.051046 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzkj9\" (UniqueName: \"kubernetes.io/projected/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-kube-api-access-zzkj9\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:43.051733 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:43.051702 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-serving-certs-ca-bundle\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:43.051922 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:43.051901 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-metrics-client-ca\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:43.051993 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:43.051926 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:43.053395 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:43.053367 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-secret-telemeter-client\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:43.053498 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:43.053435 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-telemeter-client-tls\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:43.053498 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:43.053463 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:43.053722 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:43.053706 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-federate-client-tls\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:43.069277 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:43.069255 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzkj9\" (UniqueName: \"kubernetes.io/projected/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-kube-api-access-zzkj9\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:43.175438 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:43.175402 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:43.498335 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:43.498307 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-76696fb494-9gc8p"] Oct 30 01:17:43.510700 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:17:43.510667 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc3b964_6e84_4860_a5bf_0a81f15b0d97.slice/crio-a628f193fbba8c2be9f20c3c3119c8801bbc4b36abf1144c487c708f097e4e82 WatchSource:0}: Error finding container a628f193fbba8c2be9f20c3c3119c8801bbc4b36abf1144c487c708f097e4e82: Status 404 returned error can't find the container with id a628f193fbba8c2be9f20c3c3119c8801bbc4b36abf1144c487c708f097e4e82 Oct 30 01:17:43.895206 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:43.895102 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" event={"ID":"9bc3b964-6e84-4860-a5bf-0a81f15b0d97","Type":"ContainerStarted","Data":"a628f193fbba8c2be9f20c3c3119c8801bbc4b36abf1144c487c708f097e4e82"} Oct 30 01:17:43.898928 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:43.898893 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" event={"ID":"9e829899-8a60-4792-b4cc-fabcaff0469f","Type":"ContainerStarted","Data":"bd5d93b94b166793743060f962fada329ccb231b014d06964b4fb9c0e2d0f644"} Oct 30 01:17:43.899076 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:43.898975 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" event={"ID":"9e829899-8a60-4792-b4cc-fabcaff0469f","Type":"ContainerStarted","Data":"82bc96e667d1b53726fecba10c5f4e43a840419861c99311d693294f102cff24"} Oct 30 01:17:43.899076 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:43.899022 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" event={"ID":"9e829899-8a60-4792-b4cc-fabcaff0469f","Type":"ContainerStarted","Data":"d16f468abb96b824f2411f44a8fb0222be982363c47b57bb75b8fb6e62d00017"} Oct 30 01:17:43.899076 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:43.899067 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:43.900255 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:43.900227 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-f7fkx" event={"ID":"0ae3164f-fec1-4210-863e-70e49e6bd26a","Type":"ContainerStarted","Data":"5fa780ef9495d54c00886fe917a2c6843587f55391ce8b458ae71ac8d377c8e6"} Oct 30 01:17:43.943801 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:43.943724 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" podStartSLOduration=2.136884151 podStartE2EDuration="4.943704303s" totalImagesPullingTime="2.806820152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:43.942465763 +0000 UTC m=+62.974756744" watchObservedRunningTime="2026-10-30 01:17:43.943704303 +0000 UTC m=+62.975995284" Oct 30 01:17:44.496325 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.496285 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:44.522253 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.522215 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 01:17:44.525323 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.525296 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.539126 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.539097 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 01:17:44.539534 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.539496 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 01:17:44.539665 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.539614 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-np2tl\"" Oct 30 01:17:44.539665 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.539619 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c86r3ikjcp03b\"" Oct 30 01:17:44.539787 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.539715 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 01:17:44.539961 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.539934 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 01:17:44.540200 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.540167 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 01:17:44.540402 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.540360 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 01:17:44.540402 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.540321 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 01:17:44.550759 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.550718 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 01:17:44.550759 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.550719 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 01:17:44.550998 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.550742 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 01:17:44.550998 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.550887 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 01:17:44.563308 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.563273 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 01:17:44.563460 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.563407 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 01:17:44.564678 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.564619 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bc1464ba-e4e0-4685-b7d8-c873b189f8de-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.564678 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.564672 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.564885 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.564716 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bc1464ba-e4e0-4685-b7d8-c873b189f8de-config-out\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.564885 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.564779 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.564885 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.564831 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.565035 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.564977 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.565035 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.565020 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.565115 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.565047 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.565188 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.565168 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.565234 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.565201 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-web-config\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.565234 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.565226 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.565341 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.565322 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48qh7\" (UniqueName: \"kubernetes.io/projected/bc1464ba-e4e0-4685-b7d8-c873b189f8de-kube-api-access-48qh7\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.565443 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.565377 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.565443 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.565431 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-config\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.565585 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.565488 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.565585 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.565513 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.565585 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.565558 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.566279 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.566252 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.566402 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.566316 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.566402 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.566363 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.569016 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.568993 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 01:17:44.638482 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.638394 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 01:17:44.638482 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.638394 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 01:17:44.666865 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.666823 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.666865 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.666870 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-web-config\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.667103 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.666900 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.667103 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.666928 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48qh7\" (UniqueName: \"kubernetes.io/projected/bc1464ba-e4e0-4685-b7d8-c873b189f8de-kube-api-access-48qh7\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.667103 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.666965 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.667103 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.666985 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-config\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.667103 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.667017 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.667103 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.667041 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.667103 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.667067 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.667103 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.667093 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.667453 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.667126 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.667453 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.667153 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.667453 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.667369 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.667587 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.667523 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bc1464ba-e4e0-4685-b7d8-c873b189f8de-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.667587 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.667564 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.667688 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.667618 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bc1464ba-e4e0-4685-b7d8-c873b189f8de-config-out\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.667688 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.667650 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.667882 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.667848 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.667947 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.667880 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.667947 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.667914 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.668049 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.667947 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.668049 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.667947 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.668390 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.668365 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.669017 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.668988 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.671518 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.670996 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.672068 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.672014 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-web-config\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.672180 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.672141 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.672250 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.672231 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.672420 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.672388 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.672581 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.672554 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bc1464ba-e4e0-4685-b7d8-c873b189f8de-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.673133 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.673106 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bc1464ba-e4e0-4685-b7d8-c873b189f8de-config-out\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.673305 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.673281 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.673369 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.673341 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.673992 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.673950 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.674078 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.674050 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.674206 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.674184 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.674578 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.674558 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-config\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.675158 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.675142 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.678406 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.678380 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.686660 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.686629 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48qh7\" (UniqueName: \"kubernetes.io/projected/bc1464ba-e4e0-4685-b7d8-c873b189f8de-kube-api-access-48qh7\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.722269 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.722230 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:44.847408 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.847359 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:44.908268 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.907727 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-f7fkx" event={"ID":"0ae3164f-fec1-4210-863e-70e49e6bd26a","Type":"ContainerStarted","Data":"b8c4b30cb949ef90f52ac42e8a57feced28de71316ce0d777b46291165919c03"} Oct 30 01:17:44.949301 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:44.949252 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-f7fkx" podStartSLOduration=1.535274971 podStartE2EDuration="2.949236303s" totalImagesPullingTime="1.413961332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:44.948965423 +0000 UTC m=+63.981256406" watchObservedRunningTime="2026-10-30 01:17:44.949236303 +0000 UTC m=+63.981527283" Oct 30 01:17:45.219175 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:45.219151 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 01:17:45.219692 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:17:45.219668 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc1464ba_e4e0_4685_b7d8_c873b189f8de.slice/crio-a046ba2d3bfee0e2bbdc1974b56581f8e57db6f84f40a65fc361e6f0b5b656f5 WatchSource:0}: Error finding container a046ba2d3bfee0e2bbdc1974b56581f8e57db6f84f40a65fc361e6f0b5b656f5: Status 404 returned error can't find the container with id a046ba2d3bfee0e2bbdc1974b56581f8e57db6f84f40a65fc361e6f0b5b656f5 Oct 30 01:17:45.568437 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:45.568393 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:45.792550 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:45.792509 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:45.913358 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:45.913322 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc1464ba-e4e0-4685-b7d8-c873b189f8de","Type":"ContainerStarted","Data":"a046ba2d3bfee0e2bbdc1974b56581f8e57db6f84f40a65fc361e6f0b5b656f5"} Oct 30 01:17:45.915119 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:45.915087 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" event={"ID":"9bc3b964-6e84-4860-a5bf-0a81f15b0d97","Type":"ContainerStarted","Data":"087734ba379ba0812c2260ad200f470afd2995d3353d8439767a84bc928fc8d0"} Oct 30 01:17:45.915351 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:45.915333 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-f7fkx" Oct 30 01:17:45.920736 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:45.920707 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-f7fkx" Oct 30 01:17:46.793562 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:46.793525 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:46.920927 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:46.920879 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" event={"ID":"9bc3b964-6e84-4860-a5bf-0a81f15b0d97","Type":"ContainerStarted","Data":"cd3c7b1c4c798710af05a2cef643b71bda93ef7c3ed2a871fb1279b618e67ff2"} Oct 30 01:17:46.921128 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:46.920993 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" event={"ID":"9bc3b964-6e84-4860-a5bf-0a81f15b0d97","Type":"ContainerStarted","Data":"3fc92fa9129a81b16052567ecc72034b124aaf27da289a578a491002e371d697"} Oct 30 01:17:46.922480 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:46.922449 2583 generic.go:356] "Generic (PLEG): container finished" podID="bc1464ba-e4e0-4685-b7d8-c873b189f8de" containerID="ea6bdcb072917bdd7945749f813e0f8b851cad79f7d09cb105b7a097c31f7c66" exitCode=0 Oct 30 01:17:46.922602 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:46.922537 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc1464ba-e4e0-4685-b7d8-c873b189f8de","Type":"ContainerDied","Data":"ea6bdcb072917bdd7945749f813e0f8b851cad79f7d09cb105b7a097c31f7c66"} Oct 30 01:17:46.990936 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:46.990881 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" podStartSLOduration=2.345991441 podStartE2EDuration="4.990867695s" totalImagesPullingTime="2.644876254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:46.978065493 +0000 UTC m=+66.010356484" watchObservedRunningTime="2026-10-30 01:17:46.990867695 +0000 UTC m=+66.023158675" Oct 30 01:17:47.008107 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:47.008070 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:47.296195 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:47.296155 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b29936-72de-4ddc-b7f1-a964a614ff61-metrics-certs\") pod \"network-metrics-daemon-64xhb\" (UID: \"f4b29936-72de-4ddc-b7f1-a964a614ff61\") " pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:17:47.296380 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:47.296230 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/88f5d337-fa16-4430-a82e-5f433354c164-original-pull-secret\") pod \"global-pull-secret-syncer-j7l6k\" (UID: \"88f5d337-fa16-4430-a82e-5f433354c164\") " pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:17:47.309764 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:47.309704 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 01:17:47.309900 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:47.309881 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 01:17:47.319336 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:47.319306 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b29936-72de-4ddc-b7f1-a964a614ff61-metrics-certs\") pod \"network-metrics-daemon-64xhb\" (UID: \"f4b29936-72de-4ddc-b7f1-a964a614ff61\") " pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:17:47.319439 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:47.319408 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/88f5d337-fa16-4430-a82e-5f433354c164-original-pull-secret\") pod \"global-pull-secret-syncer-j7l6k\" (UID: \"88f5d337-fa16-4430-a82e-5f433354c164\") " pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:17:47.397586 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:47.397545 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czkw9\" (UniqueName: \"kubernetes.io/projected/1c5a490f-f8db-4afb-9a1f-b1accc9eadea-kube-api-access-czkw9\") pod \"network-check-target-dmnzr\" (UID: \"1c5a490f-f8db-4afb-9a1f-b1accc9eadea\") " pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:17:47.410980 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:47.410953 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 01:17:47.431466 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:47.431436 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 01:17:47.440989 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:47.440962 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czkw9\" (UniqueName: \"kubernetes.io/projected/1c5a490f-f8db-4afb-9a1f-b1accc9eadea-kube-api-access-czkw9\") pod \"network-check-target-dmnzr\" (UID: \"1c5a490f-f8db-4afb-9a1f-b1accc9eadea\") " pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:17:47.616285 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:47.616198 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xp5gx\"" Oct 30 01:17:47.623529 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:47.623495 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:17:47.624386 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:47.624360 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wjscp\"" Oct 30 01:17:47.630764 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:47.630719 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-n79rv\"" Oct 30 01:17:47.631809 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:47.631788 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:17:47.637530 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:47.637504 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:17:47.936618 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:47.936554 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-64xhb"] Oct 30 01:17:47.939408 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:17:47.939369 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b29936_72de_4ddc_b7f1_a964a614ff61.slice/crio-441f6293c35e074c06fc5dab94106eec6cbe639f8d9182f14b539714cae612fc WatchSource:0}: Error finding container 441f6293c35e074c06fc5dab94106eec6cbe639f8d9182f14b539714cae612fc: Status 404 returned error can't find the container with id 441f6293c35e074c06fc5dab94106eec6cbe639f8d9182f14b539714cae612fc Oct 30 01:17:47.948804 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:47.948779 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dmnzr"] Oct 30 01:17:47.949817 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:17:47.949782 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c5a490f_f8db_4afb_9a1f_b1accc9eadea.slice/crio-1116cb567c792d34865bb572924c8f794fcb4aae2cedd9963945cc57f6ae84f3 WatchSource:0}: Error finding container 1116cb567c792d34865bb572924c8f794fcb4aae2cedd9963945cc57f6ae84f3: Status 404 returned error can't find the container with id 1116cb567c792d34865bb572924c8f794fcb4aae2cedd9963945cc57f6ae84f3 Oct 30 01:17:47.973224 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:47.973191 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j7l6k"] Oct 30 01:17:47.976389 ip-10-0-138-223 kubenswrapper[2583]: W1030 01:17:47.976358 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88f5d337_fa16_4430_a82e_5f433354c164.slice/crio-946b2a1dcdb0ea41b11209253f4b624b3b8c0260d683ee44df668300d93a16c4 WatchSource:0}: Error finding container 946b2a1dcdb0ea41b11209253f4b624b3b8c0260d683ee44df668300d93a16c4: Status 404 returned error can't find the container with id 946b2a1dcdb0ea41b11209253f4b624b3b8c0260d683ee44df668300d93a16c4 Oct 30 01:17:48.932455 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:48.932367 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-64xhb" event={"ID":"f4b29936-72de-4ddc-b7f1-a964a614ff61","Type":"ContainerStarted","Data":"441f6293c35e074c06fc5dab94106eec6cbe639f8d9182f14b539714cae612fc"} Oct 30 01:17:48.934948 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:48.934844 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dmnzr" event={"ID":"1c5a490f-f8db-4afb-9a1f-b1accc9eadea","Type":"ContainerStarted","Data":"1116cb567c792d34865bb572924c8f794fcb4aae2cedd9963945cc57f6ae84f3"} Oct 30 01:17:48.936776 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:48.936669 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j7l6k" event={"ID":"88f5d337-fa16-4430-a82e-5f433354c164","Type":"ContainerStarted","Data":"946b2a1dcdb0ea41b11209253f4b624b3b8c0260d683ee44df668300d93a16c4"} Oct 30 01:17:49.920112 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:49.920066 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:49.944576 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:49.943915 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-64xhb" event={"ID":"f4b29936-72de-4ddc-b7f1-a964a614ff61","Type":"ContainerStarted","Data":"78d9f4597440a4ca1f24256dafce8fc83050e856137304e1afe596c6c81c96fc"} Oct 30 01:17:49.944576 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:49.943983 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-64xhb" event={"ID":"f4b29936-72de-4ddc-b7f1-a964a614ff61","Type":"ContainerStarted","Data":"039f0a4eb890b65e99b46768de980754fe5c350ac643e85bc376119fd91a9b68"} Oct 30 01:17:50.823922 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:50.823844 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:51.082012 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:51.081889 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:51.341004 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:51.340802 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:51.559520 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:51.559482 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:51.969774 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:51.969323 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc1464ba-e4e0-4685-b7d8-c873b189f8de","Type":"ContainerStarted","Data":"a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23"} Oct 30 01:17:51.969774 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:51.969451 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc1464ba-e4e0-4685-b7d8-c873b189f8de","Type":"ContainerStarted","Data":"2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73"} Oct 30 01:17:51.969774 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:51.969497 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc1464ba-e4e0-4685-b7d8-c873b189f8de","Type":"ContainerStarted","Data":"fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef"} Oct 30 01:17:52.145960 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:52.145924 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:52.347221 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:52.347167 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:52.974624 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:52.974581 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dmnzr" event={"ID":"1c5a490f-f8db-4afb-9a1f-b1accc9eadea","Type":"ContainerStarted","Data":"090d61781c5b16e235904614388c6bd9f88319e9a8b5e7ffba901c6cb3c2fd35"} Oct 30 01:17:52.974825 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:52.974692 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:17:52.978614 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:52.978583 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc1464ba-e4e0-4685-b7d8-c873b189f8de","Type":"ContainerStarted","Data":"31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1"} Oct 30 01:17:52.978808 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:52.978644 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc1464ba-e4e0-4685-b7d8-c873b189f8de","Type":"ContainerStarted","Data":"3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3"} Oct 30 01:17:52.978808 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:52.978677 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc1464ba-e4e0-4685-b7d8-c873b189f8de","Type":"ContainerStarted","Data":"e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07"} Oct 30 01:17:53.012351 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:53.012291 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dmnzr" podStartSLOduration=67.330603488 podStartE2EDuration="1m12.012272063s" totalImagesPullingTime="4.681668575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:53.011502263 +0000 UTC m=+72.043793246" watchObservedRunningTime="2026-10-30 01:17:53.012272063 +0000 UTC m=+72.044563046" Oct 30 01:17:53.012795 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:53.012741 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-64xhb" podStartSLOduration=70.817245556 podStartE2EDuration="1m12.012731184s" totalImagesPullingTime="1.195485628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:50.008051204 +0000 UTC m=+69.040342186" watchObservedRunningTime="2026-10-30 01:17:53.012731184 +0000 UTC m=+72.045022166" Oct 30 01:17:53.072958 ip-10-0-138-223 kubenswrapper[2583]: I1030 01:17:53.072902 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.438807768 podStartE2EDuration="9.072887669s" totalImagesPullingTime="5.61200623s" totalInitContainerRuntime="22.073671ms" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:53.072032131 +0000 UTC m=+72.104323151" watchObservedRunningTime="2026-10-30 01:17:53.072887669 +0000 UTC m=+72.105178713" Oct 30 01:17:54.358208 ip-10-0-138-223 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 01:17:54.373304 ip-10-0-138-223 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 01:17:54.373485 ip-10-0-138-223 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 01:17:54.373543 ip-10-0-138-223 systemd[1]: kubelet.service: Consumed 4.542s CPU time. Oct 30 01:17:54.388372 ip-10-0-138-223 systemd[1]: Starting Kubernetes Kubelet... Oct 30 01:17:54.456483 ip-10-0-138-223 kubenswrapper[6572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 01:17:54.456483 ip-10-0-138-223 kubenswrapper[6572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 01:17:54.456483 ip-10-0-138-223 kubenswrapper[6572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 01:17:54.456483 ip-10-0-138-223 kubenswrapper[6572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 01:17:54.458565 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458470 6572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:17:54.458565 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458564 6572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:17:54.458565 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458568 6572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:17:54.458565 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458571 6572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:17:54.458685 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458574 6572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:17:54.458685 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458577 6572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:17:54.458685 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458579 6572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:17:54.458685 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458582 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:17:54.458685 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458585 6572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:17:54.458685 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458588 6572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:17:54.458685 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458590 6572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:17:54.458685 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458593 6572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:17:54.458685 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458596 6572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:17:54.458685 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458598 6572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:17:54.458685 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458601 6572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:17:54.458685 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458611 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:17:54.458685 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458614 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:17:54.458685 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458617 6572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:17:54.458685 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458619 6572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:17:54.458685 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458622 6572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:17:54.458685 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458624 6572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:17:54.458685 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458627 6572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:17:54.458685 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458631 6572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:17:54.458685 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458634 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:17:54.459184 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458636 6572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:17:54.459184 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458640 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:17:54.459184 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458643 6572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:17:54.459184 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458646 6572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:17:54.459184 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458649 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:17:54.459184 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458651 6572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:17:54.459184 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458654 6572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:17:54.459184 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458657 6572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:17:54.459184 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458659 6572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:17:54.459184 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458662 6572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:17:54.459184 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458665 6572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:17:54.459184 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458667 6572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:17:54.459184 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458670 6572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:17:54.459184 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458673 6572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:17:54.459184 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458676 6572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:17:54.459184 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458679 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:17:54.459184 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458681 6572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:17:54.459184 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458684 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:17:54.459184 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458687 6572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:17:54.459644 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458690 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:17:54.459644 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458692 6572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:17:54.459644 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458695 6572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:17:54.459644 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458698 6572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:17:54.459644 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458701 6572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:17:54.459644 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458704 6572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:17:54.459644 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458707 6572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:17:54.459644 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458710 6572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:17:54.459644 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458712 6572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:17:54.459644 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458715 6572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:17:54.459644 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458718 6572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:17:54.459644 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458722 6572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:17:54.459644 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458728 6572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:17:54.459644 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458731 6572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:17:54.459644 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458735 6572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:17:54.459644 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458737 6572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:17:54.459644 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458741 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:17:54.459644 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458764 6572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:17:54.459644 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458767 6572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:17:54.460111 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458770 6572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:17:54.460111 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458772 6572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:17:54.460111 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458775 6572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:17:54.460111 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458778 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:17:54.460111 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458780 6572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:17:54.460111 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458783 6572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:17:54.460111 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458786 6572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:17:54.460111 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458789 6572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:17:54.460111 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458791 6572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:17:54.460111 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458794 6572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:17:54.460111 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458796 6572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:17:54.460111 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458799 6572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:17:54.460111 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458802 6572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:17:54.460111 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458804 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:17:54.460111 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458807 6572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:17:54.460111 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458809 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:17:54.460111 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458813 6572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:17:54.460111 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458815 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:17:54.460111 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458819 6572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:17:54.460111 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458822 6572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:17:54.460634 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458824 6572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:17:54.460634 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458827 6572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:17:54.460634 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458829 6572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:17:54.460634 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458832 6572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:17:54.460634 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458835 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:17:54.460634 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458838 6572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:17:54.460634 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458841 6572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:17:54.460634 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458843 6572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:17:54.460634 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458846 6572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:17:54.460634 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458849 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:17:54.460634 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458851 6572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:17:54.460634 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458854 6572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:17:54.460634 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458856 6572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:17:54.460634 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458859 6572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:17:54.460634 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458862 6572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:17:54.460634 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458865 6572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:17:54.460634 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458867 6572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:17:54.460634 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458871 6572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:17:54.460634 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458875 6572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:17:54.461161 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458879 6572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:17:54.461161 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458883 6572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:17:54.461161 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458886 6572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:17:54.461161 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458889 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:17:54.461161 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.458892 6572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:17:54.461161 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459661 6572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:17:54.461161 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459667 6572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:17:54.461161 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459670 6572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:17:54.461161 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459673 6572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:17:54.461161 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459676 6572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:17:54.461161 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459679 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:17:54.461161 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459683 6572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:17:54.461161 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459686 6572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:17:54.461161 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459689 6572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:17:54.461161 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459692 6572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:17:54.461161 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459694 6572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:17:54.461161 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459697 6572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:17:54.461161 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459700 6572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:17:54.461161 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459703 6572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:17:54.461161 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459706 6572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:17:54.461657 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459709 6572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:17:54.461657 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459711 6572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:17:54.461657 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459714 6572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:17:54.461657 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459717 6572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:17:54.461657 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459719 6572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:17:54.461657 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459722 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:17:54.461657 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459724 6572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:17:54.461657 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459727 6572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:17:54.461657 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459729 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:17:54.461657 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459732 6572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:17:54.461657 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459734 6572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:17:54.461657 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459737 6572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:17:54.461657 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459739 6572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:17:54.461657 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459742 6572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:17:54.461657 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459760 6572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:17:54.461657 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459763 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:17:54.461657 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459765 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:17:54.461657 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459768 6572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:17:54.461657 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459771 6572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:17:54.461657 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459774 6572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:17:54.462154 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459776 6572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:17:54.462154 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459779 6572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:17:54.462154 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459782 6572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:17:54.462154 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459786 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:17:54.462154 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459789 6572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:17:54.462154 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459791 6572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:17:54.462154 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459795 6572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:17:54.462154 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459798 6572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:17:54.462154 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459801 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:17:54.462154 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459803 6572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:17:54.462154 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459806 6572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:17:54.462154 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459808 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:17:54.462154 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459811 6572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:17:54.462154 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459814 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:17:54.462154 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459817 6572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:17:54.462154 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459819 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:17:54.462154 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459822 6572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:17:54.462154 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459825 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:17:54.462154 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459827 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:17:54.462621 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459829 6572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:17:54.462621 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459832 6572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:17:54.462621 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459835 6572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:17:54.462621 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459837 6572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:17:54.462621 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459840 6572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:17:54.462621 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459843 6572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:17:54.462621 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459845 6572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:17:54.462621 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459848 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:17:54.462621 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459851 6572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:17:54.462621 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459854 6572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:17:54.462621 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459857 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:17:54.462621 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459859 6572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:17:54.462621 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459862 6572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:17:54.462621 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459864 6572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:17:54.462621 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459867 6572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:17:54.462621 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459876 6572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:17:54.462621 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459879 6572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:17:54.462621 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459882 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:17:54.462621 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459885 6572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:17:54.462621 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459888 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:17:54.463117 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459891 6572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:17:54.463117 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459893 6572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:17:54.463117 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459896 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:17:54.463117 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459899 6572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:17:54.463117 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459901 6572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:17:54.463117 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459904 6572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:17:54.463117 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459906 6572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:17:54.463117 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459909 6572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:17:54.463117 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459912 6572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:17:54.463117 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459915 6572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:17:54.463117 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459917 6572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:17:54.463117 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459920 6572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:17:54.463117 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459922 6572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:17:54.463117 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459925 6572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:17:54.463117 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459928 6572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:17:54.463117 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459931 6572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:17:54.463117 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459934 6572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:17:54.463117 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459936 6572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:17:54.463117 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459939 6572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:17:54.463117 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459942 6572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:17:54.463596 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459945 6572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:17:54.463596 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459949 6572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:17:54.463596 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459952 6572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:17:54.463596 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459954 6572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:17:54.463596 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459958 6572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:17:54.463596 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459962 6572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:17:54.463596 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459965 6572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:17:54.463596 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459969 6572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:17:54.463596 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459971 6572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:17:54.463596 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459974 6572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:17:54.463596 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459978 6572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:17:54.463596 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.459982 6572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:17:54.463596 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460513 6572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:17:54.463596 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460518 6572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:17:54.463596 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460522 6572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:17:54.463596 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460525 6572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:17:54.463596 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460528 6572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:17:54.463596 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460533 6572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:17:54.464048 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460537 6572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:17:54.464048 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460540 6572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:17:54.464048 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460543 6572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:17:54.464048 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460546 6572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:17:54.464048 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460548 6572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:17:54.464048 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460552 6572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:17:54.464048 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460556 6572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:17:54.464048 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460561 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:17:54.464048 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460564 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:17:54.464048 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460567 6572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:17:54.464048 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460570 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:17:54.464048 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460573 6572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:17:54.464048 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460575 6572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:17:54.464048 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460578 6572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:17:54.464048 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460580 6572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:17:54.464048 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460583 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:17:54.464048 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460586 6572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:17:54.464048 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460588 6572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:17:54.464048 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460591 6572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:17:54.464515 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460594 6572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:17:54.464515 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460596 6572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:17:54.464515 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460599 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:17:54.464515 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460602 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:17:54.464515 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460605 6572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:17:54.464515 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460608 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:17:54.464515 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460611 6572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:17:54.464515 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460613 6572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:17:54.464515 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460616 6572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:17:54.464515 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460618 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:17:54.464515 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460621 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:17:54.464515 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460624 6572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:17:54.464515 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460627 6572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:17:54.464515 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460629 6572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:17:54.464515 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460632 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:17:54.464515 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460635 6572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:17:54.464515 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460637 6572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:17:54.464515 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460640 6572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:17:54.464515 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460643 6572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:17:54.465012 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460645 6572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:17:54.465012 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460648 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:17:54.465012 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460650 6572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:17:54.465012 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460653 6572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:17:54.465012 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460655 6572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:17:54.465012 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460658 6572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:17:54.465012 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460661 6572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:17:54.465012 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460663 6572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:17:54.465012 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460667 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:17:54.465012 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460669 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:17:54.465012 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460672 6572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:17:54.465012 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460675 6572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:17:54.465012 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460677 6572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:17:54.465012 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460680 6572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:17:54.465012 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460682 6572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:17:54.465012 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460685 6572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:17:54.465012 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460688 6572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:17:54.465012 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460690 6572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:17:54.465012 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460693 6572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:17:54.465012 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460696 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:17:54.465513 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460698 6572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:17:54.465513 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460702 6572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:17:54.465513 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460704 6572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:17:54.465513 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460706 6572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:17:54.465513 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460709 6572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:17:54.465513 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460711 6572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:17:54.465513 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460714 6572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:17:54.465513 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460717 6572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:17:54.465513 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460719 6572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:17:54.465513 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460722 6572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:17:54.465513 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460724 6572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:17:54.465513 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460727 6572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:17:54.465513 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460729 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:17:54.465513 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460732 6572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:17:54.465513 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460734 6572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:17:54.465513 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460737 6572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:17:54.465513 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460740 6572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:17:54.465513 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460742 6572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:17:54.465513 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460758 6572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:17:54.465513 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460772 6572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:17:54.466000 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460775 6572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:17:54.466000 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460778 6572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:17:54.466000 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460780 6572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:17:54.466000 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460783 6572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:17:54.466000 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460786 6572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:17:54.466000 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460788 6572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:17:54.466000 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460791 6572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:17:54.466000 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460793 6572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:17:54.466000 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460796 6572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:17:54.466000 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460799 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:17:54.466000 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460802 6572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:17:54.466000 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460804 6572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:17:54.466000 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460808 6572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:17:54.466000 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460810 6572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:17:54.466000 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460813 6572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:17:54.466000 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460816 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:17:54.466000 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460818 6572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:17:54.466000 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460821 6572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:17:54.466000 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460823 6572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:17:54.466000 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460827 6572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:17:54.466512 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460830 6572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:17:54.466512 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.460832 6572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:17:54.466512 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.460925 6572 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 01:17:54.466512 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.460939 6572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 01:17:54.466512 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.460949 6572 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 01:17:54.466512 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.460954 6572 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 01:17:54.466512 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.460959 6572 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 01:17:54.466512 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.460962 6572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 01:17:54.466512 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.460967 6572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 01:17:54.466512 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.460971 6572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 01:17:54.466512 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.460975 6572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 01:17:54.466512 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.460978 6572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 01:17:54.466512 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.460981 6572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 01:17:54.466512 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.460985 6572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 01:17:54.466512 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.460988 6572 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 01:17:54.466512 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.460991 6572 flags.go:64] FLAG: --cgroup-root="" Oct 30 01:17:54.466512 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.460994 6572 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 01:17:54.466512 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.460998 6572 flags.go:64] FLAG: --client-ca-file="" Oct 30 01:17:54.466512 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461001 6572 flags.go:64] FLAG: --cloud-provider="external" Oct 30 01:17:54.466512 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461004 6572 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 01:17:54.466512 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461009 6572 flags.go:64] FLAG: --cluster-domain="" Oct 30 01:17:54.466512 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461012 6572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 01:17:54.466512 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461015 6572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 01:17:54.467072 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461018 6572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 01:17:54.467072 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461022 6572 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 01:17:54.467072 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461027 6572 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 01:17:54.467072 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461030 6572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 01:17:54.467072 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461035 6572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 01:17:54.467072 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461038 6572 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 01:17:54.467072 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461041 6572 flags.go:64] FLAG: --contention-profiling="false" Oct 30 01:17:54.467072 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461045 6572 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 01:17:54.467072 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461048 6572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 01:17:54.467072 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461051 6572 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 01:17:54.467072 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461054 6572 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 01:17:54.467072 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461059 6572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 01:17:54.467072 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461062 6572 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 01:17:54.467072 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461067 6572 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 01:17:54.467072 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461070 6572 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 01:17:54.467072 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461073 6572 flags.go:64] FLAG: --enable-server="true" Oct 30 01:17:54.467072 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461077 6572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 01:17:54.467072 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461081 6572 flags.go:64] FLAG: --event-burst="100" Oct 30 01:17:54.467072 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461085 6572 flags.go:64] FLAG: --event-qps="50" Oct 30 01:17:54.467072 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461088 6572 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 01:17:54.467072 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461091 6572 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 01:17:54.467072 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461094 6572 flags.go:64] FLAG: --eviction-hard="" Oct 30 01:17:54.467072 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461098 6572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 01:17:54.467072 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461102 6572 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 01:17:54.467664 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461105 6572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 01:17:54.467664 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461108 6572 flags.go:64] FLAG: --eviction-soft="" Oct 30 01:17:54.467664 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461111 6572 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 01:17:54.467664 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461114 6572 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 01:17:54.467664 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461117 6572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 01:17:54.467664 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461120 6572 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 01:17:54.467664 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461123 6572 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 01:17:54.467664 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461126 6572 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 01:17:54.467664 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461129 6572 flags.go:64] FLAG: --feature-gates="" Oct 30 01:17:54.467664 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461133 6572 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 01:17:54.467664 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461137 6572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 01:17:54.467664 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461140 6572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 01:17:54.467664 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461146 6572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 01:17:54.467664 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461149 6572 flags.go:64] FLAG: --healthz-port="10248" Oct 30 01:17:54.467664 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461153 6572 flags.go:64] FLAG: --help="false" Oct 30 01:17:54.467664 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461156 6572 flags.go:64] FLAG: --hostname-override="ip-10-0-138-223.ec2.internal" Oct 30 01:17:54.467664 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461159 6572 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 01:17:54.467664 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461162 6572 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 01:17:54.467664 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461166 6572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 01:17:54.467664 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461169 6572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 01:17:54.467664 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461173 6572 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 01:17:54.467664 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461178 6572 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 01:17:54.467664 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461181 6572 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 01:17:54.468254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461184 6572 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 01:17:54.468254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461187 6572 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 01:17:54.468254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461190 6572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 01:17:54.468254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461193 6572 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 01:17:54.468254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461196 6572 flags.go:64] FLAG: --kube-reserved="" Oct 30 01:17:54.468254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461199 6572 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 01:17:54.468254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461202 6572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 01:17:54.468254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461205 6572 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 01:17:54.468254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461208 6572 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 01:17:54.468254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461211 6572 flags.go:64] FLAG: --lock-file="" Oct 30 01:17:54.468254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461214 6572 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 01:17:54.468254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461217 6572 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 01:17:54.468254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461220 6572 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 01:17:54.468254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461226 6572 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 01:17:54.468254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461229 6572 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 01:17:54.468254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461232 6572 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 01:17:54.468254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461235 6572 flags.go:64] FLAG: --logging-format="text" Oct 30 01:17:54.468254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461238 6572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 01:17:54.468254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461241 6572 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 01:17:54.468254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461244 6572 flags.go:64] FLAG: --manifest-url="" Oct 30 01:17:54.468254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461247 6572 flags.go:64] FLAG: --manifest-url-header="" Oct 30 01:17:54.468254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461253 6572 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 01:17:54.468254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461257 6572 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 01:17:54.468254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461261 6572 flags.go:64] FLAG: --max-pods="110" Oct 30 01:17:54.468254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461264 6572 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 01:17:54.468925 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461268 6572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 01:17:54.468925 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461271 6572 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 01:17:54.468925 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461274 6572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 01:17:54.468925 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461277 6572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 01:17:54.468925 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461279 6572 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 01:17:54.468925 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461285 6572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 01:17:54.468925 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461292 6572 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 01:17:54.468925 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461295 6572 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 01:17:54.468925 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461298 6572 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 01:17:54.468925 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461301 6572 flags.go:64] FLAG: --pod-cidr="" Oct 30 01:17:54.468925 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461304 6572 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 01:17:54.468925 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461307 6572 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 01:17:54.468925 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461310 6572 flags.go:64] FLAG: --pods-per-core="0" Oct 30 01:17:54.468925 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461313 6572 flags.go:64] FLAG: --port="10250" Oct 30 01:17:54.468925 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461316 6572 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 01:17:54.468925 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461319 6572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f0e1a6b194e4cd23" Oct 30 01:17:54.468925 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461323 6572 flags.go:64] FLAG: --qos-reserved="" Oct 30 01:17:54.468925 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461326 6572 flags.go:64] FLAG: --read-only-port="10255" Oct 30 01:17:54.468925 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461329 6572 flags.go:64] FLAG: --register-node="true" Oct 30 01:17:54.468925 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461332 6572 flags.go:64] FLAG: --register-with-taints="" Oct 30 01:17:54.468925 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461336 6572 flags.go:64] FLAG: --registry-burst="10" Oct 30 01:17:54.468925 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461339 6572 flags.go:64] FLAG: --registry-qps="5" Oct 30 01:17:54.468925 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461342 6572 flags.go:64] FLAG: --reserved-cpus="" Oct 30 01:17:54.468925 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461345 6572 flags.go:64] FLAG: --reserved-memory="" Oct 30 01:17:54.468925 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461349 6572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 01:17:54.468925 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461352 6572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 01:17:54.469617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461356 6572 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 01:17:54.469617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461359 6572 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 01:17:54.469617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461363 6572 flags.go:64] FLAG: --runonce="false" Oct 30 01:17:54.469617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461367 6572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 01:17:54.469617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461370 6572 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 01:17:54.469617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461373 6572 flags.go:64] FLAG: --seccomp-default="false" Oct 30 01:17:54.469617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461376 6572 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 01:17:54.469617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461379 6572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 01:17:54.469617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461382 6572 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 01:17:54.469617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461386 6572 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 01:17:54.469617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461389 6572 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 01:17:54.469617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461394 6572 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 01:17:54.469617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461397 6572 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 01:17:54.469617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461400 6572 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 01:17:54.469617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461403 6572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 01:17:54.469617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461406 6572 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 01:17:54.469617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461410 6572 flags.go:64] FLAG: --system-cgroups="" Oct 30 01:17:54.469617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461412 6572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 01:17:54.469617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461418 6572 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 01:17:54.469617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461421 6572 flags.go:64] FLAG: --tls-cert-file="" Oct 30 01:17:54.469617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461424 6572 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 01:17:54.469617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461429 6572 flags.go:64] FLAG: --tls-min-version="" Oct 30 01:17:54.469617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461432 6572 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 01:17:54.469617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461435 6572 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 01:17:54.469617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461438 6572 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 01:17:54.470550 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461441 6572 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 01:17:54.470550 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461444 6572 flags.go:64] FLAG: --v="2" Oct 30 01:17:54.470550 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461448 6572 flags.go:64] FLAG: --version="false" Oct 30 01:17:54.470550 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461452 6572 flags.go:64] FLAG: --vmodule="" Oct 30 01:17:54.470550 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461457 6572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 01:17:54.470550 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461460 6572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 01:17:54.470550 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461604 6572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:17:54.470550 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461608 6572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:17:54.470550 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461612 6572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:17:54.470550 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461615 6572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:17:54.470550 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461618 6572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:17:54.470550 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461622 6572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:17:54.470550 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461624 6572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:17:54.470550 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461628 6572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:17:54.470550 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461630 6572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:17:54.470550 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461633 6572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:17:54.470550 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461636 6572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:17:54.470550 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461639 6572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:17:54.470550 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461643 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:17:54.470550 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461646 6572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:17:54.470550 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461649 6572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:17:54.471404 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461651 6572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:17:54.471404 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461654 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:17:54.471404 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461657 6572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:17:54.471404 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461660 6572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:17:54.471404 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461663 6572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:17:54.471404 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461666 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:17:54.471404 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461669 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:17:54.471404 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461671 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:17:54.471404 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461674 6572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:17:54.471404 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461677 6572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:17:54.471404 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461680 6572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:17:54.471404 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461682 6572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:17:54.471404 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461685 6572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:17:54.471404 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461688 6572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:17:54.471404 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461691 6572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:17:54.471404 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461693 6572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:17:54.471404 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461696 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:17:54.471404 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461699 6572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:17:54.471404 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461701 6572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:17:54.472223 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461705 6572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:17:54.472223 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461710 6572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:17:54.472223 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461713 6572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:17:54.472223 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461716 6572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:17:54.472223 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461719 6572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:17:54.472223 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461722 6572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:17:54.472223 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461725 6572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:17:54.472223 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461727 6572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:17:54.472223 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461730 6572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:17:54.472223 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461733 6572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:17:54.472223 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461737 6572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:17:54.472223 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461740 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:17:54.472223 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461742 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:17:54.472223 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461758 6572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:17:54.472223 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461761 6572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:17:54.472223 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461763 6572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:17:54.472223 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461766 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:17:54.472223 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461769 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:17:54.472223 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461772 6572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:17:54.472942 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461774 6572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:17:54.472942 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461777 6572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:17:54.472942 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461780 6572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:17:54.472942 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461783 6572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:17:54.472942 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461785 6572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:17:54.472942 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461788 6572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:17:54.472942 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461790 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:17:54.472942 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461793 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:17:54.472942 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461795 6572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:17:54.472942 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461798 6572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:17:54.472942 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461801 6572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:17:54.472942 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461803 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:17:54.472942 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461806 6572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:17:54.472942 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461808 6572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:17:54.472942 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461812 6572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:17:54.472942 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461815 6572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:17:54.472942 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461818 6572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:17:54.472942 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461820 6572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:17:54.472942 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461824 6572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:17:54.473461 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461827 6572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:17:54.473461 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461830 6572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:17:54.473461 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461832 6572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:17:54.473461 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461835 6572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:17:54.473461 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461839 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:17:54.473461 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461842 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:17:54.473461 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461844 6572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:17:54.473461 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461847 6572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:17:54.473461 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461849 6572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:17:54.473461 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461852 6572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:17:54.473461 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461855 6572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:17:54.473461 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461858 6572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:17:54.473461 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461860 6572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:17:54.473461 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461863 6572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:17:54.473461 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461865 6572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:17:54.473461 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461868 6572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:17:54.473461 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461871 6572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:17:54.473461 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461874 6572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:17:54.473461 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461876 6572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:17:54.473461 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461879 6572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:17:54.474041 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461882 6572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:17:54.474041 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461884 6572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:17:54.474041 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461888 6572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:17:54.474041 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461892 6572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:17:54.474041 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461895 6572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:17:54.474041 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461897 6572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:17:54.474041 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461900 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:17:54.474041 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461905 6572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:17:54.474041 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461907 6572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:17:54.474041 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461910 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:17:54.474041 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461913 6572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:17:54.474041 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461916 6572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:17:54.474041 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461919 6572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:17:54.474041 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461922 6572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:17:54.474041 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.461950 6572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 01:17:54.474041 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461987 6572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:17:54.474437 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461993 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:17:54.474437 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461996 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:17:54.474437 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.461999 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:17:54.474437 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462003 6572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:17:54.474437 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462005 6572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:17:54.474437 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462008 6572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:17:54.474437 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462011 6572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:17:54.474437 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462014 6572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:17:54.474437 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462016 6572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:17:54.474437 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462019 6572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:17:54.474437 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462022 6572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:17:54.474437 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462024 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:17:54.474437 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462027 6572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:17:54.474437 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462029 6572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:17:54.474437 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462033 6572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:17:54.474437 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462037 6572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:17:54.474437 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462041 6572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:17:54.474437 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462043 6572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:17:54.474437 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462046 6572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:17:54.474930 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462048 6572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:17:54.474930 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462051 6572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:17:54.474930 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462054 6572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:17:54.474930 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462056 6572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:17:54.474930 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462059 6572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:17:54.474930 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462063 6572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:17:54.474930 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462065 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:17:54.474930 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462068 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:17:54.474930 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462071 6572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:17:54.474930 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462074 6572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:17:54.474930 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462077 6572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:17:54.474930 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462080 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:17:54.474930 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462083 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:17:54.474930 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462087 6572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:17:54.474930 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462090 6572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:17:54.474930 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462093 6572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:17:54.474930 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462095 6572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:17:54.474930 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462098 6572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:17:54.474930 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462100 6572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:17:54.474930 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462103 6572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:17:54.475396 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462106 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:17:54.475396 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462108 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:17:54.475396 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462111 6572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:17:54.475396 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462113 6572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:17:54.475396 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462116 6572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:17:54.475396 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462118 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:17:54.475396 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462121 6572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:17:54.475396 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462124 6572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:17:54.475396 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462126 6572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:17:54.475396 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462129 6572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:17:54.475396 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462131 6572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:17:54.475396 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462134 6572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:17:54.475396 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462137 6572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:17:54.475396 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462139 6572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:17:54.475396 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462142 6572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:17:54.475396 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462146 6572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:17:54.475396 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462148 6572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:17:54.475396 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462153 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:17:54.475396 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462156 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:17:54.475965 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462159 6572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:17:54.475965 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462162 6572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:17:54.475965 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462165 6572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:17:54.475965 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462167 6572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:17:54.475965 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462171 6572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:17:54.475965 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462174 6572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:17:54.475965 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462178 6572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:17:54.475965 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462180 6572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:17:54.475965 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462183 6572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:17:54.475965 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462186 6572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:17:54.475965 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462189 6572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:17:54.475965 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462191 6572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:17:54.475965 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462194 6572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:17:54.475965 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462197 6572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:17:54.475965 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462199 6572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:17:54.475965 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462202 6572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:17:54.475965 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462206 6572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:17:54.475965 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462209 6572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:17:54.475965 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462212 6572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:17:54.475965 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462214 6572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:17:54.476463 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462217 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:17:54.476463 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462220 6572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:17:54.476463 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462223 6572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:17:54.476463 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462225 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:17:54.476463 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462228 6572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:17:54.476463 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462230 6572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:17:54.476463 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462233 6572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:17:54.476463 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462235 6572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:17:54.476463 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462238 6572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:17:54.476463 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462240 6572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:17:54.476463 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462244 6572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:17:54.476463 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462247 6572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:17:54.476463 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462250 6572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:17:54.476463 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462252 6572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:17:54.476463 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462255 6572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:17:54.476463 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462258 6572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:17:54.476463 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462260 6572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:17:54.476463 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462263 6572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:17:54.476463 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462268 6572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:17:54.476463 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462270 6572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:17:54.476967 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462273 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:17:54.476967 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462276 6572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:17:54.476967 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462278 6572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:17:54.476967 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462281 6572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:17:54.476967 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462284 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:17:54.476967 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462286 6572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:17:54.476967 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.462290 6572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:17:54.476967 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.466088 6572 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 01:17:54.476967 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.466105 6572 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 01:17:54.476967 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.466124 6572 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 01:17:54.476967 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.466128 6572 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 01:17:54.476967 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466190 6572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:17:54.476967 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466196 6572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:17:54.476967 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466200 6572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:17:54.476967 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466203 6572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:17:54.476967 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466206 6572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:17:54.476967 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466209 6572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:17:54.476967 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466212 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:17:54.476967 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466215 6572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:17:54.476967 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466218 6572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:17:54.477456 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466220 6572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:17:54.477456 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466223 6572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:17:54.477456 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466225 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:17:54.477456 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466229 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:17:54.477456 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466231 6572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:17:54.477456 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466234 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:17:54.477456 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466237 6572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:17:54.477456 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466241 6572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:17:54.477456 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466245 6572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:17:54.477456 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466249 6572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:17:54.477456 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466252 6572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:17:54.477456 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466255 6572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:17:54.477456 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466258 6572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:17:54.477456 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466261 6572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:17:54.477456 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466264 6572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:17:54.477456 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466266 6572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:17:54.477456 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466269 6572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:17:54.477456 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466272 6572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:17:54.477456 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466275 6572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:17:54.478119 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466278 6572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:17:54.478119 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466281 6572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:17:54.478119 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466283 6572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:17:54.478119 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466286 6572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:17:54.478119 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466289 6572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:17:54.478119 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466291 6572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:17:54.478119 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466294 6572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:17:54.478119 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466297 6572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:17:54.478119 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466300 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:17:54.478119 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466302 6572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:17:54.478119 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466305 6572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:17:54.478119 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466307 6572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:17:54.478119 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466310 6572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:17:54.478119 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466313 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:17:54.478119 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466315 6572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:17:54.478119 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466319 6572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:17:54.478119 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466322 6572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:17:54.478119 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466325 6572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:17:54.478119 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466327 6572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:17:54.478119 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466330 6572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:17:54.478647 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466333 6572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:17:54.478647 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466335 6572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:17:54.478647 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466338 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:17:54.478647 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466342 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:17:54.478647 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466344 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:17:54.478647 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466347 6572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:17:54.478647 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466349 6572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:17:54.478647 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466353 6572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:17:54.478647 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466357 6572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:17:54.478647 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466360 6572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:17:54.478647 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466363 6572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:17:54.478647 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466366 6572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:17:54.478647 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466369 6572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:17:54.478647 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466371 6572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:17:54.478647 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466374 6572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:17:54.478647 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466376 6572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:17:54.478647 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466379 6572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:17:54.478647 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466382 6572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:17:54.478647 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466384 6572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:17:54.479116 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466387 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:17:54.479116 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466389 6572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:17:54.479116 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466392 6572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:17:54.479116 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466395 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:17:54.479116 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466398 6572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:17:54.479116 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466400 6572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:17:54.479116 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466404 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:17:54.479116 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466407 6572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:17:54.479116 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466409 6572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:17:54.479116 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466412 6572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:17:54.479116 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466415 6572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:17:54.479116 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466418 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:17:54.479116 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466420 6572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:17:54.479116 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466423 6572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:17:54.479116 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466425 6572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:17:54.479116 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466428 6572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:17:54.479116 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466431 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:17:54.479116 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466434 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:17:54.479116 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466437 6572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:17:54.479568 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466439 6572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:17:54.479568 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466442 6572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:17:54.479568 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466444 6572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:17:54.479568 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466447 6572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:17:54.479568 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466449 6572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:17:54.479568 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466452 6572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:17:54.479568 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466455 6572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:17:54.479568 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466457 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:17:54.479568 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466459 6572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:17:54.479568 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466462 6572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:17:54.479568 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466465 6572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:17:54.479568 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466467 6572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:17:54.479568 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466470 6572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:17:54.479568 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466472 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:17:54.479568 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466475 6572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:17:54.479568 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466478 6572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:17:54.479568 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466481 6572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:17:54.479568 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466484 6572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:17:54.479568 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466487 6572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:17:54.479568 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466490 6572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:17:54.480076 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.466518 6572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 01:17:54.480076 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466659 6572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:17:54.480076 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466664 6572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:17:54.480076 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466667 6572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:17:54.480076 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466670 6572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:17:54.480076 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466672 6572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:17:54.480076 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466675 6572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:17:54.480076 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466679 6572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:17:54.480076 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466681 6572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:17:54.480076 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466684 6572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:17:54.480076 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466687 6572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:17:54.480076 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466690 6572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:17:54.480076 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466693 6572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:17:54.480076 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466695 6572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:17:54.480076 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466698 6572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:17:54.480076 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466701 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:17:54.480459 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466703 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:17:54.480459 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466706 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:17:54.480459 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466709 6572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:17:54.480459 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466712 6572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:17:54.480459 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466714 6572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:17:54.480459 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466717 6572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:17:54.480459 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466720 6572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:17:54.480459 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466722 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:17:54.480459 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466725 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:17:54.480459 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466727 6572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:17:54.480459 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466730 6572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:17:54.480459 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466733 6572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:17:54.480459 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466735 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:17:54.480459 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466738 6572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:17:54.480459 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466741 6572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:17:54.480459 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466758 6572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:17:54.480459 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466762 6572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:17:54.480459 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466765 6572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:17:54.480459 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466768 6572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:17:54.480959 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466771 6572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:17:54.480959 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466774 6572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:17:54.480959 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466776 6572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:17:54.480959 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466779 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:17:54.480959 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466781 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:17:54.480959 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466784 6572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:17:54.480959 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466786 6572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:17:54.480959 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466789 6572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:17:54.480959 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466792 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:17:54.480959 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466795 6572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:17:54.480959 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466798 6572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:17:54.480959 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466800 6572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:17:54.480959 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466804 6572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:17:54.480959 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466808 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:17:54.480959 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466811 6572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:17:54.480959 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466813 6572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:17:54.480959 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466816 6572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:17:54.480959 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466819 6572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:17:54.480959 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466821 6572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:17:54.480959 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466824 6572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:17:54.481531 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466826 6572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:17:54.481531 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466829 6572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:17:54.481531 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466831 6572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:17:54.481531 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466834 6572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:17:54.481531 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466836 6572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:17:54.481531 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466839 6572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:17:54.481531 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466842 6572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:17:54.481531 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466844 6572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:17:54.481531 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466847 6572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:17:54.481531 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466850 6572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:17:54.481531 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466852 6572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:17:54.481531 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466855 6572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:17:54.481531 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466857 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:17:54.481531 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466860 6572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:17:54.481531 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466864 6572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:17:54.481531 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466867 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:17:54.481531 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466870 6572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:17:54.481531 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466873 6572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:17:54.481531 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466876 6572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:17:54.482019 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466879 6572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:17:54.482019 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466881 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:17:54.482019 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466884 6572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:17:54.482019 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466887 6572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:17:54.482019 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466890 6572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:17:54.482019 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466893 6572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:17:54.482019 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466895 6572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:17:54.482019 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466898 6572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:17:54.482019 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466901 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:17:54.482019 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466903 6572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:17:54.482019 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466906 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:17:54.482019 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466908 6572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:17:54.482019 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466911 6572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:17:54.482019 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466914 6572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:17:54.482019 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466916 6572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:17:54.482019 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466919 6572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:17:54.482019 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466922 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:17:54.482019 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466924 6572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:17:54.482019 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466927 6572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:17:54.482481 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466930 6572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:17:54.482481 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466933 6572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:17:54.482481 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466935 6572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:17:54.482481 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466937 6572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:17:54.482481 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466940 6572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:17:54.482481 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466943 6572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:17:54.482481 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466945 6572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:17:54.482481 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466948 6572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:17:54.482481 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466950 6572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:17:54.482481 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466953 6572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:17:54.482481 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466955 6572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:17:54.482481 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466958 6572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:17:54.482481 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466960 6572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:17:54.482481 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.466963 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:17:54.482481 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.466987 6572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 01:17:54.482481 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467020 6572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:17:54.482933 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467025 6572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:17:54.482933 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467028 6572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:17:54.482933 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467032 6572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:17:54.482933 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467034 6572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:17:54.482933 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467037 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:17:54.482933 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467041 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:17:54.482933 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467043 6572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:17:54.482933 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467046 6572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:17:54.482933 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467049 6572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:17:54.482933 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467051 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:17:54.482933 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467057 6572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:17:54.482933 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467060 6572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:17:54.482933 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467063 6572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:17:54.482933 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467065 6572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:17:54.482933 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467068 6572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:17:54.482933 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467071 6572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:17:54.482933 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467073 6572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:17:54.482933 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467076 6572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:17:54.482933 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467079 6572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:17:54.483384 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467081 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:17:54.483384 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467085 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:17:54.483384 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467087 6572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:17:54.483384 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467090 6572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:17:54.483384 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467093 6572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:17:54.483384 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467095 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:17:54.483384 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467098 6572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:17:54.483384 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467101 6572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:17:54.483384 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467103 6572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:17:54.483384 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467107 6572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:17:54.483384 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467111 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:17:54.483384 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467114 6572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:17:54.483384 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467116 6572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:17:54.483384 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467119 6572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:17:54.483384 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467121 6572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:17:54.483384 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467125 6572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:17:54.483384 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467127 6572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:17:54.483384 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467130 6572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:17:54.483384 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467133 6572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:17:54.483929 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467136 6572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:17:54.483929 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467138 6572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:17:54.483929 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467141 6572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:17:54.483929 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467144 6572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:17:54.483929 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467150 6572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:17:54.483929 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467153 6572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:17:54.483929 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467155 6572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:17:54.483929 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467158 6572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:17:54.483929 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467160 6572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:17:54.483929 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467163 6572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:17:54.483929 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467166 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:17:54.483929 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467169 6572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:17:54.483929 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467173 6572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:17:54.483929 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467177 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:17:54.483929 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467180 6572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:17:54.483929 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467183 6572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:17:54.483929 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467186 6572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:17:54.483929 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467188 6572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:17:54.483929 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467191 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:17:54.484415 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467194 6572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:17:54.484415 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467196 6572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:17:54.484415 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467199 6572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:17:54.484415 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467201 6572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:17:54.484415 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467204 6572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:17:54.484415 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467207 6572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:17:54.484415 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467210 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:17:54.484415 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467212 6572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:17:54.484415 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467215 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:17:54.484415 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467217 6572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:17:54.484415 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467221 6572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:17:54.484415 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467224 6572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:17:54.484415 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467227 6572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:17:54.484415 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467229 6572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:17:54.484415 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467232 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:17:54.484415 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467235 6572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:17:54.484415 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467237 6572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:17:54.484415 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467241 6572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:17:54.484415 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467244 6572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:17:54.484898 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467246 6572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:17:54.484898 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467249 6572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:17:54.484898 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467252 6572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:17:54.484898 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467254 6572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:17:54.484898 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467257 6572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:17:54.484898 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467259 6572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:17:54.484898 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467262 6572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:17:54.484898 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467264 6572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:17:54.484898 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467267 6572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:17:54.484898 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467269 6572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:17:54.484898 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467272 6572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:17:54.484898 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467274 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:17:54.484898 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467277 6572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:17:54.484898 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467280 6572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:17:54.484898 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467282 6572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:17:54.484898 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467285 6572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:17:54.484898 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467287 6572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:17:54.484898 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467290 6572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:17:54.484898 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467293 6572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:17:54.484898 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467295 6572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:17:54.485377 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467298 6572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:17:54.485377 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467300 6572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:17:54.485377 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467303 6572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:17:54.485377 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467306 6572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:17:54.485377 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467309 6572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:17:54.485377 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467311 6572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:17:54.485377 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467314 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:17:54.485377 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467316 6572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:17:54.485377 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:17:54.467320 6572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:17:54.485377 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.467373 6572 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 01:17:54.485377 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.468414 6572 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 01:17:54.485377 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.468476 6572 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 01:17:54.485377 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.468664 6572 server.go:1014] "Starting client certificate rotation" Oct 30 01:17:54.485377 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.468816 6572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 01:17:54.485377 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.468898 6572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 01:11:41 +0000 UTC" deadline="2028-06-02 08:02:37.150913737 +0000 UTC" Oct 30 01:17:54.485377 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.468948 6572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13950h44m42.681969301s" Oct 30 01:17:54.485793 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.469196 6572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 01:17:54.485793 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.471658 6572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 01:17:54.485793 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.473095 6572 log.go:25] "Validated CRI v1 runtime API" Oct 30 01:17:54.485793 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.474959 6572 log.go:25] "Validated CRI v1 image API" Oct 30 01:17:54.485793 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.475742 6572 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 01:17:54.485793 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.477983 6572 fs.go:135] Filesystem UUIDs: map[18798d15-a7a2-4adb-b7e4-7f663ea9309b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e41383ef-8ce0-477f-8523-ffd98ae4edd0:/dev/nvme0n1p3] Oct 30 01:17:54.485994 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.478004 6572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1116cb567c792d34865bb572924c8f794fcb4aae2cedd9963945cc57f6ae84f3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1116cb567c792d34865bb572924c8f794fcb4aae2cedd9963945cc57f6ae84f3/userdata/shm major:0 minor:413 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/16b8d3afdfe0b8d1782acbd5b377b2fa9b2f56255bc2a6aba928ce62ab88f227/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/16b8d3afdfe0b8d1782acbd5b377b2fa9b2f56255bc2a6aba928ce62ab88f227/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/27a1569120759fae8a09aba518f9bcfd8599fa09d5a4a3d906ef8ac73a041c05/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/27a1569120759fae8a09aba518f9bcfd8599fa09d5a4a3d906ef8ac73a041c05/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2a28007d9fd8ecf4d20c4694935fff740ce2fa87946bb4f514fab6cbcef3b920/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2a28007d9fd8ecf4d20c4694935fff740ce2fa87946bb4f514fab6cbcef3b920/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e7b0dcd5c52d7ed1b1ab65a474ce3b34b5a528207d6a62d0901cc211d6c1daf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e7b0dcd5c52d7ed1b1ab65a474ce3b34b5a528207d6a62d0901cc211d6c1daf/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3bd77351638fccdbe592f3c0c30162240591236be59aff6b515940657a064a2b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3bd77351638fccdbe592f3c0c30162240591236be59aff6b515940657a064a2b/userdata/shm major:0 minor:265 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/441f6293c35e074c06fc5dab94106eec6cbe639f8d9182f14b539714cae612fc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/441f6293c35e074c06fc5dab94106eec6cbe639f8d9182f14b539714cae612fc/userdata/shm major:0 minor:411 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4804e601006c9694ff3e71642609d0091e993141294c870ed271be987c4c434a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4804e601006c9694ff3e71642609d0091e993141294c870ed271be987c4c434a/userdata/shm major:0 minor:320 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5fa780ef9495d54c00886fe917a2c6843587f55391ce8b458ae71ac8d377c8e6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5fa780ef9495d54c00886fe917a2c6843587f55391ce8b458ae71ac8d377c8e6/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/61c3f93c78077742aa562727b783c60bb0f3a3341a17838f2669945a71f08ca1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/61c3f93c78077742aa562727b783c60bb0f3a3341a17838f2669945a71f08ca1/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6f4aa6724eabbbabd00cb2f9343dc873cf56f83d7f5a7b511923590f68455e50/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6f4aa6724eabbbabd00cb2f9343dc873cf56f83d7f5a7b511923590f68455e50/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/758ba053dd70b7aa4ba6e97a8652ecd512db79c8dfeeb0a29de1f494a16fdc23/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/758ba053dd70b7aa4ba6e97a8652ecd512db79c8dfeeb0a29de1f494a16fdc23/userdata/shm major:0 minor:261 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/76d6428545389bf939771ec6760306f141d7037d715bab0edee0308cd2a9cf45/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/76d6428545389bf939771ec6760306f141d7037d715bab0edee0308cd2a9cf45/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7ce030618253c5c68cb56a995163f8a4d99ae28ad868ecbc8cc2464bf3595ab6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7ce030618253c5c68cb56a995163f8a4d99ae28ad868ecbc8cc2464bf3595ab6/userdata/shm major:0 minor:259 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8ae82ecd7a980f22b2e052f7bae608beaa4523926642ab8ca614d120bc8cb0bb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8ae82ecd7a980f22b2e052f7bae608beaa4523926642ab8ca614d120bc8cb0bb/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/946b2a1dcdb0ea41b11209253f4b624b3b8c0260d683ee44df668300d93a16c4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/946b2a1dcdb0ea41b11209253f4b624b3b8c0260d683ee44df668300d93a16c4/userdata/shm major:0 minor:415 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a046ba2d3bfee0e2bbdc1974b56581f8e57db6f84f40a65fc361e6f0b5b656f5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a046ba2d3bfee0e2bbdc1974b56581f8e57db6f84f40a65fc361e6f0b5b656f5/userdata/shm major:0 minor:388 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a1d852b890b43a7fcf9884a14d90e565e73bf0e32d149b3ef12324d8e15319b8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a1d852b890b43a7fcf9884a14d90e565e73bf0e32d149b3ef12324d8e15319b8/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a628f193fbba8c2be9f20c3c3119c8801bbc4b36abf1144c487c708f097e4e82/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a628f193fbba8c2be9f20c3c3119c8801bbc4b36abf1144c487c708f097e4e82/userdata/shm major:0 minor:351 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a89f78f9ff48ec09e65d336ba89ab39eba3f6e47e0e47b47d563d8370fb90a98/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a89f78f9ff48ec09e65d336ba89ab39eba3f6e47e0e47b47d563d8370fb90a98/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b2039c18dcbb8325a89eeee1c2021801fdc09d8b18b5cb888aaec7e98a9ee3d7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b2039c18dcbb8325a89eeee1c2021801fdc09d8b18b5cb888aaec7e98a9ee3d7/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eb32738b0ac5a00b52a65f77ffc436810dabbd33e62e70360a962b704a1b28ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eb32738b0ac5a00b52a65f77ffc436810dabbd33e62e70360a962b704a1b28ca/userdata/shm major:0 minor:211 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f19bfb393c5bc8f29de77632302d48be4377d9623cb8c2a1d62d72b200c4e4e5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f19bfb393c5bc8f29de77632302d48be4377d9623cb8c2a1d62d72b200c4e4e5/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00c444f0-9313-4c0a-b9c8-79a953eaba17/volumes/kubernetes.io~projected/kube-api-access-j5bjt:{mountpoint:/var/lib/kubelet/pods/00c444f0-9313-4c0a-b9c8-79a953eaba17/volumes/kubernetes.io~projected/kube-api-access-j5bjt major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00c444f0-9313-4c0a-b9c8-79a953eaba17/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/00c444f0-9313-4c0a-b9c8-79a953eaba17/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03772f27-6bc6-4048-a515-d8d4e9abf431/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/03772f27-6bc6-4048-a515-d8d4e9abf431/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03772f27-6bc6-4048-a515-d8d4e9abf431/volumes/kubernetes.io~projected/kube-api-access-46pfs:{mountpoint:/var/lib/kubelet/pods/03772f27-6bc6-4048-a515-d8d4e9abf431/volumes/kubernetes.io~projected/kube-api-access-46pfs major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03772f27-6bc6-4048-a515-d8d4e9abf431/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/03772f27-6bc6-4048-a515-d8d4e9abf431/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/04631a42-b08e-4aff-9fc4-78dd7392d59d/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/04631a42-b08e-4aff-9fc4-78dd7392d59d/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/04631a42-b08e-4aff-9fc4-78dd7392d59d/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/04631a42-b08e-4aff-9fc4-78dd7392d59d/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/04631a42-b08e-4aff-9fc4-78dd7392d59d/volumes/kubernetes.io~projected/kube-api-access-9jnnc:{mountpoint:/var/lib/kubelet/pods/04631a42-b08e-4aff-9fc4-78dd7392d59d/volumes/kubernetes.io~projected/kube-api-access-9jnnc major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ae3164f-fec1-4210-863e-70e49e6bd26a/volumes/kubernetes.io~projected/kube-api-access-2ljcm:{mountpoint:/var/lib/kubelet/pods/0ae3164f-fec1-4210-863e-70e49e6bd26a/volumes/kubernetes.io~projected/kube-api-access-2ljcm major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ae3164f-fec1-4210-863e-70e49e6bd26a/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/0ae3164f-fec1-4210-863e-70e49e6bd26a/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c5a490f-f8db-4afb-9a1f-b1accc9eadea/volumes/kubernetes.io~projected/kube-api-access-czkw9:{mountpoint:/var/lib/kubelet/pods/1c5a490f-f8db-4afb-9a1f-b1accc9eadea/volumes/kubernetes.io~projected/kube-api-access-czkw9 major:0 minor:410 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/262c01e0-8469-4fbe-aaa4-8a8c50551fce/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/262c01e0-8469-4fbe-aaa4-8a8c50551fce/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3efd0b05-2dcd-4e6c-a24e-9e955a58fa9b/volumes/kubernetes.io~projected/kube-api-access-q2d66:{mountpoint:/var/lib/kubelet/pods/3efd0b05-2dcd-4e6c-a24e-9e955a58fa9b/volumes/kubernetes.io~projected/kube-api-access-q2d66 major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3efd0b05-2dcd-4e6c-a24e-9e955a58fa9b/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/3efd0b05-2dcd-4e6c-a24e-9e955a58fa9b/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b97aea6-bab9-498d-bce7-95e2a47c8c05/volumes/kubernetes.io~projected/kube-api-access-gc85r:{mountpoint:/var/lib/kubelet/pods/4b97aea6-bab9-498d-bce7-95e2a47c8c05/volumes/kubernetes.io~projected/kube-api-access-gc85r major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88f5d337-fa16-4430-a82e-5f433354c164/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/88f5d337-fa16-4430-a82e-5f433354c164/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:409 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b00a159-66f4-4b9d-bcf1-560094dd69bb/volumes/kubernetes.io~projected/kube-api-access-fm4x7:{mountpoint:/var/lib/kubelet/pods/8b00a159-66f4-4b9d-bcf1-560094dd69bb/volumes/kubernetes.io~projected/kube-api-access-fm4x7 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9bc3b964-6e84-4860-a5bf-0a81f15b0d97/volumes/kubernetes.io~projected/kube-api-access-zzkj9:{mountpoint:/var/lib/kubelet/pods/9bc3b964-6e84-4860-a5bf-0a81f15b0d97/volumes/kubernetes.io~projected/kube-api-access-zzkj9 major:0 minor:350 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9bc3b964-6e84-4860-a5bf-0a81f15b0d97/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/9bc3b964-6e84-4860-a5bf-0a81f15b0d97/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9bc3b964-6e84-4860-a5bf-0a81f15b0d97/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/9bc3b964-6e84-4860-a5bf-0a81f15b0d97/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9bc3b964-6e84-4860-a5bf-0a81f15b0d97/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/9bc3b964-6e84-4860-a5bf-0a81f15b0d97/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9bc3b964-6e84-4860-a5bf-0a81f15b0d97/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/9bc3b964-6e84-4860-a5bf-0a81f15b0d97/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ceda047-30e9-496c-b854-af4043a83656/volumes/kubernetes.io~projected/kube-api-access-tqww5:{mountpoint:/var/lib/kubelet/pods/9ceda047-30e9-496c-b854-af4043a83656/volumes/kubernetes.io~projected/kube-api-access-tqww5 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e829899-8a60-4792-b4cc-fabcaff0469f/volumes/kubernetes.io~projected/kube-api-access-n85mv:{mountpoint:/var/lib/kubelet/pods/9e829899-8a60-4792-b4cc-fabcaff0469f/volumes/kubernetes.io~projected/kube-api-access-n85mv major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e829899-8a60-4792-b4cc-fabcaff0469f/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/9e829899-8a60-4792-b4cc-fabcaff0469f/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e829899-8a60-4792-b4cc-fabcaff0469f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/9e829899-8a60-4792-b4cc-fabcaff0469f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e829899-8a60-4792-b4cc-fabcaff0469f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/9e829899-8a60-4792-b4cc-fabcaff0469f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e829899-8a60-4792-b4cc-fabcaff0469f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/9e829899-8a60-4792-b4cc-fabcaff0469f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e829899-8a60-4792-b4cc-fabcaff0469f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/9e829899-8a60-4792-b4cc-fabcaff0469f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e829899-8a60-4792-b4cc-fabcaff0469f/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/9e829899-8a60-4792-b4cc-fabcaff0469f/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6babafa-f85a-4a1b-9fe9-8bac54198754/volumes/kubernetes.io~projected/kube-api-access-lm5kk:{mountpoint:/var/lib/kubelet/pods/a6babafa-f85a-4a1b-9fe9-8bac54198754/volumes/kubernetes.io~projected/kube-api-access-lm5kk major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab/volumes/kubernetes.io~projected/kube-api-access-wwz2j:{mountpoint:/var/lib/kubelet/pods/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab/volumes/kubernetes.io~projected/kube-api-access-wwz2j major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2/volumes/kubernetes.io~projected/kube-api-access-sx5fw:{mountpoint:/var/lib/kubelet/pods/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2/volumes/kubernetes.io~projected/kube-api-access-sx5fw major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volume-subpaths/web-config/config-reloader/2 major:0 minor:381 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:381 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volume-subpaths/web-config/prometheus/15 major:0 minor:381 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volumes/kubernetes.io~empty-dir/config-out major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volumes/kubernetes.io~projected/kube-api-access-48qh7:{mountpoint:/var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volumes/kubernetes.io~projected/kube-api-access-48qh7 major:0 minor:387 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volumes/kubernetes.io~projected/tls-assets major:0 minor:379 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volumes/kubernetes.io~secret/config major:0 minor:385 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:383 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:377 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:386 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:378 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:382 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:380 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:384 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volumes/kubernetes.io~secret/web-config major:0 minor:381 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf531672-6ee1-4188-b38d-2899bf9d6a0f/volumes/kubernetes.io~projected/kube-api-access-6b2bg:{mountpoint:/var/lib/kubelet/pods/cf531672-6ee1-4188-b38d-2899bf9d6a0f/volumes/kubernetes.io~projected/kube-api-access-6b2bg major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf531672-6ee1-4188-b38d-2899bf9d6a0f/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/cf531672-6ee1-4188-b38d-2899bf9d6a0f/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf531672-6ee1-4188-b38d-2899bf9d6a0f/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/cf531672-6ee1-4188-b38d-2899bf9d6a0f/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7922f23-5ba7-4931-8d19-598e2c4fe041/volumes/kubernetes.io~projected/kube-api-access-z6wcg:{mountpoint:/var/lib/kubelet/pods/e7922f23-5ba7-4931-8d19-598e2c4fe041/volumes/kubernetes.io~projected/kube-api-access-z6wcg major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3928614-3b83-4bf6-9e3f-04a71bae52bd/volumes/kubernetes.io~projected/kube-api-access-d6w4k:{mountpoint:/var/lib/kubelet/pods/f3928614-3b83-4bf6-9e3f-04a71bae52bd/volumes/kubernetes.io~projected/kube-api-access-d6w4k major:0 minor:256 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3928614-3b83-4bf6-9e3f-04a71bae52bd/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/f3928614-3b83-4bf6-9e3f-04a71bae52bd/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3928614-3b83-4bf6-9e3f-04a71bae52bd/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/f3928614-3b83-4bf6-9e3f-04a71bae52bd/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4b29936-72de-4ddc-b7f1-a964a614ff61/volumes/kubernetes.io~projected/kube-api-access-hqpz6:{mountpoint:/var/lib/kubelet/pods/f4b29936-72de-4ddc-b7f1-a964a614ff61/volumes/kubernetes.io~projected/kube-api-access-hqpz6 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4b29936-72de-4ddc-b7f1-a964a614ff61/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/f4b29936-72de-4ddc-b7f1-a964a614ff61/volumes/kubernetes.io~secret/metrics-certs major:0 minor:400 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fad7d0da-535f-447c-89bd-ab36ac1861a8/volumes/kubernetes.io~projected/kube-api-access-njqrk:{mountpoint:/var/lib/kubelet/pods/fad7d0da-535f-447c-89bd-ab36ac1861a8/volumes/kubernetes.io~projected/kube-api-access-njqrk major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fad7d0da-535f-447c-89bd-ab36ac1861a8/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/fad7d0da-535f-447c-89bd-ab36ac1861a8/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:202 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/6350ae371b657ed95f0447d3119712994446e9d2b91b8c26ddf84ab4391c6597/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/54080148ec9577b33d9e67ba2d2ad6d7cde91c9d6d06d94a8b1ac240d21ded63/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/70dd04d2337b3176535a3c4cbda9c5fc600e2b29f90b827be3f43fe2a358415c/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/b59bd83cb16f8582e0f76c09eac7dd557067c9fb024ca4e1ef8b6ed3c064e783/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/42efa339d3d145353fe03b7a4415bb4187aa263f228a051977e57d915d925de2/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/284969f8239ce386f420394b51a990a989622a74e078bf9d750b8098d920fbed/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/3ed3847ea22c37c24f3b8a692ec3fdb8d5550ed7328310c3c0a4438634ea88b6/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/7cf80d5a8d77f9ed8cfdf65f2a2d6805ec3ab02ebf94a474fc08d7621e090535/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/dfadb956c8754b7afe00ef09d9a5f5c74ba833b935e39e9caa9d952db75d0a70/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/8f54674905247b4c375c133665839c8ef09fe39dfb7359981fd86b26d9741eab/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/c5141dc6716bb4c79113f90d7f6af83f0561b9515f5d8edf20f4146dd9cef8ec/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/e65e05f8f7cb50ada2c94f41b32d9da8948dda39a9737b3666f920605a24dd7b/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/b0b73ea51f9fe01753a65b035e0a581c3b1da6a574ec7e05ea3c5de411c345fe/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/c137045e3d4663e3a17f89e83324851c5e482782f41f340feda5957d508638e1/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/a39c01a8cd794ff2ea4c8d797003b99c110a4fb2bf76c267855432b5e5f9c695/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/52ada650bfb0cd3d074d7f01ad943a32a4316658128c9afaa2f960f5738b5895/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/8192a85bd7f9cc127b21ae3ffb2afce9dcc7b928154119c3beb9f1c26f436a4c/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/f676a5fb4d68c9a9312a2772425713dfe2f3a056618e660b67efb035b906e025/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/9f659301fe510fc807ff7e6622ae26cd19c55687c49b9acabe97e31852dfb21c/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/c7c552494a185150003fc677f971952f68118b77422d5dc5127cb75148e506a2/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/0988d01b2489fb56273b215af0280e84ed9343159c9fd251a17b61f5ea6a0210/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/b0b3ebe14e512bdb0ed607b0ba16826c4934bb9ea24aefe3ce3abded4c7150f5/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/be35b63c2a96c0d38e109ca6b38b0c1850471d453211cc29cf9bd5cfee7ad900/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/bc0bce35b2a4b0650af4e8837a9143c2ccd48c7864230900afbed9e6c10ecaaf/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/e0ae469ed43c74563db43b0ded8c7a716e41029f5536edc086be996102410ad1/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/eed7275e4de7f56864dbca6e12f8eb60ea478705248522b4e74e1c14ed43d189/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/645582e25761ca1cffbb8f2deb6d1ff78c370efab6f4cb871dcc682f6d869fc3/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/dff5c01f2111d342347efc918c676ddc4c71f1670ef9928dfd428bdb5fce2c80/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/3066d346c8d522d658375ce8bb1a61b25154e2f4409b21eee37f569f53eb5a57/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/0dcd7afdf07d2d17b3d9da410d7eeccab351c863f084a43711e37e42a8176bfa/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/eb6017cea24ff54c633401830736861ef9a44735d2495d81899f35f7329cc31e/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/0e0bf5dcee5d320b5fb74d18744eb2be19795c6b1a7dc39011989feb708c5458/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/2e9ae13b0e31e6b52691e7c77ad410a81dd9d0418c4ce35bb7ec775192338286/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/7e84c9b157e1a6dece3e4b9c2b389c2d073b0608ee03962d8adff1180e48fe00/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/c678c476a22216839698eb5071346559e42fee53fc88d858f3604530c3c14544/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/2ce8cc7aa929b0bea2f171e07fcc5eae5af408f7a820734bf56a22644f913a1b/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/ee63932670e9c6120e599a6103559f71410fdc50243e0baf09e6f5bcd0b7c0dc/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/85abb0b441080c4933940a68aa6d459f49c74fa91f1d784c5cba6dbfca7d7633/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/583d17502d2168c43d7eb14bbf3f95dc8592eb308a2228b2b42f6a7951a30b15/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/5b27e0b0a020633d0432af8fd7c27a13b26c88b65bb69fe33688ec099535ff8a/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/d9f04403b87f9e460892b835bdf6e1da2b5de3b7cc64968cba0033c01fd1a143/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/2243695d8d23d24aecce0a05b2d857022567c7c2ff7e9507778e02b08af3efbe/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/dd75c13deb85974dc983b21ed95a1fb3f8d84f8dcde3e54f7532be1849aaf162/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/4a01e2ace5e113e6f32ac288e6d814437011a7e1e98b70418e4d15ac09eb9015/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/9f14ee4f210cdd2dc21f9d5ce96091fa9e14a6cdd16e22834216215d33609129/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/67c7324bd02bc848979b4b76c9f9a434456d76dda31c37065911e6ef8644c0d9/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/67225441d38fec7e152a79d287b4a3b96e8e8505cc394eaa0393b5b4799ab8cb/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-355:{mountpoint:/var/lib/containers/storage/overlay/d745a38f3b96115ae287551aaf0cdb90b4ea9b71dc7baa0ee77d7809c82b82b0/merged major:0 minor:355 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/7081a5b18a8027fa08f1feb9c9d2926ad10aea8824210ee4d63ec57da7ef054f/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/e4c8b48ab423a74b317065a21bd76e463e367cd4d0020d038eaee22a23974f01/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/beb754b36fb74b754fa157db2d3100935354b4b49e17d1d3d04dfbd28b0315d9/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/b4d92764d71421f0ecd6bdd76bf7346da3e6535a244b4439c1cb24b8fee8a850/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-392:{mountpoint:/var/lib/containers/storage/overlay/7f6712cc732d62e82d9beb9ded2ef4250e45631eba5c189819e80c2cff510b2b/merged major:0 minor:392 fsType:overlay blockSize:0} overlay_0-397:{mountpoint:/var/lib/containers/storage/overlay/7da862b624df70877d4378b24f3b878197dc83f6ed709554b1731e1bcb898fb5/merged major:0 minor:397 fsType:overlay blockSize:0} overlay_0-402:{mountpoint:/var/lib/containers/storage/overlay/f2ac40fe8364da7d6d48a297129c8f214f97d4f84d163514dd5d9a2c86675675/merged major:0 minor:402 fsType:overlay blockSize:0} overlay_0-417:{mountpoint:/var/lib/containers/storage/overlay/db846a83a6bf8f78d065ea729cb2b44b8d9b294ed526cf0e58b19fb90fd1c67b/merged major:0 minor:417 fsType:overlay blockSize:0} overlay_0-419:{mountpoint:/var/lib/containers/storage/overlay/9b1cdd1fee706fb065fe669f9b999921295fd556a8f6a81dda0c1618e9d36454/merged major:0 minor:419 fsType:overlay blockSize:0} overlay_0-421:{mountpoint:/var/lib/containers/storage/overlay/64f9300510067f465caaf5776613081a57c36156dd49a3174ec6adc38603b7a4/merged major:0 minor:421 fsType:overlay blockSize:0} overlay_0-423:{mountpoint:/var/lib/containers/storage/overlay/74bf63ca16e0d8e9f740dd82087a15ddaef53169da124f9ae6c384d25f0c24a4/merged major:0 minor:423 fsType:overlay blockSize:0} overlay_0-425:{mountpoint:/var/lib/containers/storage/overlay/57ad7c388d0f7b8819bfb890db9021f8bf2d5b68f49342447241d502a3a1fb32/merged major:0 minor:425 fsType:overlay blockSize:0} overlay_0-431:{mountpoint:/var/lib/containers/storage/overlay/447540e3a07a7f8eb35bd340ac06792b1f365717881c11b3537a9fd77087f1f7/merged major:0 minor:431 fsType:overlay blockSize:0} overlay_0-436:{mountpoint:/var/lib/containers/storage/overlay/589868e69b5adc106fd5d2fcfb89539485dfeb659449cbc81134190be54b8777/merged major:0 minor:436 fsType:overlay blockSize:0} overlay_0-442:{mountpoint:/var/lib/containers/storage/overlay/13b66c253da130ab0b8bcdd5d7799d513af8d86b798f29fcb0f94d69cf91b85f/merged major:0 minor:442 fsType:overlay blockSize:0} overlay_0-447:{mountpoint:/var/lib/containers/storage/overlay/b921bd373e58492af87c5330d10b78a6f8a1579b4dcc42eafad83389788dbdf8/merged major:0 minor:447 fsType:overlay blockSize:0} overlay_0-452:{mountpoint:/var/lib/containers/storage/overlay/3d5b49c7d33f03d8ae4f9d79259b589e69dcb25b0dc7d9226a88684d371ff6a8/merged major:0 minor:452 fsType:overlay blockSize:0} overlay_0-457:{mountpoint:/var/lib/containers/storage/overlay/f1a5ba601891eca137e48449f428d77b1da36a869f3a0d03c18f6f48697fe46c/merged major:0 minor:457 fsType:overlay blockSize:0} overlay_0-462:{mountpoint:/var/lib/containers/storage/overlay/4da4fbecd270580979fdbd815069b44f2962f676fbab773b580c7e31836b1b34/merged major:0 minor:462 fsType:overlay blockSize:0} overlay_0-467:{mountpoint:/var/lib/containers/storage/overlay/2c0a966aef55c9309a1f88d3c93cfe49f2e5e1e237f1c5c944db79eda8ed212d/merged major:0 minor:467 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/33d616103e86723dce9a64c0cf64193f1c89910f21b60b9d49fca08c7de1b5cf/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/9b0ba67d6501c4fbadb79efab5c931122a49b029543322959185811137f04f1f/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/abce55161c0a333cb0c8675a74f135077a256dae90401b4414afc7108b056e65/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/60c9e9e6a9abaa60a38927e48cea4e9f6e0b1da024d32882966a7624f4f86dff/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/7b2d56edafc8cd10e3e724cbdc68fc113d19e64bdded19c505aad4adbd9659d8/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/3caf04afc2edaddad9f32d290cec02d75f2257ca2d70f2668e3cc60d5cffbc77/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/81769e719062d0364df1a63fe8f3dabead67464e683c3557978e9b09a1cad750/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/60f5a536d601798252186e4bc3095f369eb15d14c0f9f3820b267a42353d9242/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/c349f5516f9611f7577731933419e2416b1bee2348bfabd40e70326ce3ce347b/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/ae727f3bdeb81221826e271c60e1b5f32b7a3e354c6dbbf2b92444c6193f25a8/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 01:17:54.486932 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.486504 6572 manager.go:217] Machine: {Timestamp:2026-10-30 01:17:54.485957159 +0000 UTC m=+0.062382554 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099956 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec281e2b4acea78f00f4ee495cc050e5 SystemUUID:ec281e2b-4ace-a78f-00f4-ee495cc050e5 BootID:463af845-f3b9-4975-a32a-ae96d317f323 Filesystems:[{Device:/var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:385 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-419 DeviceMajor:0 DeviceMinor:419 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-442 DeviceMajor:0 DeviceMinor:442 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/262c01e0-8469-4fbe-aaa4-8a8c50551fce/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8b00a159-66f4-4b9d-bcf1-560094dd69bb/volumes/kubernetes.io~projected/kube-api-access-fm4x7 DeviceMajor:0 DeviceMinor:64 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/00c444f0-9313-4c0a-b9c8-79a953eaba17/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fad7d0da-535f-447c-89bd-ab36ac1861a8/volumes/kubernetes.io~projected/kube-api-access-njqrk DeviceMajor:0 DeviceMinor:205 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f4b29936-72de-4ddc-b7f1-a964a614ff61/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:400 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab/volumes/kubernetes.io~projected/kube-api-access-wwz2j DeviceMajor:0 DeviceMinor:257 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0ae3164f-fec1-4210-863e-70e49e6bd26a/volumes/kubernetes.io~projected/kube-api-access-2ljcm DeviceMajor:0 DeviceMinor:341 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/27a1569120759fae8a09aba518f9bcfd8599fa09d5a4a3d906ef8ac73a041c05/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:382 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:381 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-462 DeviceMajor:0 DeviceMinor:462 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-447 DeviceMajor:0 DeviceMinor:447 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cf531672-6ee1-4188-b38d-2899bf9d6a0f/volumes/kubernetes.io~projected/kube-api-access-6b2bg DeviceMajor:0 DeviceMinor:258 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9bc3b964-6e84-4860-a5bf-0a81f15b0d97/volumes/kubernetes.io~projected/kube-api-access-zzkj9 DeviceMajor:0 DeviceMinor:350 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1c5a490f-f8db-4afb-9a1f-b1accc9eadea/volumes/kubernetes.io~projected/kube-api-access-czkw9 DeviceMajor:0 DeviceMinor:410 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3efd0b05-2dcd-4e6c-a24e-9e955a58fa9b/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9e829899-8a60-4792-b4cc-fabcaff0469f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:314 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/03772f27-6bc6-4048-a515-d8d4e9abf431/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8ae82ecd7a980f22b2e052f7bae608beaa4523926642ab8ca614d120bc8cb0bb/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:372 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:386 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-431 DeviceMajor:0 DeviceMinor:431 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/03772f27-6bc6-4048-a515-d8d4e9abf431/volumes/kubernetes.io~projected/kube-api-access-46pfs DeviceMajor:0 DeviceMinor:59 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:252 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5fa780ef9495d54c00886fe917a2c6843587f55391ce8b458ae71ac8d377c8e6/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9bc3b964-6e84-4860-a5bf-0a81f15b0d97/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:346 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/03772f27-6bc6-4048-a515-d8d4e9abf431/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/00c444f0-9313-4c0a-b9c8-79a953eaba17/volumes/kubernetes.io~projected/kube-api-access-j5bjt DeviceMajor:0 DeviceMinor:197 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f3928614-3b83-4bf6-9e3f-04a71bae52bd/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:253 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:255 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9bc3b964-6e84-4860-a5bf-0a81f15b0d97/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:349 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:383 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:381 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-457 DeviceMajor:0 DeviceMinor:457 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:378 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volumes/kubernetes.io~projected/kube-api-access-48qh7 DeviceMajor:0 DeviceMinor:387 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/441f6293c35e074c06fc5dab94106eec6cbe639f8d9182f14b539714cae612fc/userdata/shm DeviceMajor:0 DeviceMinor:411 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/946b2a1dcdb0ea41b11209253f4b624b3b8c0260d683ee44df668300d93a16c4/userdata/shm DeviceMajor:0 DeviceMinor:415 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a6babafa-f85a-4a1b-9fe9-8bac54198754/volumes/kubernetes.io~projected/kube-api-access-lm5kk DeviceMajor:0 DeviceMinor:65 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f3928614-3b83-4bf6-9e3f-04a71bae52bd/volumes/kubernetes.io~projected/kube-api-access-d6w4k DeviceMajor:0 DeviceMinor:256 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9e829899-8a60-4792-b4cc-fabcaff0469f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:315 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f4b29936-72de-4ddc-b7f1-a964a614ff61/volumes/kubernetes.io~projected/kube-api-access-hqpz6 DeviceMajor:0 DeviceMinor:63 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0ae3164f-fec1-4210-863e-70e49e6bd26a/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:337 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9bc3b964-6e84-4860-a5bf-0a81f15b0d97/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:348 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-425 DeviceMajor:0 DeviceMinor:425 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eb32738b0ac5a00b52a65f77ffc436810dabbd33e62e70360a962b704a1b28ca/userdata/shm DeviceMajor:0 DeviceMinor:211 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a628f193fbba8c2be9f20c3c3119c8801bbc4b36abf1144c487c708f097e4e82/userdata/shm DeviceMajor:0 DeviceMinor:351 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-397 DeviceMajor:0 DeviceMinor:397 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/e7922f23-5ba7-4931-8d19-598e2c4fe041/volumes/kubernetes.io~projected/kube-api-access-z6wcg DeviceMajor:0 DeviceMinor:60 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f3928614-3b83-4bf6-9e3f-04a71bae52bd/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:251 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:377 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-436 DeviceMajor:0 DeviceMinor:436 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/04631a42-b08e-4aff-9fc4-78dd7392d59d/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fad7d0da-535f-447c-89bd-ab36ac1861a8/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:202 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9e829899-8a60-4792-b4cc-fabcaff0469f/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:316 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-467 DeviceMajor:0 DeviceMinor:467 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cf531672-6ee1-4188-b38d-2899bf9d6a0f/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:254 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3bd77351638fccdbe592f3c0c30162240591236be59aff6b515940657a064a2b/userdata/shm DeviceMajor:0 DeviceMinor:265 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:381 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cf531672-6ee1-4188-b38d-2899bf9d6a0f/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:246 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7ce030618253c5c68cb56a995163f8a4d99ae28ad868ecbc8cc2464bf3595ab6/userdata/shm DeviceMajor:0 DeviceMinor:259 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:381 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-417 DeviceMajor:0 DeviceMinor:417 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a1d852b890b43a7fcf9884a14d90e565e73bf0e32d149b3ef12324d8e15319b8/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e7b0dcd5c52d7ed1b1ab65a474ce3b34b5a528207d6a62d0901cc211d6c1daf/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-423 DeviceMajor:0 DeviceMinor:423 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4b97aea6-bab9-498d-bce7-95e2a47c8c05/volumes/kubernetes.io~projected/kube-api-access-gc85r DeviceMajor:0 DeviceMinor:66 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9e829899-8a60-4792-b4cc-fabcaff0469f/volumes/kubernetes.io~projected/kube-api-access-n85mv DeviceMajor:0 DeviceMinor:319 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9e829899-8a60-4792-b4cc-fabcaff0469f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:318 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3efd0b05-2dcd-4e6c-a24e-9e955a58fa9b/volumes/kubernetes.io~projected/kube-api-access-q2d66 DeviceMajor:0 DeviceMinor:206 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:379 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-421 DeviceMajor:0 DeviceMinor:421 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9e829899-8a60-4792-b4cc-fabcaff0469f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:317 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/04631a42-b08e-4aff-9fc4-78dd7392d59d/volumes/kubernetes.io~projected/kube-api-access-9jnnc DeviceMajor:0 DeviceMinor:62 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/758ba053dd70b7aa4ba6e97a8652ecd512db79c8dfeeb0a29de1f494a16fdc23/userdata/shm DeviceMajor:0 DeviceMinor:261 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f19bfb393c5bc8f29de77632302d48be4377d9623cb8c2a1d62d72b200c4e4e5/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-392 DeviceMajor:0 DeviceMinor:392 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4804e601006c9694ff3e71642609d0091e993141294c870ed271be987c4c434a/userdata/shm DeviceMajor:0 DeviceMinor:320 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:380 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-402 DeviceMajor:0 DeviceMinor:402 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9bc3b964-6e84-4860-a5bf-0a81f15b0d97/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:347 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a046ba2d3bfee0e2bbdc1974b56581f8e57db6f84f40a65fc361e6f0b5b656f5/userdata/shm DeviceMajor:0 DeviceMinor:388 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2a28007d9fd8ecf4d20c4694935fff740ce2fa87946bb4f514fab6cbcef3b920/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/88f5d337-fa16-4430-a82e-5f433354c164/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:409 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-452 DeviceMajor:0 DeviceMinor:452 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/61c3f93c78077742aa562727b783c60bb0f3a3341a17838f2669945a71f08ca1/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9ceda047-30e9-496c-b854-af4043a83656/volumes/kubernetes.io~projected/kube-api-access-tqww5 DeviceMajor:0 DeviceMinor:61 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2/volumes/kubernetes.io~projected/kube-api-access-sx5fw DeviceMajor:0 DeviceMinor:67 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1116cb567c792d34865bb572924c8f794fcb4aae2cedd9963945cc57f6ae84f3/userdata/shm DeviceMajor:0 DeviceMinor:413 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/16b8d3afdfe0b8d1782acbd5b377b2fa9b2f56255bc2a6aba928ce62ab88f227/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/76d6428545389bf939771ec6760306f141d7037d715bab0edee0308cd2a9cf45/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9e829899-8a60-4792-b4cc-fabcaff0469f/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:310 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-355 DeviceMajor:0 DeviceMinor:355 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:384 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/04631a42-b08e-4aff-9fc4-78dd7392d59d/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a89f78f9ff48ec09e65d336ba89ab39eba3f6e47e0e47b47d563d8370fb90a98/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6f4aa6724eabbbabd00cb2f9343dc873cf56f83d7f5a7b511923590f68455e50/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b2039c18dcbb8325a89eeee1c2021801fdc09d8b18b5cb888aaec7e98a9ee3d7/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1116cb567c792d3 MacAddress:f6:5c:e6:df:ad:ec Speed:10000 Mtu:8901} {Name:3bd77351638fccd MacAddress:82:06:2e:a0:a2:1d Speed:10000 Mtu:8901} {Name:441f6293c35e074 MacAddress:46:9d:f9:65:b0:1a Speed:10000 Mtu:8901} {Name:4804e601006c969 MacAddress:7e:c7:75:25:ca:68 Speed:10000 Mtu:8901} {Name:5fa780ef9495d54 MacAddress:16:6c:f4:26:40:38 Speed:10000 Mtu:8901} {Name:76d6428545389bf MacAddress:2e:e3:f6:15:5b:ce Speed:10000 Mtu:8901} {Name:7ce030618253c5c MacAddress:f6:0b:25:21:7b:77 Speed:10000 Mtu:8901} {Name:8ae82ecd7a980f2 MacAddress:7e:7e:7d:f4:fd:07 Speed:10000 Mtu:8901} {Name:946b2a1dcdb0ea4 MacAddress:3a:90:9c:37:c8:f8 Speed:10000 Mtu:8901} {Name:a046ba2d3bfee0e MacAddress:b2:18:7c:73:dd:00 Speed:10000 Mtu:8901} {Name:a628f193fbba8c2 MacAddress:c6:73:2a:b6:74:d1 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ee:f6:e3:43 Speed:0 Mtu:9001} {Name:br-int MacAddress:26:14:e5:61:77:27 Speed:0 Mtu:8901} {Name:eb32738b0ac5a00 MacAddress:86:16:82:5c:87:1a Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ee:f6:e3:43 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:7a:59:02:cc:bf:58 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:4a:ea:a6:10:04:04 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 01:17:54.487178 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.486936 6572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 01:17:54.487178 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.486986 6572 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 01:17:54.487285 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.487260 6572 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 01:17:54.487433 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.487286 6572 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-223.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 01:17:54.487478 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.487436 6572 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 01:17:54.487478 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.487443 6572 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 01:17:54.487478 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.487453 6572 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 01:17:54.487478 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.487471 6572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 01:17:54.487580 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.487571 6572 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 01:17:54.487838 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.487770 6572 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 01:17:54.487902 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.487893 6572 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 01:17:54.487945 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.487937 6572 kubelet.go:493] "Attempting to sync node with API server" Oct 30 01:17:54.487975 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.487949 6572 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 01:17:54.487975 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.487959 6572 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 01:17:54.487975 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.487971 6572 kubelet.go:405] "Adding apiserver pod source" Oct 30 01:17:54.488097 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.487980 6572 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 01:17:54.490257 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.489552 6572 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 01:17:54.490257 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.489573 6572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 01:17:54.490654 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.490633 6572 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 01:17:54.493407 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.493391 6572 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 01:17:54.493407 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.493405 6572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 01:17:54.493525 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.493519 6572 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 01:17:54.493725 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.493713 6572 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 01:17:54.493895 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.493884 6572 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 01:17:54.494014 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.494005 6572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 01:17:54.494043 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.494020 6572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 01:17:54.494043 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.494027 6572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 01:17:54.494043 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.494035 6572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 01:17:54.494121 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.494047 6572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 01:17:54.494121 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.494057 6572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 01:17:54.494121 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.494064 6572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 01:17:54.494121 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.494069 6572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 01:17:54.494121 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.494076 6572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 01:17:54.494121 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.494082 6572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 01:17:54.494121 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.494090 6572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 01:17:54.494121 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.494099 6572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 01:17:54.494121 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.494125 6572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 01:17:54.494451 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.494134 6572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 01:17:54.497585 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.497570 6572 server.go:1263] "Started kubelet" Oct 30 01:17:54.497711 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.497650 6572 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 01:17:54.497792 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.497690 6572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 01:17:54.497852 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.497820 6572 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 01:17:54.498165 ip-10-0-138-223 systemd[1]: Started Kubernetes Kubelet. Oct 30 01:17:54.498315 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.498295 6572 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 01:17:54.499076 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.499057 6572 server.go:317] "Adding debug handlers to kubelet server" Oct 30 01:17:54.501940 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.501883 6572 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 01:17:54.501940 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.501899 6572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 01:17:54.502222 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.501960 6572 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 01:17:54.502222 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.501945 6572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 01:11:43 +0000 UTC" deadline="2028-06-24 13:34:03.320003092 +0000 UTC" Oct 30 01:17:54.502222 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.501979 6572 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 01:17:54.502222 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.501978 6572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14484h16m8.818027351s" Oct 30 01:17:54.502222 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.501972 6572 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 01:17:54.502222 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:17:54.502017 6572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-223.ec2.internal\" not found" Oct 30 01:17:54.505590 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.504521 6572 factory.go:55] Registering systemd factory Oct 30 01:17:54.505590 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.504549 6572 factory.go:223] Registration of the systemd container factory successfully Oct 30 01:17:54.506123 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.506104 6572 factory.go:153] Registering CRI-O factory Oct 30 01:17:54.506349 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.506126 6572 factory.go:223] Registration of the crio container factory successfully Oct 30 01:17:54.506349 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.506186 6572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 01:17:54.506349 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.506208 6572 factory.go:103] Registering Raw factory Oct 30 01:17:54.506349 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.506226 6572 manager.go:1199] Started watching for new ooms in manager Oct 30 01:17:54.506580 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.506566 6572 manager.go:319] Starting recovery of all containers Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.512832 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03772f27-6bc6-4048-a515-d8d4e9abf431" volumeName="kubernetes.io/configmap/03772f27-6bc6-4048-a515-d8d4e9abf431-ovnkube-config" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.512888 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ae3164f-fec1-4210-863e-70e49e6bd26a" volumeName="kubernetes.io/projected/0ae3164f-fec1-4210-863e-70e49e6bd26a-kube-api-access-2ljcm" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.512906 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3efd0b05-2dcd-4e6c-a24e-9e955a58fa9b" volumeName="kubernetes.io/projected/3efd0b05-2dcd-4e6c-a24e-9e955a58fa9b-kube-api-access-q2d66" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.512920 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3efd0b05-2dcd-4e6c-a24e-9e955a58fa9b" volumeName="kubernetes.io/secret/3efd0b05-2dcd-4e6c-a24e-9e955a58fa9b-cert" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.512934 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e829899-8a60-4792-b4cc-fabcaff0469f" volumeName="kubernetes.io/secret/9e829899-8a60-4792-b4cc-fabcaff0469f-secret-grpc-tls" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.512947 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e829899-8a60-4792-b4cc-fabcaff0469f" volumeName="kubernetes.io/secret/9e829899-8a60-4792-b4cc-fabcaff0469f-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.512960 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc1464ba-e4e0-4685-b7d8-c873b189f8de" volumeName="kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.512973 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc1464ba-e4e0-4685-b7d8-c873b189f8de" volumeName="kubernetes.io/empty-dir/bc1464ba-e4e0-4685-b7d8-c873b189f8de-config-out" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513002 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="262c01e0-8469-4fbe-aaa4-8a8c50551fce" volumeName="kubernetes.io/secret/262c01e0-8469-4fbe-aaa4-8a8c50551fce-agent-certs" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513015 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bc3b964-6e84-4860-a5bf-0a81f15b0d97" volumeName="kubernetes.io/secret/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-telemeter-client-tls" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513027 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc1464ba-e4e0-4685-b7d8-c873b189f8de" volumeName="kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-grpc-tls" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513038 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4b29936-72de-4ddc-b7f1-a964a614ff61" volumeName="kubernetes.io/projected/f4b29936-72de-4ddc-b7f1-a964a614ff61-kube-api-access-hqpz6" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513051 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4b29936-72de-4ddc-b7f1-a964a614ff61" volumeName="kubernetes.io/secret/f4b29936-72de-4ddc-b7f1-a964a614ff61-metrics-certs" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513063 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="262c01e0-8469-4fbe-aaa4-8a8c50551fce" volumeName="kubernetes.io/configmap/262c01e0-8469-4fbe-aaa4-8a8c50551fce-konnectivity-ca" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513094 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bc3b964-6e84-4860-a5bf-0a81f15b0d97" volumeName="kubernetes.io/configmap/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-telemeter-trusted-ca-bundle" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513106 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bc3b964-6e84-4860-a5bf-0a81f15b0d97" volumeName="kubernetes.io/secret/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-secret-telemeter-client" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513120 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bc3b964-6e84-4860-a5bf-0a81f15b0d97" volumeName="kubernetes.io/secret/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513132 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bc3b964-6e84-4860-a5bf-0a81f15b0d97" volumeName="kubernetes.io/configmap/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513143 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bc3b964-6e84-4860-a5bf-0a81f15b0d97" volumeName="kubernetes.io/projected/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-kube-api-access-zzkj9" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513155 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc1464ba-e4e0-4685-b7d8-c873b189f8de" volumeName="kubernetes.io/projected/bc1464ba-e4e0-4685-b7d8-c873b189f8de-kube-api-access-48qh7" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513168 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf531672-6ee1-4188-b38d-2899bf9d6a0f" volumeName="kubernetes.io/projected/cf531672-6ee1-4188-b38d-2899bf9d6a0f-kube-api-access-6b2bg" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513179 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf531672-6ee1-4188-b38d-2899bf9d6a0f" volumeName="kubernetes.io/secret/cf531672-6ee1-4188-b38d-2899bf9d6a0f-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513192 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ae3164f-fec1-4210-863e-70e49e6bd26a" volumeName="kubernetes.io/secret/0ae3164f-fec1-4210-863e-70e49e6bd26a-monitoring-plugin-cert" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513205 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88f5d337-fa16-4430-a82e-5f433354c164" volumeName="kubernetes.io/secret/88f5d337-fa16-4430-a82e-5f433354c164-original-pull-secret" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513217 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e829899-8a60-4792-b4cc-fabcaff0469f" volumeName="kubernetes.io/secret/9e829899-8a60-4792-b4cc-fabcaff0469f-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513228 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc1464ba-e4e0-4685-b7d8-c873b189f8de" volumeName="kubernetes.io/projected/bc1464ba-e4e0-4685-b7d8-c873b189f8de-tls-assets" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513241 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7922f23-5ba7-4931-8d19-598e2c4fe041" volumeName="kubernetes.io/projected/e7922f23-5ba7-4931-8d19-598e2c4fe041-kube-api-access-z6wcg" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513252 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fad7d0da-535f-447c-89bd-ab36ac1861a8" volumeName="kubernetes.io/empty-dir/fad7d0da-535f-447c-89bd-ab36ac1861a8-data-volume" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513270 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2" volumeName="kubernetes.io/projected/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-kube-api-access-sx5fw" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513282 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00c444f0-9313-4c0a-b9c8-79a953eaba17" volumeName="kubernetes.io/projected/00c444f0-9313-4c0a-b9c8-79a953eaba17-kube-api-access-j5bjt" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513301 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00c444f0-9313-4c0a-b9c8-79a953eaba17" volumeName="kubernetes.io/secret/00c444f0-9313-4c0a-b9c8-79a953eaba17-metrics-tls" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513313 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04631a42-b08e-4aff-9fc4-78dd7392d59d" volumeName="kubernetes.io/empty-dir/04631a42-b08e-4aff-9fc4-78dd7392d59d-tmp" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513340 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e829899-8a60-4792-b4cc-fabcaff0469f" volumeName="kubernetes.io/secret/9e829899-8a60-4792-b4cc-fabcaff0469f-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513354 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2" volumeName="kubernetes.io/configmap/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-cni-binary-copy" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513368 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc1464ba-e4e0-4685-b7d8-c873b189f8de" volumeName="kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-web-config" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513384 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc1464ba-e4e0-4685-b7d8-c873b189f8de" volumeName="kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513396 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bc3b964-6e84-4860-a5bf-0a81f15b0d97" volumeName="kubernetes.io/secret/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-federate-client-tls" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513409 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e829899-8a60-4792-b4cc-fabcaff0469f" volumeName="kubernetes.io/secret/9e829899-8a60-4792-b4cc-fabcaff0469f-secret-thanos-querier-tls" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513422 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc1464ba-e4e0-4685-b7d8-c873b189f8de" volumeName="kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513435 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fad7d0da-535f-447c-89bd-ab36ac1861a8" volumeName="kubernetes.io/configmap/fad7d0da-535f-447c-89bd-ab36ac1861a8-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513447 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fad7d0da-535f-447c-89bd-ab36ac1861a8" volumeName="kubernetes.io/secret/fad7d0da-535f-447c-89bd-ab36ac1861a8-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513461 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03772f27-6bc6-4048-a515-d8d4e9abf431" volumeName="kubernetes.io/secret/03772f27-6bc6-4048-a515-d8d4e9abf431-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513473 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04631a42-b08e-4aff-9fc4-78dd7392d59d" volumeName="kubernetes.io/empty-dir/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-tuned" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513485 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b00a159-66f4-4b9d-bcf1-560094dd69bb" volumeName="kubernetes.io/projected/8b00a159-66f4-4b9d-bcf1-560094dd69bb-kube-api-access-fm4x7" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513499 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e829899-8a60-4792-b4cc-fabcaff0469f" volumeName="kubernetes.io/configmap/9e829899-8a60-4792-b4cc-fabcaff0469f-metrics-client-ca" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513511 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc1464ba-e4e0-4685-b7d8-c873b189f8de" volumeName="kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-configmap-metrics-client-ca" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513523 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc1464ba-e4e0-4685-b7d8-c873b189f8de" volumeName="kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513536 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fad7d0da-535f-447c-89bd-ab36ac1861a8" volumeName="kubernetes.io/projected/fad7d0da-535f-447c-89bd-ab36ac1861a8-kube-api-access-njqrk" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513548 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00c444f0-9313-4c0a-b9c8-79a953eaba17" volumeName="kubernetes.io/empty-dir/00c444f0-9313-4c0a-b9c8-79a953eaba17-tmp-dir" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513563 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b97aea6-bab9-498d-bce7-95e2a47c8c05" volumeName="kubernetes.io/projected/4b97aea6-bab9-498d-bce7-95e2a47c8c05-kube-api-access-gc85r" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513577 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6babafa-f85a-4a1b-9fe9-8bac54198754" volumeName="kubernetes.io/empty-dir/a6babafa-f85a-4a1b-9fe9-8bac54198754-tmp-dir" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513590 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab" volumeName="kubernetes.io/projected/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab-kube-api-access-wwz2j" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513602 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2" volumeName="kubernetes.io/configmap/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513616 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00c444f0-9313-4c0a-b9c8-79a953eaba17" volumeName="kubernetes.io/configmap/00c444f0-9313-4c0a-b9c8-79a953eaba17-config-volume" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513628 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b97aea6-bab9-498d-bce7-95e2a47c8c05" volumeName="kubernetes.io/configmap/4b97aea6-bab9-498d-bce7-95e2a47c8c05-multus-daemon-config" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513641 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab" volumeName="kubernetes.io/configmap/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513663 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab" volumeName="kubernetes.io/configmap/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab-metrics-client-ca" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513675 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc1464ba-e4e0-4685-b7d8-c873b189f8de" volumeName="kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513689 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3928614-3b83-4bf6-9e3f-04a71bae52bd" volumeName="kubernetes.io/configmap/f3928614-3b83-4bf6-9e3f-04a71bae52bd-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513703 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03772f27-6bc6-4048-a515-d8d4e9abf431" volumeName="kubernetes.io/projected/03772f27-6bc6-4048-a515-d8d4e9abf431-kube-api-access-46pfs" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513717 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c5a490f-f8db-4afb-9a1f-b1accc9eadea" volumeName="kubernetes.io/projected/1c5a490f-f8db-4afb-9a1f-b1accc9eadea-kube-api-access-czkw9" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513730 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b97aea6-bab9-498d-bce7-95e2a47c8c05" volumeName="kubernetes.io/configmap/4b97aea6-bab9-498d-bce7-95e2a47c8c05-cni-binary-copy" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513759 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc1464ba-e4e0-4685-b7d8-c873b189f8de" volumeName="kubernetes.io/empty-dir/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-db" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513772 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf531672-6ee1-4188-b38d-2899bf9d6a0f" volumeName="kubernetes.io/configmap/cf531672-6ee1-4188-b38d-2899bf9d6a0f-metrics-client-ca" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513785 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3928614-3b83-4bf6-9e3f-04a71bae52bd" volumeName="kubernetes.io/empty-dir/f3928614-3b83-4bf6-9e3f-04a71bae52bd-node-exporter-textfile" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513798 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc1464ba-e4e0-4685-b7d8-c873b189f8de" volumeName="kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513811 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc1464ba-e4e0-4685-b7d8-c873b189f8de" volumeName="kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513824 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf531672-6ee1-4188-b38d-2899bf9d6a0f" volumeName="kubernetes.io/secret/cf531672-6ee1-4188-b38d-2899bf9d6a0f-openshift-state-metrics-tls" seLinuxMountContext="" Oct 30 01:17:54.513776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513837 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2" volumeName="kubernetes.io/configmap/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 01:17:54.517183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513849 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bc3b964-6e84-4860-a5bf-0a81f15b0d97" volumeName="kubernetes.io/configmap/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-metrics-client-ca" seLinuxMountContext="" Oct 30 01:17:54.517183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513861 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03772f27-6bc6-4048-a515-d8d4e9abf431" volumeName="kubernetes.io/configmap/03772f27-6bc6-4048-a515-d8d4e9abf431-ovnkube-script-lib" seLinuxMountContext="" Oct 30 01:17:54.517183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513875 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b00a159-66f4-4b9d-bcf1-560094dd69bb" volumeName="kubernetes.io/configmap/8b00a159-66f4-4b9d-bcf1-560094dd69bb-iptables-alerter-script" seLinuxMountContext="" Oct 30 01:17:54.517183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513886 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ceda047-30e9-496c-b854-af4043a83656" volumeName="kubernetes.io/projected/9ceda047-30e9-496c-b854-af4043a83656-kube-api-access-tqww5" seLinuxMountContext="" Oct 30 01:17:54.517183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513899 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6babafa-f85a-4a1b-9fe9-8bac54198754" volumeName="kubernetes.io/projected/a6babafa-f85a-4a1b-9fe9-8bac54198754-kube-api-access-lm5kk" seLinuxMountContext="" Oct 30 01:17:54.517183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513912 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab" volumeName="kubernetes.io/empty-dir/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab-volume-directive-shadow" seLinuxMountContext="" Oct 30 01:17:54.517183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513924 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab" volumeName="kubernetes.io/secret/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 01:17:54.517183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513938 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab" volumeName="kubernetes.io/secret/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab-kube-state-metrics-tls" seLinuxMountContext="" Oct 30 01:17:54.517183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513950 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc1464ba-e4e0-4685-b7d8-c873b189f8de" volumeName="kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 01:17:54.517183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513962 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc1464ba-e4e0-4685-b7d8-c873b189f8de" volumeName="kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-metrics-client-certs" seLinuxMountContext="" Oct 30 01:17:54.517183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513975 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7922f23-5ba7-4931-8d19-598e2c4fe041" volumeName="kubernetes.io/configmap/e7922f23-5ba7-4931-8d19-598e2c4fe041-serviceca" seLinuxMountContext="" Oct 30 01:17:54.517183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.513988 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3928614-3b83-4bf6-9e3f-04a71bae52bd" volumeName="kubernetes.io/configmap/f3928614-3b83-4bf6-9e3f-04a71bae52bd-metrics-client-ca" seLinuxMountContext="" Oct 30 01:17:54.517183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.514000 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3928614-3b83-4bf6-9e3f-04a71bae52bd" volumeName="kubernetes.io/projected/f3928614-3b83-4bf6-9e3f-04a71bae52bd-kube-api-access-d6w4k" seLinuxMountContext="" Oct 30 01:17:54.517183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.514013 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3928614-3b83-4bf6-9e3f-04a71bae52bd" volumeName="kubernetes.io/secret/f3928614-3b83-4bf6-9e3f-04a71bae52bd-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 01:17:54.517183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.514025 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3928614-3b83-4bf6-9e3f-04a71bae52bd" volumeName="kubernetes.io/secret/f3928614-3b83-4bf6-9e3f-04a71bae52bd-node-exporter-tls" seLinuxMountContext="" Oct 30 01:17:54.517183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.514037 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03772f27-6bc6-4048-a515-d8d4e9abf431" volumeName="kubernetes.io/configmap/03772f27-6bc6-4048-a515-d8d4e9abf431-env-overrides" seLinuxMountContext="" Oct 30 01:17:54.517183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.514049 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e829899-8a60-4792-b4cc-fabcaff0469f" volumeName="kubernetes.io/empty-dir/9e829899-8a60-4792-b4cc-fabcaff0469f-tmp" seLinuxMountContext="" Oct 30 01:17:54.517183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.514062 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e829899-8a60-4792-b4cc-fabcaff0469f" volumeName="kubernetes.io/projected/9e829899-8a60-4792-b4cc-fabcaff0469f-kube-api-access-n85mv" seLinuxMountContext="" Oct 30 01:17:54.517183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.514075 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e829899-8a60-4792-b4cc-fabcaff0469f" volumeName="kubernetes.io/secret/9e829899-8a60-4792-b4cc-fabcaff0469f-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 30 01:17:54.517183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.514088 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc1464ba-e4e0-4685-b7d8-c873b189f8de" volumeName="kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 01:17:54.517183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.514100 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04631a42-b08e-4aff-9fc4-78dd7392d59d" volumeName="kubernetes.io/projected/04631a42-b08e-4aff-9fc4-78dd7392d59d-kube-api-access-9jnnc" seLinuxMountContext="" Oct 30 01:17:54.517183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.514112 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc1464ba-e4e0-4685-b7d8-c873b189f8de" volumeName="kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 30 01:17:54.517183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.514124 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc1464ba-e4e0-4685-b7d8-c873b189f8de" volumeName="kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 30 01:17:54.517183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.514140 6572 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc1464ba-e4e0-4685-b7d8-c873b189f8de" volumeName="kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-config" seLinuxMountContext="" Oct 30 01:17:54.517183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.514151 6572 reconstruct.go:97] "Volume reconstruction finished" Oct 30 01:17:54.517183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.514159 6572 reconciler.go:29] "Reconciler: start to sync state" Oct 30 01:17:54.531728 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.531669 6572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 01:17:54.534259 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.534095 6572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 01:17:54.534259 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.534122 6572 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 01:17:54.534259 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.534147 6572 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 01:17:54.534259 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:17:54.534186 6572 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 01:17:54.546342 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.546301 6572 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:17:54.546901 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.546876 6572 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 01:17:54.547020 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.546928 6572 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:17:54.562334 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.562287 6572 generic.go:356] "Generic (PLEG): container finished" podID="f3928614-3b83-4bf6-9e3f-04a71bae52bd" containerID="7368301157ab752ac1c80a8f7a2c2d3ed82e4574beb3098fff5de35e30d293a1" exitCode=0 Oct 30 01:17:54.565110 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.565080 6572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:17:54.565255 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.565237 6572 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:17:54.584534 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.584496 6572 generic.go:356] "Generic (PLEG): container finished" podID="bc1464ba-e4e0-4685-b7d8-c873b189f8de" containerID="ea6bdcb072917bdd7945749f813e0f8b851cad79f7d09cb105b7a097c31f7c66" exitCode=0 Oct 30 01:17:54.595607 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.595569 6572 generic.go:356] "Generic (PLEG): container finished" podID="ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2" containerID="eb3e06c8feb81e01ff8c945c440fc1ea63d771c4f9c11b04b035dfafd5451ff8" exitCode=0 Oct 30 01:17:54.595607 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.595604 6572 generic.go:356] "Generic (PLEG): container finished" podID="ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2" containerID="264f25e08b0d9ad778d6058b1c16806d36cbf76db794849973247a5cefd19f87" exitCode=0 Oct 30 01:17:54.595607 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.595615 6572 generic.go:356] "Generic (PLEG): container finished" podID="ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2" containerID="5cebc9daa6072033ea315d8df77b9800bb019244088ca3cfcf0a1411ea202a12" exitCode=0 Oct 30 01:17:54.595863 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.595625 6572 generic.go:356] "Generic (PLEG): container finished" podID="ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2" containerID="6e9f4186c23bd5392346aa525b7164dc3dee4b04bdf92be033d606f7596d0ec2" exitCode=0 Oct 30 01:17:54.595863 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.595637 6572 generic.go:356] "Generic (PLEG): container finished" podID="ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2" containerID="42f27733be2f82c99d03d96478b5b25cdb5855f889c8c1211bb23a75d34a55ee" exitCode=0 Oct 30 01:17:54.595863 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.595646 6572 generic.go:356] "Generic (PLEG): container finished" podID="ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2" containerID="ae1e4e6158ca6d91aef2ee5636da23e0f68e574f7bc2ddc22994e5efe2f25ee9" exitCode=0 Oct 30 01:17:54.603806 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.603776 6572 generic.go:356] "Generic (PLEG): container finished" podID="b222e074fa1fff6b3c88cd830a3f1d7d" containerID="a21211a8ecfda058796f06998ebf0b00316e2b4441b7fd62b0d237432acd4d62" exitCode=0 Oct 30 01:17:54.629260 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.629189 6572 manager.go:324] Recovery completed Oct 30 01:17:54.634304 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:17:54.634279 6572 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 01:17:54.653901 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.653869 6572 cpu_manager.go:226] "Starting" policy="none" Oct 30 01:17:54.653901 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.653889 6572 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 01:17:54.653901 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.653912 6572 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 01:17:54.654139 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.654039 6572 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 01:17:54.654139 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.654050 6572 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 01:17:54.654139 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.654067 6572 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 01:17:54.654139 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.654072 6572 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 01:17:54.654139 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.654077 6572 policy_none.go:50] "Start" Oct 30 01:17:54.654139 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.654086 6572 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 01:17:54.654139 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.654095 6572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 01:17:54.654347 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.654222 6572 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 01:17:54.654347 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.654229 6572 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 01:17:54.654347 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.654235 6572 policy_none.go:44] "Start" Oct 30 01:17:54.659036 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.659001 6572 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 01:17:54.661768 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.661739 6572 manager.go:341] "Starting Device Plugin manager" Oct 30 01:17:54.662247 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:17:54.661782 6572 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 01:17:54.662289 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.662249 6572 server.go:87] "Starting device plugin registration server" Oct 30 01:17:54.662532 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.662522 6572 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 01:17:54.662563 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.662534 6572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 01:17:54.662657 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.662630 6572 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 01:17:54.662786 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.662738 6572 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 01:17:54.662836 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.662799 6572 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 01:17:54.662836 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.662804 6572 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 01:17:54.662947 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.662854 6572 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T01:17:54.662784107Z","UUID":"8becacef-9573-4158-b181-c511e12fcc6d","Handler":null,"Name":"","Endpoint":""} Oct 30 01:17:54.664282 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.664227 6572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 01:17:54.664282 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.664255 6572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 01:17:54.664519 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:17:54.664499 6572 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 01:17:54.763271 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.763226 6572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:17:54.766118 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.766093 6572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:17:54.766218 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.766132 6572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:17:54.766218 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.766147 6572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:17:54.766218 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.766192 6572 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-223.ec2.internal" Oct 30 01:17:54.796589 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.796549 6572 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-223.ec2.internal" Oct 30 01:17:54.796821 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.796655 6572 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-223.ec2.internal" Oct 30 01:17:54.835175 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.835113 6572 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal"] Oct 30 01:17:54.835908 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.835869 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal" event={"ID":"129d2d6479043c5ffd993e1a7481e29a","Type":"ContainerStarted","Data":"6ff6924d1949f8a7539a748cd3e6453202f29f76629a80f14200e88a7ff0b70f"} Oct 30 01:17:54.836051 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.835913 6572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal" Oct 30 01:17:54.836051 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.835917 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal" event={"ID":"129d2d6479043c5ffd993e1a7481e29a","Type":"ContainerStarted","Data":"16b8d3afdfe0b8d1782acbd5b377b2fa9b2f56255bc2a6aba928ce62ab88f227"} Oct 30 01:17:54.836051 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.836037 6572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal" Oct 30 01:17:54.836199 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.836064 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal" event={"ID":"b222e074fa1fff6b3c88cd830a3f1d7d","Type":"ContainerStarted","Data":"a8bd72ff55c6dc61f27c0ed60e4e5b8cc138b011e4ee066c7d411b98098fbb4d"} Oct 30 01:17:54.836199 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.836091 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal" event={"ID":"b222e074fa1fff6b3c88cd830a3f1d7d","Type":"ContainerDied","Data":"a21211a8ecfda058796f06998ebf0b00316e2b4441b7fd62b0d237432acd4d62"} Oct 30 01:17:54.836199 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.836112 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal" event={"ID":"b222e074fa1fff6b3c88cd830a3f1d7d","Type":"ContainerStarted","Data":"61c3f93c78077742aa562727b783c60bb0f3a3341a17838f2669945a71f08ca1"} Oct 30 01:17:54.857563 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.857532 6572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 01:17:54.857662 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:17:54.857590 6572 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-223.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal" Oct 30 01:17:54.857948 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.857934 6572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 01:17:54.857987 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:17:54.857964 6572 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal" Oct 30 01:17:54.949334 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.949297 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/129d2d6479043c5ffd993e1a7481e29a-config\") pod \"kube-apiserver-proxy-ip-10-0-138-223.ec2.internal\" (UID: \"129d2d6479043c5ffd993e1a7481e29a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal" Oct 30 01:17:54.949334 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.949336 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b222e074fa1fff6b3c88cd830a3f1d7d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal\" (UID: \"b222e074fa1fff6b3c88cd830a3f1d7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal" Oct 30 01:17:54.949507 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:54.949355 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b222e074fa1fff6b3c88cd830a3f1d7d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal\" (UID: \"b222e074fa1fff6b3c88cd830a3f1d7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal" Oct 30 01:17:55.049988 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.049949 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b222e074fa1fff6b3c88cd830a3f1d7d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal\" (UID: \"b222e074fa1fff6b3c88cd830a3f1d7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal" Oct 30 01:17:55.050126 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.050004 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b222e074fa1fff6b3c88cd830a3f1d7d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal\" (UID: \"b222e074fa1fff6b3c88cd830a3f1d7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal" Oct 30 01:17:55.050126 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.050021 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/129d2d6479043c5ffd993e1a7481e29a-config\") pod \"kube-apiserver-proxy-ip-10-0-138-223.ec2.internal\" (UID: \"129d2d6479043c5ffd993e1a7481e29a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal" Oct 30 01:17:55.050126 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.049964 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b222e074fa1fff6b3c88cd830a3f1d7d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal\" (UID: \"b222e074fa1fff6b3c88cd830a3f1d7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal" Oct 30 01:17:55.050126 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.050059 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/129d2d6479043c5ffd993e1a7481e29a-config\") pod \"kube-apiserver-proxy-ip-10-0-138-223.ec2.internal\" (UID: \"129d2d6479043c5ffd993e1a7481e29a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal" Oct 30 01:17:55.050282 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.050106 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b222e074fa1fff6b3c88cd830a3f1d7d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal\" (UID: \"b222e074fa1fff6b3c88cd830a3f1d7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal" Oct 30 01:17:55.488417 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.488379 6572 apiserver.go:52] "Watching apiserver" Oct 30 01:17:55.530705 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.530667 6572 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 01:17:55.531402 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.531374 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-cnw75","openshift-multus/multus-m8h9g","openshift-ovn-kubernetes/ovnkube-node-6zn5n","kube-system/global-pull-secret-syncer-j7l6k","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4","openshift-dns/dns-default-98r9d","kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal","openshift-monitoring/telemeter-client-76696fb494-9gc8p","openshift-insights/insights-runtime-extractor-6vtnc","openshift-monitoring/thanos-querier-578df9b8f5-9z7nm","openshift-cluster-node-tuning-operator/tuned-cmcx6","openshift-dns/node-resolver-4dl5p","openshift-image-registry/node-ca-54hlq","openshift-multus/multus-additional-cni-plugins-5wdsg","openshift-monitoring/monitoring-plugin-77f4f76594-f7fkx","kube-system/konnectivity-agent-67v6w","openshift-ingress-canary/ingress-canary-42jd7","openshift-monitoring/prometheus-k8s-0","openshift-network-diagnostics/network-check-target-dmnzr","openshift-network-operator/iptables-alerter-xv9wf","openshift-multus/network-metrics-daemon-64xhb","openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9","openshift-monitoring/node-exporter-tqcs7"] Oct 30 01:17:55.544431 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.544390 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fsgft\"" Oct 30 01:17:55.544431 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.544410 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pcscp\"" Oct 30 01:17:55.544687 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.544586 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7dsj7\"" Oct 30 01:17:55.544687 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.544618 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 01:17:55.544818 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.544774 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 01:17:55.544871 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.544815 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-n79rv\"" Oct 30 01:17:55.544992 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.544962 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 01:17:55.545106 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.544998 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 01:17:55.545106 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.545023 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 01:17:55.545478 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.545459 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 01:17:55.545651 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.545464 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 01:17:55.545759 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.545472 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 01:17:55.545759 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.545506 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 01:17:55.545861 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.545573 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 01:17:55.545861 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.545582 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 01:17:55.545861 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.545599 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 01:17:55.545861 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.545655 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 01:17:55.545984 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.545707 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 01:17:55.546129 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.546113 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 01:17:55.551691 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.551673 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 01:17:55.551691 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.551692 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 01:17:55.554206 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.554189 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p2b59\"" Oct 30 01:17:55.554582 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.554568 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xmk59\"" Oct 30 01:17:55.554853 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.554838 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 01:17:55.554949 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.554933 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hb7tb\"" Oct 30 01:17:55.555336 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.555309 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 01:17:55.555433 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.555350 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:17:55.555433 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.555385 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 01:17:55.555521 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.555491 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 01:17:55.555521 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.555495 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 01:17:55.555616 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.555549 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 01:17:55.555616 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.555553 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 01:17:55.555701 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.555616 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 01:17:55.555701 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.555621 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-58lhf\"" Oct 30 01:17:55.555701 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.555645 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 01:17:55.555701 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.555677 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 01:17:55.555895 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.555719 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 01:17:55.555895 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.555736 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:17:55.555895 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.555774 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 01:17:55.555895 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.555794 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 01:17:55.555895 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.555805 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 01:17:55.555895 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.555849 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 01:17:55.556141 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.555906 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 01:17:55.556141 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.555918 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 01:17:55.556141 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.555929 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 01:17:55.556141 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.555960 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-p4hjp\"" Oct 30 01:17:55.556141 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.555978 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j8bsj\"" Oct 30 01:17:55.556141 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.556029 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wjscp\"" Oct 30 01:17:55.556141 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.556048 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xp5gx\"" Oct 30 01:17:55.556141 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.556049 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 01:17:55.556141 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.556068 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-srg4h\"" Oct 30 01:17:55.556477 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.556465 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ms58v\"" Oct 30 01:17:55.556641 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.556574 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 01:17:55.556641 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.556603 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 01:17:55.556834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.556664 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 01:17:55.556834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.556682 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 01:17:55.556834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.556684 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 01:17:55.556834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.556704 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-ntp2h\"" Oct 30 01:17:55.556834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.556731 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 01:17:55.557121 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.556903 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 01:17:55.557121 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.556980 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k4mfm\"" Oct 30 01:17:55.557121 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.557004 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 01:17:55.557121 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.557061 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 01:17:55.557121 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.557094 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-xqnkh\"" Oct 30 01:17:55.557121 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.557118 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 01:17:55.557385 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.557225 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 01:17:55.557385 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.557277 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 01:17:55.557506 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.557458 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 01:17:55.557919 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.557900 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 01:17:55.558063 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.558050 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 01:17:55.558123 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.558095 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 01:17:55.558173 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.558121 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-hgx58\"" Oct 30 01:17:55.558173 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.558133 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 01:17:55.558173 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.558153 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-2817d7ip80efo\"" Oct 30 01:17:55.560014 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.559996 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vthnr\"" Oct 30 01:17:55.560062 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.560014 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 01:17:55.561649 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.561636 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 01:17:55.565729 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.565701 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 01:17:55.565857 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.565794 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 01:17:55.565981 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.565961 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 01:17:55.566084 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.566035 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 01:17:55.566195 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.566178 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 01:17:55.566280 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.566219 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 01:17:55.566280 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.566248 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 01:17:55.566411 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.566284 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-wz8qb\"" Oct 30 01:17:55.566411 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.566330 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 01:17:55.566411 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.566351 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 01:17:55.566411 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.566373 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 01:17:55.566591 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.566406 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 01:17:55.566591 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.566424 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 01:17:55.566591 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.566459 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 01:17:55.566707 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.566593 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 01:17:55.566801 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.566780 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-ng7jx\"" Oct 30 01:17:55.566952 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.566933 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 01:17:55.566993 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.566962 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 01:17:55.567391 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.567374 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c86r3ikjcp03b\"" Oct 30 01:17:55.567449 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.567403 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 01:17:55.567487 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.567450 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-np2tl\"" Oct 30 01:17:55.567571 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.567557 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 01:17:55.567613 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.567562 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 01:17:55.572040 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.572024 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 01:17:55.575383 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.573824 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 01:17:55.604290 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.604246 6572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 01:17:55.608482 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.608460 6572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal" Oct 30 01:17:55.608637 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.608612 6572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal" Oct 30 01:17:55.628310 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.628283 6572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 01:17:55.628382 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:17:55.628341 6572 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-223.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal" Oct 30 01:17:55.632262 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.632240 6572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 01:17:55.632309 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:17:55.632295 6572 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal" Oct 30 01:17:55.652458 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.652418 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-run-openvswitch\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.652458 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.652462 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e829899-8a60-4792-b4cc-fabcaff0469f-tmp\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:55.652727 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.652487 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7922f23-5ba7-4931-8d19-598e2c4fe041-host\") pod \"node-ca-54hlq\" (UID: \"e7922f23-5ba7-4931-8d19-598e2c4fe041\") " pod="openshift-image-registry/node-ca-54hlq" Oct 30 01:17:55.652727 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.652548 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f3928614-3b83-4bf6-9e3f-04a71bae52bd-node-exporter-wtmp\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:55.652727 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.652583 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f3928614-3b83-4bf6-9e3f-04a71bae52bd-node-exporter-accelerators-collector-config\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:55.652727 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.652604 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f3928614-3b83-4bf6-9e3f-04a71bae52bd-root\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:55.652727 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.652608 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e829899-8a60-4792-b4cc-fabcaff0469f-tmp\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:55.652727 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.652621 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/88f5d337-fa16-4430-a82e-5f433354c164-kubelet-config\") pod \"global-pull-secret-syncer-j7l6k\" (UID: \"88f5d337-fa16-4430-a82e-5f433354c164\") " pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:17:55.652727 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.652640 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e829899-8a60-4792-b4cc-fabcaff0469f-metrics-client-ca\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:55.652727 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.652658 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-system-cni-dir\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.652727 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.652702 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.653102 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.652740 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-4gtf9\" (UID: \"a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9" Oct 30 01:17:55.653102 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.652804 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-host-run-netns\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.653102 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.652833 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-multus-conf-dir\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.653102 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.652836 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.653102 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.652843 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-4gtf9\" (UID: \"a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9" Oct 30 01:17:55.653102 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.652849 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f3928614-3b83-4bf6-9e3f-04a71bae52bd-node-exporter-accelerators-collector-config\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:55.653102 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.652858 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-system-cni-dir\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:17:55.653102 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.652894 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqww5\" (UniqueName: \"kubernetes.io/projected/9ceda047-30e9-496c-b854-af4043a83656-kube-api-access-tqww5\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:17:55.653102 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.652921 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-4gtf9\" (UID: \"a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9" Oct 30 01:17:55.653102 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.652931 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e829899-8a60-4792-b4cc-fabcaff0469f-metrics-client-ca\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:55.653102 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.653034 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwz2j\" (UniqueName: \"kubernetes.io/projected/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab-kube-api-access-wwz2j\") pod \"kube-state-metrics-5dc8d7d89b-4gtf9\" (UID: \"a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9" Oct 30 01:17:55.653102 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.653095 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.653503 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.653192 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf531672-6ee1-4188-b38d-2899bf9d6a0f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-cnw75\" (UID: \"cf531672-6ee1-4188-b38d-2899bf9d6a0f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cnw75" Oct 30 01:17:55.653503 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.653211 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-4gtf9\" (UID: \"a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9" Oct 30 01:17:55.653503 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.653272 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-multus-cni-dir\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.653503 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.653354 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-systemd-units\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.653503 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.653386 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.653503 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.653396 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9ceda047-30e9-496c-b854-af4043a83656-etc-selinux\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:17:55.653503 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.653400 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf531672-6ee1-4188-b38d-2899bf9d6a0f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-cnw75\" (UID: \"cf531672-6ee1-4188-b38d-2899bf9d6a0f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cnw75" Oct 30 01:17:55.653503 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.653430 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5bjt\" (UniqueName: \"kubernetes.io/projected/00c444f0-9313-4c0a-b9c8-79a953eaba17-kube-api-access-j5bjt\") pod \"dns-default-98r9d\" (UID: \"00c444f0-9313-4c0a-b9c8-79a953eaba17\") " pod="openshift-dns/dns-default-98r9d" Oct 30 01:17:55.653503 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.653460 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.653503 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.653477 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b2bg\" (UniqueName: \"kubernetes.io/projected/cf531672-6ee1-4188-b38d-2899bf9d6a0f-kube-api-access-6b2bg\") pod \"openshift-state-metrics-7bd7489957-cnw75\" (UID: \"cf531672-6ee1-4188-b38d-2899bf9d6a0f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cnw75" Oct 30 01:17:55.653503 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.653493 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9e829899-8a60-4792-b4cc-fabcaff0469f-secret-thanos-querier-tls\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:55.654008 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.653574 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9e829899-8a60-4792-b4cc-fabcaff0469f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:55.654008 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.653605 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9e829899-8a60-4792-b4cc-fabcaff0469f-secret-grpc-tls\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:55.654008 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.653633 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-host-var-lib-cni-multus\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.654008 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.653660 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-slash\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.654008 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.653685 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.654008 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.653702 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.654008 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.653711 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.654008 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.653770 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9e829899-8a60-4792-b4cc-fabcaff0469f-secret-thanos-querier-tls\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:55.654008 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.653839 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/88f5d337-fa16-4430-a82e-5f433354c164-original-pull-secret\") pod \"global-pull-secret-syncer-j7l6k\" (UID: \"88f5d337-fa16-4430-a82e-5f433354c164\") " pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:17:55.654008 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.653844 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9e829899-8a60-4792-b4cc-fabcaff0469f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:55.654008 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.653870 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9e829899-8a60-4792-b4cc-fabcaff0469f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:55.654008 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.653892 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-host-var-lib-cni-bin\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.654008 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.653908 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-cni-netd\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.654008 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.653924 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzkj9\" (UniqueName: \"kubernetes.io/projected/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-kube-api-access-zzkj9\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:55.654008 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.653931 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.654008 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.653921 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9e829899-8a60-4792-b4cc-fabcaff0469f-secret-grpc-tls\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:55.654008 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.653958 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.654008 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.653976 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:55.654008 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.654003 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3928614-3b83-4bf6-9e3f-04a71bae52bd-sys\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:55.654721 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.654028 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqpz6\" (UniqueName: \"kubernetes.io/projected/f4b29936-72de-4ddc-b7f1-a964a614ff61-kube-api-access-hqpz6\") pod \"network-metrics-daemon-64xhb\" (UID: \"f4b29936-72de-4ddc-b7f1-a964a614ff61\") " pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:17:55.654721 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.654051 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm5kk\" (UniqueName: \"kubernetes.io/projected/a6babafa-f85a-4a1b-9fe9-8bac54198754-kube-api-access-lm5kk\") pod \"node-resolver-4dl5p\" (UID: \"a6babafa-f85a-4a1b-9fe9-8bac54198754\") " pod="openshift-dns/node-resolver-4dl5p" Oct 30 01:17:55.654721 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.654074 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bc1464ba-e4e0-4685-b7d8-c873b189f8de-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.654721 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.654101 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-cni-binary-copy\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:17:55.654721 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.654128 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.654721 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.654145 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:55.654721 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.654179 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/88f5d337-fa16-4430-a82e-5f433354c164-original-pull-secret\") pod \"global-pull-secret-syncer-j7l6k\" (UID: \"88f5d337-fa16-4430-a82e-5f433354c164\") " pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:17:55.654721 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.654190 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b97aea6-bab9-498d-bce7-95e2a47c8c05-cni-binary-copy\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.654721 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.654221 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9e829899-8a60-4792-b4cc-fabcaff0469f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:55.654721 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.654244 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-var-lib-openvswitch\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.654721 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.654313 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-run-ovn-kubernetes\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.654721 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.654324 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-cni-binary-copy\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:17:55.654721 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.654374 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/262c01e0-8469-4fbe-aaa4-8a8c50551fce-konnectivity-ca\") pod \"konnectivity-agent-67v6w\" (UID: \"262c01e0-8469-4fbe-aaa4-8a8c50551fce\") " pod="kube-system/konnectivity-agent-67v6w" Oct 30 01:17:55.654721 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.654398 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b97aea6-bab9-498d-bce7-95e2a47c8c05-cni-binary-copy\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.654721 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.654400 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.654721 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.654405 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9e829899-8a60-4792-b4cc-fabcaff0469f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:55.654721 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.654432 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bc1464ba-e4e0-4685-b7d8-c873b189f8de-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.654721 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.654464 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czkw9\" (UniqueName: \"kubernetes.io/projected/1c5a490f-f8db-4afb-9a1f-b1accc9eadea-kube-api-access-czkw9\") pod \"network-check-target-dmnzr\" (UID: \"1c5a490f-f8db-4afb-9a1f-b1accc9eadea\") " pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:17:55.654721 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.654521 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/262c01e0-8469-4fbe-aaa4-8a8c50551fce-konnectivity-ca\") pod \"konnectivity-agent-67v6w\" (UID: \"262c01e0-8469-4fbe-aaa4-8a8c50551fce\") " pod="kube-system/konnectivity-agent-67v6w" Oct 30 01:17:55.654721 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.654622 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fad7d0da-535f-447c-89bd-ab36ac1861a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6vtnc\" (UID: \"fad7d0da-535f-447c-89bd-ab36ac1861a8\") " pod="openshift-insights/insights-runtime-extractor-6vtnc" Oct 30 01:17:55.654721 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.654645 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njqrk\" (UniqueName: \"kubernetes.io/projected/fad7d0da-535f-447c-89bd-ab36ac1861a8-kube-api-access-njqrk\") pod \"insights-runtime-extractor-6vtnc\" (UID: \"fad7d0da-535f-447c-89bd-ab36ac1861a8\") " pod="openshift-insights/insights-runtime-extractor-6vtnc" Oct 30 01:17:55.654721 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.654646 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9e829899-8a60-4792-b4cc-fabcaff0469f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:55.654721 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.654661 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00c444f0-9313-4c0a-b9c8-79a953eaba17-tmp-dir\") pod \"dns-default-98r9d\" (UID: \"00c444f0-9313-4c0a-b9c8-79a953eaba17\") " pod="openshift-dns/dns-default-98r9d" Oct 30 01:17:55.654721 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.654677 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-web-config\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.654721 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.654725 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-run-systemd\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.654769 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.654802 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-modprobe-d\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.654828 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-lib-modules\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.654853 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fad7d0da-535f-447c-89bd-ab36ac1861a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6vtnc\" (UID: \"fad7d0da-535f-447c-89bd-ab36ac1861a8\") " pod="openshift-insights/insights-runtime-extractor-6vtnc" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.654878 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-web-config\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.654886 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00c444f0-9313-4c0a-b9c8-79a953eaba17-metrics-tls\") pod \"dns-default-98r9d\" (UID: \"00c444f0-9313-4c0a-b9c8-79a953eaba17\") " pod="openshift-dns/dns-default-98r9d" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.654925 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.654947 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.654955 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00c444f0-9313-4c0a-b9c8-79a953eaba17-tmp-dir\") pod \"dns-default-98r9d\" (UID: \"00c444f0-9313-4c0a-b9c8-79a953eaba17\") " pod="openshift-dns/dns-default-98r9d" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.654965 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6babafa-f85a-4a1b-9fe9-8bac54198754-tmp-dir\") pod \"node-resolver-4dl5p\" (UID: \"a6babafa-f85a-4a1b-9fe9-8bac54198754\") " pod="openshift-dns/node-resolver-4dl5p" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.654991 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-telemeter-client-tls\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655010 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda047-30e9-496c-b854-af4043a83656-kubelet-dir\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655027 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6wcg\" (UniqueName: \"kubernetes.io/projected/e7922f23-5ba7-4931-8d19-598e2c4fe041-kube-api-access-z6wcg\") pod \"node-ca-54hlq\" (UID: \"e7922f23-5ba7-4931-8d19-598e2c4fe041\") " pod="openshift-image-registry/node-ca-54hlq" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655054 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-hostroot\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655094 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655098 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655115 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00c444f0-9313-4c0a-b9c8-79a953eaba17-metrics-tls\") pod \"dns-default-98r9d\" (UID: \"00c444f0-9313-4c0a-b9c8-79a953eaba17\") " pod="openshift-dns/dns-default-98r9d" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655116 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-etc-kubernetes\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655131 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6babafa-f85a-4a1b-9fe9-8bac54198754-tmp-dir\") pod \"node-resolver-4dl5p\" (UID: \"a6babafa-f85a-4a1b-9fe9-8bac54198754\") " pod="openshift-dns/node-resolver-4dl5p" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655198 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-etc-openvswitch\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655219 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b00a159-66f4-4b9d-bcf1-560094dd69bb-host-slash\") pod \"iptables-alerter-xv9wf\" (UID: \"8b00a159-66f4-4b9d-bcf1-560094dd69bb\") " pod="openshift-network-operator/iptables-alerter-xv9wf" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655247 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655269 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-telemeter-client-tls\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655275 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-metrics-client-ca\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655304 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655317 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-host-var-lib-kubelet\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655350 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4b97aea6-bab9-498d-bce7-95e2a47c8c05-multus-daemon-config\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655384 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46pfs\" (UniqueName: \"kubernetes.io/projected/03772f27-6bc6-4048-a515-d8d4e9abf431-kube-api-access-46pfs\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655409 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-systemd\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655468 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655483 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-metrics-client-ca\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655502 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fad7d0da-535f-447c-89bd-ab36ac1861a8-data-volume\") pod \"insights-runtime-extractor-6vtnc\" (UID: \"fad7d0da-535f-447c-89bd-ab36ac1861a8\") " pod="openshift-insights/insights-runtime-extractor-6vtnc" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655520 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b29936-72de-4ddc-b7f1-a964a614ff61-metrics-certs\") pod \"network-metrics-daemon-64xhb\" (UID: \"f4b29936-72de-4ddc-b7f1-a964a614ff61\") " pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655537 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4b97aea6-bab9-498d-bce7-95e2a47c8c05-multus-daemon-config\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655555 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf531672-6ee1-4188-b38d-2899bf9d6a0f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-cnw75\" (UID: \"cf531672-6ee1-4188-b38d-2899bf9d6a0f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cnw75" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655642 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-host-run-multus-certs\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655655 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fad7d0da-535f-447c-89bd-ab36ac1861a8-data-volume\") pod \"insights-runtime-extractor-6vtnc\" (UID: \"fad7d0da-535f-447c-89bd-ab36ac1861a8\") " pod="openshift-insights/insights-runtime-extractor-6vtnc" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655678 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/03772f27-6bc6-4048-a515-d8d4e9abf431-env-overrides\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.655681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655710 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b29936-72de-4ddc-b7f1-a964a614ff61-metrics-certs\") pod \"network-metrics-daemon-64xhb\" (UID: \"f4b29936-72de-4ddc-b7f1-a964a614ff61\") " pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655782 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-tuned\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655818 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fad7d0da-535f-447c-89bd-ab36ac1861a8-crio-socket\") pod \"insights-runtime-extractor-6vtnc\" (UID: \"fad7d0da-535f-447c-89bd-ab36ac1861a8\") " pod="openshift-insights/insights-runtime-extractor-6vtnc" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655824 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-tuned\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655841 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf531672-6ee1-4188-b38d-2899bf9d6a0f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-cnw75\" (UID: \"cf531672-6ee1-4188-b38d-2899bf9d6a0f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cnw75" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655851 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/00c444f0-9313-4c0a-b9c8-79a953eaba17-config-volume\") pod \"dns-default-98r9d\" (UID: \"00c444f0-9313-4c0a-b9c8-79a953eaba17\") " pod="openshift-dns/dns-default-98r9d" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655852 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/03772f27-6bc6-4048-a515-d8d4e9abf431-env-overrides\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655882 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bc1464ba-e4e0-4685-b7d8-c873b189f8de-config-out\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655900 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48qh7\" (UniqueName: \"kubernetes.io/projected/bc1464ba-e4e0-4685-b7d8-c873b189f8de-kube-api-access-48qh7\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655915 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655930 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fad7d0da-535f-447c-89bd-ab36ac1861a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6vtnc\" (UID: \"fad7d0da-535f-447c-89bd-ab36ac1861a8\") " pod="openshift-insights/insights-runtime-extractor-6vtnc" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655938 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bc1464ba-e4e0-4685-b7d8-c873b189f8de-config-out\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655947 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655968 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/00c444f0-9313-4c0a-b9c8-79a953eaba17-config-volume\") pod \"dns-default-98r9d\" (UID: \"00c444f0-9313-4c0a-b9c8-79a953eaba17\") " pod="openshift-dns/dns-default-98r9d" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.655968 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.656019 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-4gtf9\" (UID: \"a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.656048 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-host-run-k8s-cni-cncf-io\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.656073 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-log-socket\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.656098 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-sysctl-conf\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.656114 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fad7d0da-535f-447c-89bd-ab36ac1861a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6vtnc\" (UID: \"fad7d0da-535f-447c-89bd-ab36ac1861a8\") " pod="openshift-insights/insights-runtime-extractor-6vtnc" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.656125 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3928614-3b83-4bf6-9e3f-04a71bae52bd-metrics-client-ca\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.656114 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.656164 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-config\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.656174 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.656213 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.656213 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-4gtf9\" (UID: \"a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.656238 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-node-log\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.656275 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.656324 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-run-netns\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.656367 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.656383 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-config\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.656412 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3928614-3b83-4bf6-9e3f-04a71bae52bd-metrics-client-ca\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.656430 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda047-30e9-496c-b854-af4043a83656-registration-dir\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.656462 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04631a42-b08e-4aff-9fc4-78dd7392d59d-tmp\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.656515 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jnnc\" (UniqueName: \"kubernetes.io/projected/04631a42-b08e-4aff-9fc4-78dd7392d59d-kube-api-access-9jnnc\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.656546 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f3928614-3b83-4bf6-9e3f-04a71bae52bd-node-exporter-tls\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.656556 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.656547 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04631a42-b08e-4aff-9fc4-78dd7392d59d-tmp\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.656569 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.656642 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.656669 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e7922f23-5ba7-4931-8d19-598e2c4fe041-serviceca\") pod \"node-ca-54hlq\" (UID: \"e7922f23-5ba7-4931-8d19-598e2c4fe041\") " pod="openshift-image-registry/node-ca-54hlq" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.656695 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc85r\" (UniqueName: \"kubernetes.io/projected/4b97aea6-bab9-498d-bce7-95e2a47c8c05-kube-api-access-gc85r\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.656708 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f3928614-3b83-4bf6-9e3f-04a71bae52bd-node-exporter-tls\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.656719 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/03772f27-6bc6-4048-a515-d8d4e9abf431-ovnkube-config\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.656775 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-cnibin\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.656805 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-var-lib-kubelet\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.656833 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a6babafa-f85a-4a1b-9fe9-8bac54198754-hosts-file\") pod \"node-resolver-4dl5p\" (UID: \"a6babafa-f85a-4a1b-9fe9-8bac54198754\") " pod="openshift-dns/node-resolver-4dl5p" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.656868 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.656914 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/03772f27-6bc6-4048-a515-d8d4e9abf431-ovnkube-script-lib\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.656927 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.656953 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/03772f27-6bc6-4048-a515-d8d4e9abf431-ovnkube-config\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.656994 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6w4k\" (UniqueName: \"kubernetes.io/projected/f3928614-3b83-4bf6-9e3f-04a71bae52bd-kube-api-access-d6w4k\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.657088 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/03772f27-6bc6-4048-a515-d8d4e9abf431-ovnkube-script-lib\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.657127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.657143 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda047-30e9-496c-b854-af4043a83656-socket-dir\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.657165 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9ceda047-30e9-496c-b854-af4043a83656-sys-fs\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.657209 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-sysconfig\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.657246 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-sys\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.657276 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3efd0b05-2dcd-4e6c-a24e-9e955a58fa9b-cert\") pod \"ingress-canary-42jd7\" (UID: \"3efd0b05-2dcd-4e6c-a24e-9e955a58fa9b\") " pod="openshift-ingress-canary/ingress-canary-42jd7" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.657334 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e7922f23-5ba7-4931-8d19-598e2c4fe041-serviceca\") pod \"node-ca-54hlq\" (UID: \"e7922f23-5ba7-4931-8d19-598e2c4fe041\") " pod="openshift-image-registry/node-ca-54hlq" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.657350 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n85mv\" (UniqueName: \"kubernetes.io/projected/9e829899-8a60-4792-b4cc-fabcaff0469f-kube-api-access-n85mv\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.657380 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-kubernetes\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.657409 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f3928614-3b83-4bf6-9e3f-04a71bae52bd-node-exporter-textfile\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.657438 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2d66\" (UniqueName: \"kubernetes.io/projected/3efd0b05-2dcd-4e6c-a24e-9e955a58fa9b-kube-api-access-q2d66\") pod \"ingress-canary-42jd7\" (UID: \"3efd0b05-2dcd-4e6c-a24e-9e955a58fa9b\") " pod="openshift-ingress-canary/ingress-canary-42jd7" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.657472 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-serving-certs-ca-bundle\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.657479 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3efd0b05-2dcd-4e6c-a24e-9e955a58fa9b-cert\") pod \"ingress-canary-42jd7\" (UID: \"3efd0b05-2dcd-4e6c-a24e-9e955a58fa9b\") " pod="openshift-ingress-canary/ingress-canary-42jd7" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.657498 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-secret-telemeter-client\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.657502 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f3928614-3b83-4bf6-9e3f-04a71bae52bd-node-exporter-textfile\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.657522 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-kubelet\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.657559 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-cni-bin\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.657616 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm4x7\" (UniqueName: \"kubernetes.io/projected/8b00a159-66f4-4b9d-bcf1-560094dd69bb-kube-api-access-fm4x7\") pod \"iptables-alerter-xv9wf\" (UID: \"8b00a159-66f4-4b9d-bcf1-560094dd69bb\") " pod="openshift-network-operator/iptables-alerter-xv9wf" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.657656 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-serving-certs-ca-bundle\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.657678 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.657769 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-secret-telemeter-client\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.657772 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8b00a159-66f4-4b9d-bcf1-560094dd69bb-iptables-alerter-script\") pod \"iptables-alerter-xv9wf\" (UID: \"8b00a159-66f4-4b9d-bcf1-560094dd69bb\") " pod="openshift-network-operator/iptables-alerter-xv9wf" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.657822 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/262c01e0-8469-4fbe-aaa4-8a8c50551fce-agent-certs\") pod \"konnectivity-agent-67v6w\" (UID: \"262c01e0-8469-4fbe-aaa4-8a8c50551fce\") " pod="kube-system/konnectivity-agent-67v6w" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.657861 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx5fw\" (UniqueName: \"kubernetes.io/projected/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-kube-api-access-sx5fw\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.657894 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda047-30e9-496c-b854-af4043a83656-device-dir\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.657904 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8b00a159-66f4-4b9d-bcf1-560094dd69bb-iptables-alerter-script\") pod \"iptables-alerter-xv9wf\" (UID: \"8b00a159-66f4-4b9d-bcf1-560094dd69bb\") " pod="openshift-network-operator/iptables-alerter-xv9wf" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.657920 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-federate-client-tls\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.657946 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-sysctl-d\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.657976 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/88f5d337-fa16-4430-a82e-5f433354c164-dbus\") pod \"global-pull-secret-syncer-j7l6k\" (UID: \"88f5d337-fa16-4430-a82e-5f433354c164\") " pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.657997 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-4gtf9\" (UID: \"a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.658016 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-cnibin\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.658054 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-run-ovn\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.658107 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/03772f27-6bc6-4048-a515-d8d4e9abf431-ovn-node-metrics-cert\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.658109 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/262c01e0-8469-4fbe-aaa4-8a8c50551fce-agent-certs\") pod \"konnectivity-agent-67v6w\" (UID: \"262c01e0-8469-4fbe-aaa4-8a8c50551fce\") " pod="kube-system/konnectivity-agent-67v6w" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.658128 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-federate-client-tls\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.658150 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ljcm\" (UniqueName: \"kubernetes.io/projected/0ae3164f-fec1-4210-863e-70e49e6bd26a-kube-api-access-2ljcm\") pod \"monitoring-plugin-77f4f76594-f7fkx\" (UID: \"0ae3164f-fec1-4210-863e-70e49e6bd26a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-f7fkx" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.658193 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9e829899-8a60-4792-b4cc-fabcaff0469f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.658220 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-run\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.658240 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3928614-3b83-4bf6-9e3f-04a71bae52bd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.658244 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-4gtf9\" (UID: \"a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.658256 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.658278 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-4gtf9\" (UID: \"a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.658293 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-host\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.658311 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0ae3164f-fec1-4210-863e-70e49e6bd26a-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-f7fkx\" (UID: \"0ae3164f-fec1-4210-863e-70e49e6bd26a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-f7fkx" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.658326 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.658342 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf531672-6ee1-4188-b38d-2899bf9d6a0f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-cnw75\" (UID: \"cf531672-6ee1-4188-b38d-2899bf9d6a0f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cnw75" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.658359 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-multus-socket-dir-parent\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.658373 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9e829899-8a60-4792-b4cc-fabcaff0469f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.658390 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/03772f27-6bc6-4048-a515-d8d4e9abf431-ovn-node-metrics-cert\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.658443 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3928614-3b83-4bf6-9e3f-04a71bae52bd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.658480 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-4gtf9\" (UID: \"a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.658556 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.658636 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0ae3164f-fec1-4210-863e-70e49e6bd26a-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-f7fkx\" (UID: \"0ae3164f-fec1-4210-863e-70e49e6bd26a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-f7fkx" Oct 30 01:17:55.658987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.658639 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf531672-6ee1-4188-b38d-2899bf9d6a0f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-cnw75\" (UID: \"cf531672-6ee1-4188-b38d-2899bf9d6a0f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cnw75" Oct 30 01:17:55.660444 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.659382 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.668174 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.668150 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5bjt\" (UniqueName: \"kubernetes.io/projected/00c444f0-9313-4c0a-b9c8-79a953eaba17-kube-api-access-j5bjt\") pod \"dns-default-98r9d\" (UID: \"00c444f0-9313-4c0a-b9c8-79a953eaba17\") " pod="openshift-dns/dns-default-98r9d" Oct 30 01:17:55.668602 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.668585 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwz2j\" (UniqueName: \"kubernetes.io/projected/a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab-kube-api-access-wwz2j\") pod \"kube-state-metrics-5dc8d7d89b-4gtf9\" (UID: \"a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4gtf9" Oct 30 01:17:55.668701 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.668683 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqww5\" (UniqueName: \"kubernetes.io/projected/9ceda047-30e9-496c-b854-af4043a83656-kube-api-access-tqww5\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:17:55.669911 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.669895 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm5kk\" (UniqueName: \"kubernetes.io/projected/a6babafa-f85a-4a1b-9fe9-8bac54198754-kube-api-access-lm5kk\") pod \"node-resolver-4dl5p\" (UID: \"a6babafa-f85a-4a1b-9fe9-8bac54198754\") " pod="openshift-dns/node-resolver-4dl5p" Oct 30 01:17:55.670559 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.670542 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b2bg\" (UniqueName: \"kubernetes.io/projected/cf531672-6ee1-4188-b38d-2899bf9d6a0f-kube-api-access-6b2bg\") pod \"openshift-state-metrics-7bd7489957-cnw75\" (UID: \"cf531672-6ee1-4188-b38d-2899bf9d6a0f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cnw75" Oct 30 01:17:55.674599 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.674581 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njqrk\" (UniqueName: \"kubernetes.io/projected/fad7d0da-535f-447c-89bd-ab36ac1861a8-kube-api-access-njqrk\") pod \"insights-runtime-extractor-6vtnc\" (UID: \"fad7d0da-535f-447c-89bd-ab36ac1861a8\") " pod="openshift-insights/insights-runtime-extractor-6vtnc" Oct 30 01:17:55.675276 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.675257 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czkw9\" (UniqueName: \"kubernetes.io/projected/1c5a490f-f8db-4afb-9a1f-b1accc9eadea-kube-api-access-czkw9\") pod \"network-check-target-dmnzr\" (UID: \"1c5a490f-f8db-4afb-9a1f-b1accc9eadea\") " pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:17:55.675383 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.675368 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqpz6\" (UniqueName: \"kubernetes.io/projected/f4b29936-72de-4ddc-b7f1-a964a614ff61-kube-api-access-hqpz6\") pod \"network-metrics-daemon-64xhb\" (UID: \"f4b29936-72de-4ddc-b7f1-a964a614ff61\") " pod="openshift-multus/network-metrics-daemon-64xhb" Oct 30 01:17:55.675550 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.675534 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzkj9\" (UniqueName: \"kubernetes.io/projected/9bc3b964-6e84-4860-a5bf-0a81f15b0d97-kube-api-access-zzkj9\") pod \"telemeter-client-76696fb494-9gc8p\" (UID: \"9bc3b964-6e84-4860-a5bf-0a81f15b0d97\") " pod="openshift-monitoring/telemeter-client-76696fb494-9gc8p" Oct 30 01:17:55.677319 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.677301 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx5fw\" (UniqueName: \"kubernetes.io/projected/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-kube-api-access-sx5fw\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:17:55.678006 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.677973 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6wcg\" (UniqueName: \"kubernetes.io/projected/e7922f23-5ba7-4931-8d19-598e2c4fe041-kube-api-access-z6wcg\") pod \"node-ca-54hlq\" (UID: \"e7922f23-5ba7-4931-8d19-598e2c4fe041\") " pod="openshift-image-registry/node-ca-54hlq" Oct 30 01:17:55.678078 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.677980 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2d66\" (UniqueName: \"kubernetes.io/projected/3efd0b05-2dcd-4e6c-a24e-9e955a58fa9b-kube-api-access-q2d66\") pod \"ingress-canary-42jd7\" (UID: \"3efd0b05-2dcd-4e6c-a24e-9e955a58fa9b\") " pod="openshift-ingress-canary/ingress-canary-42jd7" Oct 30 01:17:55.678124 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.678077 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc85r\" (UniqueName: \"kubernetes.io/projected/4b97aea6-bab9-498d-bce7-95e2a47c8c05-kube-api-access-gc85r\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.678124 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.678102 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jnnc\" (UniqueName: \"kubernetes.io/projected/04631a42-b08e-4aff-9fc4-78dd7392d59d-kube-api-access-9jnnc\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.678362 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.678343 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6w4k\" (UniqueName: \"kubernetes.io/projected/f3928614-3b83-4bf6-9e3f-04a71bae52bd-kube-api-access-d6w4k\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:55.679075 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.679057 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ljcm\" (UniqueName: \"kubernetes.io/projected/0ae3164f-fec1-4210-863e-70e49e6bd26a-kube-api-access-2ljcm\") pod \"monitoring-plugin-77f4f76594-f7fkx\" (UID: \"0ae3164f-fec1-4210-863e-70e49e6bd26a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-f7fkx" Oct 30 01:17:55.679627 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.679612 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm4x7\" (UniqueName: \"kubernetes.io/projected/8b00a159-66f4-4b9d-bcf1-560094dd69bb-kube-api-access-fm4x7\") pod \"iptables-alerter-xv9wf\" (UID: \"8b00a159-66f4-4b9d-bcf1-560094dd69bb\") " pod="openshift-network-operator/iptables-alerter-xv9wf" Oct 30 01:17:55.679716 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.679700 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46pfs\" (UniqueName: \"kubernetes.io/projected/03772f27-6bc6-4048-a515-d8d4e9abf431-kube-api-access-46pfs\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.680538 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.680520 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n85mv\" (UniqueName: \"kubernetes.io/projected/9e829899-8a60-4792-b4cc-fabcaff0469f-kube-api-access-n85mv\") pod \"thanos-querier-578df9b8f5-9z7nm\" (UID: \"9e829899-8a60-4792-b4cc-fabcaff0469f\") " pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:55.680916 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.680900 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48qh7\" (UniqueName: \"kubernetes.io/projected/bc1464ba-e4e0-4685-b7d8-c873b189f8de-kube-api-access-48qh7\") pod \"prometheus-k8s-0\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:55.758876 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.758787 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fad7d0da-535f-447c-89bd-ab36ac1861a8-crio-socket\") pod \"insights-runtime-extractor-6vtnc\" (UID: \"fad7d0da-535f-447c-89bd-ab36ac1861a8\") " pod="openshift-insights/insights-runtime-extractor-6vtnc" Oct 30 01:17:55.758876 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.758825 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-host-run-k8s-cni-cncf-io\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.759146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.758843 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-log-socket\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.759146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.758906 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-sysctl-conf\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.759146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.758908 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fad7d0da-535f-447c-89bd-ab36ac1861a8-crio-socket\") pod \"insights-runtime-extractor-6vtnc\" (UID: \"fad7d0da-535f-447c-89bd-ab36ac1861a8\") " pod="openshift-insights/insights-runtime-extractor-6vtnc" Oct 30 01:17:55.759146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.758925 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-node-log\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.759146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.758931 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-log-socket\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.759146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.758904 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-host-run-k8s-cni-cncf-io\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.759146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.758954 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-node-log\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.759146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.758964 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-run-netns\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.759146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.758994 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda047-30e9-496c-b854-af4043a83656-registration-dir\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:17:55.759146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759015 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-cnibin\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:17:55.759146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759036 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-var-lib-kubelet\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.759146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759043 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-run-netns\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.759146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759057 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a6babafa-f85a-4a1b-9fe9-8bac54198754-hosts-file\") pod \"node-resolver-4dl5p\" (UID: \"a6babafa-f85a-4a1b-9fe9-8bac54198754\") " pod="openshift-dns/node-resolver-4dl5p" Oct 30 01:17:55.759146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759074 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.759146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759074 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-cnibin\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:17:55.759146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759088 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-var-lib-kubelet\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.759146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759094 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda047-30e9-496c-b854-af4043a83656-socket-dir\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:17:55.759146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759073 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda047-30e9-496c-b854-af4043a83656-registration-dir\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:17:55.759146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759115 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9ceda047-30e9-496c-b854-af4043a83656-sys-fs\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:17:55.759146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759079 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-sysctl-conf\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.759146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759122 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a6babafa-f85a-4a1b-9fe9-8bac54198754-hosts-file\") pod \"node-resolver-4dl5p\" (UID: \"a6babafa-f85a-4a1b-9fe9-8bac54198754\") " pod="openshift-dns/node-resolver-4dl5p" Oct 30 01:17:55.759146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759127 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.759146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759131 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-sysconfig\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.759146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759146 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda047-30e9-496c-b854-af4043a83656-socket-dir\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759173 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-sys\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759172 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9ceda047-30e9-496c-b854-af4043a83656-sys-fs\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759204 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-sysconfig\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759211 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-sys\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759201 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-kubernetes\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759242 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-kubernetes\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759249 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-kubelet\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759265 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-cni-bin\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759287 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759292 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-kubelet\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759309 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-cni-bin\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759315 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda047-30e9-496c-b854-af4043a83656-device-dir\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759331 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759338 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-sysctl-d\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759360 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/88f5d337-fa16-4430-a82e-5f433354c164-dbus\") pod \"global-pull-secret-syncer-j7l6k\" (UID: \"88f5d337-fa16-4430-a82e-5f433354c164\") " pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759380 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-cnibin\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759395 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-run-ovn\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759401 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda047-30e9-496c-b854-af4043a83656-device-dir\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759412 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-run\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759436 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-host\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759442 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-sysctl-d\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759435 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/88f5d337-fa16-4430-a82e-5f433354c164-dbus\") pod \"global-pull-secret-syncer-j7l6k\" (UID: \"88f5d337-fa16-4430-a82e-5f433354c164\") " pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759450 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-cnibin\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759451 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-run-ovn\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759464 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-multus-socket-dir-parent\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759484 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-host\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759488 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-run\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759496 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-run-openvswitch\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759495 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-multus-socket-dir-parent\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759513 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7922f23-5ba7-4931-8d19-598e2c4fe041-host\") pod \"node-ca-54hlq\" (UID: \"e7922f23-5ba7-4931-8d19-598e2c4fe041\") " pod="openshift-image-registry/node-ca-54hlq" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759538 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f3928614-3b83-4bf6-9e3f-04a71bae52bd-node-exporter-wtmp\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759549 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-run-openvswitch\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759554 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f3928614-3b83-4bf6-9e3f-04a71bae52bd-root\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759571 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/88f5d337-fa16-4430-a82e-5f433354c164-kubelet-config\") pod \"global-pull-secret-syncer-j7l6k\" (UID: \"88f5d337-fa16-4430-a82e-5f433354c164\") " pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759577 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7922f23-5ba7-4931-8d19-598e2c4fe041-host\") pod \"node-ca-54hlq\" (UID: \"e7922f23-5ba7-4931-8d19-598e2c4fe041\") " pod="openshift-image-registry/node-ca-54hlq" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759586 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f3928614-3b83-4bf6-9e3f-04a71bae52bd-node-exporter-wtmp\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759590 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-system-cni-dir\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759612 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/88f5d337-fa16-4430-a82e-5f433354c164-kubelet-config\") pod \"global-pull-secret-syncer-j7l6k\" (UID: \"88f5d337-fa16-4430-a82e-5f433354c164\") " pod="kube-system/global-pull-secret-syncer-j7l6k" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759617 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-host-run-netns\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759624 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f3928614-3b83-4bf6-9e3f-04a71bae52bd-root\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759644 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-system-cni-dir\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759655 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-multus-conf-dir\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759658 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-host-run-netns\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759686 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-multus-conf-dir\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759682 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-system-cni-dir\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759722 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-multus-cni-dir\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759741 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-systemd-units\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759741 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2-system-cni-dir\") pod \"multus-additional-cni-plugins-5wdsg\" (UID: \"ba9644c4-ec1e-4bb8-ae9a-d47acaa0dbb2\") " pod="openshift-multus/multus-additional-cni-plugins-5wdsg" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759777 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9ceda047-30e9-496c-b854-af4043a83656-etc-selinux\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759797 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-multus-cni-dir\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759801 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-systemd-units\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759804 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-host-var-lib-cni-multus\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759824 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-slash\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759829 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-host-var-lib-cni-multus\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759839 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9ceda047-30e9-496c-b854-af4043a83656-etc-selinux\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759849 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-host-var-lib-cni-bin\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759863 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-slash\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759865 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-cni-netd\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759883 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-cni-netd\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759889 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3928614-3b83-4bf6-9e3f-04a71bae52bd-sys\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759915 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-host-var-lib-cni-bin\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759918 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3928614-3b83-4bf6-9e3f-04a71bae52bd-sys\") pod \"node-exporter-tqcs7\" (UID: \"f3928614-3b83-4bf6-9e3f-04a71bae52bd\") " pod="openshift-monitoring/node-exporter-tqcs7" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759923 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-var-lib-openvswitch\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759958 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-run-ovn-kubernetes\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759994 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-var-lib-openvswitch\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.760004 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-run-systemd\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.759998 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-host-run-ovn-kubernetes\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.759979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.760023 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-modprobe-d\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.762023 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.760038 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-lib-modules\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.762023 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.760057 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda047-30e9-496c-b854-af4043a83656-kubelet-dir\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:17:55.762023 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.760076 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-hostroot\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.762023 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.760090 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-etc-kubernetes\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.762023 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.760105 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-etc-openvswitch\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.762023 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.760115 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9ceda047-30e9-496c-b854-af4043a83656-kubelet-dir\") pod \"aws-ebs-csi-driver-node-62pt4\" (UID: \"9ceda047-30e9-496c-b854-af4043a83656\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-62pt4" Oct 30 01:17:55.762023 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.760121 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-hostroot\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.762023 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.760153 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-lib-modules\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.762023 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.760150 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-etc-kubernetes\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.762023 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.760174 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-modprobe-d\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.762023 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.760159 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-run-systemd\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.762023 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.760187 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/03772f27-6bc6-4048-a515-d8d4e9abf431-etc-openvswitch\") pod \"ovnkube-node-6zn5n\" (UID: \"03772f27-6bc6-4048-a515-d8d4e9abf431\") " pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:55.762023 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.760207 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b00a159-66f4-4b9d-bcf1-560094dd69bb-host-slash\") pod \"iptables-alerter-xv9wf\" (UID: \"8b00a159-66f4-4b9d-bcf1-560094dd69bb\") " pod="openshift-network-operator/iptables-alerter-xv9wf" Oct 30 01:17:55.762023 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.760229 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-host-var-lib-kubelet\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.762023 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.760253 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-systemd\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.762023 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.760263 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b00a159-66f4-4b9d-bcf1-560094dd69bb-host-slash\") pod \"iptables-alerter-xv9wf\" (UID: \"8b00a159-66f4-4b9d-bcf1-560094dd69bb\") " pod="openshift-network-operator/iptables-alerter-xv9wf" Oct 30 01:17:55.762023 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.760269 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-host-var-lib-kubelet\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.762023 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.760278 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-host-run-multus-certs\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:55.762023 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.760303 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/04631a42-b08e-4aff-9fc4-78dd7392d59d-etc-systemd\") pod \"tuned-cmcx6\" (UID: \"04631a42-b08e-4aff-9fc4-78dd7392d59d\") " pod="openshift-cluster-node-tuning-operator/tuned-cmcx6" Oct 30 01:17:55.762023 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:55.760327 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4b97aea6-bab9-498d-bce7-95e2a47c8c05-host-run-multus-certs\") pod \"multus-m8h9g\" (UID: \"4b97aea6-bab9-498d-bce7-95e2a47c8c05\") " pod="openshift-multus/multus-m8h9g" Oct 30 01:17:56.416163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:56.416127 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-f7fkx" Oct 30 01:17:57.227865 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:57.227834 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:17:57.292908 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:57.292866 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:57.486682 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:57.486588 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:57.963013 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:57.962979 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-578df9b8f5-9z7nm" Oct 30 01:17:58.266611 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:58.266529 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-98r9d" Oct 30 01:17:59.114619 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:59.114581 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:17:59.130310 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:17:59.130283 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:18:00.565838 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:18:00.565795 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:00.773116 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:18:00.773079 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:01.202772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:18:01.202714 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:18:01.823329 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:18:01.823291 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:02.166024 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:18:02.165984 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:02.809304 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:18:02.809265 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:02.915495 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:18:02.915443 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:18:04.209350 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:18:04.209276 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:18:04.209781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:18:04.209549 6572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 01:18:11.219912 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:18:11.219877 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:11.409707 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:18:11.409670 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:12.814053 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:18:12.814011 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:13.383428 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:18:13.383398 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6zn5n" Oct 30 01:18:14.689152 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:18:14.689119 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:18:15.550833 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:18:15.550796 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:15.731030 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:18:15.730996 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:15.908277 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:18:15.908190 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:18.918036 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:18:18.918002 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-67v6w" Oct 30 01:18:22.917930 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:18:22.917896 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dmnzr" Oct 30 01:18:24.229013 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:18:24.228978 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:24.428308 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:18:24.428264 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:44.687101 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:18:44.687069 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:18:57.905941 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:18:57.905900 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:58.095717 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:18:58.095680 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:01.218985 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:01.218956 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:01.937249 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:01.937210 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:04.315184 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:04.315146 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:04.570004 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:04.569906 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:05.863396 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:05.863362 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:06.052223 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:06.052190 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:06.257380 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:06.257339 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:13.792091 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:13.792029 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 01:19:13.793101 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:13.793064 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bc1464ba-e4e0-4685-b7d8-c873b189f8de" containerName="prometheus" containerID="cri-o://fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef" gracePeriod=600 Oct 30 01:19:13.793255 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:13.793119 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bc1464ba-e4e0-4685-b7d8-c873b189f8de" containerName="thanos-sidecar" containerID="cri-o://a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23" gracePeriod=600 Oct 30 01:19:13.793255 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:13.793136 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bc1464ba-e4e0-4685-b7d8-c873b189f8de" containerName="kube-rbac-proxy-web" containerID="cri-o://e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07" gracePeriod=600 Oct 30 01:19:13.793255 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:13.793110 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bc1464ba-e4e0-4685-b7d8-c873b189f8de" containerName="kube-rbac-proxy" containerID="cri-o://3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3" gracePeriod=600 Oct 30 01:19:13.793255 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:13.793173 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bc1464ba-e4e0-4685-b7d8-c873b189f8de" containerName="config-reloader" containerID="cri-o://2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73" gracePeriod=600 Oct 30 01:19:13.793255 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:13.793191 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bc1464ba-e4e0-4685-b7d8-c873b189f8de" containerName="kube-rbac-proxy-thanos" containerID="cri-o://31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1" gracePeriod=600 Oct 30 01:19:14.031213 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.031185 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:14.195248 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.195213 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-configmap-kubelet-serving-ca-bundle\") pod \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " Oct 30 01:19:14.195413 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.195264 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc1464ba-e4e0-4685-b7d8-c873b189f8de-kube-api-access-48qh7\" (UniqueName: \"kubernetes.io/projected/bc1464ba-e4e0-4685-b7d8-c873b189f8de-kube-api-access-48qh7\") pod \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " Oct 30 01:19:14.195413 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.195286 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-configmap-serving-certs-ca-bundle\") pod \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " Oct 30 01:19:14.195413 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.195308 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-prometheus-k8s-tls\") pod \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " Oct 30 01:19:14.195413 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.195340 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-trusted-ca-bundle\") pod \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " Oct 30 01:19:14.195413 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.195364 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-metrics-client-certs\") pod \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " Oct 30 01:19:14.195413 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.195391 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc1464ba-e4e0-4685-b7d8-c873b189f8de-tls-assets\" (UniqueName: \"kubernetes.io/projected/bc1464ba-e4e0-4685-b7d8-c873b189f8de-tls-assets\") pod \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " Oct 30 01:19:14.195724 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.195435 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-rulefiles-1\") pod \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " Oct 30 01:19:14.195724 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.195469 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-rulefiles-2\") pod \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " Oct 30 01:19:14.195724 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.195507 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " Oct 30 01:19:14.195724 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.195535 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-web-config\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-web-config\") pod \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " Oct 30 01:19:14.195724 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.195575 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " Oct 30 01:19:14.195724 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.195609 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-kube-rbac-proxy\") pod \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " Oct 30 01:19:14.195724 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.195657 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-rulefiles-0\") pod \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " Oct 30 01:19:14.195724 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.195686 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-configmap-metrics-client-ca\") pod \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " Oct 30 01:19:14.195724 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.195726 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc1464ba-e4e0-4685-b7d8-c873b189f8de-config-out\" (UniqueName: \"kubernetes.io/empty-dir/bc1464ba-e4e0-4685-b7d8-c873b189f8de-config-out\") pod \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " Oct 30 01:19:14.196182 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.195732 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-configmap-kubelet-serving-ca-bundle" pod "bc1464ba-e4e0-4685-b7d8-c873b189f8de" (UID: "bc1464ba-e4e0-4685-b7d8-c873b189f8de"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:14.196182 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.195777 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-thanos-prometheus-http-client-file\") pod \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " Oct 30 01:19:14.196182 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.195808 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-grpc-tls\") pod \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " Oct 30 01:19:14.196182 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.195845 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-db\") pod \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " Oct 30 01:19:14.196182 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.195881 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-config\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-config\") pod \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\" (UID: \"bc1464ba-e4e0-4685-b7d8-c873b189f8de\") " Oct 30 01:19:14.196182 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.195880 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-configmap-serving-certs-ca-bundle" pod "bc1464ba-e4e0-4685-b7d8-c873b189f8de" (UID: "bc1464ba-e4e0-4685-b7d8-c873b189f8de"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:14.196182 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.195965 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-rulefiles-1" pod "bc1464ba-e4e0-4685-b7d8-c873b189f8de" (UID: "bc1464ba-e4e0-4685-b7d8-c873b189f8de"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:14.196182 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.196075 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-trusted-ca-bundle" pod "bc1464ba-e4e0-4685-b7d8-c873b189f8de" (UID: "bc1464ba-e4e0-4685-b7d8-c873b189f8de"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:14.196182 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.196167 6572 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:19:14.196650 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.196189 6572 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:19:14.196650 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.196204 6572 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:19:14.196650 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.196221 6572 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:19:14.197255 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.197222 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-configmap-metrics-client-ca" pod "bc1464ba-e4e0-4685-b7d8-c873b189f8de" (UID: "bc1464ba-e4e0-4685-b7d8-c873b189f8de"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:14.197911 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.197884 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-rulefiles-2" pod "bc1464ba-e4e0-4685-b7d8-c873b189f8de" (UID: "bc1464ba-e4e0-4685-b7d8-c873b189f8de"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:14.198894 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.198632 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-rulefiles-0" pod "bc1464ba-e4e0-4685-b7d8-c873b189f8de" (UID: "bc1464ba-e4e0-4685-b7d8-c873b189f8de"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:14.198894 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.198769 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-prometheus-k8s-kube-rbac-proxy-web" pod "bc1464ba-e4e0-4685-b7d8-c873b189f8de" (UID: "bc1464ba-e4e0-4685-b7d8-c873b189f8de"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:19:14.198894 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.198777 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1464ba-e4e0-4685-b7d8-c873b189f8de-kube-api-access-48qh7" pod "bc1464ba-e4e0-4685-b7d8-c873b189f8de" (UID: "bc1464ba-e4e0-4685-b7d8-c873b189f8de"). InnerVolumeSpecName "kube-api-access-48qh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:14.199226 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.198924 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1464ba-e4e0-4685-b7d8-c873b189f8de-config-out" pod "bc1464ba-e4e0-4685-b7d8-c873b189f8de" (UID: "bc1464ba-e4e0-4685-b7d8-c873b189f8de"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:14.199226 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.199130 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1464ba-e4e0-4685-b7d8-c873b189f8de-tls-assets" pod "bc1464ba-e4e0-4685-b7d8-c873b189f8de" (UID: "bc1464ba-e4e0-4685-b7d8-c873b189f8de"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:14.199339 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.199287 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-kube-rbac-proxy" pod "bc1464ba-e4e0-4685-b7d8-c873b189f8de" (UID: "bc1464ba-e4e0-4685-b7d8-c873b189f8de"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:19:14.199603 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.199566 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-config" pod "bc1464ba-e4e0-4685-b7d8-c873b189f8de" (UID: "bc1464ba-e4e0-4685-b7d8-c873b189f8de"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:19:14.200443 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.200416 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-db" pod "bc1464ba-e4e0-4685-b7d8-c873b189f8de" (UID: "bc1464ba-e4e0-4685-b7d8-c873b189f8de"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:14.200522 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.200484 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-metrics-client-certs" pod "bc1464ba-e4e0-4685-b7d8-c873b189f8de" (UID: "bc1464ba-e4e0-4685-b7d8-c873b189f8de"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:19:14.200815 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.200781 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-thanos-prometheus-http-client-file" pod "bc1464ba-e4e0-4685-b7d8-c873b189f8de" (UID: "bc1464ba-e4e0-4685-b7d8-c873b189f8de"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:19:14.201088 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.201067 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-prometheus-k8s-tls" pod "bc1464ba-e4e0-4685-b7d8-c873b189f8de" (UID: "bc1464ba-e4e0-4685-b7d8-c873b189f8de"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:19:14.201183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.201134 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-grpc-tls" pod "bc1464ba-e4e0-4685-b7d8-c873b189f8de" (UID: "bc1464ba-e4e0-4685-b7d8-c873b189f8de"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:19:14.201318 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.201301 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-prometheus-k8s-thanos-sidecar-tls" pod "bc1464ba-e4e0-4685-b7d8-c873b189f8de" (UID: "bc1464ba-e4e0-4685-b7d8-c873b189f8de"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:19:14.210151 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.210130 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-web-config" pod "bc1464ba-e4e0-4685-b7d8-c873b189f8de" (UID: "bc1464ba-e4e0-4685-b7d8-c873b189f8de"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:19:14.297533 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.297499 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48qh7\" (UniqueName: \"kubernetes.io/projected/bc1464ba-e4e0-4685-b7d8-c873b189f8de-kube-api-access-48qh7\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:19:14.297533 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.297527 6572 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:19:14.297533 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.297536 6572 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-metrics-client-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:19:14.297818 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.297547 6572 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bc1464ba-e4e0-4685-b7d8-c873b189f8de-tls-assets\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:19:14.297818 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.297558 6572 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:19:14.297818 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.297568 6572 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:19:14.297818 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.297577 6572 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-web-config\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:19:14.297818 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.297587 6572 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:19:14.297818 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.297595 6572 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-kube-rbac-proxy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:19:14.297818 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.297605 6572 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:19:14.297818 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.297614 6572 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc1464ba-e4e0-4685-b7d8-c873b189f8de-configmap-metrics-client-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:19:14.297818 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.297623 6572 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bc1464ba-e4e0-4685-b7d8-c873b189f8de-config-out\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:19:14.297818 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.297633 6572 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:19:14.297818 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.297641 6572 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-secret-grpc-tls\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:19:14.297818 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.297650 6572 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bc1464ba-e4e0-4685-b7d8-c873b189f8de-prometheus-k8s-db\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:19:14.297818 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.297658 6572 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/bc1464ba-e4e0-4685-b7d8-c873b189f8de-config\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:19:14.686592 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:19:14.686553 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:19:14.887250 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.887216 6572 generic.go:356] "Generic (PLEG): container finished" podID="bc1464ba-e4e0-4685-b7d8-c873b189f8de" containerID="31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1" exitCode=0 Oct 30 01:19:14.887250 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.887241 6572 generic.go:356] "Generic (PLEG): container finished" podID="bc1464ba-e4e0-4685-b7d8-c873b189f8de" containerID="3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3" exitCode=0 Oct 30 01:19:14.887250 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.887248 6572 generic.go:356] "Generic (PLEG): container finished" podID="bc1464ba-e4e0-4685-b7d8-c873b189f8de" containerID="e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07" exitCode=0 Oct 30 01:19:14.887250 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.887253 6572 generic.go:356] "Generic (PLEG): container finished" podID="bc1464ba-e4e0-4685-b7d8-c873b189f8de" containerID="a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23" exitCode=0 Oct 30 01:19:14.887250 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.887259 6572 generic.go:356] "Generic (PLEG): container finished" podID="bc1464ba-e4e0-4685-b7d8-c873b189f8de" containerID="2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73" exitCode=0 Oct 30 01:19:14.887840 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.887264 6572 generic.go:356] "Generic (PLEG): container finished" podID="bc1464ba-e4e0-4685-b7d8-c873b189f8de" containerID="fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef" exitCode=0 Oct 30 01:19:14.887840 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.887308 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc1464ba-e4e0-4685-b7d8-c873b189f8de","Type":"ContainerDied","Data":"31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1"} Oct 30 01:19:14.887840 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.887387 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc1464ba-e4e0-4685-b7d8-c873b189f8de","Type":"ContainerDied","Data":"3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3"} Oct 30 01:19:14.887840 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.887322 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:14.887840 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.887429 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc1464ba-e4e0-4685-b7d8-c873b189f8de","Type":"ContainerDied","Data":"e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07"} Oct 30 01:19:14.887840 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.887468 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc1464ba-e4e0-4685-b7d8-c873b189f8de","Type":"ContainerDied","Data":"a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23"} Oct 30 01:19:14.887840 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.887479 6572 scope.go:122] "RemoveContainer" containerID="31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1" Oct 30 01:19:14.887840 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.887510 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc1464ba-e4e0-4685-b7d8-c873b189f8de","Type":"ContainerDied","Data":"2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73"} Oct 30 01:19:14.887840 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.887553 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc1464ba-e4e0-4685-b7d8-c873b189f8de","Type":"ContainerDied","Data":"fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef"} Oct 30 01:19:14.887840 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.887596 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bc1464ba-e4e0-4685-b7d8-c873b189f8de","Type":"ContainerDied","Data":"a046ba2d3bfee0e2bbdc1974b56581f8e57db6f84f40a65fc361e6f0b5b656f5"} Oct 30 01:19:14.895224 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.895205 6572 scope.go:122] "RemoveContainer" containerID="3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3" Oct 30 01:19:14.902395 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.902369 6572 scope.go:122] "RemoveContainer" containerID="e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07" Oct 30 01:19:14.910016 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.909994 6572 scope.go:122] "RemoveContainer" containerID="a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23" Oct 30 01:19:14.916987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.916968 6572 scope.go:122] "RemoveContainer" containerID="2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73" Oct 30 01:19:14.924257 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.924233 6572 scope.go:122] "RemoveContainer" containerID="fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef" Oct 30 01:19:14.931527 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.931502 6572 scope.go:122] "RemoveContainer" containerID="ea6bdcb072917bdd7945749f813e0f8b851cad79f7d09cb105b7a097c31f7c66" Oct 30 01:19:14.938594 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.938571 6572 scope.go:122] "RemoveContainer" containerID="31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1" Oct 30 01:19:14.938929 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:19:14.938905 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1\": container with ID starting with 31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1 not found: ID does not exist" containerID="31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1" Oct 30 01:19:14.939012 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.938943 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1"} err="failed to get container status \"31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1\": rpc error: code = NotFound desc = could not find container \"31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1\": container with ID starting with 31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1 not found: ID does not exist" Oct 30 01:19:14.939012 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.938992 6572 scope.go:122] "RemoveContainer" containerID="3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3" Oct 30 01:19:14.939302 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:19:14.939283 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3\": container with ID starting with 3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3 not found: ID does not exist" containerID="3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3" Oct 30 01:19:14.939345 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.939309 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3"} err="failed to get container status \"3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3\": rpc error: code = NotFound desc = could not find container \"3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3\": container with ID starting with 3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3 not found: ID does not exist" Oct 30 01:19:14.939345 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.939324 6572 scope.go:122] "RemoveContainer" containerID="e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07" Oct 30 01:19:14.939541 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:19:14.939521 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07\": container with ID starting with e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07 not found: ID does not exist" containerID="e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07" Oct 30 01:19:14.939582 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.939547 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07"} err="failed to get container status \"e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07\": rpc error: code = NotFound desc = could not find container \"e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07\": container with ID starting with e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07 not found: ID does not exist" Oct 30 01:19:14.939582 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.939564 6572 scope.go:122] "RemoveContainer" containerID="a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23" Oct 30 01:19:14.939901 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:19:14.939882 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23\": container with ID starting with a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23 not found: ID does not exist" containerID="a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23" Oct 30 01:19:14.939950 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.939906 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23"} err="failed to get container status \"a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23\": rpc error: code = NotFound desc = could not find container \"a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23\": container with ID starting with a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23 not found: ID does not exist" Oct 30 01:19:14.939950 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.939920 6572 scope.go:122] "RemoveContainer" containerID="2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73" Oct 30 01:19:14.940172 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:19:14.940154 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73\": container with ID starting with 2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73 not found: ID does not exist" containerID="2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73" Oct 30 01:19:14.940208 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.940176 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73"} err="failed to get container status \"2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73\": rpc error: code = NotFound desc = could not find container \"2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73\": container with ID starting with 2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73 not found: ID does not exist" Oct 30 01:19:14.940208 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.940191 6572 scope.go:122] "RemoveContainer" containerID="fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef" Oct 30 01:19:14.940406 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:19:14.940389 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef\": container with ID starting with fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef not found: ID does not exist" containerID="fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef" Oct 30 01:19:14.940445 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.940410 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef"} err="failed to get container status \"fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef\": rpc error: code = NotFound desc = could not find container \"fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef\": container with ID starting with fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef not found: ID does not exist" Oct 30 01:19:14.940445 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.940421 6572 scope.go:122] "RemoveContainer" containerID="ea6bdcb072917bdd7945749f813e0f8b851cad79f7d09cb105b7a097c31f7c66" Oct 30 01:19:14.940662 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:19:14.940646 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea6bdcb072917bdd7945749f813e0f8b851cad79f7d09cb105b7a097c31f7c66\": container with ID starting with ea6bdcb072917bdd7945749f813e0f8b851cad79f7d09cb105b7a097c31f7c66 not found: ID does not exist" containerID="ea6bdcb072917bdd7945749f813e0f8b851cad79f7d09cb105b7a097c31f7c66" Oct 30 01:19:14.940710 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.940664 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea6bdcb072917bdd7945749f813e0f8b851cad79f7d09cb105b7a097c31f7c66"} err="failed to get container status \"ea6bdcb072917bdd7945749f813e0f8b851cad79f7d09cb105b7a097c31f7c66\": rpc error: code = NotFound desc = could not find container \"ea6bdcb072917bdd7945749f813e0f8b851cad79f7d09cb105b7a097c31f7c66\": container with ID starting with ea6bdcb072917bdd7945749f813e0f8b851cad79f7d09cb105b7a097c31f7c66 not found: ID does not exist" Oct 30 01:19:14.940710 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.940676 6572 scope.go:122] "RemoveContainer" containerID="31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1" Oct 30 01:19:14.940931 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.940913 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1"} err="failed to get container status \"31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1\": rpc error: code = NotFound desc = could not find container \"31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1\": container with ID starting with 31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1 not found: ID does not exist" Oct 30 01:19:14.940931 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.940930 6572 scope.go:122] "RemoveContainer" containerID="3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3" Oct 30 01:19:14.941126 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.941110 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3"} err="failed to get container status \"3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3\": rpc error: code = NotFound desc = could not find container \"3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3\": container with ID starting with 3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3 not found: ID does not exist" Oct 30 01:19:14.941126 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.941125 6572 scope.go:122] "RemoveContainer" containerID="e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07" Oct 30 01:19:14.941301 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.941283 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07"} err="failed to get container status \"e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07\": rpc error: code = NotFound desc = could not find container \"e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07\": container with ID starting with e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07 not found: ID does not exist" Oct 30 01:19:14.941338 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.941301 6572 scope.go:122] "RemoveContainer" containerID="a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23" Oct 30 01:19:14.941495 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.941476 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23"} err="failed to get container status \"a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23\": rpc error: code = NotFound desc = could not find container \"a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23\": container with ID starting with a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23 not found: ID does not exist" Oct 30 01:19:14.941536 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.941496 6572 scope.go:122] "RemoveContainer" containerID="2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73" Oct 30 01:19:14.941716 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.941700 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73"} err="failed to get container status \"2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73\": rpc error: code = NotFound desc = could not find container \"2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73\": container with ID starting with 2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73 not found: ID does not exist" Oct 30 01:19:14.941716 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.941715 6572 scope.go:122] "RemoveContainer" containerID="fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef" Oct 30 01:19:14.941940 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.941923 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef"} err="failed to get container status \"fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef\": rpc error: code = NotFound desc = could not find container \"fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef\": container with ID starting with fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef not found: ID does not exist" Oct 30 01:19:14.941940 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.941939 6572 scope.go:122] "RemoveContainer" containerID="ea6bdcb072917bdd7945749f813e0f8b851cad79f7d09cb105b7a097c31f7c66" Oct 30 01:19:14.942112 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.942090 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea6bdcb072917bdd7945749f813e0f8b851cad79f7d09cb105b7a097c31f7c66"} err="failed to get container status \"ea6bdcb072917bdd7945749f813e0f8b851cad79f7d09cb105b7a097c31f7c66\": rpc error: code = NotFound desc = could not find container \"ea6bdcb072917bdd7945749f813e0f8b851cad79f7d09cb105b7a097c31f7c66\": container with ID starting with ea6bdcb072917bdd7945749f813e0f8b851cad79f7d09cb105b7a097c31f7c66 not found: ID does not exist" Oct 30 01:19:14.942155 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.942113 6572 scope.go:122] "RemoveContainer" containerID="31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1" Oct 30 01:19:14.942284 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.942269 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1"} err="failed to get container status \"31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1\": rpc error: code = NotFound desc = could not find container \"31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1\": container with ID starting with 31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1 not found: ID does not exist" Oct 30 01:19:14.942324 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.942284 6572 scope.go:122] "RemoveContainer" containerID="3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3" Oct 30 01:19:14.942498 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.942481 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3"} err="failed to get container status \"3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3\": rpc error: code = NotFound desc = could not find container \"3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3\": container with ID starting with 3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3 not found: ID does not exist" Oct 30 01:19:14.942539 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.942499 6572 scope.go:122] "RemoveContainer" containerID="e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07" Oct 30 01:19:14.942694 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.942679 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07"} err="failed to get container status \"e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07\": rpc error: code = NotFound desc = could not find container \"e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07\": container with ID starting with e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07 not found: ID does not exist" Oct 30 01:19:14.942737 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.942693 6572 scope.go:122] "RemoveContainer" containerID="a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23" Oct 30 01:19:14.942884 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.942869 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23"} err="failed to get container status \"a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23\": rpc error: code = NotFound desc = could not find container \"a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23\": container with ID starting with a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23 not found: ID does not exist" Oct 30 01:19:14.942884 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.942883 6572 scope.go:122] "RemoveContainer" containerID="2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73" Oct 30 01:19:14.943115 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.943096 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73"} err="failed to get container status \"2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73\": rpc error: code = NotFound desc = could not find container \"2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73\": container with ID starting with 2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73 not found: ID does not exist" Oct 30 01:19:14.943169 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.943114 6572 scope.go:122] "RemoveContainer" containerID="fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef" Oct 30 01:19:14.943365 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.943348 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef"} err="failed to get container status \"fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef\": rpc error: code = NotFound desc = could not find container \"fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef\": container with ID starting with fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef not found: ID does not exist" Oct 30 01:19:14.943409 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.943366 6572 scope.go:122] "RemoveContainer" containerID="ea6bdcb072917bdd7945749f813e0f8b851cad79f7d09cb105b7a097c31f7c66" Oct 30 01:19:14.943610 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.943585 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea6bdcb072917bdd7945749f813e0f8b851cad79f7d09cb105b7a097c31f7c66"} err="failed to get container status \"ea6bdcb072917bdd7945749f813e0f8b851cad79f7d09cb105b7a097c31f7c66\": rpc error: code = NotFound desc = could not find container \"ea6bdcb072917bdd7945749f813e0f8b851cad79f7d09cb105b7a097c31f7c66\": container with ID starting with ea6bdcb072917bdd7945749f813e0f8b851cad79f7d09cb105b7a097c31f7c66 not found: ID does not exist" Oct 30 01:19:14.943610 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.943609 6572 scope.go:122] "RemoveContainer" containerID="31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1" Oct 30 01:19:14.943911 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.943887 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1"} err="failed to get container status \"31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1\": rpc error: code = NotFound desc = could not find container \"31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1\": container with ID starting with 31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1 not found: ID does not exist" Oct 30 01:19:14.943911 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.943909 6572 scope.go:122] "RemoveContainer" containerID="3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3" Oct 30 01:19:14.944173 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.944154 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3"} err="failed to get container status \"3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3\": rpc error: code = NotFound desc = could not find container \"3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3\": container with ID starting with 3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3 not found: ID does not exist" Oct 30 01:19:14.944249 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.944174 6572 scope.go:122] "RemoveContainer" containerID="e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07" Oct 30 01:19:14.944400 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.944383 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07"} err="failed to get container status \"e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07\": rpc error: code = NotFound desc = could not find container \"e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07\": container with ID starting with e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07 not found: ID does not exist" Oct 30 01:19:14.944445 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.944401 6572 scope.go:122] "RemoveContainer" containerID="a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23" Oct 30 01:19:14.944511 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.944491 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 01:19:14.944623 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.944605 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23"} err="failed to get container status \"a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23\": rpc error: code = NotFound desc = could not find container \"a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23\": container with ID starting with a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23 not found: ID does not exist" Oct 30 01:19:14.944691 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.944624 6572 scope.go:122] "RemoveContainer" containerID="2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73" Oct 30 01:19:14.944950 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.944929 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73"} err="failed to get container status \"2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73\": rpc error: code = NotFound desc = could not find container \"2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73\": container with ID starting with 2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73 not found: ID does not exist" Oct 30 01:19:14.945023 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.944949 6572 scope.go:122] "RemoveContainer" containerID="fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef" Oct 30 01:19:14.945196 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.945180 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef"} err="failed to get container status \"fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef\": rpc error: code = NotFound desc = could not find container \"fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef\": container with ID starting with fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef not found: ID does not exist" Oct 30 01:19:14.945196 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.945195 6572 scope.go:122] "RemoveContainer" containerID="ea6bdcb072917bdd7945749f813e0f8b851cad79f7d09cb105b7a097c31f7c66" Oct 30 01:19:14.945413 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.945396 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea6bdcb072917bdd7945749f813e0f8b851cad79f7d09cb105b7a097c31f7c66"} err="failed to get container status \"ea6bdcb072917bdd7945749f813e0f8b851cad79f7d09cb105b7a097c31f7c66\": rpc error: code = NotFound desc = could not find container \"ea6bdcb072917bdd7945749f813e0f8b851cad79f7d09cb105b7a097c31f7c66\": container with ID starting with ea6bdcb072917bdd7945749f813e0f8b851cad79f7d09cb105b7a097c31f7c66 not found: ID does not exist" Oct 30 01:19:14.945467 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.945413 6572 scope.go:122] "RemoveContainer" containerID="31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1" Oct 30 01:19:14.945608 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.945592 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1"} err="failed to get container status \"31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1\": rpc error: code = NotFound desc = could not find container \"31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1\": container with ID starting with 31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1 not found: ID does not exist" Oct 30 01:19:14.945608 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.945607 6572 scope.go:122] "RemoveContainer" containerID="3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3" Oct 30 01:19:14.945879 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.945859 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3"} err="failed to get container status \"3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3\": rpc error: code = NotFound desc = could not find container \"3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3\": container with ID starting with 3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3 not found: ID does not exist" Oct 30 01:19:14.945926 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.945879 6572 scope.go:122] "RemoveContainer" containerID="e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07" Oct 30 01:19:14.946110 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.946094 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07"} err="failed to get container status \"e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07\": rpc error: code = NotFound desc = could not find container \"e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07\": container with ID starting with e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07 not found: ID does not exist" Oct 30 01:19:14.946147 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.946109 6572 scope.go:122] "RemoveContainer" containerID="a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23" Oct 30 01:19:14.946347 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.946327 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23"} err="failed to get container status \"a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23\": rpc error: code = NotFound desc = could not find container \"a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23\": container with ID starting with a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23 not found: ID does not exist" Oct 30 01:19:14.946388 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.946349 6572 scope.go:122] "RemoveContainer" containerID="2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73" Oct 30 01:19:14.946564 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.946546 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73"} err="failed to get container status \"2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73\": rpc error: code = NotFound desc = could not find container \"2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73\": container with ID starting with 2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73 not found: ID does not exist" Oct 30 01:19:14.946609 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.946564 6572 scope.go:122] "RemoveContainer" containerID="fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef" Oct 30 01:19:14.946796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.946772 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef"} err="failed to get container status \"fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef\": rpc error: code = NotFound desc = could not find container \"fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef\": container with ID starting with fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef not found: ID does not exist" Oct 30 01:19:14.946796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.946789 6572 scope.go:122] "RemoveContainer" containerID="ea6bdcb072917bdd7945749f813e0f8b851cad79f7d09cb105b7a097c31f7c66" Oct 30 01:19:14.947012 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.946992 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea6bdcb072917bdd7945749f813e0f8b851cad79f7d09cb105b7a097c31f7c66"} err="failed to get container status \"ea6bdcb072917bdd7945749f813e0f8b851cad79f7d09cb105b7a097c31f7c66\": rpc error: code = NotFound desc = could not find container \"ea6bdcb072917bdd7945749f813e0f8b851cad79f7d09cb105b7a097c31f7c66\": container with ID starting with ea6bdcb072917bdd7945749f813e0f8b851cad79f7d09cb105b7a097c31f7c66 not found: ID does not exist" Oct 30 01:19:14.947058 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.947013 6572 scope.go:122] "RemoveContainer" containerID="31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1" Oct 30 01:19:14.947211 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.947191 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1"} err="failed to get container status \"31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1\": rpc error: code = NotFound desc = could not find container \"31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1\": container with ID starting with 31cdf547817539344eaf9a882d56eb4d2dbf70fb7039e0b9927b76e6f446c8c1 not found: ID does not exist" Oct 30 01:19:14.947269 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.947212 6572 scope.go:122] "RemoveContainer" containerID="3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3" Oct 30 01:19:14.947424 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.947404 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3"} err="failed to get container status \"3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3\": rpc error: code = NotFound desc = could not find container \"3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3\": container with ID starting with 3d6d834d52b56f4e7a777cac4c395e7d0827371adf99bb8258aba3ed508218e3 not found: ID does not exist" Oct 30 01:19:14.947477 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.947424 6572 scope.go:122] "RemoveContainer" containerID="e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07" Oct 30 01:19:14.947646 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.947618 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07"} err="failed to get container status \"e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07\": rpc error: code = NotFound desc = could not find container \"e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07\": container with ID starting with e37c46a3ead69f8d79d9f6841b4912f3fc12fc511044da5e32191c973109fd07 not found: ID does not exist" Oct 30 01:19:14.947692 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.947646 6572 scope.go:122] "RemoveContainer" containerID="a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23" Oct 30 01:19:14.947886 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.947869 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23"} err="failed to get container status \"a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23\": rpc error: code = NotFound desc = could not find container \"a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23\": container with ID starting with a15ea66e7731fefd8169ab4a4f13ab1eb3547345b049fcbea8fec803cf810b23 not found: ID does not exist" Oct 30 01:19:14.947932 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.947886 6572 scope.go:122] "RemoveContainer" containerID="2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73" Oct 30 01:19:14.948097 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.948080 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73"} err="failed to get container status \"2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73\": rpc error: code = NotFound desc = could not find container \"2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73\": container with ID starting with 2791bc6ef49aee0b0354d45cfcb7d86c366ad3eff0a2e45eaca85c6548f2ad73 not found: ID does not exist" Oct 30 01:19:14.948138 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.948098 6572 scope.go:122] "RemoveContainer" containerID="fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef" Oct 30 01:19:14.948299 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.948281 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef"} err="failed to get container status \"fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef\": rpc error: code = NotFound desc = could not find container \"fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef\": container with ID starting with fcc907459234e23da7546f5b562e329f1a472cfba59118a200f50abf62c6e0ef not found: ID does not exist" Oct 30 01:19:14.948337 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.948299 6572 scope.go:122] "RemoveContainer" containerID="ea6bdcb072917bdd7945749f813e0f8b851cad79f7d09cb105b7a097c31f7c66" Oct 30 01:19:14.948512 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.948494 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea6bdcb072917bdd7945749f813e0f8b851cad79f7d09cb105b7a097c31f7c66"} err="failed to get container status \"ea6bdcb072917bdd7945749f813e0f8b851cad79f7d09cb105b7a097c31f7c66\": rpc error: code = NotFound desc = could not find container \"ea6bdcb072917bdd7945749f813e0f8b851cad79f7d09cb105b7a097c31f7c66\": container with ID starting with ea6bdcb072917bdd7945749f813e0f8b851cad79f7d09cb105b7a097c31f7c66 not found: ID does not exist" Oct 30 01:19:14.954556 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.954529 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 01:19:14.973833 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.973807 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 01:19:14.974110 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.974096 6572 cpu_manager.go:414] "containerMap: removing container" podUID="bc1464ba-e4e0-4685-b7d8-c873b189f8de" containerName="kube-rbac-proxy" Oct 30 01:19:14.974149 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.974113 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc1464ba-e4e0-4685-b7d8-c873b189f8de" containerName="kube-rbac-proxy" Oct 30 01:19:14.974149 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.974122 6572 cpu_manager.go:414] "containerMap: removing container" podUID="bc1464ba-e4e0-4685-b7d8-c873b189f8de" containerName="kube-rbac-proxy-web" Oct 30 01:19:14.974149 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.974127 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc1464ba-e4e0-4685-b7d8-c873b189f8de" containerName="kube-rbac-proxy-web" Oct 30 01:19:14.974149 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.974137 6572 cpu_manager.go:414] "containerMap: removing container" podUID="bc1464ba-e4e0-4685-b7d8-c873b189f8de" containerName="thanos-sidecar" Oct 30 01:19:14.974149 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.974142 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc1464ba-e4e0-4685-b7d8-c873b189f8de" containerName="thanos-sidecar" Oct 30 01:19:14.974149 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.974148 6572 cpu_manager.go:414] "containerMap: removing container" podUID="bc1464ba-e4e0-4685-b7d8-c873b189f8de" containerName="prometheus" Oct 30 01:19:14.974321 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.974153 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc1464ba-e4e0-4685-b7d8-c873b189f8de" containerName="prometheus" Oct 30 01:19:14.974321 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.974168 6572 cpu_manager.go:414] "containerMap: removing container" podUID="bc1464ba-e4e0-4685-b7d8-c873b189f8de" containerName="config-reloader" Oct 30 01:19:14.974321 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.974203 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc1464ba-e4e0-4685-b7d8-c873b189f8de" containerName="config-reloader" Oct 30 01:19:14.974321 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.974210 6572 cpu_manager.go:414] "containerMap: removing container" podUID="bc1464ba-e4e0-4685-b7d8-c873b189f8de" containerName="init-config-reloader" Oct 30 01:19:14.974321 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.974214 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc1464ba-e4e0-4685-b7d8-c873b189f8de" containerName="init-config-reloader" Oct 30 01:19:14.974321 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.974221 6572 cpu_manager.go:414] "containerMap: removing container" podUID="bc1464ba-e4e0-4685-b7d8-c873b189f8de" containerName="kube-rbac-proxy-thanos" Oct 30 01:19:14.974321 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.974225 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc1464ba-e4e0-4685-b7d8-c873b189f8de" containerName="kube-rbac-proxy-thanos" Oct 30 01:19:14.974321 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.974283 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc1464ba-e4e0-4685-b7d8-c873b189f8de" containerName="kube-rbac-proxy" Oct 30 01:19:14.974321 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.974289 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc1464ba-e4e0-4685-b7d8-c873b189f8de" containerName="kube-rbac-proxy-web" Oct 30 01:19:14.974321 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.974296 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc1464ba-e4e0-4685-b7d8-c873b189f8de" containerName="thanos-sidecar" Oct 30 01:19:14.974321 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.974301 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc1464ba-e4e0-4685-b7d8-c873b189f8de" containerName="prometheus" Oct 30 01:19:14.974321 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.974312 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc1464ba-e4e0-4685-b7d8-c873b189f8de" containerName="config-reloader" Oct 30 01:19:14.974321 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.974318 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc1464ba-e4e0-4685-b7d8-c873b189f8de" containerName="init-config-reloader" Oct 30 01:19:14.974321 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.974324 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc1464ba-e4e0-4685-b7d8-c873b189f8de" containerName="kube-rbac-proxy-thanos" Oct 30 01:19:14.978084 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:14.978064 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.001902 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.001874 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 01:19:15.002117 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.002102 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 01:19:15.002420 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.002396 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 01:19:15.002635 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.002623 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 01:19:15.002868 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.002855 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 01:19:15.002955 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.002884 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 01:19:15.003179 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.003165 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 01:19:15.003229 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.003216 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 01:19:15.003394 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.003382 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c86r3ikjcp03b\"" Oct 30 01:19:15.003442 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.003382 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 01:19:15.003442 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.003397 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-np2tl\"" Oct 30 01:19:15.003907 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.003895 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 01:19:15.003963 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.003950 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 01:19:15.004013 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.003964 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 01:19:15.004207 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.004188 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 01:19:15.013918 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.013893 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 01:19:15.015105 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.015086 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 01:19:15.015699 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.015680 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 01:19:15.104265 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.104173 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/326173d8-e9a0-4c44-8af9-d400b03828ac-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.104265 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.104221 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/326173d8-e9a0-4c44-8af9-d400b03828ac-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.104265 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.104241 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cv2m\" (UniqueName: \"kubernetes.io/projected/326173d8-e9a0-4c44-8af9-d400b03828ac-kube-api-access-2cv2m\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.104475 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.104369 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/326173d8-e9a0-4c44-8af9-d400b03828ac-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.104475 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.104401 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/326173d8-e9a0-4c44-8af9-d400b03828ac-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.104475 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.104465 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/326173d8-e9a0-4c44-8af9-d400b03828ac-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.104569 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.104507 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/326173d8-e9a0-4c44-8af9-d400b03828ac-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.104569 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.104531 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/326173d8-e9a0-4c44-8af9-d400b03828ac-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.104569 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.104554 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/326173d8-e9a0-4c44-8af9-d400b03828ac-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.104655 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.104569 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/326173d8-e9a0-4c44-8af9-d400b03828ac-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.104655 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.104594 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/326173d8-e9a0-4c44-8af9-d400b03828ac-config-out\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.104655 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.104633 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/326173d8-e9a0-4c44-8af9-d400b03828ac-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.104772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.104731 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/326173d8-e9a0-4c44-8af9-d400b03828ac-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.104830 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.104781 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/326173d8-e9a0-4c44-8af9-d400b03828ac-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.104867 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.104827 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/326173d8-e9a0-4c44-8af9-d400b03828ac-web-config\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.104867 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.104845 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/326173d8-e9a0-4c44-8af9-d400b03828ac-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.104927 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.104889 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/326173d8-e9a0-4c44-8af9-d400b03828ac-config\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.104927 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.104918 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/326173d8-e9a0-4c44-8af9-d400b03828ac-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.104989 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.104934 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/326173d8-e9a0-4c44-8af9-d400b03828ac-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.104989 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.104952 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/326173d8-e9a0-4c44-8af9-d400b03828ac-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.206068 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.206008 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/326173d8-e9a0-4c44-8af9-d400b03828ac-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.206068 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.206066 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/326173d8-e9a0-4c44-8af9-d400b03828ac-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.206068 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.206086 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/326173d8-e9a0-4c44-8af9-d400b03828ac-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.206380 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.206114 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/326173d8-e9a0-4c44-8af9-d400b03828ac-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.206380 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.206143 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/326173d8-e9a0-4c44-8af9-d400b03828ac-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.206380 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.206183 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/326173d8-e9a0-4c44-8af9-d400b03828ac-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.206380 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.206209 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/326173d8-e9a0-4c44-8af9-d400b03828ac-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.206380 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.206233 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/326173d8-e9a0-4c44-8af9-d400b03828ac-config-out\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.206380 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.206256 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/326173d8-e9a0-4c44-8af9-d400b03828ac-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.206380 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.206297 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/326173d8-e9a0-4c44-8af9-d400b03828ac-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.206699 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.206541 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/326173d8-e9a0-4c44-8af9-d400b03828ac-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.206699 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.206533 6572 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 01:19:15.206699 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.206589 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/326173d8-e9a0-4c44-8af9-d400b03828ac-web-config\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.206699 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.206615 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/326173d8-e9a0-4c44-8af9-d400b03828ac-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.206699 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.206657 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/326173d8-e9a0-4c44-8af9-d400b03828ac-config\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.207035 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.206701 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/326173d8-e9a0-4c44-8af9-d400b03828ac-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.207035 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.206725 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/326173d8-e9a0-4c44-8af9-d400b03828ac-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.207035 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.206775 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/326173d8-e9a0-4c44-8af9-d400b03828ac-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.207035 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.206808 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/326173d8-e9a0-4c44-8af9-d400b03828ac-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.207035 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.206836 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/326173d8-e9a0-4c44-8af9-d400b03828ac-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.207035 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.206859 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cv2m\" (UniqueName: \"kubernetes.io/projected/326173d8-e9a0-4c44-8af9-d400b03828ac-kube-api-access-2cv2m\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.207035 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.206971 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/326173d8-e9a0-4c44-8af9-d400b03828ac-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.207035 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.206997 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/326173d8-e9a0-4c44-8af9-d400b03828ac-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.207450 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.207152 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/326173d8-e9a0-4c44-8af9-d400b03828ac-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.207522 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.207497 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/326173d8-e9a0-4c44-8af9-d400b03828ac-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.207629 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.207597 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/326173d8-e9a0-4c44-8af9-d400b03828ac-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.209635 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.209591 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/326173d8-e9a0-4c44-8af9-d400b03828ac-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.210323 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.210295 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/326173d8-e9a0-4c44-8af9-d400b03828ac-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.210439 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.210318 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/326173d8-e9a0-4c44-8af9-d400b03828ac-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.210514 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.210494 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/326173d8-e9a0-4c44-8af9-d400b03828ac-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.210666 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.210632 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/326173d8-e9a0-4c44-8af9-d400b03828ac-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.211109 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.211090 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/326173d8-e9a0-4c44-8af9-d400b03828ac-config-out\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.211217 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.211181 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/326173d8-e9a0-4c44-8af9-d400b03828ac-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.211504 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.211485 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/326173d8-e9a0-4c44-8af9-d400b03828ac-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.211929 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.211893 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/326173d8-e9a0-4c44-8af9-d400b03828ac-config\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.212104 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.212082 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/326173d8-e9a0-4c44-8af9-d400b03828ac-web-config\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.212183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.212108 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/326173d8-e9a0-4c44-8af9-d400b03828ac-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.212183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.212143 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/326173d8-e9a0-4c44-8af9-d400b03828ac-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.212303 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.212279 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/326173d8-e9a0-4c44-8af9-d400b03828ac-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.213043 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.213026 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/326173d8-e9a0-4c44-8af9-d400b03828ac-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.223402 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.223381 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cv2m\" (UniqueName: \"kubernetes.io/projected/326173d8-e9a0-4c44-8af9-d400b03828ac-kube-api-access-2cv2m\") pod \"prometheus-k8s-0\" (UID: \"326173d8-e9a0-4c44-8af9-d400b03828ac\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.295043 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.294990 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:15.609725 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.609666 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 01:19:15.622701 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:19:15.622660 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod326173d8_e9a0_4c44_8af9_d400b03828ac.slice/crio-fcc81449d7782f5a6ebe12e8025dc8f09c7f6fbf61d20bb9b37249e63f09001f WatchSource:0}: Error finding container fcc81449d7782f5a6ebe12e8025dc8f09c7f6fbf61d20bb9b37249e63f09001f: Status 404 returned error can't find the container with id fcc81449d7782f5a6ebe12e8025dc8f09c7f6fbf61d20bb9b37249e63f09001f Oct 30 01:19:15.898389 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.898291 6572 generic.go:356] "Generic (PLEG): container finished" podID="326173d8-e9a0-4c44-8af9-d400b03828ac" containerID="3b6fece5a7831a9470a3f7c54af1ba76ca0677303b2563ec3188e456f8f0c00f" exitCode=0 Oct 30 01:19:15.898389 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.898335 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"326173d8-e9a0-4c44-8af9-d400b03828ac","Type":"ContainerDied","Data":"3b6fece5a7831a9470a3f7c54af1ba76ca0677303b2563ec3188e456f8f0c00f"} Oct 30 01:19:15.898940 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:15.898400 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"326173d8-e9a0-4c44-8af9-d400b03828ac","Type":"ContainerStarted","Data":"fcc81449d7782f5a6ebe12e8025dc8f09c7f6fbf61d20bb9b37249e63f09001f"} Oct 30 01:19:16.539667 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:16.539589 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc1464ba-e4e0-4685-b7d8-c873b189f8de" path="/var/lib/kubelet/pods/bc1464ba-e4e0-4685-b7d8-c873b189f8de/volumes" Oct 30 01:19:16.905673 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:16.905590 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"326173d8-e9a0-4c44-8af9-d400b03828ac","Type":"ContainerStarted","Data":"e4772796040930c9474e6679fe97dbe54fee0e9d17ee321f6eeda4ec96572f71"} Oct 30 01:19:16.905673 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:16.905649 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"326173d8-e9a0-4c44-8af9-d400b03828ac","Type":"ContainerStarted","Data":"a09422054fcc4b52f1fe8a593007490ba15ea445cec1b2379e6410d29b17f8f5"} Oct 30 01:19:16.906186 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:16.905676 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"326173d8-e9a0-4c44-8af9-d400b03828ac","Type":"ContainerStarted","Data":"fab781aaa5eba4f80390ae01cfe35540ddadf9010c5a0b8099371a2406fb11f2"} Oct 30 01:19:16.906186 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:16.905700 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"326173d8-e9a0-4c44-8af9-d400b03828ac","Type":"ContainerStarted","Data":"d55cdad4e9fb53ba00a2946126017081535d2fa7141712c43c635c2efc043b40"} Oct 30 01:19:16.906186 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:16.905725 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"326173d8-e9a0-4c44-8af9-d400b03828ac","Type":"ContainerStarted","Data":"3d6f45b5650e076ecf4456b350af93f670c4bd3e328ae97a4f18720afdb78eef"} Oct 30 01:19:16.906186 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:16.905764 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"326173d8-e9a0-4c44-8af9-d400b03828ac","Type":"ContainerStarted","Data":"bfff07c24822a4d43707a9ba62d8536efd33c9fa92a2b0e0c784b0ee969a0482"} Oct 30 01:19:16.966544 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:16.966453 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.950521093 podStartE2EDuration="2.966434241s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.913148ms" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:16.966001796 +0000 UTC m=+82.542427198" watchObservedRunningTime="2026-10-30 01:19:16.966434241 +0000 UTC m=+82.542859852" Oct 30 01:19:17.045093 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:17.045061 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:17.306814 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:17.306780 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:17.632892 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:17.632794 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:17.819261 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:17.819224 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:20.295668 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:20.295631 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:19:30.396760 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:30.396707 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:30.583854 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:30.583820 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:30.762401 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:19:30.762372 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:44.688377 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:19:44.688281 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:20:02.049599 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:20:02.049556 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:14.686457 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:20:14.686419 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:20:15.295454 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:20:15.295419 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:20:15.311093 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:20:15.311061 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:20:16.137388 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:20:16.137355 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:20:44.686158 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:20:44.686116 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:21:02.166138 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:02.166094 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:02.353854 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:02.353806 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:14.686476 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:21:14.686440 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:21:21.416227 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:21.416189 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:21.649775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:21.649726 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:21.869601 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:21.869509 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:22.865312 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:22.865273 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:23.053973 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.053938 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:23.610534 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.610501 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c85qml"] Oct 30 01:21:23.613904 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.613885 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c85qml" Oct 30 01:21:23.638125 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.638093 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 01:21:23.638125 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.638101 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 01:21:23.638258 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.638166 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-t2w6h\"" Oct 30 01:21:23.639334 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.639317 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07e39324-5f99-4c0e-9968-cb65e2c5eea2-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c85qml\" (UID: \"07e39324-5f99-4c0e-9968-cb65e2c5eea2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c85qml" Oct 30 01:21:23.639388 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.639343 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thbqj\" (UniqueName: \"kubernetes.io/projected/07e39324-5f99-4c0e-9968-cb65e2c5eea2-kube-api-access-thbqj\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c85qml\" (UID: \"07e39324-5f99-4c0e-9968-cb65e2c5eea2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c85qml" Oct 30 01:21:23.639434 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.639421 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07e39324-5f99-4c0e-9968-cb65e2c5eea2-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c85qml\" (UID: \"07e39324-5f99-4c0e-9968-cb65e2c5eea2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c85qml" Oct 30 01:21:23.642405 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.642385 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c85qml"] Oct 30 01:21:23.698445 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.698422 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rl9jj"] Oct 30 01:21:23.701847 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.701828 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rl9jj" Oct 30 01:21:23.716626 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.716609 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-vdnp6\"" Oct 30 01:21:23.721652 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.721633 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rl9jj"] Oct 30 01:21:23.740171 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.740147 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07e39324-5f99-4c0e-9968-cb65e2c5eea2-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c85qml\" (UID: \"07e39324-5f99-4c0e-9968-cb65e2c5eea2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c85qml" Oct 30 01:21:23.740258 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.740181 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thbqj\" (UniqueName: \"kubernetes.io/projected/07e39324-5f99-4c0e-9968-cb65e2c5eea2-kube-api-access-thbqj\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c85qml\" (UID: \"07e39324-5f99-4c0e-9968-cb65e2c5eea2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c85qml" Oct 30 01:21:23.740258 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.740210 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/094b28a0-e057-4a0e-8583-089b04c76127-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rl9jj\" (UID: \"094b28a0-e057-4a0e-8583-089b04c76127\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rl9jj" Oct 30 01:21:23.740258 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.740245 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grxth\" (UniqueName: \"kubernetes.io/projected/094b28a0-e057-4a0e-8583-089b04c76127-kube-api-access-grxth\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rl9jj\" (UID: \"094b28a0-e057-4a0e-8583-089b04c76127\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rl9jj" Oct 30 01:21:23.740392 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.740284 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07e39324-5f99-4c0e-9968-cb65e2c5eea2-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c85qml\" (UID: \"07e39324-5f99-4c0e-9968-cb65e2c5eea2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c85qml" Oct 30 01:21:23.740459 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.740441 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/094b28a0-e057-4a0e-8583-089b04c76127-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rl9jj\" (UID: \"094b28a0-e057-4a0e-8583-089b04c76127\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rl9jj" Oct 30 01:21:23.740521 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.740478 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07e39324-5f99-4c0e-9968-cb65e2c5eea2-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c85qml\" (UID: \"07e39324-5f99-4c0e-9968-cb65e2c5eea2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c85qml" Oct 30 01:21:23.740578 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.740562 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07e39324-5f99-4c0e-9968-cb65e2c5eea2-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c85qml\" (UID: \"07e39324-5f99-4c0e-9968-cb65e2c5eea2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c85qml" Oct 30 01:21:23.756763 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.756721 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thbqj\" (UniqueName: \"kubernetes.io/projected/07e39324-5f99-4c0e-9968-cb65e2c5eea2-kube-api-access-thbqj\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c85qml\" (UID: \"07e39324-5f99-4c0e-9968-cb65e2c5eea2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c85qml" Oct 30 01:21:23.791549 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.791519 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9x9v9"] Oct 30 01:21:23.795028 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.795012 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9x9v9" Oct 30 01:21:23.812945 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.812909 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-rfsbk\"" Oct 30 01:21:23.813872 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.813850 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9x9v9"] Oct 30 01:21:23.841439 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.841405 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/094b28a0-e057-4a0e-8583-089b04c76127-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rl9jj\" (UID: \"094b28a0-e057-4a0e-8583-089b04c76127\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rl9jj" Oct 30 01:21:23.841559 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.841456 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/094b28a0-e057-4a0e-8583-089b04c76127-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rl9jj\" (UID: \"094b28a0-e057-4a0e-8583-089b04c76127\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rl9jj" Oct 30 01:21:23.841559 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.841477 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea59f9b0-3567-40e9-84a0-412e683a68c3-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9x9v9\" (UID: \"ea59f9b0-3567-40e9-84a0-412e683a68c3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9x9v9" Oct 30 01:21:23.841559 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.841497 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtfpl\" (UniqueName: \"kubernetes.io/projected/ea59f9b0-3567-40e9-84a0-412e683a68c3-kube-api-access-wtfpl\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9x9v9\" (UID: \"ea59f9b0-3567-40e9-84a0-412e683a68c3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9x9v9" Oct 30 01:21:23.841559 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.841520 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grxth\" (UniqueName: \"kubernetes.io/projected/094b28a0-e057-4a0e-8583-089b04c76127-kube-api-access-grxth\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rl9jj\" (UID: \"094b28a0-e057-4a0e-8583-089b04c76127\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rl9jj" Oct 30 01:21:23.841559 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.841551 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea59f9b0-3567-40e9-84a0-412e683a68c3-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9x9v9\" (UID: \"ea59f9b0-3567-40e9-84a0-412e683a68c3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9x9v9" Oct 30 01:21:23.841853 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.841787 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/094b28a0-e057-4a0e-8583-089b04c76127-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rl9jj\" (UID: \"094b28a0-e057-4a0e-8583-089b04c76127\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rl9jj" Oct 30 01:21:23.841912 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.841860 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/094b28a0-e057-4a0e-8583-089b04c76127-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rl9jj\" (UID: \"094b28a0-e057-4a0e-8583-089b04c76127\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rl9jj" Oct 30 01:21:23.858963 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.858935 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grxth\" (UniqueName: \"kubernetes.io/projected/094b28a0-e057-4a0e-8583-089b04c76127-kube-api-access-grxth\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rl9jj\" (UID: \"094b28a0-e057-4a0e-8583-089b04c76127\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rl9jj" Oct 30 01:21:23.891805 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.891729 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qk2gf"] Oct 30 01:21:23.899138 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.899117 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qk2gf" Oct 30 01:21:23.911807 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.911787 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-b28pp\"" Oct 30 01:21:23.912669 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.912640 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qk2gf"] Oct 30 01:21:23.929053 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.929029 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c85qml" Oct 30 01:21:23.930702 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.930678 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:23.943159 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.943134 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea59f9b0-3567-40e9-84a0-412e683a68c3-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9x9v9\" (UID: \"ea59f9b0-3567-40e9-84a0-412e683a68c3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9x9v9" Oct 30 01:21:23.943245 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.943179 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qk2gf\" (UID: \"5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qk2gf" Oct 30 01:21:23.943245 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.943226 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78lh4\" (UniqueName: \"kubernetes.io/projected/5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1-kube-api-access-78lh4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qk2gf\" (UID: \"5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qk2gf" Oct 30 01:21:23.943403 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.943382 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea59f9b0-3567-40e9-84a0-412e683a68c3-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9x9v9\" (UID: \"ea59f9b0-3567-40e9-84a0-412e683a68c3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9x9v9" Oct 30 01:21:23.943464 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.943420 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtfpl\" (UniqueName: \"kubernetes.io/projected/ea59f9b0-3567-40e9-84a0-412e683a68c3-kube-api-access-wtfpl\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9x9v9\" (UID: \"ea59f9b0-3567-40e9-84a0-412e683a68c3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9x9v9" Oct 30 01:21:23.943464 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.943447 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea59f9b0-3567-40e9-84a0-412e683a68c3-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9x9v9\" (UID: \"ea59f9b0-3567-40e9-84a0-412e683a68c3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9x9v9" Oct 30 01:21:23.943567 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.943467 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qk2gf\" (UID: \"5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qk2gf" Oct 30 01:21:23.943734 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.943705 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea59f9b0-3567-40e9-84a0-412e683a68c3-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9x9v9\" (UID: \"ea59f9b0-3567-40e9-84a0-412e683a68c3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9x9v9" Oct 30 01:21:23.960255 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:23.960226 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtfpl\" (UniqueName: \"kubernetes.io/projected/ea59f9b0-3567-40e9-84a0-412e683a68c3-kube-api-access-wtfpl\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9x9v9\" (UID: \"ea59f9b0-3567-40e9-84a0-412e683a68c3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9x9v9" Oct 30 01:21:24.016996 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:24.016966 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rl9jj" Oct 30 01:21:24.044798 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:24.044768 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qk2gf\" (UID: \"5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qk2gf" Oct 30 01:21:24.044899 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:24.044839 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qk2gf\" (UID: \"5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qk2gf" Oct 30 01:21:24.044899 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:24.044889 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78lh4\" (UniqueName: \"kubernetes.io/projected/5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1-kube-api-access-78lh4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qk2gf\" (UID: \"5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qk2gf" Oct 30 01:21:24.045133 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:24.045108 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qk2gf\" (UID: \"5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qk2gf" Oct 30 01:21:24.045179 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:24.045164 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qk2gf\" (UID: \"5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qk2gf" Oct 30 01:21:24.061936 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:24.061912 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78lh4\" (UniqueName: \"kubernetes.io/projected/5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1-kube-api-access-78lh4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qk2gf\" (UID: \"5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qk2gf" Oct 30 01:21:24.110020 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:24.109992 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9x9v9" Oct 30 01:21:24.214992 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:24.214761 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qk2gf" Oct 30 01:21:24.215140 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:24.215002 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c85qml"] Oct 30 01:21:24.218874 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:21:24.218852 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e39324_5f99_4c0e_9968_cb65e2c5eea2.slice/crio-65ccd7e80f592052a502f0fbde804aefe824be0fe01af7666c45f633ef3e6fee WatchSource:0}: Error finding container 65ccd7e80f592052a502f0fbde804aefe824be0fe01af7666c45f633ef3e6fee: Status 404 returned error can't find the container with id 65ccd7e80f592052a502f0fbde804aefe824be0fe01af7666c45f633ef3e6fee Oct 30 01:21:24.222338 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:24.222313 6572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:21:24.308789 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:24.308764 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rl9jj"] Oct 30 01:21:24.312013 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:21:24.311987 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod094b28a0_e057_4a0e_8583_089b04c76127.slice/crio-79a3f37f07dfd031bc418862fce94bef3f7a0e05c85723736e575a88107d4c4d WatchSource:0}: Error finding container 79a3f37f07dfd031bc418862fce94bef3f7a0e05c85723736e575a88107d4c4d: Status 404 returned error can't find the container with id 79a3f37f07dfd031bc418862fce94bef3f7a0e05c85723736e575a88107d4c4d Oct 30 01:21:24.351823 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:24.351790 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rl9jj" event={"ID":"094b28a0-e057-4a0e-8583-089b04c76127","Type":"ContainerStarted","Data":"79a3f37f07dfd031bc418862fce94bef3f7a0e05c85723736e575a88107d4c4d"} Oct 30 01:21:24.352898 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:24.352868 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c85qml" event={"ID":"07e39324-5f99-4c0e-9968-cb65e2c5eea2","Type":"ContainerStarted","Data":"65ccd7e80f592052a502f0fbde804aefe824be0fe01af7666c45f633ef3e6fee"} Oct 30 01:21:24.394063 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:24.394042 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9x9v9"] Oct 30 01:21:24.396340 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:21:24.396318 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea59f9b0_3567_40e9_84a0_412e683a68c3.slice/crio-7ef36320fa2c3c954f7c8c496ab2cf5e4d47829691c3e5cf67ff166c0d1f7ccf WatchSource:0}: Error finding container 7ef36320fa2c3c954f7c8c496ab2cf5e4d47829691c3e5cf67ff166c0d1f7ccf: Status 404 returned error can't find the container with id 7ef36320fa2c3c954f7c8c496ab2cf5e4d47829691c3e5cf67ff166c0d1f7ccf Oct 30 01:21:24.498849 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:24.498558 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qk2gf"] Oct 30 01:21:24.502255 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:21:24.502232 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f06535c_ebb4_4fc3_9479_eb2e3a55bbc1.slice/crio-20a062814a804cc8d009c0dd541ef9b77d2cf47db6e403ec0c5aaa2aa55ac370 WatchSource:0}: Error finding container 20a062814a804cc8d009c0dd541ef9b77d2cf47db6e403ec0c5aaa2aa55ac370: Status 404 returned error can't find the container with id 20a062814a804cc8d009c0dd541ef9b77d2cf47db6e403ec0c5aaa2aa55ac370 Oct 30 01:21:24.711211 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:24.710010 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f95lw"] Oct 30 01:21:24.714131 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:24.714106 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f95lw" Oct 30 01:21:24.727115 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:24.727089 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-jf7h2\"" Oct 30 01:21:24.731570 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:24.731547 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f95lw"] Oct 30 01:21:24.751340 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:24.751274 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d550adeb-0895-4f14-b00e-0d30b59b8824-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f95lw\" (UID: \"d550adeb-0895-4f14-b00e-0d30b59b8824\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f95lw" Oct 30 01:21:24.751451 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:24.751342 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rxvg\" (UniqueName: \"kubernetes.io/projected/d550adeb-0895-4f14-b00e-0d30b59b8824-kube-api-access-4rxvg\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f95lw\" (UID: \"d550adeb-0895-4f14-b00e-0d30b59b8824\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f95lw" Oct 30 01:21:24.751508 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:24.751473 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d550adeb-0895-4f14-b00e-0d30b59b8824-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f95lw\" (UID: \"d550adeb-0895-4f14-b00e-0d30b59b8824\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f95lw" Oct 30 01:21:24.852283 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:24.852237 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d550adeb-0895-4f14-b00e-0d30b59b8824-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f95lw\" (UID: \"d550adeb-0895-4f14-b00e-0d30b59b8824\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f95lw" Oct 30 01:21:24.852453 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:24.852313 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rxvg\" (UniqueName: \"kubernetes.io/projected/d550adeb-0895-4f14-b00e-0d30b59b8824-kube-api-access-4rxvg\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f95lw\" (UID: \"d550adeb-0895-4f14-b00e-0d30b59b8824\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f95lw" Oct 30 01:21:24.852453 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:24.852383 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d550adeb-0895-4f14-b00e-0d30b59b8824-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f95lw\" (UID: \"d550adeb-0895-4f14-b00e-0d30b59b8824\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f95lw" Oct 30 01:21:24.852839 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:24.852814 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d550adeb-0895-4f14-b00e-0d30b59b8824-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f95lw\" (UID: \"d550adeb-0895-4f14-b00e-0d30b59b8824\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f95lw" Oct 30 01:21:24.853071 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:24.853050 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d550adeb-0895-4f14-b00e-0d30b59b8824-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f95lw\" (UID: \"d550adeb-0895-4f14-b00e-0d30b59b8824\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f95lw" Oct 30 01:21:24.870728 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:24.870698 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rxvg\" (UniqueName: \"kubernetes.io/projected/d550adeb-0895-4f14-b00e-0d30b59b8824-kube-api-access-4rxvg\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f95lw\" (UID: \"d550adeb-0895-4f14-b00e-0d30b59b8824\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f95lw" Oct 30 01:21:25.034955 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:25.034871 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f95lw" Oct 30 01:21:25.358612 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:25.358528 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qk2gf" event={"ID":"5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1","Type":"ContainerStarted","Data":"20a062814a804cc8d009c0dd541ef9b77d2cf47db6e403ec0c5aaa2aa55ac370"} Oct 30 01:21:25.362262 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:25.361015 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f95lw"] Oct 30 01:21:25.362262 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:25.361151 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9x9v9" event={"ID":"ea59f9b0-3567-40e9-84a0-412e683a68c3","Type":"ContainerStarted","Data":"7ef36320fa2c3c954f7c8c496ab2cf5e4d47829691c3e5cf67ff166c0d1f7ccf"} Oct 30 01:21:25.367338 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:21:25.367309 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd550adeb_0895_4f14_b00e_0d30b59b8824.slice/crio-527c6005111f7c871bd6a59e0a0a579a950a0a2d020e0d5710d5d5316b8ffb0f WatchSource:0}: Error finding container 527c6005111f7c871bd6a59e0a0a579a950a0a2d020e0d5710d5d5316b8ffb0f: Status 404 returned error can't find the container with id 527c6005111f7c871bd6a59e0a0a579a950a0a2d020e0d5710d5d5316b8ffb0f Oct 30 01:21:26.141502 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:26.141467 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfz55t"] Oct 30 01:21:26.145511 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:26.145488 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfz55t" Oct 30 01:21:26.164842 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:26.164541 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0daee51-5c5b-4b93-8ac2-490ab8492d08-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfz55t\" (UID: \"b0daee51-5c5b-4b93-8ac2-490ab8492d08\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfz55t" Oct 30 01:21:26.164842 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:26.164589 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vprc\" (UniqueName: \"kubernetes.io/projected/b0daee51-5c5b-4b93-8ac2-490ab8492d08-kube-api-access-7vprc\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfz55t\" (UID: \"b0daee51-5c5b-4b93-8ac2-490ab8492d08\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfz55t" Oct 30 01:21:26.164842 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:26.164614 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0daee51-5c5b-4b93-8ac2-490ab8492d08-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfz55t\" (UID: \"b0daee51-5c5b-4b93-8ac2-490ab8492d08\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfz55t" Oct 30 01:21:26.169285 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:26.168224 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-4f68m\"" Oct 30 01:21:26.169285 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:26.169221 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfz55t"] Oct 30 01:21:26.266687 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:26.265731 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vprc\" (UniqueName: \"kubernetes.io/projected/b0daee51-5c5b-4b93-8ac2-490ab8492d08-kube-api-access-7vprc\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfz55t\" (UID: \"b0daee51-5c5b-4b93-8ac2-490ab8492d08\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfz55t" Oct 30 01:21:26.266687 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:26.265800 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0daee51-5c5b-4b93-8ac2-490ab8492d08-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfz55t\" (UID: \"b0daee51-5c5b-4b93-8ac2-490ab8492d08\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfz55t" Oct 30 01:21:26.266687 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:26.265879 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0daee51-5c5b-4b93-8ac2-490ab8492d08-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfz55t\" (UID: \"b0daee51-5c5b-4b93-8ac2-490ab8492d08\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfz55t" Oct 30 01:21:26.266687 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:26.266212 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0daee51-5c5b-4b93-8ac2-490ab8492d08-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfz55t\" (UID: \"b0daee51-5c5b-4b93-8ac2-490ab8492d08\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfz55t" Oct 30 01:21:26.266687 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:26.266646 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0daee51-5c5b-4b93-8ac2-490ab8492d08-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfz55t\" (UID: \"b0daee51-5c5b-4b93-8ac2-490ab8492d08\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfz55t" Oct 30 01:21:26.293380 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:26.293313 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vprc\" (UniqueName: \"kubernetes.io/projected/b0daee51-5c5b-4b93-8ac2-490ab8492d08-kube-api-access-7vprc\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfz55t\" (UID: \"b0daee51-5c5b-4b93-8ac2-490ab8492d08\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfz55t" Oct 30 01:21:26.371035 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:26.370866 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f95lw" event={"ID":"d550adeb-0895-4f14-b00e-0d30b59b8824","Type":"ContainerStarted","Data":"527c6005111f7c871bd6a59e0a0a579a950a0a2d020e0d5710d5d5316b8ffb0f"} Oct 30 01:21:26.465540 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:26.465269 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfz55t" Oct 30 01:21:26.791003 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:26.790148 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfz55t"] Oct 30 01:21:26.794142 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:21:26.794106 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0daee51_5c5b_4b93_8ac2_490ab8492d08.slice/crio-7d24b917d1bc74e2623dd70c245148c4944566a07de9a9481afd1e3812162eae WatchSource:0}: Error finding container 7d24b917d1bc74e2623dd70c245148c4944566a07de9a9481afd1e3812162eae: Status 404 returned error can't find the container with id 7d24b917d1bc74e2623dd70c245148c4944566a07de9a9481afd1e3812162eae Oct 30 01:21:27.073675 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:27.073592 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:27.277194 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:27.277081 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:27.379867 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:27.379661 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfz55t" event={"ID":"b0daee51-5c5b-4b93-8ac2-490ab8492d08","Type":"ContainerStarted","Data":"7d24b917d1bc74e2623dd70c245148c4944566a07de9a9481afd1e3812162eae"} Oct 30 01:21:27.825075 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:27.825019 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:28.020822 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:28.020715 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:29.005783 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:29.005578 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714qpdl"] Oct 30 01:21:29.011021 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:29.010440 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714qpdl" Oct 30 01:21:29.027690 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:29.026928 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-kz2qx\"" Oct 30 01:21:29.028995 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:29.028941 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714qpdl"] Oct 30 01:21:29.092739 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:29.092411 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjtsv\" (UniqueName: \"kubernetes.io/projected/c9456502-37f5-4f73-9403-9805f0a576a5-kube-api-access-mjtsv\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714qpdl\" (UID: \"c9456502-37f5-4f73-9403-9805f0a576a5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714qpdl" Oct 30 01:21:29.092739 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:29.092490 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c9456502-37f5-4f73-9403-9805f0a576a5-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714qpdl\" (UID: \"c9456502-37f5-4f73-9403-9805f0a576a5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714qpdl" Oct 30 01:21:29.092739 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:29.092555 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9456502-37f5-4f73-9403-9805f0a576a5-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714qpdl\" (UID: \"c9456502-37f5-4f73-9403-9805f0a576a5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714qpdl" Oct 30 01:21:29.193444 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:29.193406 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c9456502-37f5-4f73-9403-9805f0a576a5-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714qpdl\" (UID: \"c9456502-37f5-4f73-9403-9805f0a576a5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714qpdl" Oct 30 01:21:29.193596 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:29.193475 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9456502-37f5-4f73-9403-9805f0a576a5-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714qpdl\" (UID: \"c9456502-37f5-4f73-9403-9805f0a576a5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714qpdl" Oct 30 01:21:29.193596 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:29.193528 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjtsv\" (UniqueName: \"kubernetes.io/projected/c9456502-37f5-4f73-9403-9805f0a576a5-kube-api-access-mjtsv\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714qpdl\" (UID: \"c9456502-37f5-4f73-9403-9805f0a576a5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714qpdl" Oct 30 01:21:29.194279 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:29.194252 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c9456502-37f5-4f73-9403-9805f0a576a5-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714qpdl\" (UID: \"c9456502-37f5-4f73-9403-9805f0a576a5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714qpdl" Oct 30 01:21:29.194536 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:29.194516 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9456502-37f5-4f73-9403-9805f0a576a5-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714qpdl\" (UID: \"c9456502-37f5-4f73-9403-9805f0a576a5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714qpdl" Oct 30 01:21:29.216907 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:29.216879 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjtsv\" (UniqueName: \"kubernetes.io/projected/c9456502-37f5-4f73-9403-9805f0a576a5-kube-api-access-mjtsv\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714qpdl\" (UID: \"c9456502-37f5-4f73-9403-9805f0a576a5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714qpdl" Oct 30 01:21:29.333506 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:29.333427 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714qpdl" Oct 30 01:21:29.649720 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:29.649557 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714qpdl"] Oct 30 01:21:29.656582 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:21:29.656543 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9456502_37f5_4f73_9403_9805f0a576a5.slice/crio-b3007ecd5caea3238b7c3518ff97a07e549f5e3c21d1a9d4d4e789c4e4e17eb9 WatchSource:0}: Error finding container b3007ecd5caea3238b7c3518ff97a07e549f5e3c21d1a9d4d4e789c4e4e17eb9: Status 404 returned error can't find the container with id b3007ecd5caea3238b7c3518ff97a07e549f5e3c21d1a9d4d4e789c4e4e17eb9 Oct 30 01:21:30.404902 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:30.404845 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714qpdl" event={"ID":"c9456502-37f5-4f73-9403-9805f0a576a5","Type":"ContainerStarted","Data":"b3007ecd5caea3238b7c3518ff97a07e549f5e3c21d1a9d4d4e789c4e4e17eb9"} Oct 30 01:21:31.185929 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.185893 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahfpl9"] Oct 30 01:21:31.194010 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.189828 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahfpl9" Oct 30 01:21:31.214149 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.214072 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahfpl9"] Oct 30 01:21:31.223071 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.222832 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czft62"] Oct 30 01:21:31.235487 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.235418 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czft62" Oct 30 01:21:31.242913 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.242850 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czft62"] Oct 30 01:21:31.251613 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.251573 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nc8rz"] Oct 30 01:21:31.255555 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.255220 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nc8rz" Oct 30 01:21:31.281481 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.281390 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nc8rz"] Oct 30 01:21:31.316220 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.316187 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e94c6943-5c8a-497a-a1d5-83acd8232992-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czft62\" (UID: \"e94c6943-5c8a-497a-a1d5-83acd8232992\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czft62" Oct 30 01:21:31.316422 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.316261 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjhxw\" (UniqueName: \"kubernetes.io/projected/4cdf1dae-636b-4e50-b328-1f428316cb7f-kube-api-access-tjhxw\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nc8rz\" (UID: \"4cdf1dae-636b-4e50-b328-1f428316cb7f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nc8rz" Oct 30 01:21:31.316422 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.316299 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cdf1dae-636b-4e50-b328-1f428316cb7f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nc8rz\" (UID: \"4cdf1dae-636b-4e50-b328-1f428316cb7f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nc8rz" Oct 30 01:21:31.316422 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.316345 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ff0254c-bc05-4a84-b705-44d893282028-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahfpl9\" (UID: \"8ff0254c-bc05-4a84-b705-44d893282028\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahfpl9" Oct 30 01:21:31.316597 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.316428 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jgcs\" (UniqueName: \"kubernetes.io/projected/e94c6943-5c8a-497a-a1d5-83acd8232992-kube-api-access-2jgcs\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czft62\" (UID: \"e94c6943-5c8a-497a-a1d5-83acd8232992\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czft62" Oct 30 01:21:31.316597 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.316458 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ff0254c-bc05-4a84-b705-44d893282028-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahfpl9\" (UID: \"8ff0254c-bc05-4a84-b705-44d893282028\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahfpl9" Oct 30 01:21:31.316597 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.316569 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x7dj\" (UniqueName: \"kubernetes.io/projected/8ff0254c-bc05-4a84-b705-44d893282028-kube-api-access-8x7dj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahfpl9\" (UID: \"8ff0254c-bc05-4a84-b705-44d893282028\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahfpl9" Oct 30 01:21:31.316769 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.316627 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e94c6943-5c8a-497a-a1d5-83acd8232992-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czft62\" (UID: \"e94c6943-5c8a-497a-a1d5-83acd8232992\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czft62" Oct 30 01:21:31.316769 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.316675 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4cdf1dae-636b-4e50-b328-1f428316cb7f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nc8rz\" (UID: \"4cdf1dae-636b-4e50-b328-1f428316cb7f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nc8rz" Oct 30 01:21:31.323325 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.323297 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v9t5k"] Oct 30 01:21:31.323552 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.323529 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:31.326554 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.326535 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v9t5k" Oct 30 01:21:31.346851 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.346827 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v9t5k"] Oct 30 01:21:31.419422 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.418001 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e94c6943-5c8a-497a-a1d5-83acd8232992-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czft62\" (UID: \"e94c6943-5c8a-497a-a1d5-83acd8232992\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czft62" Oct 30 01:21:31.419422 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.418046 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjhxw\" (UniqueName: \"kubernetes.io/projected/4cdf1dae-636b-4e50-b328-1f428316cb7f-kube-api-access-tjhxw\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nc8rz\" (UID: \"4cdf1dae-636b-4e50-b328-1f428316cb7f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nc8rz" Oct 30 01:21:31.419422 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.418082 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cdf1dae-636b-4e50-b328-1f428316cb7f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nc8rz\" (UID: \"4cdf1dae-636b-4e50-b328-1f428316cb7f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nc8rz" Oct 30 01:21:31.419422 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.418113 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/627442d3-8a28-4c41-aeff-aff7ec981c87-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v9t5k\" (UID: \"627442d3-8a28-4c41-aeff-aff7ec981c87\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v9t5k" Oct 30 01:21:31.419422 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.418158 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ff0254c-bc05-4a84-b705-44d893282028-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahfpl9\" (UID: \"8ff0254c-bc05-4a84-b705-44d893282028\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahfpl9" Oct 30 01:21:31.419422 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.418199 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jgcs\" (UniqueName: \"kubernetes.io/projected/e94c6943-5c8a-497a-a1d5-83acd8232992-kube-api-access-2jgcs\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czft62\" (UID: \"e94c6943-5c8a-497a-a1d5-83acd8232992\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czft62" Oct 30 01:21:31.419422 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.418230 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ff0254c-bc05-4a84-b705-44d893282028-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahfpl9\" (UID: \"8ff0254c-bc05-4a84-b705-44d893282028\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahfpl9" Oct 30 01:21:31.419422 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.418257 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x7dj\" (UniqueName: \"kubernetes.io/projected/8ff0254c-bc05-4a84-b705-44d893282028-kube-api-access-8x7dj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahfpl9\" (UID: \"8ff0254c-bc05-4a84-b705-44d893282028\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahfpl9" Oct 30 01:21:31.419422 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.418301 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e94c6943-5c8a-497a-a1d5-83acd8232992-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czft62\" (UID: \"e94c6943-5c8a-497a-a1d5-83acd8232992\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czft62" Oct 30 01:21:31.419422 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.418325 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mmm5\" (UniqueName: \"kubernetes.io/projected/627442d3-8a28-4c41-aeff-aff7ec981c87-kube-api-access-4mmm5\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v9t5k\" (UID: \"627442d3-8a28-4c41-aeff-aff7ec981c87\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v9t5k" Oct 30 01:21:31.419422 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.418362 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4cdf1dae-636b-4e50-b328-1f428316cb7f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nc8rz\" (UID: \"4cdf1dae-636b-4e50-b328-1f428316cb7f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nc8rz" Oct 30 01:21:31.419422 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.418414 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/627442d3-8a28-4c41-aeff-aff7ec981c87-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v9t5k\" (UID: \"627442d3-8a28-4c41-aeff-aff7ec981c87\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v9t5k" Oct 30 01:21:31.422864 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.420901 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e94c6943-5c8a-497a-a1d5-83acd8232992-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czft62\" (UID: \"e94c6943-5c8a-497a-a1d5-83acd8232992\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czft62" Oct 30 01:21:31.422864 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.421202 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cdf1dae-636b-4e50-b328-1f428316cb7f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nc8rz\" (UID: \"4cdf1dae-636b-4e50-b328-1f428316cb7f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nc8rz" Oct 30 01:21:31.422864 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.422127 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4cdf1dae-636b-4e50-b328-1f428316cb7f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nc8rz\" (UID: \"4cdf1dae-636b-4e50-b328-1f428316cb7f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nc8rz" Oct 30 01:21:31.422864 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.422486 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ff0254c-bc05-4a84-b705-44d893282028-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahfpl9\" (UID: \"8ff0254c-bc05-4a84-b705-44d893282028\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahfpl9" Oct 30 01:21:31.422864 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.422523 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ff0254c-bc05-4a84-b705-44d893282028-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahfpl9\" (UID: \"8ff0254c-bc05-4a84-b705-44d893282028\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahfpl9" Oct 30 01:21:31.422864 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.422717 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e94c6943-5c8a-497a-a1d5-83acd8232992-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czft62\" (UID: \"e94c6943-5c8a-497a-a1d5-83acd8232992\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czft62" Oct 30 01:21:31.444743 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.444283 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjhxw\" (UniqueName: \"kubernetes.io/projected/4cdf1dae-636b-4e50-b328-1f428316cb7f-kube-api-access-tjhxw\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nc8rz\" (UID: \"4cdf1dae-636b-4e50-b328-1f428316cb7f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nc8rz" Oct 30 01:21:31.444743 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.444699 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jgcs\" (UniqueName: \"kubernetes.io/projected/e94c6943-5c8a-497a-a1d5-83acd8232992-kube-api-access-2jgcs\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czft62\" (UID: \"e94c6943-5c8a-497a-a1d5-83acd8232992\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czft62" Oct 30 01:21:31.446671 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.446599 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x7dj\" (UniqueName: \"kubernetes.io/projected/8ff0254c-bc05-4a84-b705-44d893282028-kube-api-access-8x7dj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahfpl9\" (UID: \"8ff0254c-bc05-4a84-b705-44d893282028\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahfpl9" Oct 30 01:21:31.511264 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.510991 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahfpl9" Oct 30 01:21:31.519781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.518923 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/627442d3-8a28-4c41-aeff-aff7ec981c87-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v9t5k\" (UID: \"627442d3-8a28-4c41-aeff-aff7ec981c87\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v9t5k" Oct 30 01:21:31.519781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.518979 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mmm5\" (UniqueName: \"kubernetes.io/projected/627442d3-8a28-4c41-aeff-aff7ec981c87-kube-api-access-4mmm5\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v9t5k\" (UID: \"627442d3-8a28-4c41-aeff-aff7ec981c87\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v9t5k" Oct 30 01:21:31.519781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.519023 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/627442d3-8a28-4c41-aeff-aff7ec981c87-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v9t5k\" (UID: \"627442d3-8a28-4c41-aeff-aff7ec981c87\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v9t5k" Oct 30 01:21:31.519781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.519302 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/627442d3-8a28-4c41-aeff-aff7ec981c87-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v9t5k\" (UID: \"627442d3-8a28-4c41-aeff-aff7ec981c87\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v9t5k" Oct 30 01:21:31.519781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.519475 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/627442d3-8a28-4c41-aeff-aff7ec981c87-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v9t5k\" (UID: \"627442d3-8a28-4c41-aeff-aff7ec981c87\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v9t5k" Oct 30 01:21:31.549124 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.549098 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mmm5\" (UniqueName: \"kubernetes.io/projected/627442d3-8a28-4c41-aeff-aff7ec981c87-kube-api-access-4mmm5\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v9t5k\" (UID: \"627442d3-8a28-4c41-aeff-aff7ec981c87\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v9t5k" Oct 30 01:21:31.562731 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.562703 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czft62" Oct 30 01:21:31.588781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.588717 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nc8rz" Oct 30 01:21:31.645220 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.645127 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v9t5k" Oct 30 01:21:31.876327 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.876213 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahfpl9"] Oct 30 01:21:31.882850 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:21:31.882818 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ff0254c_bc05_4a84_b705_44d893282028.slice/crio-fc3aff306e3e8ab294e4c8a28aa9601e17674865ed66347f9dbf97ae7c663d10 WatchSource:0}: Error finding container fc3aff306e3e8ab294e4c8a28aa9601e17674865ed66347f9dbf97ae7c663d10: Status 404 returned error can't find the container with id fc3aff306e3e8ab294e4c8a28aa9601e17674865ed66347f9dbf97ae7c663d10 Oct 30 01:21:31.907582 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.907547 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czft62"] Oct 30 01:21:31.918354 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:21:31.918322 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94c6943_5c8a_497a_a1d5_83acd8232992.slice/crio-5a54d3ea55de4a51b5402149e36904d1c5588af32e08775dfa81d40d02f48b90 WatchSource:0}: Error finding container 5a54d3ea55de4a51b5402149e36904d1c5588af32e08775dfa81d40d02f48b90: Status 404 returned error can't find the container with id 5a54d3ea55de4a51b5402149e36904d1c5588af32e08775dfa81d40d02f48b90 Oct 30 01:21:31.933500 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.933353 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nc8rz"] Oct 30 01:21:31.937456 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:21:31.937416 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cdf1dae_636b_4e50_b328_1f428316cb7f.slice/crio-b673ff80d6e0c8bd4cd5ad7640ea31b7c7c91b14df678f0429942bf1e6658204 WatchSource:0}: Error finding container b673ff80d6e0c8bd4cd5ad7640ea31b7c7c91b14df678f0429942bf1e6658204: Status 404 returned error can't find the container with id b673ff80d6e0c8bd4cd5ad7640ea31b7c7c91b14df678f0429942bf1e6658204 Oct 30 01:21:31.959388 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:31.959316 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v9t5k"] Oct 30 01:21:31.966587 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:21:31.966556 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod627442d3_8a28_4c41_aeff_aff7ec981c87.slice/crio-c2ca2363b8580882de7422d4f4c837b04a1be5feeaef5a2afe21c8e74d66013b WatchSource:0}: Error finding container c2ca2363b8580882de7422d4f4c837b04a1be5feeaef5a2afe21c8e74d66013b: Status 404 returned error can't find the container with id c2ca2363b8580882de7422d4f4c837b04a1be5feeaef5a2afe21c8e74d66013b Oct 30 01:21:32.440466 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:32.440395 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahfpl9" event={"ID":"8ff0254c-bc05-4a84-b705-44d893282028","Type":"ContainerStarted","Data":"fc3aff306e3e8ab294e4c8a28aa9601e17674865ed66347f9dbf97ae7c663d10"} Oct 30 01:21:32.445976 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:32.445888 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czft62" event={"ID":"e94c6943-5c8a-497a-a1d5-83acd8232992","Type":"ContainerStarted","Data":"5a54d3ea55de4a51b5402149e36904d1c5588af32e08775dfa81d40d02f48b90"} Oct 30 01:21:32.450786 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:32.450743 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nc8rz" event={"ID":"4cdf1dae-636b-4e50-b328-1f428316cb7f","Type":"ContainerStarted","Data":"b673ff80d6e0c8bd4cd5ad7640ea31b7c7c91b14df678f0429942bf1e6658204"} Oct 30 01:21:32.452611 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:32.452547 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v9t5k" event={"ID":"627442d3-8a28-4c41-aeff-aff7ec981c87","Type":"ContainerStarted","Data":"c2ca2363b8580882de7422d4f4c837b04a1be5feeaef5a2afe21c8e74d66013b"} Oct 30 01:21:32.861692 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:32.861559 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:33.044193 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:33.044144 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:37.108267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:37.108138 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:41.558839 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:41.558480 6572 generic.go:356] "Generic (PLEG): container finished" podID="094b28a0-e057-4a0e-8583-089b04c76127" containerID="e02856e7730fcdfe6f9690fd731c22efa97990d59f6bee1b89c8e79812999589" exitCode=0 Oct 30 01:21:41.558839 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:41.558600 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rl9jj" event={"ID":"094b28a0-e057-4a0e-8583-089b04c76127","Type":"ContainerDied","Data":"e02856e7730fcdfe6f9690fd731c22efa97990d59f6bee1b89c8e79812999589"} Oct 30 01:21:41.564092 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:41.563203 6572 generic.go:356] "Generic (PLEG): container finished" podID="07e39324-5f99-4c0e-9968-cb65e2c5eea2" containerID="eb85dd696967f1bce478bd5ee64b009b142926199bb42f59731f0f0888e97dc0" exitCode=0 Oct 30 01:21:41.564092 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:41.563272 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c85qml" event={"ID":"07e39324-5f99-4c0e-9968-cb65e2c5eea2","Type":"ContainerDied","Data":"eb85dd696967f1bce478bd5ee64b009b142926199bb42f59731f0f0888e97dc0"} Oct 30 01:21:42.573787 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:42.573685 6572 generic.go:356] "Generic (PLEG): container finished" podID="5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1" containerID="4b05a2db66abbc7136dea232c337a0f0ffd0d23a0810d08c540e496c0c1e7064" exitCode=0 Oct 30 01:21:42.574239 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:42.573819 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qk2gf" event={"ID":"5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1","Type":"ContainerDied","Data":"4b05a2db66abbc7136dea232c337a0f0ffd0d23a0810d08c540e496c0c1e7064"} Oct 30 01:21:42.580186 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:42.580142 6572 generic.go:356] "Generic (PLEG): container finished" podID="c9456502-37f5-4f73-9403-9805f0a576a5" containerID="4d98431be2bfdaa98e0ad13d21c23f9c1c3059e3219b98e1e60204f288b53e30" exitCode=0 Oct 30 01:21:42.580300 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:42.580242 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714qpdl" event={"ID":"c9456502-37f5-4f73-9403-9805f0a576a5","Type":"ContainerDied","Data":"4d98431be2bfdaa98e0ad13d21c23f9c1c3059e3219b98e1e60204f288b53e30"} Oct 30 01:21:42.586039 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:42.585920 6572 generic.go:356] "Generic (PLEG): container finished" podID="ea59f9b0-3567-40e9-84a0-412e683a68c3" containerID="8648df85ed8ab995985a6928334d4769e5d7640663c8892a11ef293490b86dde" exitCode=0 Oct 30 01:21:42.586039 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:42.585982 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9x9v9" event={"ID":"ea59f9b0-3567-40e9-84a0-412e683a68c3","Type":"ContainerDied","Data":"8648df85ed8ab995985a6928334d4769e5d7640663c8892a11ef293490b86dde"} Oct 30 01:21:43.592915 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:43.592883 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f95lw" event={"ID":"d550adeb-0895-4f14-b00e-0d30b59b8824","Type":"ContainerStarted","Data":"f3102cbb8400c41c4422d570f2fc447c1408e7e718bb9b713907d9720e1ab666"} Oct 30 01:21:43.595341 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:43.595310 6572 generic.go:356] "Generic (PLEG): container finished" podID="094b28a0-e057-4a0e-8583-089b04c76127" containerID="db300615f40f70643ff371f98a85f0c6a0d2e33be05e8a161568f5f193abbf2f" exitCode=0 Oct 30 01:21:43.595468 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:43.595388 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rl9jj" event={"ID":"094b28a0-e057-4a0e-8583-089b04c76127","Type":"ContainerDied","Data":"db300615f40f70643ff371f98a85f0c6a0d2e33be05e8a161568f5f193abbf2f"} Oct 30 01:21:43.599485 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:43.599457 6572 generic.go:356] "Generic (PLEG): container finished" podID="07e39324-5f99-4c0e-9968-cb65e2c5eea2" containerID="96711a4631cd8eb288b940fb65dcdc7db34836a6361c2cd610e2ce918f01585a" exitCode=0 Oct 30 01:21:43.599612 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:43.599546 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c85qml" event={"ID":"07e39324-5f99-4c0e-9968-cb65e2c5eea2","Type":"ContainerDied","Data":"96711a4631cd8eb288b940fb65dcdc7db34836a6361c2cd610e2ce918f01585a"} Oct 30 01:21:43.601453 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:43.601427 6572 generic.go:356] "Generic (PLEG): container finished" podID="e94c6943-5c8a-497a-a1d5-83acd8232992" containerID="8ba47d600b9af1a27b3c924a4a7b04673beddf407d91f9bd2288caa47f401eef" exitCode=0 Oct 30 01:21:43.601555 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:43.601518 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czft62" event={"ID":"e94c6943-5c8a-497a-a1d5-83acd8232992","Type":"ContainerDied","Data":"8ba47d600b9af1a27b3c924a4a7b04673beddf407d91f9bd2288caa47f401eef"} Oct 30 01:21:43.603946 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:43.603921 6572 generic.go:356] "Generic (PLEG): container finished" podID="4cdf1dae-636b-4e50-b328-1f428316cb7f" containerID="47084c0ff1090d152b9294d3c89cee41030f6b5979536f23a120fb76dc52ad31" exitCode=0 Oct 30 01:21:43.604051 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:43.604009 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nc8rz" event={"ID":"4cdf1dae-636b-4e50-b328-1f428316cb7f","Type":"ContainerDied","Data":"47084c0ff1090d152b9294d3c89cee41030f6b5979536f23a120fb76dc52ad31"} Oct 30 01:21:43.606803 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:43.606357 6572 generic.go:356] "Generic (PLEG): container finished" podID="627442d3-8a28-4c41-aeff-aff7ec981c87" containerID="0b33280bc567c83d21f865276a537ef4ff9e4d3c73d6f1f1e4e6bdd3b37c3cb0" exitCode=0 Oct 30 01:21:43.606803 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:43.606441 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v9t5k" event={"ID":"627442d3-8a28-4c41-aeff-aff7ec981c87","Type":"ContainerDied","Data":"0b33280bc567c83d21f865276a537ef4ff9e4d3c73d6f1f1e4e6bdd3b37c3cb0"} Oct 30 01:21:44.614390 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:44.614310 6572 generic.go:356] "Generic (PLEG): container finished" podID="ea59f9b0-3567-40e9-84a0-412e683a68c3" containerID="46aec4f3d247221af3461670d173e8cdefcd7be8944cc18d2b6a782ee09b276b" exitCode=0 Oct 30 01:21:44.614809 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:44.614617 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9x9v9" event={"ID":"ea59f9b0-3567-40e9-84a0-412e683a68c3","Type":"ContainerDied","Data":"46aec4f3d247221af3461670d173e8cdefcd7be8944cc18d2b6a782ee09b276b"} Oct 30 01:21:44.618623 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:44.618002 6572 generic.go:356] "Generic (PLEG): container finished" podID="e94c6943-5c8a-497a-a1d5-83acd8232992" containerID="c80663c6493586ddc54e26a6420febc0aa63fc0f938e5bc9257314ee7883fa62" exitCode=0 Oct 30 01:21:44.618623 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:44.618083 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czft62" event={"ID":"e94c6943-5c8a-497a-a1d5-83acd8232992","Type":"ContainerDied","Data":"c80663c6493586ddc54e26a6420febc0aa63fc0f938e5bc9257314ee7883fa62"} Oct 30 01:21:44.621487 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:44.620872 6572 generic.go:356] "Generic (PLEG): container finished" podID="4cdf1dae-636b-4e50-b328-1f428316cb7f" containerID="54d71a9aa9fcab78be940db9d5c3a849b647e7012a3dc9a98ea5fc46a72188d2" exitCode=0 Oct 30 01:21:44.621487 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:44.620975 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nc8rz" event={"ID":"4cdf1dae-636b-4e50-b328-1f428316cb7f","Type":"ContainerDied","Data":"54d71a9aa9fcab78be940db9d5c3a849b647e7012a3dc9a98ea5fc46a72188d2"} Oct 30 01:21:44.624771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:44.624726 6572 generic.go:356] "Generic (PLEG): container finished" podID="627442d3-8a28-4c41-aeff-aff7ec981c87" containerID="a7eec94ecd9aa32a28f7a581896e06c8e5751dbd7a61c374e77fcd30f7718a4c" exitCode=0 Oct 30 01:21:44.624918 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:44.624896 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v9t5k" event={"ID":"627442d3-8a28-4c41-aeff-aff7ec981c87","Type":"ContainerDied","Data":"a7eec94ecd9aa32a28f7a581896e06c8e5751dbd7a61c374e77fcd30f7718a4c"} Oct 30 01:21:44.626917 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:44.626892 6572 generic.go:356] "Generic (PLEG): container finished" podID="b0daee51-5c5b-4b93-8ac2-490ab8492d08" containerID="3ec601887363ebd46b88f26fcc40fb96a293eabd62e6c3c5b604a6d406f449a7" exitCode=0 Oct 30 01:21:44.627006 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:44.626988 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfz55t" event={"ID":"b0daee51-5c5b-4b93-8ac2-490ab8492d08","Type":"ContainerDied","Data":"3ec601887363ebd46b88f26fcc40fb96a293eabd62e6c3c5b604a6d406f449a7"} Oct 30 01:21:44.631063 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:44.631041 6572 generic.go:356] "Generic (PLEG): container finished" podID="5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1" containerID="bda7eea0229ec7aa570be7dcfebd8540b1499911478e8270eb3de45aaedead87" exitCode=0 Oct 30 01:21:44.631321 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:44.631303 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qk2gf" event={"ID":"5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1","Type":"ContainerDied","Data":"bda7eea0229ec7aa570be7dcfebd8540b1499911478e8270eb3de45aaedead87"} Oct 30 01:21:44.636533 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:44.636469 6572 generic.go:356] "Generic (PLEG): container finished" podID="d550adeb-0895-4f14-b00e-0d30b59b8824" containerID="f3102cbb8400c41c4422d570f2fc447c1408e7e718bb9b713907d9720e1ab666" exitCode=0 Oct 30 01:21:44.636613 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:44.636574 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f95lw" event={"ID":"d550adeb-0895-4f14-b00e-0d30b59b8824","Type":"ContainerDied","Data":"f3102cbb8400c41c4422d570f2fc447c1408e7e718bb9b713907d9720e1ab666"} Oct 30 01:21:44.640700 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:44.640678 6572 generic.go:356] "Generic (PLEG): container finished" podID="c9456502-37f5-4f73-9403-9805f0a576a5" containerID="188917cc505deb595ede479ddcbc51ecb73ed23760d1aef354455695375907cc" exitCode=0 Oct 30 01:21:44.640969 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:44.640937 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714qpdl" event={"ID":"c9456502-37f5-4f73-9403-9805f0a576a5","Type":"ContainerDied","Data":"188917cc505deb595ede479ddcbc51ecb73ed23760d1aef354455695375907cc"} Oct 30 01:21:44.643435 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:44.643409 6572 generic.go:356] "Generic (PLEG): container finished" podID="8ff0254c-bc05-4a84-b705-44d893282028" containerID="3a47acc7d2a2ea8491b95bf158311dbaacb5917aeb4a5a875d33fd7fed4faace" exitCode=0 Oct 30 01:21:44.643525 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:44.643456 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahfpl9" event={"ID":"8ff0254c-bc05-4a84-b705-44d893282028","Type":"ContainerDied","Data":"3a47acc7d2a2ea8491b95bf158311dbaacb5917aeb4a5a875d33fd7fed4faace"} Oct 30 01:21:44.702482 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:21:44.702454 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:21:45.664797 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:45.664671 6572 generic.go:356] "Generic (PLEG): container finished" podID="8ff0254c-bc05-4a84-b705-44d893282028" containerID="0b5e13173f0c6652816e906b645125918335b2b933ed5ba9083d07fc9fc65d5c" exitCode=0 Oct 30 01:21:45.665287 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:45.664868 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahfpl9" event={"ID":"8ff0254c-bc05-4a84-b705-44d893282028","Type":"ContainerDied","Data":"0b5e13173f0c6652816e906b645125918335b2b933ed5ba9083d07fc9fc65d5c"} Oct 30 01:21:51.751026 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:51.750986 6572 generic.go:356] "Generic (PLEG): container finished" podID="b0daee51-5c5b-4b93-8ac2-490ab8492d08" containerID="40558631a7f55f7704dd2b365919abe9625b30b0e3b2b16d79e17fa172c822dd" exitCode=0 Oct 30 01:21:51.751606 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:21:51.751098 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfz55t" event={"ID":"b0daee51-5c5b-4b93-8ac2-490ab8492d08","Type":"ContainerDied","Data":"40558631a7f55f7704dd2b365919abe9625b30b0e3b2b16d79e17fa172c822dd"} Oct 30 01:22:00.869799 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:00.869704 6572 generic.go:356] "Generic (PLEG): container finished" podID="d550adeb-0895-4f14-b00e-0d30b59b8824" containerID="16c74789805b690389f81a605d5b15f9e6b2ddfb5190048d8a93f4b489d9eadb" exitCode=0 Oct 30 01:22:00.869799 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:00.869767 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f95lw" event={"ID":"d550adeb-0895-4f14-b00e-0d30b59b8824","Type":"ContainerDied","Data":"16c74789805b690389f81a605d5b15f9e6b2ddfb5190048d8a93f4b489d9eadb"} Oct 30 01:22:02.189976 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:02.189765 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:03.916850 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:03.916811 6572 generic.go:356] "Generic (PLEG): container finished" podID="094b28a0-e057-4a0e-8583-089b04c76127" containerID="754a49c9cbe07befa70ca5f5eae7eb8bf4d21499414b82e086fedac0a6234ae8" exitCode=0 Oct 30 01:22:03.917324 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:03.916982 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rl9jj" event={"ID":"094b28a0-e057-4a0e-8583-089b04c76127","Type":"ContainerDied","Data":"754a49c9cbe07befa70ca5f5eae7eb8bf4d21499414b82e086fedac0a6234ae8"} Oct 30 01:22:04.947833 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:04.946181 6572 generic.go:356] "Generic (PLEG): container finished" podID="07e39324-5f99-4c0e-9968-cb65e2c5eea2" containerID="a1bdb5fcfd121353e5988d6d34fbdefb87c6ac29afb27061e18895620a5ebdfb" exitCode=0 Oct 30 01:22:04.947833 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:04.946969 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c85qml" event={"ID":"07e39324-5f99-4c0e-9968-cb65e2c5eea2","Type":"ContainerDied","Data":"a1bdb5fcfd121353e5988d6d34fbdefb87c6ac29afb27061e18895620a5ebdfb"} Oct 30 01:22:05.147905 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:05.147869 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rl9jj" Oct 30 01:22:05.281477 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:05.281442 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/094b28a0-e057-4a0e-8583-089b04c76127-bundle\" (UniqueName: \"kubernetes.io/empty-dir/094b28a0-e057-4a0e-8583-089b04c76127-bundle\") pod \"094b28a0-e057-4a0e-8583-089b04c76127\" (UID: \"094b28a0-e057-4a0e-8583-089b04c76127\") " Oct 30 01:22:05.281477 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:05.281477 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/094b28a0-e057-4a0e-8583-089b04c76127-util\" (UniqueName: \"kubernetes.io/empty-dir/094b28a0-e057-4a0e-8583-089b04c76127-util\") pod \"094b28a0-e057-4a0e-8583-089b04c76127\" (UID: \"094b28a0-e057-4a0e-8583-089b04c76127\") " Oct 30 01:22:05.281477 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:05.281508 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/094b28a0-e057-4a0e-8583-089b04c76127-kube-api-access-grxth\" (UniqueName: \"kubernetes.io/projected/094b28a0-e057-4a0e-8583-089b04c76127-kube-api-access-grxth\") pod \"094b28a0-e057-4a0e-8583-089b04c76127\" (UID: \"094b28a0-e057-4a0e-8583-089b04c76127\") " Oct 30 01:22:05.282553 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:05.282392 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094b28a0-e057-4a0e-8583-089b04c76127-bundle" pod "094b28a0-e057-4a0e-8583-089b04c76127" (UID: "094b28a0-e057-4a0e-8583-089b04c76127"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:05.296043 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:05.296005 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/094b28a0-e057-4a0e-8583-089b04c76127-kube-api-access-grxth" pod "094b28a0-e057-4a0e-8583-089b04c76127" (UID: "094b28a0-e057-4a0e-8583-089b04c76127"). InnerVolumeSpecName "kube-api-access-grxth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:05.300147 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:05.300088 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094b28a0-e057-4a0e-8583-089b04c76127-util" pod "094b28a0-e057-4a0e-8583-089b04c76127" (UID: "094b28a0-e057-4a0e-8583-089b04c76127"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:05.382789 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:05.382733 6572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/094b28a0-e057-4a0e-8583-089b04c76127-bundle\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:22:05.382789 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:05.382783 6572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/094b28a0-e057-4a0e-8583-089b04c76127-util\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:22:05.383038 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:05.382798 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grxth\" (UniqueName: \"kubernetes.io/projected/094b28a0-e057-4a0e-8583-089b04c76127-kube-api-access-grxth\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:22:05.962834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:05.962794 6572 generic.go:356] "Generic (PLEG): container finished" podID="b0daee51-5c5b-4b93-8ac2-490ab8492d08" containerID="a29388f4ad8aa8af6ed7aa653c6427f33a37496c7ed051925b3565acf9b36fb0" exitCode=0 Oct 30 01:22:05.963281 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:05.962957 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfz55t" event={"ID":"b0daee51-5c5b-4b93-8ac2-490ab8492d08","Type":"ContainerDied","Data":"a29388f4ad8aa8af6ed7aa653c6427f33a37496c7ed051925b3565acf9b36fb0"} Oct 30 01:22:05.966847 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:05.966816 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rl9jj" Oct 30 01:22:05.967958 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:05.967929 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7rl9jj" event={"ID":"094b28a0-e057-4a0e-8583-089b04c76127","Type":"ContainerDied","Data":"79a3f37f07dfd031bc418862fce94bef3f7a0e05c85723736e575a88107d4c4d"} Oct 30 01:22:05.968098 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:05.968001 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79a3f37f07dfd031bc418862fce94bef3f7a0e05c85723736e575a88107d4c4d" Oct 30 01:22:06.182040 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:06.181903 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c85qml" Oct 30 01:22:06.292160 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:06.290155 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07e39324-5f99-4c0e-9968-cb65e2c5eea2-util\" (UniqueName: \"kubernetes.io/empty-dir/07e39324-5f99-4c0e-9968-cb65e2c5eea2-util\") pod \"07e39324-5f99-4c0e-9968-cb65e2c5eea2\" (UID: \"07e39324-5f99-4c0e-9968-cb65e2c5eea2\") " Oct 30 01:22:06.292160 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:06.290212 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07e39324-5f99-4c0e-9968-cb65e2c5eea2-kube-api-access-thbqj\" (UniqueName: \"kubernetes.io/projected/07e39324-5f99-4c0e-9968-cb65e2c5eea2-kube-api-access-thbqj\") pod \"07e39324-5f99-4c0e-9968-cb65e2c5eea2\" (UID: \"07e39324-5f99-4c0e-9968-cb65e2c5eea2\") " Oct 30 01:22:06.292160 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:06.290439 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07e39324-5f99-4c0e-9968-cb65e2c5eea2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/07e39324-5f99-4c0e-9968-cb65e2c5eea2-bundle\") pod \"07e39324-5f99-4c0e-9968-cb65e2c5eea2\" (UID: \"07e39324-5f99-4c0e-9968-cb65e2c5eea2\") " Oct 30 01:22:06.297198 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:06.297126 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e39324-5f99-4c0e-9968-cb65e2c5eea2-bundle" pod "07e39324-5f99-4c0e-9968-cb65e2c5eea2" (UID: "07e39324-5f99-4c0e-9968-cb65e2c5eea2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:06.300990 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:06.300953 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e39324-5f99-4c0e-9968-cb65e2c5eea2-util" pod "07e39324-5f99-4c0e-9968-cb65e2c5eea2" (UID: "07e39324-5f99-4c0e-9968-cb65e2c5eea2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:06.304163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:06.304128 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e39324-5f99-4c0e-9968-cb65e2c5eea2-kube-api-access-thbqj" pod "07e39324-5f99-4c0e-9968-cb65e2c5eea2" (UID: "07e39324-5f99-4c0e-9968-cb65e2c5eea2"). InnerVolumeSpecName "kube-api-access-thbqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:06.391941 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:06.391854 6572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07e39324-5f99-4c0e-9968-cb65e2c5eea2-bundle\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:22:06.391941 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:06.391896 6572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07e39324-5f99-4c0e-9968-cb65e2c5eea2-util\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:22:06.391941 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:06.391910 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thbqj\" (UniqueName: \"kubernetes.io/projected/07e39324-5f99-4c0e-9968-cb65e2c5eea2-kube-api-access-thbqj\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:22:06.980583 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:06.980428 6572 generic.go:356] "Generic (PLEG): container finished" podID="5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1" containerID="2f30d0111c474b94c55efd0ae3737ddb211a89a9ab34807aef37bf6c9cd406d7" exitCode=0 Oct 30 01:22:06.980583 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:06.980513 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qk2gf" event={"ID":"5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1","Type":"ContainerDied","Data":"2f30d0111c474b94c55efd0ae3737ddb211a89a9ab34807aef37bf6c9cd406d7"} Oct 30 01:22:06.986830 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:06.986796 6572 generic.go:356] "Generic (PLEG): container finished" podID="c9456502-37f5-4f73-9403-9805f0a576a5" containerID="e8bdced7a846bf919baee9bc3d9db2be5baabcfb3d212f18cf9c4559de86c5fa" exitCode=0 Oct 30 01:22:06.986982 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:06.986920 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714qpdl" event={"ID":"c9456502-37f5-4f73-9403-9805f0a576a5","Type":"ContainerDied","Data":"e8bdced7a846bf919baee9bc3d9db2be5baabcfb3d212f18cf9c4559de86c5fa"} Oct 30 01:22:06.990727 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:06.990693 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c85qml" event={"ID":"07e39324-5f99-4c0e-9968-cb65e2c5eea2","Type":"ContainerDied","Data":"65ccd7e80f592052a502f0fbde804aefe824be0fe01af7666c45f633ef3e6fee"} Oct 30 01:22:06.990889 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:06.990774 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65ccd7e80f592052a502f0fbde804aefe824be0fe01af7666c45f633ef3e6fee" Oct 30 01:22:06.990889 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:06.990827 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c85qml" Oct 30 01:22:06.994807 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:06.994777 6572 generic.go:356] "Generic (PLEG): container finished" podID="ea59f9b0-3567-40e9-84a0-412e683a68c3" containerID="d449e5bee095fa1a8c09ce0ec8bbd5a703b90f125a5d59bf2563e7c2fbfb6f23" exitCode=0 Oct 30 01:22:06.994934 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:06.994866 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9x9v9" event={"ID":"ea59f9b0-3567-40e9-84a0-412e683a68c3","Type":"ContainerDied","Data":"d449e5bee095fa1a8c09ce0ec8bbd5a703b90f125a5d59bf2563e7c2fbfb6f23"} Oct 30 01:22:07.001082 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:06.999383 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czft62" event={"ID":"e94c6943-5c8a-497a-a1d5-83acd8232992","Type":"ContainerStarted","Data":"84a2f5d1a9d115d2736a37bb4c1ab2f9af012f4706ed1927db3a98fc58bb6708"} Oct 30 01:22:07.160050 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:07.159965 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czft62" podStartSLOduration=2.75562212 podStartE2EDuration="36.159951049s" totalImagesPullingTime="33.404328929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:07.15957562 +0000 UTC m=+252.736001020" watchObservedRunningTime="2026-10-30 01:22:07.159951049 +0000 UTC m=+252.736376445" Oct 30 01:22:07.243356 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:07.243335 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfz55t" Oct 30 01:22:07.403557 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:07.403525 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0daee51-5c5b-4b93-8ac2-490ab8492d08-util\" (UniqueName: \"kubernetes.io/empty-dir/b0daee51-5c5b-4b93-8ac2-490ab8492d08-util\") pod \"b0daee51-5c5b-4b93-8ac2-490ab8492d08\" (UID: \"b0daee51-5c5b-4b93-8ac2-490ab8492d08\") " Oct 30 01:22:07.403721 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:07.403568 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0daee51-5c5b-4b93-8ac2-490ab8492d08-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0daee51-5c5b-4b93-8ac2-490ab8492d08-bundle\") pod \"b0daee51-5c5b-4b93-8ac2-490ab8492d08\" (UID: \"b0daee51-5c5b-4b93-8ac2-490ab8492d08\") " Oct 30 01:22:07.403721 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:07.403604 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0daee51-5c5b-4b93-8ac2-490ab8492d08-kube-api-access-7vprc\" (UniqueName: \"kubernetes.io/projected/b0daee51-5c5b-4b93-8ac2-490ab8492d08-kube-api-access-7vprc\") pod \"b0daee51-5c5b-4b93-8ac2-490ab8492d08\" (UID: \"b0daee51-5c5b-4b93-8ac2-490ab8492d08\") " Oct 30 01:22:07.404273 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:07.404244 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0daee51-5c5b-4b93-8ac2-490ab8492d08-bundle" pod "b0daee51-5c5b-4b93-8ac2-490ab8492d08" (UID: "b0daee51-5c5b-4b93-8ac2-490ab8492d08"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:07.405788 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:07.405742 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0daee51-5c5b-4b93-8ac2-490ab8492d08-kube-api-access-7vprc" pod "b0daee51-5c5b-4b93-8ac2-490ab8492d08" (UID: "b0daee51-5c5b-4b93-8ac2-490ab8492d08"). InnerVolumeSpecName "kube-api-access-7vprc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:07.410012 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:07.409981 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0daee51-5c5b-4b93-8ac2-490ab8492d08-util" pod "b0daee51-5c5b-4b93-8ac2-490ab8492d08" (UID: "b0daee51-5c5b-4b93-8ac2-490ab8492d08"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:07.504742 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:07.504692 6572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0daee51-5c5b-4b93-8ac2-490ab8492d08-util\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:22:07.504742 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:07.504740 6572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0daee51-5c5b-4b93-8ac2-490ab8492d08-bundle\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:22:07.504742 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:07.504760 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vprc\" (UniqueName: \"kubernetes.io/projected/b0daee51-5c5b-4b93-8ac2-490ab8492d08-kube-api-access-7vprc\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:22:08.007562 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.007526 6572 generic.go:356] "Generic (PLEG): container finished" podID="e94c6943-5c8a-497a-a1d5-83acd8232992" containerID="84a2f5d1a9d115d2736a37bb4c1ab2f9af012f4706ed1927db3a98fc58bb6708" exitCode=0 Oct 30 01:22:08.008150 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.007613 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czft62" event={"ID":"e94c6943-5c8a-497a-a1d5-83acd8232992","Type":"ContainerDied","Data":"84a2f5d1a9d115d2736a37bb4c1ab2f9af012f4706ed1927db3a98fc58bb6708"} Oct 30 01:22:08.010339 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.010315 6572 generic.go:356] "Generic (PLEG): container finished" podID="4cdf1dae-636b-4e50-b328-1f428316cb7f" containerID="968bbe08e8e4061deda58a5940c34d380bbd0aba65eff0e0c57f75e2f07863f0" exitCode=0 Oct 30 01:22:08.010465 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.010408 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nc8rz" event={"ID":"4cdf1dae-636b-4e50-b328-1f428316cb7f","Type":"ContainerDied","Data":"968bbe08e8e4061deda58a5940c34d380bbd0aba65eff0e0c57f75e2f07863f0"} Oct 30 01:22:08.012862 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.012826 6572 generic.go:356] "Generic (PLEG): container finished" podID="627442d3-8a28-4c41-aeff-aff7ec981c87" containerID="bcafe49bf3f2998c57c2d5ce1350d7afc0893a528d8ebf5a6775dae51d37e290" exitCode=0 Oct 30 01:22:08.012983 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.012908 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v9t5k" event={"ID":"627442d3-8a28-4c41-aeff-aff7ec981c87","Type":"ContainerDied","Data":"bcafe49bf3f2998c57c2d5ce1350d7afc0893a528d8ebf5a6775dae51d37e290"} Oct 30 01:22:08.015025 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.015003 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfz55t" Oct 30 01:22:08.015131 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.015068 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfz55t" event={"ID":"b0daee51-5c5b-4b93-8ac2-490ab8492d08","Type":"ContainerDied","Data":"7d24b917d1bc74e2623dd70c245148c4944566a07de9a9481afd1e3812162eae"} Oct 30 01:22:08.015131 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.015103 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d24b917d1bc74e2623dd70c245148c4944566a07de9a9481afd1e3812162eae" Oct 30 01:22:08.017608 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.017528 6572 generic.go:356] "Generic (PLEG): container finished" podID="8ff0254c-bc05-4a84-b705-44d893282028" containerID="d419a3414c469cf65535c7534e13c0148c91df96d0f088cba767c1eaf4f469bf" exitCode=0 Oct 30 01:22:08.017724 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.017640 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahfpl9" event={"ID":"8ff0254c-bc05-4a84-b705-44d893282028","Type":"ContainerDied","Data":"d419a3414c469cf65535c7534e13c0148c91df96d0f088cba767c1eaf4f469bf"} Oct 30 01:22:08.176383 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.176095 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9x9v9" Oct 30 01:22:08.220217 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.220189 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714qpdl" Oct 30 01:22:08.224021 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.223998 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qk2gf" Oct 30 01:22:08.314938 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.314848 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea59f9b0-3567-40e9-84a0-412e683a68c3-kube-api-access-wtfpl\" (UniqueName: \"kubernetes.io/projected/ea59f9b0-3567-40e9-84a0-412e683a68c3-kube-api-access-wtfpl\") pod \"ea59f9b0-3567-40e9-84a0-412e683a68c3\" (UID: \"ea59f9b0-3567-40e9-84a0-412e683a68c3\") " Oct 30 01:22:08.314938 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.314919 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea59f9b0-3567-40e9-84a0-412e683a68c3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea59f9b0-3567-40e9-84a0-412e683a68c3-bundle\") pod \"ea59f9b0-3567-40e9-84a0-412e683a68c3\" (UID: \"ea59f9b0-3567-40e9-84a0-412e683a68c3\") " Oct 30 01:22:08.315163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.314980 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea59f9b0-3567-40e9-84a0-412e683a68c3-util\" (UniqueName: \"kubernetes.io/empty-dir/ea59f9b0-3567-40e9-84a0-412e683a68c3-util\") pod \"ea59f9b0-3567-40e9-84a0-412e683a68c3\" (UID: \"ea59f9b0-3567-40e9-84a0-412e683a68c3\") " Oct 30 01:22:08.315715 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.315692 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea59f9b0-3567-40e9-84a0-412e683a68c3-bundle" pod "ea59f9b0-3567-40e9-84a0-412e683a68c3" (UID: "ea59f9b0-3567-40e9-84a0-412e683a68c3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:08.317167 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.317140 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea59f9b0-3567-40e9-84a0-412e683a68c3-kube-api-access-wtfpl" pod "ea59f9b0-3567-40e9-84a0-412e683a68c3" (UID: "ea59f9b0-3567-40e9-84a0-412e683a68c3"). InnerVolumeSpecName "kube-api-access-wtfpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:08.322989 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.322956 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea59f9b0-3567-40e9-84a0-412e683a68c3-util" pod "ea59f9b0-3567-40e9-84a0-412e683a68c3" (UID: "ea59f9b0-3567-40e9-84a0-412e683a68c3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:08.416105 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.416062 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9456502-37f5-4f73-9403-9805f0a576a5-kube-api-access-mjtsv\" (UniqueName: \"kubernetes.io/projected/c9456502-37f5-4f73-9403-9805f0a576a5-kube-api-access-mjtsv\") pod \"c9456502-37f5-4f73-9403-9805f0a576a5\" (UID: \"c9456502-37f5-4f73-9403-9805f0a576a5\") " Oct 30 01:22:08.416299 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.416117 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9456502-37f5-4f73-9403-9805f0a576a5-util\" (UniqueName: \"kubernetes.io/empty-dir/c9456502-37f5-4f73-9403-9805f0a576a5-util\") pod \"c9456502-37f5-4f73-9403-9805f0a576a5\" (UID: \"c9456502-37f5-4f73-9403-9805f0a576a5\") " Oct 30 01:22:08.416299 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.416137 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1-util\" (UniqueName: \"kubernetes.io/empty-dir/5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1-util\") pod \"5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1\" (UID: \"5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1\") " Oct 30 01:22:08.416299 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.416228 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1-bundle\") pod \"5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1\" (UID: \"5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1\") " Oct 30 01:22:08.416299 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.416264 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1-kube-api-access-78lh4\" (UniqueName: \"kubernetes.io/projected/5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1-kube-api-access-78lh4\") pod \"5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1\" (UID: \"5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1\") " Oct 30 01:22:08.416299 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.416296 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9456502-37f5-4f73-9403-9805f0a576a5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9456502-37f5-4f73-9403-9805f0a576a5-bundle\") pod \"c9456502-37f5-4f73-9403-9805f0a576a5\" (UID: \"c9456502-37f5-4f73-9403-9805f0a576a5\") " Oct 30 01:22:08.416559 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.416489 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtfpl\" (UniqueName: \"kubernetes.io/projected/ea59f9b0-3567-40e9-84a0-412e683a68c3-kube-api-access-wtfpl\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:22:08.416559 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.416506 6572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea59f9b0-3567-40e9-84a0-412e683a68c3-bundle\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:22:08.416559 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.416520 6572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea59f9b0-3567-40e9-84a0-412e683a68c3-util\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:22:08.416891 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.416860 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1-bundle" pod "5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1" (UID: "5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:08.417043 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.417024 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9456502-37f5-4f73-9403-9805f0a576a5-bundle" pod "c9456502-37f5-4f73-9403-9805f0a576a5" (UID: "c9456502-37f5-4f73-9403-9805f0a576a5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:08.418619 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.418588 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9456502-37f5-4f73-9403-9805f0a576a5-kube-api-access-mjtsv" pod "c9456502-37f5-4f73-9403-9805f0a576a5" (UID: "c9456502-37f5-4f73-9403-9805f0a576a5"). InnerVolumeSpecName "kube-api-access-mjtsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:08.418853 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.418823 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1-kube-api-access-78lh4" pod "5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1" (UID: "5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1"). InnerVolumeSpecName "kube-api-access-78lh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:08.423547 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.423516 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9456502-37f5-4f73-9403-9805f0a576a5-util" pod "c9456502-37f5-4f73-9403-9805f0a576a5" (UID: "c9456502-37f5-4f73-9403-9805f0a576a5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:08.423765 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.423720 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1-util" pod "5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1" (UID: "5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:08.517506 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.517468 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjtsv\" (UniqueName: \"kubernetes.io/projected/c9456502-37f5-4f73-9403-9805f0a576a5-kube-api-access-mjtsv\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:22:08.517506 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.517500 6572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c9456502-37f5-4f73-9403-9805f0a576a5-util\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:22:08.517506 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.517511 6572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1-util\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:22:08.517764 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.517520 6572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1-bundle\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:22:08.517764 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.517529 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78lh4\" (UniqueName: \"kubernetes.io/projected/5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1-kube-api-access-78lh4\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:22:08.517764 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:08.517537 6572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9456502-37f5-4f73-9403-9805f0a576a5-bundle\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:22:09.023872 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.023832 6572 generic.go:356] "Generic (PLEG): container finished" podID="d550adeb-0895-4f14-b00e-0d30b59b8824" containerID="66dde01008a1f7e7928173b4fe5129db48e0fbb67db8121fdd8aca5c68373f4f" exitCode=0 Oct 30 01:22:09.024310 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.023918 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f95lw" event={"ID":"d550adeb-0895-4f14-b00e-0d30b59b8824","Type":"ContainerDied","Data":"66dde01008a1f7e7928173b4fe5129db48e0fbb67db8121fdd8aca5c68373f4f"} Oct 30 01:22:09.025859 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.025842 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714qpdl" Oct 30 01:22:09.025982 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.025841 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be714qpdl" event={"ID":"c9456502-37f5-4f73-9403-9805f0a576a5","Type":"ContainerDied","Data":"b3007ecd5caea3238b7c3518ff97a07e549f5e3c21d1a9d4d4e789c4e4e17eb9"} Oct 30 01:22:09.025982 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.025973 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3007ecd5caea3238b7c3518ff97a07e549f5e3c21d1a9d4d4e789c4e4e17eb9" Oct 30 01:22:09.028032 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.028005 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9x9v9" event={"ID":"ea59f9b0-3567-40e9-84a0-412e683a68c3","Type":"ContainerDied","Data":"7ef36320fa2c3c954f7c8c496ab2cf5e4d47829691c3e5cf67ff166c0d1f7ccf"} Oct 30 01:22:09.028130 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.028051 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ef36320fa2c3c954f7c8c496ab2cf5e4d47829691c3e5cf67ff166c0d1f7ccf" Oct 30 01:22:09.028130 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.028022 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9x9v9" Oct 30 01:22:09.030202 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.030178 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qk2gf" event={"ID":"5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1","Type":"ContainerDied","Data":"20a062814a804cc8d009c0dd541ef9b77d2cf47db6e403ec0c5aaa2aa55ac370"} Oct 30 01:22:09.030285 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.030218 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20a062814a804cc8d009c0dd541ef9b77d2cf47db6e403ec0c5aaa2aa55ac370" Oct 30 01:22:09.030367 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.030351 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qk2gf" Oct 30 01:22:09.174621 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.174594 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahfpl9" Oct 30 01:22:09.222003 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.221972 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ff0254c-bc05-4a84-b705-44d893282028-kube-api-access-8x7dj\" (UniqueName: \"kubernetes.io/projected/8ff0254c-bc05-4a84-b705-44d893282028-kube-api-access-8x7dj\") pod \"8ff0254c-bc05-4a84-b705-44d893282028\" (UID: \"8ff0254c-bc05-4a84-b705-44d893282028\") " Oct 30 01:22:09.222122 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.222034 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ff0254c-bc05-4a84-b705-44d893282028-util\" (UniqueName: \"kubernetes.io/empty-dir/8ff0254c-bc05-4a84-b705-44d893282028-util\") pod \"8ff0254c-bc05-4a84-b705-44d893282028\" (UID: \"8ff0254c-bc05-4a84-b705-44d893282028\") " Oct 30 01:22:09.222164 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.222118 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ff0254c-bc05-4a84-b705-44d893282028-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ff0254c-bc05-4a84-b705-44d893282028-bundle\") pod \"8ff0254c-bc05-4a84-b705-44d893282028\" (UID: \"8ff0254c-bc05-4a84-b705-44d893282028\") " Oct 30 01:22:09.224035 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.223999 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ff0254c-bc05-4a84-b705-44d893282028-kube-api-access-8x7dj" pod "8ff0254c-bc05-4a84-b705-44d893282028" (UID: "8ff0254c-bc05-4a84-b705-44d893282028"). InnerVolumeSpecName "kube-api-access-8x7dj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:09.224942 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.224913 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ff0254c-bc05-4a84-b705-44d893282028-bundle" pod "8ff0254c-bc05-4a84-b705-44d893282028" (UID: "8ff0254c-bc05-4a84-b705-44d893282028"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:09.227615 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.227591 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czft62" Oct 30 01:22:09.228534 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.228504 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ff0254c-bc05-4a84-b705-44d893282028-util" pod "8ff0254c-bc05-4a84-b705-44d893282028" (UID: "8ff0254c-bc05-4a84-b705-44d893282028"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:09.233314 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.233297 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nc8rz" Oct 30 01:22:09.236402 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.236381 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v9t5k" Oct 30 01:22:09.322667 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.322573 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cdf1dae-636b-4e50-b328-1f428316cb7f-util\" (UniqueName: \"kubernetes.io/empty-dir/4cdf1dae-636b-4e50-b328-1f428316cb7f-util\") pod \"4cdf1dae-636b-4e50-b328-1f428316cb7f\" (UID: \"4cdf1dae-636b-4e50-b328-1f428316cb7f\") " Oct 30 01:22:09.322667 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.322615 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/627442d3-8a28-4c41-aeff-aff7ec981c87-kube-api-access-4mmm5\" (UniqueName: \"kubernetes.io/projected/627442d3-8a28-4c41-aeff-aff7ec981c87-kube-api-access-4mmm5\") pod \"627442d3-8a28-4c41-aeff-aff7ec981c87\" (UID: \"627442d3-8a28-4c41-aeff-aff7ec981c87\") " Oct 30 01:22:09.322667 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.322658 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e94c6943-5c8a-497a-a1d5-83acd8232992-util\" (UniqueName: \"kubernetes.io/empty-dir/e94c6943-5c8a-497a-a1d5-83acd8232992-util\") pod \"e94c6943-5c8a-497a-a1d5-83acd8232992\" (UID: \"e94c6943-5c8a-497a-a1d5-83acd8232992\") " Oct 30 01:22:09.322971 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.322705 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e94c6943-5c8a-497a-a1d5-83acd8232992-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e94c6943-5c8a-497a-a1d5-83acd8232992-bundle\") pod \"e94c6943-5c8a-497a-a1d5-83acd8232992\" (UID: \"e94c6943-5c8a-497a-a1d5-83acd8232992\") " Oct 30 01:22:09.322971 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.322734 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/627442d3-8a28-4c41-aeff-aff7ec981c87-bundle\" (UniqueName: \"kubernetes.io/empty-dir/627442d3-8a28-4c41-aeff-aff7ec981c87-bundle\") pod \"627442d3-8a28-4c41-aeff-aff7ec981c87\" (UID: \"627442d3-8a28-4c41-aeff-aff7ec981c87\") " Oct 30 01:22:09.322971 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.322869 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/627442d3-8a28-4c41-aeff-aff7ec981c87-util\" (UniqueName: \"kubernetes.io/empty-dir/627442d3-8a28-4c41-aeff-aff7ec981c87-util\") pod \"627442d3-8a28-4c41-aeff-aff7ec981c87\" (UID: \"627442d3-8a28-4c41-aeff-aff7ec981c87\") " Oct 30 01:22:09.322971 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.322923 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cdf1dae-636b-4e50-b328-1f428316cb7f-kube-api-access-tjhxw\" (UniqueName: \"kubernetes.io/projected/4cdf1dae-636b-4e50-b328-1f428316cb7f-kube-api-access-tjhxw\") pod \"4cdf1dae-636b-4e50-b328-1f428316cb7f\" (UID: \"4cdf1dae-636b-4e50-b328-1f428316cb7f\") " Oct 30 01:22:09.323184 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.322972 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cdf1dae-636b-4e50-b328-1f428316cb7f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cdf1dae-636b-4e50-b328-1f428316cb7f-bundle\") pod \"4cdf1dae-636b-4e50-b328-1f428316cb7f\" (UID: \"4cdf1dae-636b-4e50-b328-1f428316cb7f\") " Oct 30 01:22:09.323184 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.323021 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e94c6943-5c8a-497a-a1d5-83acd8232992-kube-api-access-2jgcs\" (UniqueName: \"kubernetes.io/projected/e94c6943-5c8a-497a-a1d5-83acd8232992-kube-api-access-2jgcs\") pod \"e94c6943-5c8a-497a-a1d5-83acd8232992\" (UID: \"e94c6943-5c8a-497a-a1d5-83acd8232992\") " Oct 30 01:22:09.323288 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.323262 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/627442d3-8a28-4c41-aeff-aff7ec981c87-bundle" pod "627442d3-8a28-4c41-aeff-aff7ec981c87" (UID: "627442d3-8a28-4c41-aeff-aff7ec981c87"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:09.323288 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.323275 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x7dj\" (UniqueName: \"kubernetes.io/projected/8ff0254c-bc05-4a84-b705-44d893282028-kube-api-access-8x7dj\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:22:09.323396 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.323292 6572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ff0254c-bc05-4a84-b705-44d893282028-util\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:22:09.323396 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.323308 6572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ff0254c-bc05-4a84-b705-44d893282028-bundle\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:22:09.323396 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.323363 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e94c6943-5c8a-497a-a1d5-83acd8232992-bundle" pod "e94c6943-5c8a-497a-a1d5-83acd8232992" (UID: "e94c6943-5c8a-497a-a1d5-83acd8232992"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:09.323878 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.323853 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cdf1dae-636b-4e50-b328-1f428316cb7f-bundle" pod "4cdf1dae-636b-4e50-b328-1f428316cb7f" (UID: "4cdf1dae-636b-4e50-b328-1f428316cb7f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:09.325544 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.325522 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e94c6943-5c8a-497a-a1d5-83acd8232992-kube-api-access-2jgcs" pod "e94c6943-5c8a-497a-a1d5-83acd8232992" (UID: "e94c6943-5c8a-497a-a1d5-83acd8232992"). InnerVolumeSpecName "kube-api-access-2jgcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:09.325544 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.325530 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/627442d3-8a28-4c41-aeff-aff7ec981c87-kube-api-access-4mmm5" pod "627442d3-8a28-4c41-aeff-aff7ec981c87" (UID: "627442d3-8a28-4c41-aeff-aff7ec981c87"). InnerVolumeSpecName "kube-api-access-4mmm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:09.325715 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.325697 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cdf1dae-636b-4e50-b328-1f428316cb7f-kube-api-access-tjhxw" pod "4cdf1dae-636b-4e50-b328-1f428316cb7f" (UID: "4cdf1dae-636b-4e50-b328-1f428316cb7f"). InnerVolumeSpecName "kube-api-access-tjhxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:09.328822 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.328801 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cdf1dae-636b-4e50-b328-1f428316cb7f-util" pod "4cdf1dae-636b-4e50-b328-1f428316cb7f" (UID: "4cdf1dae-636b-4e50-b328-1f428316cb7f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:09.331940 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.331910 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/627442d3-8a28-4c41-aeff-aff7ec981c87-util" pod "627442d3-8a28-4c41-aeff-aff7ec981c87" (UID: "627442d3-8a28-4c41-aeff-aff7ec981c87"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:09.332164 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.332142 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e94c6943-5c8a-497a-a1d5-83acd8232992-util" pod "e94c6943-5c8a-497a-a1d5-83acd8232992" (UID: "e94c6943-5c8a-497a-a1d5-83acd8232992"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:09.424586 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.424531 6572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4cdf1dae-636b-4e50-b328-1f428316cb7f-util\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:22:09.424586 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.424578 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mmm5\" (UniqueName: \"kubernetes.io/projected/627442d3-8a28-4c41-aeff-aff7ec981c87-kube-api-access-4mmm5\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:22:09.424586 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.424588 6572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e94c6943-5c8a-497a-a1d5-83acd8232992-util\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:22:09.424586 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.424597 6572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e94c6943-5c8a-497a-a1d5-83acd8232992-bundle\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:22:09.424586 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.424607 6572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/627442d3-8a28-4c41-aeff-aff7ec981c87-bundle\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:22:09.424956 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.424615 6572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/627442d3-8a28-4c41-aeff-aff7ec981c87-util\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:22:09.424956 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.424623 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjhxw\" (UniqueName: \"kubernetes.io/projected/4cdf1dae-636b-4e50-b328-1f428316cb7f-kube-api-access-tjhxw\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:22:09.424956 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.424634 6572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4cdf1dae-636b-4e50-b328-1f428316cb7f-bundle\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:22:09.424956 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:09.424649 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jgcs\" (UniqueName: \"kubernetes.io/projected/e94c6943-5c8a-497a-a1d5-83acd8232992-kube-api-access-2jgcs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:22:10.036830 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:10.036798 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v9t5k" Oct 30 01:22:10.036830 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:10.036817 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v9t5k" event={"ID":"627442d3-8a28-4c41-aeff-aff7ec981c87","Type":"ContainerDied","Data":"c2ca2363b8580882de7422d4f4c837b04a1be5feeaef5a2afe21c8e74d66013b"} Oct 30 01:22:10.037329 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:10.036879 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2ca2363b8580882de7422d4f4c837b04a1be5feeaef5a2afe21c8e74d66013b" Oct 30 01:22:10.038938 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:10.038913 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahfpl9" Oct 30 01:22:10.038938 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:10.038923 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahfpl9" event={"ID":"8ff0254c-bc05-4a84-b705-44d893282028","Type":"ContainerDied","Data":"fc3aff306e3e8ab294e4c8a28aa9601e17674865ed66347f9dbf97ae7c663d10"} Oct 30 01:22:10.039112 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:10.038966 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc3aff306e3e8ab294e4c8a28aa9601e17674865ed66347f9dbf97ae7c663d10" Oct 30 01:22:10.040953 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:10.040932 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czft62" Oct 30 01:22:10.041053 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:10.040937 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czft62" event={"ID":"e94c6943-5c8a-497a-a1d5-83acd8232992","Type":"ContainerDied","Data":"5a54d3ea55de4a51b5402149e36904d1c5588af32e08775dfa81d40d02f48b90"} Oct 30 01:22:10.041116 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:10.041059 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a54d3ea55de4a51b5402149e36904d1c5588af32e08775dfa81d40d02f48b90" Oct 30 01:22:10.043154 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:10.043131 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nc8rz" event={"ID":"4cdf1dae-636b-4e50-b328-1f428316cb7f","Type":"ContainerDied","Data":"b673ff80d6e0c8bd4cd5ad7640ea31b7c7c91b14df678f0429942bf1e6658204"} Oct 30 01:22:10.043245 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:10.043173 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b673ff80d6e0c8bd4cd5ad7640ea31b7c7c91b14df678f0429942bf1e6658204" Oct 30 01:22:10.043245 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:10.043179 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nc8rz" Oct 30 01:22:10.172166 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:10.172132 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f95lw" Oct 30 01:22:10.230890 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:10.230850 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d550adeb-0895-4f14-b00e-0d30b59b8824-util\" (UniqueName: \"kubernetes.io/empty-dir/d550adeb-0895-4f14-b00e-0d30b59b8824-util\") pod \"d550adeb-0895-4f14-b00e-0d30b59b8824\" (UID: \"d550adeb-0895-4f14-b00e-0d30b59b8824\") " Oct 30 01:22:10.231090 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:10.230916 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d550adeb-0895-4f14-b00e-0d30b59b8824-kube-api-access-4rxvg\" (UniqueName: \"kubernetes.io/projected/d550adeb-0895-4f14-b00e-0d30b59b8824-kube-api-access-4rxvg\") pod \"d550adeb-0895-4f14-b00e-0d30b59b8824\" (UID: \"d550adeb-0895-4f14-b00e-0d30b59b8824\") " Oct 30 01:22:10.231090 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:10.231020 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d550adeb-0895-4f14-b00e-0d30b59b8824-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d550adeb-0895-4f14-b00e-0d30b59b8824-bundle\") pod \"d550adeb-0895-4f14-b00e-0d30b59b8824\" (UID: \"d550adeb-0895-4f14-b00e-0d30b59b8824\") " Oct 30 01:22:10.231390 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:10.231361 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d550adeb-0895-4f14-b00e-0d30b59b8824-bundle" pod "d550adeb-0895-4f14-b00e-0d30b59b8824" (UID: "d550adeb-0895-4f14-b00e-0d30b59b8824"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:10.233170 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:10.233138 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d550adeb-0895-4f14-b00e-0d30b59b8824-kube-api-access-4rxvg" pod "d550adeb-0895-4f14-b00e-0d30b59b8824" (UID: "d550adeb-0895-4f14-b00e-0d30b59b8824"). InnerVolumeSpecName "kube-api-access-4rxvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:10.237096 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:10.237064 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d550adeb-0895-4f14-b00e-0d30b59b8824-util" pod "d550adeb-0895-4f14-b00e-0d30b59b8824" (UID: "d550adeb-0895-4f14-b00e-0d30b59b8824"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:22:10.332201 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:10.332112 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rxvg\" (UniqueName: \"kubernetes.io/projected/d550adeb-0895-4f14-b00e-0d30b59b8824-kube-api-access-4rxvg\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:22:10.332201 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:10.332143 6572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d550adeb-0895-4f14-b00e-0d30b59b8824-bundle\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:22:10.332201 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:10.332156 6572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d550adeb-0895-4f14-b00e-0d30b59b8824-util\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:22:10.953693 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:10.953656 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:11.049103 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.049070 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f95lw" event={"ID":"d550adeb-0895-4f14-b00e-0d30b59b8824","Type":"ContainerDied","Data":"527c6005111f7c871bd6a59e0a0a579a950a0a2d020e0d5710d5d5316b8ffb0f"} Oct 30 01:22:11.049483 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.049114 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f95lw" Oct 30 01:22:11.049483 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.049119 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="527c6005111f7c871bd6a59e0a0a579a950a0a2d020e0d5710d5d5316b8ffb0f" Oct 30 01:22:11.150143 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.150114 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:11.650983 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.650951 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:11.710641 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.710605 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-bnhrd"] Oct 30 01:22:11.710936 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.710924 6572 cpu_manager.go:414] "containerMap: removing container" podUID="e94c6943-5c8a-497a-a1d5-83acd8232992" containerName="pull" Oct 30 01:22:11.710977 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.710939 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e94c6943-5c8a-497a-a1d5-83acd8232992" containerName="pull" Oct 30 01:22:11.710977 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.710951 6572 cpu_manager.go:414] "containerMap: removing container" podUID="094b28a0-e057-4a0e-8583-089b04c76127" containerName="util" Oct 30 01:22:11.710977 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.710956 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="094b28a0-e057-4a0e-8583-089b04c76127" containerName="util" Oct 30 01:22:11.710977 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.710964 6572 cpu_manager.go:414] "containerMap: removing container" podUID="094b28a0-e057-4a0e-8583-089b04c76127" containerName="pull" Oct 30 01:22:11.710977 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.710970 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="094b28a0-e057-4a0e-8583-089b04c76127" containerName="pull" Oct 30 01:22:11.710977 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.710976 6572 cpu_manager.go:414] "containerMap: removing container" podUID="8ff0254c-bc05-4a84-b705-44d893282028" containerName="pull" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.710981 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ff0254c-bc05-4a84-b705-44d893282028" containerName="pull" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.710990 6572 cpu_manager.go:414] "containerMap: removing container" podUID="ea59f9b0-3567-40e9-84a0-412e683a68c3" containerName="util" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.710995 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea59f9b0-3567-40e9-84a0-412e683a68c3" containerName="util" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711003 6572 cpu_manager.go:414] "containerMap: removing container" podUID="ea59f9b0-3567-40e9-84a0-412e683a68c3" containerName="pull" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711007 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea59f9b0-3567-40e9-84a0-412e683a68c3" containerName="pull" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711015 6572 cpu_manager.go:414] "containerMap: removing container" podUID="c9456502-37f5-4f73-9403-9805f0a576a5" containerName="extract" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711019 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9456502-37f5-4f73-9403-9805f0a576a5" containerName="extract" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711025 6572 cpu_manager.go:414] "containerMap: removing container" podUID="5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1" containerName="extract" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711030 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1" containerName="extract" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711036 6572 cpu_manager.go:414] "containerMap: removing container" podUID="e94c6943-5c8a-497a-a1d5-83acd8232992" containerName="extract" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711040 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e94c6943-5c8a-497a-a1d5-83acd8232992" containerName="extract" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711045 6572 cpu_manager.go:414] "containerMap: removing container" podUID="8ff0254c-bc05-4a84-b705-44d893282028" containerName="extract" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711050 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ff0254c-bc05-4a84-b705-44d893282028" containerName="extract" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711057 6572 cpu_manager.go:414] "containerMap: removing container" podUID="5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1" containerName="util" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711061 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1" containerName="util" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711071 6572 cpu_manager.go:414] "containerMap: removing container" podUID="627442d3-8a28-4c41-aeff-aff7ec981c87" containerName="pull" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711075 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="627442d3-8a28-4c41-aeff-aff7ec981c87" containerName="pull" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711081 6572 cpu_manager.go:414] "containerMap: removing container" podUID="4cdf1dae-636b-4e50-b328-1f428316cb7f" containerName="extract" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711085 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cdf1dae-636b-4e50-b328-1f428316cb7f" containerName="extract" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711092 6572 cpu_manager.go:414] "containerMap: removing container" podUID="8ff0254c-bc05-4a84-b705-44d893282028" containerName="util" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711096 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ff0254c-bc05-4a84-b705-44d893282028" containerName="util" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711102 6572 cpu_manager.go:414] "containerMap: removing container" podUID="d550adeb-0895-4f14-b00e-0d30b59b8824" containerName="extract" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711106 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d550adeb-0895-4f14-b00e-0d30b59b8824" containerName="extract" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711116 6572 cpu_manager.go:414] "containerMap: removing container" podUID="627442d3-8a28-4c41-aeff-aff7ec981c87" containerName="util" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711120 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="627442d3-8a28-4c41-aeff-aff7ec981c87" containerName="util" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711128 6572 cpu_manager.go:414] "containerMap: removing container" podUID="4cdf1dae-636b-4e50-b328-1f428316cb7f" containerName="util" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711132 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cdf1dae-636b-4e50-b328-1f428316cb7f" containerName="util" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711138 6572 cpu_manager.go:414] "containerMap: removing container" podUID="c9456502-37f5-4f73-9403-9805f0a576a5" containerName="pull" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711142 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9456502-37f5-4f73-9403-9805f0a576a5" containerName="pull" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711148 6572 cpu_manager.go:414] "containerMap: removing container" podUID="e94c6943-5c8a-497a-a1d5-83acd8232992" containerName="util" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711152 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e94c6943-5c8a-497a-a1d5-83acd8232992" containerName="util" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711160 6572 cpu_manager.go:414] "containerMap: removing container" podUID="d550adeb-0895-4f14-b00e-0d30b59b8824" containerName="pull" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711164 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d550adeb-0895-4f14-b00e-0d30b59b8824" containerName="pull" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711169 6572 cpu_manager.go:414] "containerMap: removing container" podUID="ea59f9b0-3567-40e9-84a0-412e683a68c3" containerName="extract" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711174 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea59f9b0-3567-40e9-84a0-412e683a68c3" containerName="extract" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711182 6572 cpu_manager.go:414] "containerMap: removing container" podUID="4cdf1dae-636b-4e50-b328-1f428316cb7f" containerName="pull" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711186 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cdf1dae-636b-4e50-b328-1f428316cb7f" containerName="pull" Oct 30 01:22:11.711176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711192 6572 cpu_manager.go:414] "containerMap: removing container" podUID="b0daee51-5c5b-4b93-8ac2-490ab8492d08" containerName="pull" Oct 30 01:22:11.712267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711196 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0daee51-5c5b-4b93-8ac2-490ab8492d08" containerName="pull" Oct 30 01:22:11.712267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711202 6572 cpu_manager.go:414] "containerMap: removing container" podUID="07e39324-5f99-4c0e-9968-cb65e2c5eea2" containerName="extract" Oct 30 01:22:11.712267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711206 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07e39324-5f99-4c0e-9968-cb65e2c5eea2" containerName="extract" Oct 30 01:22:11.712267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711212 6572 cpu_manager.go:414] "containerMap: removing container" podUID="07e39324-5f99-4c0e-9968-cb65e2c5eea2" containerName="util" Oct 30 01:22:11.712267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711216 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07e39324-5f99-4c0e-9968-cb65e2c5eea2" containerName="util" Oct 30 01:22:11.712267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711221 6572 cpu_manager.go:414] "containerMap: removing container" podUID="627442d3-8a28-4c41-aeff-aff7ec981c87" containerName="extract" Oct 30 01:22:11.712267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711226 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="627442d3-8a28-4c41-aeff-aff7ec981c87" containerName="extract" Oct 30 01:22:11.712267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711235 6572 cpu_manager.go:414] "containerMap: removing container" podUID="c9456502-37f5-4f73-9403-9805f0a576a5" containerName="util" Oct 30 01:22:11.712267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711239 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9456502-37f5-4f73-9403-9805f0a576a5" containerName="util" Oct 30 01:22:11.712267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711245 6572 cpu_manager.go:414] "containerMap: removing container" podUID="d550adeb-0895-4f14-b00e-0d30b59b8824" containerName="util" Oct 30 01:22:11.712267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711249 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d550adeb-0895-4f14-b00e-0d30b59b8824" containerName="util" Oct 30 01:22:11.712267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711259 6572 cpu_manager.go:414] "containerMap: removing container" podUID="07e39324-5f99-4c0e-9968-cb65e2c5eea2" containerName="pull" Oct 30 01:22:11.712267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711263 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07e39324-5f99-4c0e-9968-cb65e2c5eea2" containerName="pull" Oct 30 01:22:11.712267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711269 6572 cpu_manager.go:414] "containerMap: removing container" podUID="094b28a0-e057-4a0e-8583-089b04c76127" containerName="extract" Oct 30 01:22:11.712267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711273 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="094b28a0-e057-4a0e-8583-089b04c76127" containerName="extract" Oct 30 01:22:11.712267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711277 6572 cpu_manager.go:414] "containerMap: removing container" podUID="b0daee51-5c5b-4b93-8ac2-490ab8492d08" containerName="extract" Oct 30 01:22:11.712267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711281 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0daee51-5c5b-4b93-8ac2-490ab8492d08" containerName="extract" Oct 30 01:22:11.712267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711288 6572 cpu_manager.go:414] "containerMap: removing container" podUID="b0daee51-5c5b-4b93-8ac2-490ab8492d08" containerName="util" Oct 30 01:22:11.712267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711292 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0daee51-5c5b-4b93-8ac2-490ab8492d08" containerName="util" Oct 30 01:22:11.712267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711298 6572 cpu_manager.go:414] "containerMap: removing container" podUID="5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1" containerName="pull" Oct 30 01:22:11.712267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711301 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1" containerName="pull" Oct 30 01:22:11.712267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711339 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea59f9b0-3567-40e9-84a0-412e683a68c3" containerName="extract" Oct 30 01:22:11.712267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711347 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="07e39324-5f99-4c0e-9968-cb65e2c5eea2" containerName="extract" Oct 30 01:22:11.712267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711352 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="627442d3-8a28-4c41-aeff-aff7ec981c87" containerName="extract" Oct 30 01:22:11.712267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711359 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0daee51-5c5b-4b93-8ac2-490ab8492d08" containerName="extract" Oct 30 01:22:11.712267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711365 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="094b28a0-e057-4a0e-8583-089b04c76127" containerName="extract" Oct 30 01:22:11.712267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711371 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9456502-37f5-4f73-9403-9805f0a576a5" containerName="extract" Oct 30 01:22:11.712267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711376 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f06535c-ebb4-4fc3-9479-eb2e3a55bbc1" containerName="extract" Oct 30 01:22:11.712267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711380 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="e94c6943-5c8a-497a-a1d5-83acd8232992" containerName="extract" Oct 30 01:22:11.712267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711384 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ff0254c-bc05-4a84-b705-44d893282028" containerName="extract" Oct 30 01:22:11.712267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711390 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="d550adeb-0895-4f14-b00e-0d30b59b8824" containerName="extract" Oct 30 01:22:11.712267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.711396 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cdf1dae-636b-4e50-b328-1f428316cb7f" containerName="extract" Oct 30 01:22:11.714521 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.714502 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-bnhrd" Oct 30 01:22:11.736969 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.736941 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 01:22:11.737162 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.737061 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 01:22:11.737241 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.737165 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-pp4hc\"" Oct 30 01:22:11.737241 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.737197 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 01:22:11.737683 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.737666 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 01:22:11.737848 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.737831 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 01:22:11.744152 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.744132 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2f111ec2-c892-4a33-8d43-c671a9b33581-manifests\") pod \"opendatahub-operator-controller-manager-685895cf4f-bnhrd\" (UID: \"2f111ec2-c892-4a33-8d43-c671a9b33581\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-bnhrd" Oct 30 01:22:11.744262 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.744174 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb9vc\" (UniqueName: \"kubernetes.io/projected/2f111ec2-c892-4a33-8d43-c671a9b33581-kube-api-access-vb9vc\") pod \"opendatahub-operator-controller-manager-685895cf4f-bnhrd\" (UID: \"2f111ec2-c892-4a33-8d43-c671a9b33581\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-bnhrd" Oct 30 01:22:11.744262 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.744200 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2f111ec2-c892-4a33-8d43-c671a9b33581-webhook-cert\") pod \"opendatahub-operator-controller-manager-685895cf4f-bnhrd\" (UID: \"2f111ec2-c892-4a33-8d43-c671a9b33581\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-bnhrd" Oct 30 01:22:11.744338 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.744313 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f111ec2-c892-4a33-8d43-c671a9b33581-tmp\") pod \"opendatahub-operator-controller-manager-685895cf4f-bnhrd\" (UID: \"2f111ec2-c892-4a33-8d43-c671a9b33581\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-bnhrd" Oct 30 01:22:11.744376 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.744337 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2f111ec2-c892-4a33-8d43-c671a9b33581-apiservice-cert\") pod \"opendatahub-operator-controller-manager-685895cf4f-bnhrd\" (UID: \"2f111ec2-c892-4a33-8d43-c671a9b33581\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-bnhrd" Oct 30 01:22:11.744376 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.744370 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2f111ec2-c892-4a33-8d43-c671a9b33581-metrics-cert\") pod \"opendatahub-operator-controller-manager-685895cf4f-bnhrd\" (UID: \"2f111ec2-c892-4a33-8d43-c671a9b33581\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-bnhrd" Oct 30 01:22:11.760548 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.760517 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-bnhrd"] Oct 30 01:22:11.845706 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.845664 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2f111ec2-c892-4a33-8d43-c671a9b33581-metrics-cert\") pod \"opendatahub-operator-controller-manager-685895cf4f-bnhrd\" (UID: \"2f111ec2-c892-4a33-8d43-c671a9b33581\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-bnhrd" Oct 30 01:22:11.845898 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.845722 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2f111ec2-c892-4a33-8d43-c671a9b33581-manifests\") pod \"opendatahub-operator-controller-manager-685895cf4f-bnhrd\" (UID: \"2f111ec2-c892-4a33-8d43-c671a9b33581\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-bnhrd" Oct 30 01:22:11.845898 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.845776 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb9vc\" (UniqueName: \"kubernetes.io/projected/2f111ec2-c892-4a33-8d43-c671a9b33581-kube-api-access-vb9vc\") pod \"opendatahub-operator-controller-manager-685895cf4f-bnhrd\" (UID: \"2f111ec2-c892-4a33-8d43-c671a9b33581\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-bnhrd" Oct 30 01:22:11.845898 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.845798 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2f111ec2-c892-4a33-8d43-c671a9b33581-webhook-cert\") pod \"opendatahub-operator-controller-manager-685895cf4f-bnhrd\" (UID: \"2f111ec2-c892-4a33-8d43-c671a9b33581\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-bnhrd" Oct 30 01:22:11.845898 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.845822 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f111ec2-c892-4a33-8d43-c671a9b33581-tmp\") pod \"opendatahub-operator-controller-manager-685895cf4f-bnhrd\" (UID: \"2f111ec2-c892-4a33-8d43-c671a9b33581\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-bnhrd" Oct 30 01:22:11.845898 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.845838 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2f111ec2-c892-4a33-8d43-c671a9b33581-apiservice-cert\") pod \"opendatahub-operator-controller-manager-685895cf4f-bnhrd\" (UID: \"2f111ec2-c892-4a33-8d43-c671a9b33581\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-bnhrd" Oct 30 01:22:11.846240 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.846215 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2f111ec2-c892-4a33-8d43-c671a9b33581-manifests\") pod \"opendatahub-operator-controller-manager-685895cf4f-bnhrd\" (UID: \"2f111ec2-c892-4a33-8d43-c671a9b33581\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-bnhrd" Oct 30 01:22:11.848377 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.848352 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f111ec2-c892-4a33-8d43-c671a9b33581-tmp\") pod \"opendatahub-operator-controller-manager-685895cf4f-bnhrd\" (UID: \"2f111ec2-c892-4a33-8d43-c671a9b33581\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-bnhrd" Oct 30 01:22:11.848516 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.848494 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2f111ec2-c892-4a33-8d43-c671a9b33581-metrics-cert\") pod \"opendatahub-operator-controller-manager-685895cf4f-bnhrd\" (UID: \"2f111ec2-c892-4a33-8d43-c671a9b33581\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-bnhrd" Oct 30 01:22:11.848627 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.848603 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2f111ec2-c892-4a33-8d43-c671a9b33581-apiservice-cert\") pod \"opendatahub-operator-controller-manager-685895cf4f-bnhrd\" (UID: \"2f111ec2-c892-4a33-8d43-c671a9b33581\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-bnhrd" Oct 30 01:22:11.848948 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.848927 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2f111ec2-c892-4a33-8d43-c671a9b33581-webhook-cert\") pod \"opendatahub-operator-controller-manager-685895cf4f-bnhrd\" (UID: \"2f111ec2-c892-4a33-8d43-c671a9b33581\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-bnhrd" Oct 30 01:22:11.863761 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.863718 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb9vc\" (UniqueName: \"kubernetes.io/projected/2f111ec2-c892-4a33-8d43-c671a9b33581-kube-api-access-vb9vc\") pod \"opendatahub-operator-controller-manager-685895cf4f-bnhrd\" (UID: \"2f111ec2-c892-4a33-8d43-c671a9b33581\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-bnhrd" Oct 30 01:22:11.879130 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:11.879094 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:12.029822 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:12.029738 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-bnhrd" Oct 30 01:22:12.111617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:12.111566 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:12.303828 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:12.303736 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:12.371763 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:12.371725 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-bnhrd"] Oct 30 01:22:12.381737 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:22:12.381707 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f111ec2_c892_4a33_8d43_c671a9b33581.slice/crio-b4a4f78c118c53ad02c3f38fee59e8815941f13ad3fc7439736c981ce7bc941e WatchSource:0}: Error finding container b4a4f78c118c53ad02c3f38fee59e8815941f13ad3fc7439736c981ce7bc941e: Status 404 returned error can't find the container with id b4a4f78c118c53ad02c3f38fee59e8815941f13ad3fc7439736c981ce7bc941e Oct 30 01:22:12.495626 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:12.495572 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:13.061684 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:13.061634 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-bnhrd" event={"ID":"2f111ec2-c892-4a33-8d43-c671a9b33581","Type":"ContainerStarted","Data":"b4a4f78c118c53ad02c3f38fee59e8815941f13ad3fc7439736c981ce7bc941e"} Oct 30 01:22:14.690614 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:22:14.690575 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:22:14.724921 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:14.724819 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:14.924583 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:14.924548 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:15.143271 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:15.143188 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:16.077168 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:16.077084 6572 generic.go:356] "Generic (PLEG): container finished" podID="2f111ec2-c892-4a33-8d43-c671a9b33581" containerID="7840ce731751728c76f70d4a9b4dd5ff8131bc0801ca51c925c1cc6947c87029" exitCode=0 Oct 30 01:22:16.077544 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:16.077166 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-bnhrd" event={"ID":"2f111ec2-c892-4a33-8d43-c671a9b33581","Type":"ContainerDied","Data":"7840ce731751728c76f70d4a9b4dd5ff8131bc0801ca51c925c1cc6947c87029"} Oct 30 01:22:17.083262 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:17.083222 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-bnhrd" event={"ID":"2f111ec2-c892-4a33-8d43-c671a9b33581","Type":"ContainerStarted","Data":"f5faeaf2600b73025d4e49de34c04822d55e055c065d2b003cf4bb269f7f141f"} Oct 30 01:22:17.083694 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:17.083353 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-bnhrd" Oct 30 01:22:17.173199 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:17.173131 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-bnhrd" podStartSLOduration=2.526152239 podStartE2EDuration="6.173110538s" totalImagesPullingTime="3.126531363s" totalInitContainerRuntime="520.426936ms" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:17.161971757 +0000 UTC m=+262.738397159" watchObservedRunningTime="2026-10-30 01:22:17.173110538 +0000 UTC m=+262.749535942" Oct 30 01:22:17.244932 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:17.244887 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-j467f"] Oct 30 01:22:17.248416 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:17.248393 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-j467f" Oct 30 01:22:17.269565 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:17.269483 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-g8tbj\"" Oct 30 01:22:17.271564 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:17.271534 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-j467f"] Oct 30 01:22:17.278899 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:17.278874 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 01:22:17.279017 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:17.278921 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:22:17.286614 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:17.286582 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znmgz\" (UniqueName: \"kubernetes.io/projected/7a1c1aa5-ae8a-4847-82be-ce21c4942458-kube-api-access-znmgz\") pod \"jobset-operator-674cd59dc5-j467f\" (UID: \"7a1c1aa5-ae8a-4847-82be-ce21c4942458\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-j467f" Oct 30 01:22:17.286705 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:17.286628 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a1c1aa5-ae8a-4847-82be-ce21c4942458-tmp\") pod \"jobset-operator-674cd59dc5-j467f\" (UID: \"7a1c1aa5-ae8a-4847-82be-ce21c4942458\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-j467f" Oct 30 01:22:17.387814 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:17.387699 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znmgz\" (UniqueName: \"kubernetes.io/projected/7a1c1aa5-ae8a-4847-82be-ce21c4942458-kube-api-access-znmgz\") pod \"jobset-operator-674cd59dc5-j467f\" (UID: \"7a1c1aa5-ae8a-4847-82be-ce21c4942458\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-j467f" Oct 30 01:22:17.387814 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:17.387774 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a1c1aa5-ae8a-4847-82be-ce21c4942458-tmp\") pod \"jobset-operator-674cd59dc5-j467f\" (UID: \"7a1c1aa5-ae8a-4847-82be-ce21c4942458\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-j467f" Oct 30 01:22:17.388257 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:17.388236 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a1c1aa5-ae8a-4847-82be-ce21c4942458-tmp\") pod \"jobset-operator-674cd59dc5-j467f\" (UID: \"7a1c1aa5-ae8a-4847-82be-ce21c4942458\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-j467f" Oct 30 01:22:17.404460 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:17.404423 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znmgz\" (UniqueName: \"kubernetes.io/projected/7a1c1aa5-ae8a-4847-82be-ce21c4942458-kube-api-access-znmgz\") pod \"jobset-operator-674cd59dc5-j467f\" (UID: \"7a1c1aa5-ae8a-4847-82be-ce21c4942458\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-j467f" Oct 30 01:22:17.577009 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:17.576965 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-j467f" Oct 30 01:22:17.882674 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:17.882639 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-j467f"] Oct 30 01:22:17.885254 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:22:17.885226 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a1c1aa5_ae8a_4847_82be_ce21c4942458.slice/crio-cf7e1b032947697c64108998841b28ec709d67970adbcb4f5b1ed434ba223736 WatchSource:0}: Error finding container cf7e1b032947697c64108998841b28ec709d67970adbcb4f5b1ed434ba223736: Status 404 returned error can't find the container with id cf7e1b032947697c64108998841b28ec709d67970adbcb4f5b1ed434ba223736 Oct 30 01:22:18.093975 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:18.093933 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-j467f" event={"ID":"7a1c1aa5-ae8a-4847-82be-ce21c4942458","Type":"ContainerStarted","Data":"cf7e1b032947697c64108998841b28ec709d67970adbcb4f5b1ed434ba223736"} Oct 30 01:22:20.105050 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:20.105016 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-j467f" event={"ID":"7a1c1aa5-ae8a-4847-82be-ce21c4942458","Type":"ContainerStarted","Data":"07cfe779f93fdb7d7bbdf6e35cf30b4e67b6c4eaf794ad4cf77d6836ac4bcacb"} Oct 30 01:22:20.141111 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:20.141053 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-j467f" podStartSLOduration=1.5246935019999999 podStartE2EDuration="3.141031899s" totalImagesPullingTime="1.616338397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:20.139582642 +0000 UTC m=+265.716008045" watchObservedRunningTime="2026-10-30 01:22:20.141031899 +0000 UTC m=+265.717457301" Oct 30 01:22:20.327422 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:20.327379 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:20.532189 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:20.532151 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:23.518700 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:23.518652 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:25.541623 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:25.541585 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:27.007035 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:27.006999 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:28.097165 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:28.097130 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-bnhrd" Oct 30 01:22:28.643511 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:28.643471 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-67gth"] Oct 30 01:22:28.650604 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:28.650575 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-67gth" Oct 30 01:22:28.665048 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:28.665022 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:22:28.665444 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:28.665421 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 01:22:28.666121 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:28.666101 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-nhhz9\"" Oct 30 01:22:28.676922 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:28.676895 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-67gth"] Oct 30 01:22:28.683425 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:28.683397 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcnhg\" (UniqueName: \"kubernetes.io/projected/bcf0de1c-aa70-4b68-90ba-7be6d341156e-kube-api-access-vcnhg\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-67gth\" (UID: \"bcf0de1c-aa70-4b68-90ba-7be6d341156e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-67gth" Oct 30 01:22:28.683551 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:28.683527 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcf0de1c-aa70-4b68-90ba-7be6d341156e-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-67gth\" (UID: \"bcf0de1c-aa70-4b68-90ba-7be6d341156e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-67gth" Oct 30 01:22:28.784573 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:28.784530 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcnhg\" (UniqueName: \"kubernetes.io/projected/bcf0de1c-aa70-4b68-90ba-7be6d341156e-kube-api-access-vcnhg\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-67gth\" (UID: \"bcf0de1c-aa70-4b68-90ba-7be6d341156e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-67gth" Oct 30 01:22:28.784786 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:28.784618 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcf0de1c-aa70-4b68-90ba-7be6d341156e-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-67gth\" (UID: \"bcf0de1c-aa70-4b68-90ba-7be6d341156e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-67gth" Oct 30 01:22:28.785049 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:28.785032 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcf0de1c-aa70-4b68-90ba-7be6d341156e-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-67gth\" (UID: \"bcf0de1c-aa70-4b68-90ba-7be6d341156e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-67gth" Oct 30 01:22:28.807399 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:28.807366 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcnhg\" (UniqueName: \"kubernetes.io/projected/bcf0de1c-aa70-4b68-90ba-7be6d341156e-kube-api-access-vcnhg\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-67gth\" (UID: \"bcf0de1c-aa70-4b68-90ba-7be6d341156e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-67gth" Oct 30 01:22:28.963742 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:28.963703 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-67gth" Oct 30 01:22:29.290340 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:29.290314 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-67gth"] Oct 30 01:22:29.292618 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:22:29.292591 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf0de1c_aa70_4b68_90ba_7be6d341156e.slice/crio-633a434fe106d64738b94aebe989b519e4009de19d08da06071bb55cf04e42d4 WatchSource:0}: Error finding container 633a434fe106d64738b94aebe989b519e4009de19d08da06071bb55cf04e42d4: Status 404 returned error can't find the container with id 633a434fe106d64738b94aebe989b519e4009de19d08da06071bb55cf04e42d4 Oct 30 01:22:30.158147 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:30.158115 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-67gth" event={"ID":"bcf0de1c-aa70-4b68-90ba-7be6d341156e","Type":"ContainerStarted","Data":"633a434fe106d64738b94aebe989b519e4009de19d08da06071bb55cf04e42d4"} Oct 30 01:22:30.304215 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:30.304174 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:32.171944 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:32.171909 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-67gth" event={"ID":"bcf0de1c-aa70-4b68-90ba-7be6d341156e","Type":"ContainerStarted","Data":"75c6c0887846615338c46d9540562b130ef54c3eab4e792a788efd5f9af3c493"} Oct 30 01:22:32.232558 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:32.232507 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-67gth" podStartSLOduration=1.638608185 podStartE2EDuration="4.232491539s" totalImagesPullingTime="2.593883354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:32.221910595 +0000 UTC m=+277.798335997" watchObservedRunningTime="2026-10-30 01:22:32.232491539 +0000 UTC m=+277.808916941" Oct 30 01:22:32.835522 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:32.835488 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-vpxn7"] Oct 30 01:22:32.839080 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:32.839057 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-vpxn7" Oct 30 01:22:32.870227 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:32.870203 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-vpxn7"] Oct 30 01:22:32.882007 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:32.881979 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 01:22:32.882007 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:32.881979 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 01:22:32.882207 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:32.882006 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-gbq8v\"" Oct 30 01:22:32.921132 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:32.921097 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dmv8\" (UniqueName: \"kubernetes.io/projected/76c97fcb-f62c-4de6-899b-3c23f809d1c6-kube-api-access-2dmv8\") pod \"limitador-operator-controller-manager-5778bb4449-vpxn7\" (UID: \"76c97fcb-f62c-4de6-899b-3c23f809d1c6\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-vpxn7" Oct 30 01:22:32.938277 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:32.938244 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:33.022544 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:33.022509 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dmv8\" (UniqueName: \"kubernetes.io/projected/76c97fcb-f62c-4de6-899b-3c23f809d1c6-kube-api-access-2dmv8\") pod \"limitador-operator-controller-manager-5778bb4449-vpxn7\" (UID: \"76c97fcb-f62c-4de6-899b-3c23f809d1c6\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-vpxn7" Oct 30 01:22:33.047160 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:33.047130 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dmv8\" (UniqueName: \"kubernetes.io/projected/76c97fcb-f62c-4de6-899b-3c23f809d1c6-kube-api-access-2dmv8\") pod \"limitador-operator-controller-manager-5778bb4449-vpxn7\" (UID: \"76c97fcb-f62c-4de6-899b-3c23f809d1c6\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-vpxn7" Oct 30 01:22:33.151943 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:33.151860 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-vpxn7" Oct 30 01:22:33.250964 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:33.250926 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:33.465238 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:33.465214 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-vpxn7"] Oct 30 01:22:33.467823 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:22:33.467735 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c97fcb_f62c_4de6_899b_3c23f809d1c6.slice/crio-54a3511362ae54cbfb6f0d1b91dc1087268c8a5f7ddb1e6dc14db4bd0b578d71 WatchSource:0}: Error finding container 54a3511362ae54cbfb6f0d1b91dc1087268c8a5f7ddb1e6dc14db4bd0b578d71: Status 404 returned error can't find the container with id 54a3511362ae54cbfb6f0d1b91dc1087268c8a5f7ddb1e6dc14db4bd0b578d71 Oct 30 01:22:33.791514 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:33.791416 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:34.184496 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:34.184458 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-vpxn7" event={"ID":"76c97fcb-f62c-4de6-899b-3c23f809d1c6","Type":"ContainerStarted","Data":"54a3511362ae54cbfb6f0d1b91dc1087268c8a5f7ddb1e6dc14db4bd0b578d71"} Oct 30 01:22:35.338953 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:35.338919 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:35.529876 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:35.529844 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:35.769052 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:35.769018 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-5dk87"] Oct 30 01:22:35.772194 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:35.772175 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5dk87" Oct 30 01:22:35.790600 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:35.790570 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 01:22:35.790600 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:35.790586 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 01:22:35.797444 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:35.797419 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-dh7fn\"" Oct 30 01:22:35.799163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:35.799140 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-5dk87"] Oct 30 01:22:35.843191 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:35.843157 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/45c5211a-de2a-40fb-8139-b068a9108b16-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-5dk87\" (UID: \"45c5211a-de2a-40fb-8139-b068a9108b16\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5dk87" Oct 30 01:22:35.843320 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:35.843229 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6flbv\" (UniqueName: \"kubernetes.io/projected/45c5211a-de2a-40fb-8139-b068a9108b16-kube-api-access-6flbv\") pod \"cert-manager-cainjector-5c57bbd4fc-5dk87\" (UID: \"45c5211a-de2a-40fb-8139-b068a9108b16\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5dk87" Oct 30 01:22:35.944479 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:35.944446 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6flbv\" (UniqueName: \"kubernetes.io/projected/45c5211a-de2a-40fb-8139-b068a9108b16-kube-api-access-6flbv\") pod \"cert-manager-cainjector-5c57bbd4fc-5dk87\" (UID: \"45c5211a-de2a-40fb-8139-b068a9108b16\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5dk87" Oct 30 01:22:35.944642 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:35.944512 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/45c5211a-de2a-40fb-8139-b068a9108b16-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-5dk87\" (UID: \"45c5211a-de2a-40fb-8139-b068a9108b16\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5dk87" Oct 30 01:22:35.965537 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:35.965509 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6flbv\" (UniqueName: \"kubernetes.io/projected/45c5211a-de2a-40fb-8139-b068a9108b16-kube-api-access-6flbv\") pod \"cert-manager-cainjector-5c57bbd4fc-5dk87\" (UID: \"45c5211a-de2a-40fb-8139-b068a9108b16\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5dk87" Oct 30 01:22:35.965642 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:35.965513 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/45c5211a-de2a-40fb-8139-b068a9108b16-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-5dk87\" (UID: \"45c5211a-de2a-40fb-8139-b068a9108b16\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5dk87" Oct 30 01:22:36.083206 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:36.083120 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5dk87" Oct 30 01:22:36.375281 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:36.375225 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-5dk87"] Oct 30 01:22:36.377966 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:22:36.377936 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45c5211a_de2a_40fb_8139_b068a9108b16.slice/crio-5828034579f4ffe6170d201ebe92418b53a5d015f3925bac0cb61033e9fe3a94 WatchSource:0}: Error finding container 5828034579f4ffe6170d201ebe92418b53a5d015f3925bac0cb61033e9fe3a94: Status 404 returned error can't find the container with id 5828034579f4ffe6170d201ebe92418b53a5d015f3925bac0cb61033e9fe3a94 Oct 30 01:22:36.773791 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:36.770023 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:37.038390 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:37.038302 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:37.200215 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:37.200180 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5dk87" event={"ID":"45c5211a-de2a-40fb-8139-b068a9108b16","Type":"ContainerStarted","Data":"5828034579f4ffe6170d201ebe92418b53a5d015f3925bac0cb61033e9fe3a94"} Oct 30 01:22:37.707379 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:37.707345 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:38.323671 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:38.323632 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:38.765902 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:38.765866 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:40.219467 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:40.219430 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5dk87" event={"ID":"45c5211a-de2a-40fb-8139-b068a9108b16","Type":"ContainerStarted","Data":"b141837626ea54be0d82209121c42cef306581efa048123ca9c2b67ef13ca526"} Oct 30 01:22:40.262844 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:40.262772 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-5dk87" podStartSLOduration=2.309784717 podStartE2EDuration="5.26273784s" totalImagesPullingTime="2.952953123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:40.262107579 +0000 UTC m=+285.838532981" watchObservedRunningTime="2026-10-30 01:22:40.26273784 +0000 UTC m=+285.839163242" Oct 30 01:22:40.731770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:40.731712 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-g7r2k"] Oct 30 01:22:40.734064 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:40.734048 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g7r2k" Oct 30 01:22:40.747842 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:40.747814 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-rlqzz\"" Oct 30 01:22:40.748613 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:40.748589 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:40.759314 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:40.759281 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-g7r2k"] Oct 30 01:22:40.786298 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:40.786258 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwsfx\" (UniqueName: \"kubernetes.io/projected/2743c917-17a4-4569-a364-2aad7e855aab-kube-api-access-rwsfx\") pod \"cert-manager-webhook-5dbbbf75cc-g7r2k\" (UID: \"2743c917-17a4-4569-a364-2aad7e855aab\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g7r2k" Oct 30 01:22:40.786469 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:40.786313 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2743c917-17a4-4569-a364-2aad7e855aab-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-g7r2k\" (UID: \"2743c917-17a4-4569-a364-2aad7e855aab\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g7r2k" Oct 30 01:22:40.887632 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:40.887593 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwsfx\" (UniqueName: \"kubernetes.io/projected/2743c917-17a4-4569-a364-2aad7e855aab-kube-api-access-rwsfx\") pod \"cert-manager-webhook-5dbbbf75cc-g7r2k\" (UID: \"2743c917-17a4-4569-a364-2aad7e855aab\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g7r2k" Oct 30 01:22:40.887832 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:40.887639 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2743c917-17a4-4569-a364-2aad7e855aab-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-g7r2k\" (UID: \"2743c917-17a4-4569-a364-2aad7e855aab\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g7r2k" Oct 30 01:22:40.904667 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:40.904631 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2743c917-17a4-4569-a364-2aad7e855aab-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-g7r2k\" (UID: \"2743c917-17a4-4569-a364-2aad7e855aab\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g7r2k" Oct 30 01:22:40.904829 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:40.904726 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwsfx\" (UniqueName: \"kubernetes.io/projected/2743c917-17a4-4569-a364-2aad7e855aab-kube-api-access-rwsfx\") pod \"cert-manager-webhook-5dbbbf75cc-g7r2k\" (UID: \"2743c917-17a4-4569-a364-2aad7e855aab\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g7r2k" Oct 30 01:22:41.043791 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:41.043686 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g7r2k" Oct 30 01:22:41.179511 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:41.179474 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:41.337139 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:41.337108 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-g7r2k"] Oct 30 01:22:41.339518 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:22:41.339493 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2743c917_17a4_4569_a364_2aad7e855aab.slice/crio-26a2e0b667e71e1acd0afc15c89f7f2c046dfde3b4753ede7e9aa6a05e37a1c7 WatchSource:0}: Error finding container 26a2e0b667e71e1acd0afc15c89f7f2c046dfde3b4753ede7e9aa6a05e37a1c7: Status 404 returned error can't find the container with id 26a2e0b667e71e1acd0afc15c89f7f2c046dfde3b4753ede7e9aa6a05e37a1c7 Oct 30 01:22:41.365572 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:41.365538 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:41.731964 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:41.731929 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:41.920414 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:41.920375 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:42.190529 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:42.190485 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:42.231160 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:42.231128 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g7r2k" event={"ID":"2743c917-17a4-4569-a364-2aad7e855aab","Type":"ContainerStarted","Data":"97c15ab0ce1d758fe633b49d3e020888774fbd52c93096eb6655b4436daea72f"} Oct 30 01:22:42.231160 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:42.231166 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g7r2k" event={"ID":"2743c917-17a4-4569-a364-2aad7e855aab","Type":"ContainerStarted","Data":"26a2e0b667e71e1acd0afc15c89f7f2c046dfde3b4753ede7e9aa6a05e37a1c7"} Oct 30 01:22:42.231379 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:42.231267 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g7r2k" Oct 30 01:22:42.265394 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:42.265340 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g7r2k" podStartSLOduration=2.265324954 podStartE2EDuration="2.265324954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:42.264366843 +0000 UTC m=+287.840792247" watchObservedRunningTime="2026-10-30 01:22:42.265324954 +0000 UTC m=+287.841750356" Oct 30 01:22:42.389299 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:42.389257 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:43.311259 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:43.311221 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:44.690907 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:22:44.690867 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:22:44.755442 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:44.755407 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:45.016931 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:45.016835 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:46.563011 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:46.562973 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:46.761291 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:46.761182 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:46.973836 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:46.973800 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:47.135125 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:47.135045 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-66f46d7b58-bszt4"] Oct 30 01:22:47.137899 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:47.137878 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-66f46d7b58-bszt4" Oct 30 01:22:47.155465 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:47.155390 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-66f46d7b58-bszt4"] Oct 30 01:22:47.163915 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:47.163817 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 01:22:47.163915 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:47.163848 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-c6x4g\"" Oct 30 01:22:47.163915 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:47.163857 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 01:22:47.163915 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:47.163870 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 01:22:47.164166 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:47.163944 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 01:22:47.216415 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:47.216373 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:47.247810 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:47.247769 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d65df0f-5ee7-4774-b73d-2f8adb718fad-tmp\") pod \"kube-auth-proxy-66f46d7b58-bszt4\" (UID: \"3d65df0f-5ee7-4774-b73d-2f8adb718fad\") " pod="openshift-ingress/kube-auth-proxy-66f46d7b58-bszt4" Oct 30 01:22:47.247983 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:47.247845 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs6l5\" (UniqueName: \"kubernetes.io/projected/3d65df0f-5ee7-4774-b73d-2f8adb718fad-kube-api-access-vs6l5\") pod \"kube-auth-proxy-66f46d7b58-bszt4\" (UID: \"3d65df0f-5ee7-4774-b73d-2f8adb718fad\") " pod="openshift-ingress/kube-auth-proxy-66f46d7b58-bszt4" Oct 30 01:22:47.247983 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:47.247942 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d65df0f-5ee7-4774-b73d-2f8adb718fad-tls-certs\") pod \"kube-auth-proxy-66f46d7b58-bszt4\" (UID: \"3d65df0f-5ee7-4774-b73d-2f8adb718fad\") " pod="openshift-ingress/kube-auth-proxy-66f46d7b58-bszt4" Oct 30 01:22:47.262308 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:47.262272 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-vpxn7" event={"ID":"76c97fcb-f62c-4de6-899b-3c23f809d1c6","Type":"ContainerStarted","Data":"ab5458f542b2810f0b2200d0f10ce8897c2471c3b1306953b614f46d2f992858"} Oct 30 01:22:47.262503 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:47.262485 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-vpxn7" Oct 30 01:22:47.348805 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:47.348764 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d65df0f-5ee7-4774-b73d-2f8adb718fad-tmp\") pod \"kube-auth-proxy-66f46d7b58-bszt4\" (UID: \"3d65df0f-5ee7-4774-b73d-2f8adb718fad\") " pod="openshift-ingress/kube-auth-proxy-66f46d7b58-bszt4" Oct 30 01:22:47.349003 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:47.348834 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs6l5\" (UniqueName: \"kubernetes.io/projected/3d65df0f-5ee7-4774-b73d-2f8adb718fad-kube-api-access-vs6l5\") pod \"kube-auth-proxy-66f46d7b58-bszt4\" (UID: \"3d65df0f-5ee7-4774-b73d-2f8adb718fad\") " pod="openshift-ingress/kube-auth-proxy-66f46d7b58-bszt4" Oct 30 01:22:47.349003 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:47.348863 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d65df0f-5ee7-4774-b73d-2f8adb718fad-tls-certs\") pod \"kube-auth-proxy-66f46d7b58-bszt4\" (UID: \"3d65df0f-5ee7-4774-b73d-2f8adb718fad\") " pod="openshift-ingress/kube-auth-proxy-66f46d7b58-bszt4" Oct 30 01:22:47.351264 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:47.351229 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d65df0f-5ee7-4774-b73d-2f8adb718fad-tmp\") pod \"kube-auth-proxy-66f46d7b58-bszt4\" (UID: \"3d65df0f-5ee7-4774-b73d-2f8adb718fad\") " pod="openshift-ingress/kube-auth-proxy-66f46d7b58-bszt4" Oct 30 01:22:47.351460 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:47.351439 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d65df0f-5ee7-4774-b73d-2f8adb718fad-tls-certs\") pod \"kube-auth-proxy-66f46d7b58-bszt4\" (UID: \"3d65df0f-5ee7-4774-b73d-2f8adb718fad\") " pod="openshift-ingress/kube-auth-proxy-66f46d7b58-bszt4" Oct 30 01:22:47.377727 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:47.377685 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs6l5\" (UniqueName: \"kubernetes.io/projected/3d65df0f-5ee7-4774-b73d-2f8adb718fad-kube-api-access-vs6l5\") pod \"kube-auth-proxy-66f46d7b58-bszt4\" (UID: \"3d65df0f-5ee7-4774-b73d-2f8adb718fad\") " pod="openshift-ingress/kube-auth-proxy-66f46d7b58-bszt4" Oct 30 01:22:47.450156 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:47.450123 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-66f46d7b58-bszt4" Oct 30 01:22:47.486105 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:47.486050 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:47.698459 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:47.698423 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:47.749406 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:47.749219 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-vpxn7" podStartSLOduration=2.78681375 podStartE2EDuration="15.749194835s" totalImagesPullingTime="12.962381085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:47.337473309 +0000 UTC m=+292.913898710" watchObservedRunningTime="2026-10-30 01:22:47.749194835 +0000 UTC m=+293.325620245" Oct 30 01:22:47.750858 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:47.750833 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-66f46d7b58-bszt4"] Oct 30 01:22:47.753114 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:22:47.753088 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d65df0f_5ee7_4774_b73d_2f8adb718fad.slice/crio-064a26ba61561d9e1137d98d74d970a6aec3adcd1710e4f7a54876f63128c26c WatchSource:0}: Error finding container 064a26ba61561d9e1137d98d74d970a6aec3adcd1710e4f7a54876f63128c26c: Status 404 returned error can't find the container with id 064a26ba61561d9e1137d98d74d970a6aec3adcd1710e4f7a54876f63128c26c Oct 30 01:22:47.904691 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:47.904653 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:48.131359 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:48.131271 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:48.237824 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:48.237796 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-g7r2k" Oct 30 01:22:48.268886 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:48.268851 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-66f46d7b58-bszt4" event={"ID":"3d65df0f-5ee7-4774-b73d-2f8adb718fad","Type":"ContainerStarted","Data":"064a26ba61561d9e1137d98d74d970a6aec3adcd1710e4f7a54876f63128c26c"} Oct 30 01:22:48.360201 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:48.360167 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:48.901569 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:48.901460 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:50.298050 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:50.298013 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:51.286264 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:51.286225 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-66f46d7b58-bszt4" event={"ID":"3d65df0f-5ee7-4774-b73d-2f8adb718fad","Type":"ContainerStarted","Data":"5860eb8421f3aa743944792f7d84145984151cae6afc5a09bb3a584a322c4d18"} Oct 30 01:22:51.321717 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:51.321664 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-66f46d7b58-bszt4" podStartSLOduration=1.833691593 podStartE2EDuration="4.32165055s" totalImagesPullingTime="2.487958957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:51.320212586 +0000 UTC m=+296.896637987" watchObservedRunningTime="2026-10-30 01:22:51.32165055 +0000 UTC m=+296.898075952" Oct 30 01:22:52.975404 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:52.975350 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:53.157615 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:53.157583 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:53.527454 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:53.527414 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:53.721258 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:53.721218 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:56.726468 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:56.726432 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-hf2jx"] Oct 30 01:22:56.732037 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:56.732015 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hf2jx" Oct 30 01:22:56.747966 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:56.747935 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 01:22:56.748100 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:56.748014 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 01:22:56.779062 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:56.779012 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 01:22:56.779062 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:56.779041 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-4zth5\"" Oct 30 01:22:56.780806 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:56.780791 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-hf2jx"] Oct 30 01:22:56.833434 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:56.833394 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/296d1c8b-da1d-4e25-8b3e-7a2c50337818-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-hf2jx\" (UID: \"296d1c8b-da1d-4e25-8b3e-7a2c50337818\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hf2jx" Oct 30 01:22:56.833617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:56.833446 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/296d1c8b-da1d-4e25-8b3e-7a2c50337818-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-hf2jx\" (UID: \"296d1c8b-da1d-4e25-8b3e-7a2c50337818\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hf2jx" Oct 30 01:22:56.833617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:56.833553 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/296d1c8b-da1d-4e25-8b3e-7a2c50337818-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-hf2jx\" (UID: \"296d1c8b-da1d-4e25-8b3e-7a2c50337818\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hf2jx" Oct 30 01:22:56.833617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:56.833597 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fslw\" (UniqueName: \"kubernetes.io/projected/296d1c8b-da1d-4e25-8b3e-7a2c50337818-kube-api-access-6fslw\") pod \"kserve-module-controller-manager-7566cf5fc8-hf2jx\" (UID: \"296d1c8b-da1d-4e25-8b3e-7a2c50337818\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hf2jx" Oct 30 01:22:56.934573 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:56.934530 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/296d1c8b-da1d-4e25-8b3e-7a2c50337818-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-hf2jx\" (UID: \"296d1c8b-da1d-4e25-8b3e-7a2c50337818\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hf2jx" Oct 30 01:22:56.934807 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:56.934579 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/296d1c8b-da1d-4e25-8b3e-7a2c50337818-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-hf2jx\" (UID: \"296d1c8b-da1d-4e25-8b3e-7a2c50337818\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hf2jx" Oct 30 01:22:56.934807 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:56.934624 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/296d1c8b-da1d-4e25-8b3e-7a2c50337818-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-hf2jx\" (UID: \"296d1c8b-da1d-4e25-8b3e-7a2c50337818\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hf2jx" Oct 30 01:22:56.934807 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:56.934657 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fslw\" (UniqueName: \"kubernetes.io/projected/296d1c8b-da1d-4e25-8b3e-7a2c50337818-kube-api-access-6fslw\") pod \"kserve-module-controller-manager-7566cf5fc8-hf2jx\" (UID: \"296d1c8b-da1d-4e25-8b3e-7a2c50337818\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hf2jx" Oct 30 01:22:56.934998 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:56.934974 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/296d1c8b-da1d-4e25-8b3e-7a2c50337818-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-hf2jx\" (UID: \"296d1c8b-da1d-4e25-8b3e-7a2c50337818\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hf2jx" Oct 30 01:22:56.935070 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:56.935002 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/296d1c8b-da1d-4e25-8b3e-7a2c50337818-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-hf2jx\" (UID: \"296d1c8b-da1d-4e25-8b3e-7a2c50337818\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hf2jx" Oct 30 01:22:56.935525 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:56.935504 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/296d1c8b-da1d-4e25-8b3e-7a2c50337818-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-hf2jx\" (UID: \"296d1c8b-da1d-4e25-8b3e-7a2c50337818\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hf2jx" Oct 30 01:22:56.952427 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:56.952399 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fslw\" (UniqueName: \"kubernetes.io/projected/296d1c8b-da1d-4e25-8b3e-7a2c50337818-kube-api-access-6fslw\") pod \"kserve-module-controller-manager-7566cf5fc8-hf2jx\" (UID: \"296d1c8b-da1d-4e25-8b3e-7a2c50337818\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hf2jx" Oct 30 01:22:57.046566 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:57.046476 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hf2jx" Oct 30 01:22:57.351262 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:57.351232 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-hf2jx"] Oct 30 01:22:57.353555 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:22:57.353531 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod296d1c8b_da1d_4e25_8b3e_7a2c50337818.slice/crio-b1930934abf5b829ee20a502d3749f28007d4a27cafc0aaf0a570318debca82c WatchSource:0}: Error finding container b1930934abf5b829ee20a502d3749f28007d4a27cafc0aaf0a570318debca82c: Status 404 returned error can't find the container with id b1930934abf5b829ee20a502d3749f28007d4a27cafc0aaf0a570318debca82c Oct 30 01:22:58.273926 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:58.273885 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-vpxn7" Oct 30 01:22:58.322333 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:58.322291 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hf2jx" event={"ID":"296d1c8b-da1d-4e25-8b3e-7a2c50337818","Type":"ContainerStarted","Data":"b1930934abf5b829ee20a502d3749f28007d4a27cafc0aaf0a570318debca82c"} Oct 30 01:22:58.950870 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:58.950827 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:59.160759 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:59.160724 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:59.329171 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:59.329138 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hf2jx" event={"ID":"296d1c8b-da1d-4e25-8b3e-7a2c50337818","Type":"ContainerStarted","Data":"c51987b6b99aab2817d9ed10d1559f065a31038f0eb729cee12b0e3d7ef4169a"} Oct 30 01:22:59.329552 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:59.329233 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hf2jx" Oct 30 01:22:59.512508 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:59.512436 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:59.706462 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:22:59.706420 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:00.109415 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:00.109381 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:00.326252 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:00.326215 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:02.384066 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:02.384028 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:04.935429 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:04.935388 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:05.124177 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:05.124139 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:05.710948 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:05.710892 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hf2jx" podStartSLOduration=8.266522673 podStartE2EDuration="9.710876497s" totalImagesPullingTime="1.444353824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:59.386844529 +0000 UTC m=+304.963269925" watchObservedRunningTime="2026-10-30 01:23:05.710876497 +0000 UTC m=+311.287301915" Oct 30 01:23:05.711227 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:05.711212 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pz5gn"] Oct 30 01:23:05.714469 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:05.714447 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-pz5gn" Oct 30 01:23:05.729333 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:05.729307 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pz5gn"] Oct 30 01:23:05.737658 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:05.737637 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-q9rn8\"" Oct 30 01:23:05.822088 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:05.822049 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pbd2\" (UniqueName: \"kubernetes.io/projected/617b8c34-7390-4da4-9167-f236c8c3eb00-kube-api-access-8pbd2\") pod \"authorino-57f7f8659f-pz5gn\" (UID: \"617b8c34-7390-4da4-9167-f236c8c3eb00\") " pod="kuadrant-system/authorino-57f7f8659f-pz5gn" Oct 30 01:23:05.864979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:05.864942 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:05.923106 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:05.923076 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pbd2\" (UniqueName: \"kubernetes.io/projected/617b8c34-7390-4da4-9167-f236c8c3eb00-kube-api-access-8pbd2\") pod \"authorino-57f7f8659f-pz5gn\" (UID: \"617b8c34-7390-4da4-9167-f236c8c3eb00\") " pod="kuadrant-system/authorino-57f7f8659f-pz5gn" Oct 30 01:23:05.940007 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:05.939983 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pbd2\" (UniqueName: \"kubernetes.io/projected/617b8c34-7390-4da4-9167-f236c8c3eb00-kube-api-access-8pbd2\") pod \"authorino-57f7f8659f-pz5gn\" (UID: \"617b8c34-7390-4da4-9167-f236c8c3eb00\") " pod="kuadrant-system/authorino-57f7f8659f-pz5gn" Oct 30 01:23:06.026267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:06.026190 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-pz5gn" Oct 30 01:23:06.067459 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:06.067360 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:06.318204 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:06.318176 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pz5gn"] Oct 30 01:23:06.320820 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:23:06.320791 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod617b8c34_7390_4da4_9167_f236c8c3eb00.slice/crio-46d5fe7bfaa4ce38fbc5b115f463bf057e852bb7ab4ee6f71b51d2997848685a WatchSource:0}: Error finding container 46d5fe7bfaa4ce38fbc5b115f463bf057e852bb7ab4ee6f71b51d2997848685a: Status 404 returned error can't find the container with id 46d5fe7bfaa4ce38fbc5b115f463bf057e852bb7ab4ee6f71b51d2997848685a Oct 30 01:23:06.366490 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:06.366456 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-pz5gn" event={"ID":"617b8c34-7390-4da4-9167-f236c8c3eb00","Type":"ContainerStarted","Data":"46d5fe7bfaa4ce38fbc5b115f463bf057e852bb7ab4ee6f71b51d2997848685a"} Oct 30 01:23:07.524453 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:07.524416 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:07.701700 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:07.701631 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:09.384600 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:09.384565 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-pz5gn" event={"ID":"617b8c34-7390-4da4-9167-f236c8c3eb00","Type":"ContainerStarted","Data":"9c4015e4048522eb1079db773b26058f832ca4101db0fced85c6fa2f88a20901"} Oct 30 01:23:09.418208 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:09.418160 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-pz5gn" podStartSLOduration=1.496772981 podStartE2EDuration="4.418142092s" totalImagesPullingTime="2.921369111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:09.41719821 +0000 UTC m=+314.993623611" watchObservedRunningTime="2026-10-30 01:23:09.418142092 +0000 UTC m=+314.994567495" Oct 30 01:23:10.568379 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:10.568349 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pz5gn"] Oct 30 01:23:10.968015 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:10.967977 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:11.139481 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:11.139448 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:11.394265 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:11.394182 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-pz5gn" podUID="617b8c34-7390-4da4-9167-f236c8c3eb00" containerName="authorino" containerID="cri-o://9c4015e4048522eb1079db773b26058f832ca4101db0fced85c6fa2f88a20901" gracePeriod=30 Oct 30 01:23:11.642013 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:11.641985 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-pz5gn" Oct 30 01:23:11.668341 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:11.668316 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/617b8c34-7390-4da4-9167-f236c8c3eb00-kube-api-access-8pbd2\" (UniqueName: \"kubernetes.io/projected/617b8c34-7390-4da4-9167-f236c8c3eb00-kube-api-access-8pbd2\") pod \"617b8c34-7390-4da4-9167-f236c8c3eb00\" (UID: \"617b8c34-7390-4da4-9167-f236c8c3eb00\") " Oct 30 01:23:11.670432 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:11.670408 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/617b8c34-7390-4da4-9167-f236c8c3eb00-kube-api-access-8pbd2" pod "617b8c34-7390-4da4-9167-f236c8c3eb00" (UID: "617b8c34-7390-4da4-9167-f236c8c3eb00"). InnerVolumeSpecName "kube-api-access-8pbd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:11.769035 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:11.769003 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pbd2\" (UniqueName: \"kubernetes.io/projected/617b8c34-7390-4da4-9167-f236c8c3eb00-kube-api-access-8pbd2\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:23:12.399984 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:12.399951 6572 generic.go:356] "Generic (PLEG): container finished" podID="617b8c34-7390-4da4-9167-f236c8c3eb00" containerID="9c4015e4048522eb1079db773b26058f832ca4101db0fced85c6fa2f88a20901" exitCode=0 Oct 30 01:23:12.400157 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:12.400003 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-pz5gn" Oct 30 01:23:12.400157 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:12.400019 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-pz5gn" event={"ID":"617b8c34-7390-4da4-9167-f236c8c3eb00","Type":"ContainerDied","Data":"9c4015e4048522eb1079db773b26058f832ca4101db0fced85c6fa2f88a20901"} Oct 30 01:23:12.400157 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:12.400065 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-pz5gn" event={"ID":"617b8c34-7390-4da4-9167-f236c8c3eb00","Type":"ContainerDied","Data":"46d5fe7bfaa4ce38fbc5b115f463bf057e852bb7ab4ee6f71b51d2997848685a"} Oct 30 01:23:12.400157 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:12.400089 6572 scope.go:122] "RemoveContainer" containerID="9c4015e4048522eb1079db773b26058f832ca4101db0fced85c6fa2f88a20901" Oct 30 01:23:12.409418 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:12.409397 6572 scope.go:122] "RemoveContainer" containerID="9c4015e4048522eb1079db773b26058f832ca4101db0fced85c6fa2f88a20901" Oct 30 01:23:12.409669 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:23:12.409650 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c4015e4048522eb1079db773b26058f832ca4101db0fced85c6fa2f88a20901\": container with ID starting with 9c4015e4048522eb1079db773b26058f832ca4101db0fced85c6fa2f88a20901 not found: ID does not exist" containerID="9c4015e4048522eb1079db773b26058f832ca4101db0fced85c6fa2f88a20901" Oct 30 01:23:12.409804 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:12.409683 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c4015e4048522eb1079db773b26058f832ca4101db0fced85c6fa2f88a20901"} err="failed to get container status \"9c4015e4048522eb1079db773b26058f832ca4101db0fced85c6fa2f88a20901\": rpc error: code = NotFound desc = could not find container \"9c4015e4048522eb1079db773b26058f832ca4101db0fced85c6fa2f88a20901\": container with ID starting with 9c4015e4048522eb1079db773b26058f832ca4101db0fced85c6fa2f88a20901 not found: ID does not exist" Oct 30 01:23:12.443834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:12.443807 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pz5gn"] Oct 30 01:23:12.447197 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:12.447175 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pz5gn"] Oct 30 01:23:12.540227 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:12.540195 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="617b8c34-7390-4da4-9167-f236c8c3eb00" path="/var/lib/kubelet/pods/617b8c34-7390-4da4-9167-f236c8c3eb00/volumes" Oct 30 01:23:13.988729 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:13.988695 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:14.168069 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:14.168027 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:14.480248 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:14.480206 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-56b49d854f-f7d45"] Oct 30 01:23:14.480808 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:14.480783 6572 cpu_manager.go:414] "containerMap: removing container" podUID="617b8c34-7390-4da4-9167-f236c8c3eb00" containerName="authorino" Oct 30 01:23:14.480808 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:14.480805 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="617b8c34-7390-4da4-9167-f236c8c3eb00" containerName="authorino" Oct 30 01:23:14.480937 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:14.480919 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="617b8c34-7390-4da4-9167-f236c8c3eb00" containerName="authorino" Oct 30 01:23:14.484049 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:14.484029 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-56b49d854f-f7d45" Oct 30 01:23:14.500263 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:14.500236 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-8z2kf\"" Oct 30 01:23:14.502735 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:14.502710 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-56b49d854f-f7d45"] Oct 30 01:23:14.510471 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:14.510450 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 30 01:23:14.593154 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:14.593113 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8593fddc-4cf8-4d98-ae2f-c6d0a553c954-cert\") pod \"jobset-controller-manager-56b49d854f-f7d45\" (UID: \"8593fddc-4cf8-4d98-ae2f-c6d0a553c954\") " pod="openshift-jobset-operator/jobset-controller-manager-56b49d854f-f7d45" Oct 30 01:23:14.593325 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:14.593182 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8593fddc-4cf8-4d98-ae2f-c6d0a553c954-metrics-certs\") pod \"jobset-controller-manager-56b49d854f-f7d45\" (UID: \"8593fddc-4cf8-4d98-ae2f-c6d0a553c954\") " pod="openshift-jobset-operator/jobset-controller-manager-56b49d854f-f7d45" Oct 30 01:23:14.593325 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:14.593234 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skzxt\" (UniqueName: \"kubernetes.io/projected/8593fddc-4cf8-4d98-ae2f-c6d0a553c954-kube-api-access-skzxt\") pod \"jobset-controller-manager-56b49d854f-f7d45\" (UID: \"8593fddc-4cf8-4d98-ae2f-c6d0a553c954\") " pod="openshift-jobset-operator/jobset-controller-manager-56b49d854f-f7d45" Oct 30 01:23:14.593397 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:14.593362 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8593fddc-4cf8-4d98-ae2f-c6d0a553c954-manager-config\") pod \"jobset-controller-manager-56b49d854f-f7d45\" (UID: \"8593fddc-4cf8-4d98-ae2f-c6d0a553c954\") " pod="openshift-jobset-operator/jobset-controller-manager-56b49d854f-f7d45" Oct 30 01:23:14.601037 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:14.601017 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 30 01:23:14.601139 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:14.601123 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 30 01:23:14.687460 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:23:14.687413 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:23:14.694662 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:14.694618 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8593fddc-4cf8-4d98-ae2f-c6d0a553c954-cert\") pod \"jobset-controller-manager-56b49d854f-f7d45\" (UID: \"8593fddc-4cf8-4d98-ae2f-c6d0a553c954\") " pod="openshift-jobset-operator/jobset-controller-manager-56b49d854f-f7d45" Oct 30 01:23:14.694868 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:14.694708 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8593fddc-4cf8-4d98-ae2f-c6d0a553c954-metrics-certs\") pod \"jobset-controller-manager-56b49d854f-f7d45\" (UID: \"8593fddc-4cf8-4d98-ae2f-c6d0a553c954\") " pod="openshift-jobset-operator/jobset-controller-manager-56b49d854f-f7d45" Oct 30 01:23:14.694868 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:14.694740 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skzxt\" (UniqueName: \"kubernetes.io/projected/8593fddc-4cf8-4d98-ae2f-c6d0a553c954-kube-api-access-skzxt\") pod \"jobset-controller-manager-56b49d854f-f7d45\" (UID: \"8593fddc-4cf8-4d98-ae2f-c6d0a553c954\") " pod="openshift-jobset-operator/jobset-controller-manager-56b49d854f-f7d45" Oct 30 01:23:14.694978 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:14.694880 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8593fddc-4cf8-4d98-ae2f-c6d0a553c954-manager-config\") pod \"jobset-controller-manager-56b49d854f-f7d45\" (UID: \"8593fddc-4cf8-4d98-ae2f-c6d0a553c954\") " pod="openshift-jobset-operator/jobset-controller-manager-56b49d854f-f7d45" Oct 30 01:23:14.695712 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:14.695681 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8593fddc-4cf8-4d98-ae2f-c6d0a553c954-manager-config\") pod \"jobset-controller-manager-56b49d854f-f7d45\" (UID: \"8593fddc-4cf8-4d98-ae2f-c6d0a553c954\") " pod="openshift-jobset-operator/jobset-controller-manager-56b49d854f-f7d45" Oct 30 01:23:14.697890 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:14.697791 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8593fddc-4cf8-4d98-ae2f-c6d0a553c954-metrics-certs\") pod \"jobset-controller-manager-56b49d854f-f7d45\" (UID: \"8593fddc-4cf8-4d98-ae2f-c6d0a553c954\") " pod="openshift-jobset-operator/jobset-controller-manager-56b49d854f-f7d45" Oct 30 01:23:14.698379 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:14.698354 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8593fddc-4cf8-4d98-ae2f-c6d0a553c954-cert\") pod \"jobset-controller-manager-56b49d854f-f7d45\" (UID: \"8593fddc-4cf8-4d98-ae2f-c6d0a553c954\") " pod="openshift-jobset-operator/jobset-controller-manager-56b49d854f-f7d45" Oct 30 01:23:14.712884 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:14.712832 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skzxt\" (UniqueName: \"kubernetes.io/projected/8593fddc-4cf8-4d98-ae2f-c6d0a553c954-kube-api-access-skzxt\") pod \"jobset-controller-manager-56b49d854f-f7d45\" (UID: \"8593fddc-4cf8-4d98-ae2f-c6d0a553c954\") " pod="openshift-jobset-operator/jobset-controller-manager-56b49d854f-f7d45" Oct 30 01:23:14.796845 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:14.794557 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-56b49d854f-f7d45" Oct 30 01:23:15.086996 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:15.086958 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-56b49d854f-f7d45"] Oct 30 01:23:15.089343 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:23:15.089312 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8593fddc_4cf8_4d98_ae2f_c6d0a553c954.slice/crio-2577d6056d7b6e45976fcf74f8771e03af2b11d9193fc90a5502251aa301962b WatchSource:0}: Error finding container 2577d6056d7b6e45976fcf74f8771e03af2b11d9193fc90a5502251aa301962b: Status 404 returned error can't find the container with id 2577d6056d7b6e45976fcf74f8771e03af2b11d9193fc90a5502251aa301962b Oct 30 01:23:15.418306 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:15.418275 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-56b49d854f-f7d45" event={"ID":"8593fddc-4cf8-4d98-ae2f-c6d0a553c954","Type":"ContainerStarted","Data":"2577d6056d7b6e45976fcf74f8771e03af2b11d9193fc90a5502251aa301962b"} Oct 30 01:23:16.310388 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:16.310345 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:16.491984 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:16.491945 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:16.907086 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:16.907045 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:17.092220 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:17.092181 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:19.441769 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:19.441708 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-56b49d854f-f7d45" event={"ID":"8593fddc-4cf8-4d98-ae2f-c6d0a553c954","Type":"ContainerStarted","Data":"a8d14fa837c30ab136af20dcb777b55a41bdf6514c7e8860254518cdf763c3e8"} Oct 30 01:23:19.442267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:19.441802 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-56b49d854f-f7d45" Oct 30 01:23:19.476092 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:19.476029 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-56b49d854f-f7d45" podStartSLOduration=1.745042691 podStartE2EDuration="5.476009264s" totalImagesPullingTime="3.730966573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:19.475033179 +0000 UTC m=+325.051458595" watchObservedRunningTime="2026-10-30 01:23:19.476009264 +0000 UTC m=+325.052434668" Oct 30 01:23:20.335368 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:20.335336 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hf2jx" Oct 30 01:23:20.405818 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:20.405743 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:20.609781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:20.609672 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:24.489407 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:24.489365 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:24.676073 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:24.676034 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:25.317253 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:25.317216 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:30.454597 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:30.454559 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-56b49d854f-f7d45" Oct 30 01:23:44.686936 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:23:44.686895 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:23:44.781053 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:44.781013 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:45.018726 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:45.018632 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:45.254712 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:45.254663 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:48.634022 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:48.633981 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:48.864994 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:48.864960 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:51.851898 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:51.851862 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:52.073440 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:52.073385 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:52.289060 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:52.289012 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:52.576315 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:52.576216 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:53.415111 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:53.415075 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:53.679954 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:53.679862 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:53.979560 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:53.979472 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:54.469951 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:54.469919 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:54.660464 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:54.660421 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:54.863919 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:23:54.863826 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:02.480425 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:02.480386 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:03.525328 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:03.525243 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:10.209642 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:10.209601 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:14.686535 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:24:14.686478 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:24:18.349509 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:18.349468 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:18.586588 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:18.586551 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:19.836402 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:19.836364 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:20.041492 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:20.041452 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:29.484201 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:29.484158 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:29.682426 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:29.682388 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-w6w9n"] Oct 30 01:24:29.686091 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:29.686066 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-w6w9n" Oct 30 01:24:29.692615 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:29.692571 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:29.701832 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:29.701796 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-w6w9n"] Oct 30 01:24:29.709732 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:29.709698 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-q9rn8\"" Oct 30 01:24:29.760900 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:29.760808 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2pv9\" (UniqueName: \"kubernetes.io/projected/fadcc370-28b6-45e4-8124-841f69ad119e-kube-api-access-b2pv9\") pod \"authorino-6f6944bcdd-w6w9n\" (UID: \"fadcc370-28b6-45e4-8124-841f69ad119e\") " pod="kuadrant-system/authorino-6f6944bcdd-w6w9n" Oct 30 01:24:29.858400 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:29.858363 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-w6w9n"] Oct 30 01:24:29.858591 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:24:29.858573 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-b2pv9], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-w6w9n" podUID="fadcc370-28b6-45e4-8124-841f69ad119e" Oct 30 01:24:29.861906 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:29.861876 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2pv9\" (UniqueName: \"kubernetes.io/projected/fadcc370-28b6-45e4-8124-841f69ad119e-kube-api-access-b2pv9\") pod \"authorino-6f6944bcdd-w6w9n\" (UID: \"fadcc370-28b6-45e4-8124-841f69ad119e\") " pod="kuadrant-system/authorino-6f6944bcdd-w6w9n" Oct 30 01:24:29.884240 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:29.884204 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2pv9\" (UniqueName: \"kubernetes.io/projected/fadcc370-28b6-45e4-8124-841f69ad119e-kube-api-access-b2pv9\") pod \"authorino-6f6944bcdd-w6w9n\" (UID: \"fadcc370-28b6-45e4-8124-841f69ad119e\") " pod="kuadrant-system/authorino-6f6944bcdd-w6w9n" Oct 30 01:24:30.066212 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:30.066122 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59bfd57dc7-xnhkc"] Oct 30 01:24:30.069806 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:30.069777 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59bfd57dc7-xnhkc" Oct 30 01:24:30.087163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:30.086862 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 01:24:30.089052 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:30.089024 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59bfd57dc7-xnhkc"] Oct 30 01:24:30.165180 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:30.165138 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/407180e4-1f73-42c3-b39c-d92378649332-tls-cert\") pod \"authorino-59bfd57dc7-xnhkc\" (UID: \"407180e4-1f73-42c3-b39c-d92378649332\") " pod="kuadrant-system/authorino-59bfd57dc7-xnhkc" Oct 30 01:24:30.165374 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:30.165207 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftr99\" (UniqueName: \"kubernetes.io/projected/407180e4-1f73-42c3-b39c-d92378649332-kube-api-access-ftr99\") pod \"authorino-59bfd57dc7-xnhkc\" (UID: \"407180e4-1f73-42c3-b39c-d92378649332\") " pod="kuadrant-system/authorino-59bfd57dc7-xnhkc" Oct 30 01:24:30.266506 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:30.266469 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/407180e4-1f73-42c3-b39c-d92378649332-tls-cert\") pod \"authorino-59bfd57dc7-xnhkc\" (UID: \"407180e4-1f73-42c3-b39c-d92378649332\") " pod="kuadrant-system/authorino-59bfd57dc7-xnhkc" Oct 30 01:24:30.266690 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:30.266525 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftr99\" (UniqueName: \"kubernetes.io/projected/407180e4-1f73-42c3-b39c-d92378649332-kube-api-access-ftr99\") pod \"authorino-59bfd57dc7-xnhkc\" (UID: \"407180e4-1f73-42c3-b39c-d92378649332\") " pod="kuadrant-system/authorino-59bfd57dc7-xnhkc" Oct 30 01:24:30.269055 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:30.269028 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/407180e4-1f73-42c3-b39c-d92378649332-tls-cert\") pod \"authorino-59bfd57dc7-xnhkc\" (UID: \"407180e4-1f73-42c3-b39c-d92378649332\") " pod="kuadrant-system/authorino-59bfd57dc7-xnhkc" Oct 30 01:24:30.289457 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:30.289420 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftr99\" (UniqueName: \"kubernetes.io/projected/407180e4-1f73-42c3-b39c-d92378649332-kube-api-access-ftr99\") pod \"authorino-59bfd57dc7-xnhkc\" (UID: \"407180e4-1f73-42c3-b39c-d92378649332\") " pod="kuadrant-system/authorino-59bfd57dc7-xnhkc" Oct 30 01:24:30.381741 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:30.381652 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59bfd57dc7-xnhkc" Oct 30 01:24:30.676303 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:30.676267 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59bfd57dc7-xnhkc"] Oct 30 01:24:30.679031 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:24:30.678989 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod407180e4_1f73_42c3_b39c_d92378649332.slice/crio-81616d1f66e7e449753dada95fc010e3c49d0dd60ec8a6ae7135a68fa143f276 WatchSource:0}: Error finding container 81616d1f66e7e449753dada95fc010e3c49d0dd60ec8a6ae7135a68fa143f276: Status 404 returned error can't find the container with id 81616d1f66e7e449753dada95fc010e3c49d0dd60ec8a6ae7135a68fa143f276 Oct 30 01:24:30.814838 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:30.814801 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59bfd57dc7-xnhkc" event={"ID":"407180e4-1f73-42c3-b39c-d92378649332","Type":"ContainerStarted","Data":"81616d1f66e7e449753dada95fc010e3c49d0dd60ec8a6ae7135a68fa143f276"} Oct 30 01:24:30.814838 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:30.814814 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-w6w9n" Oct 30 01:24:30.823110 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:30.823084 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-w6w9n" Oct 30 01:24:30.872682 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:30.872650 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fadcc370-28b6-45e4-8124-841f69ad119e-kube-api-access-b2pv9\" (UniqueName: \"kubernetes.io/projected/fadcc370-28b6-45e4-8124-841f69ad119e-kube-api-access-b2pv9\") pod \"fadcc370-28b6-45e4-8124-841f69ad119e\" (UID: \"fadcc370-28b6-45e4-8124-841f69ad119e\") " Oct 30 01:24:30.874692 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:30.874663 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fadcc370-28b6-45e4-8124-841f69ad119e-kube-api-access-b2pv9" pod "fadcc370-28b6-45e4-8124-841f69ad119e" (UID: "fadcc370-28b6-45e4-8124-841f69ad119e"). InnerVolumeSpecName "kube-api-access-b2pv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:30.973628 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:30.973532 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2pv9\" (UniqueName: \"kubernetes.io/projected/fadcc370-28b6-45e4-8124-841f69ad119e-kube-api-access-b2pv9\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:24:31.824534 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:31.824489 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59bfd57dc7-xnhkc" event={"ID":"407180e4-1f73-42c3-b39c-d92378649332","Type":"ContainerStarted","Data":"d19189bbdf2a700bf7fdd9aed2220aaa969c153ed99a3b4f7dbc3555a0da05d9"} Oct 30 01:24:31.824959 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:31.824538 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-w6w9n" Oct 30 01:24:31.862367 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:31.862309 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-59bfd57dc7-xnhkc" podStartSLOduration=1.257941961 podStartE2EDuration="1.862295057s" totalImagesPullingTime="604.353096ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:24:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:24:31.860487275 +0000 UTC m=+397.436912677" watchObservedRunningTime="2026-10-30 01:24:31.862295057 +0000 UTC m=+397.438720515" Oct 30 01:24:31.940788 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:31.940734 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-w6w9n"] Oct 30 01:24:31.951354 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:31.951316 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-w6w9n"] Oct 30 01:24:32.540263 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:32.540229 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fadcc370-28b6-45e4-8124-841f69ad119e" path="/var/lib/kubelet/pods/fadcc370-28b6-45e4-8124-841f69ad119e/volumes" Oct 30 01:24:36.832103 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:24:36.832069 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:44.686722 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:24:44.686683 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:25:02.594686 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:25:02.594640 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:14.686884 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:25:14.686845 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:25:26.484969 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:25:26.484929 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:26.777562 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:25:26.777456 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:29.701123 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:25:29.701083 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:29.892698 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:25:29.892659 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:44.687294 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:25:44.687209 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:25:55.046288 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:25:55.046249 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:55.511219 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:25:55.511185 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:56.736743 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:25:56.736703 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:56.945363 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:25:56.945310 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:01.252106 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:01.252071 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:02.697852 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:02.697812 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:03.586122 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:03.586082 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:04.956894 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:04.956855 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-clwkt"] Oct 30 01:26:04.960908 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:04.960886 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-clwkt" Oct 30 01:26:04.974468 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:04.974436 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 01:26:04.987504 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:04.987469 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-clwkt"] Oct 30 01:26:04.987672 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:04.987593 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 01:26:04.988187 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:04.988161 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xzxn7\"" Oct 30 01:26:04.988311 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:04.988237 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 01:26:04.990526 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:04.990501 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c92a422d-2da3-4ce1-9155-a23b8eed9c90-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-clwkt\" (UID: \"c92a422d-2da3-4ce1-9155-a23b8eed9c90\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-clwkt" Oct 30 01:26:04.990634 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:04.990540 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c92a422d-2da3-4ce1-9155-a23b8eed9c90-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-clwkt\" (UID: \"c92a422d-2da3-4ce1-9155-a23b8eed9c90\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-clwkt" Oct 30 01:26:04.990634 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:04.990626 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c92a422d-2da3-4ce1-9155-a23b8eed9c90-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-clwkt\" (UID: \"c92a422d-2da3-4ce1-9155-a23b8eed9c90\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-clwkt" Oct 30 01:26:04.990765 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:04.990687 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c92a422d-2da3-4ce1-9155-a23b8eed9c90-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-clwkt\" (UID: \"c92a422d-2da3-4ce1-9155-a23b8eed9c90\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-clwkt" Oct 30 01:26:04.990765 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:04.990722 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pgk7\" (UniqueName: \"kubernetes.io/projected/c92a422d-2da3-4ce1-9155-a23b8eed9c90-kube-api-access-7pgk7\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-clwkt\" (UID: \"c92a422d-2da3-4ce1-9155-a23b8eed9c90\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-clwkt" Oct 30 01:26:04.990878 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:04.990766 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c92a422d-2da3-4ce1-9155-a23b8eed9c90-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-clwkt\" (UID: \"c92a422d-2da3-4ce1-9155-a23b8eed9c90\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-clwkt" Oct 30 01:26:05.091268 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:05.091225 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c92a422d-2da3-4ce1-9155-a23b8eed9c90-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-clwkt\" (UID: \"c92a422d-2da3-4ce1-9155-a23b8eed9c90\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-clwkt" Oct 30 01:26:05.091465 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:05.091282 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c92a422d-2da3-4ce1-9155-a23b8eed9c90-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-clwkt\" (UID: \"c92a422d-2da3-4ce1-9155-a23b8eed9c90\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-clwkt" Oct 30 01:26:05.091465 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:05.091315 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c92a422d-2da3-4ce1-9155-a23b8eed9c90-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-clwkt\" (UID: \"c92a422d-2da3-4ce1-9155-a23b8eed9c90\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-clwkt" Oct 30 01:26:05.091465 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:05.091362 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c92a422d-2da3-4ce1-9155-a23b8eed9c90-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-clwkt\" (UID: \"c92a422d-2da3-4ce1-9155-a23b8eed9c90\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-clwkt" Oct 30 01:26:05.091618 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:05.091499 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c92a422d-2da3-4ce1-9155-a23b8eed9c90-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-clwkt\" (UID: \"c92a422d-2da3-4ce1-9155-a23b8eed9c90\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-clwkt" Oct 30 01:26:05.091618 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:05.091562 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pgk7\" (UniqueName: \"kubernetes.io/projected/c92a422d-2da3-4ce1-9155-a23b8eed9c90-kube-api-access-7pgk7\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-clwkt\" (UID: \"c92a422d-2da3-4ce1-9155-a23b8eed9c90\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-clwkt" Oct 30 01:26:05.091823 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:05.091797 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c92a422d-2da3-4ce1-9155-a23b8eed9c90-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-clwkt\" (UID: \"c92a422d-2da3-4ce1-9155-a23b8eed9c90\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-clwkt" Oct 30 01:26:05.091915 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:05.091822 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c92a422d-2da3-4ce1-9155-a23b8eed9c90-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-clwkt\" (UID: \"c92a422d-2da3-4ce1-9155-a23b8eed9c90\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-clwkt" Oct 30 01:26:05.091915 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:05.091858 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c92a422d-2da3-4ce1-9155-a23b8eed9c90-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-clwkt\" (UID: \"c92a422d-2da3-4ce1-9155-a23b8eed9c90\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-clwkt" Oct 30 01:26:05.093785 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:05.093737 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c92a422d-2da3-4ce1-9155-a23b8eed9c90-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-clwkt\" (UID: \"c92a422d-2da3-4ce1-9155-a23b8eed9c90\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-clwkt" Oct 30 01:26:05.094017 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:05.093999 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c92a422d-2da3-4ce1-9155-a23b8eed9c90-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-clwkt\" (UID: \"c92a422d-2da3-4ce1-9155-a23b8eed9c90\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-clwkt" Oct 30 01:26:05.108834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:05.108793 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pgk7\" (UniqueName: \"kubernetes.io/projected/c92a422d-2da3-4ce1-9155-a23b8eed9c90-kube-api-access-7pgk7\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-clwkt\" (UID: \"c92a422d-2da3-4ce1-9155-a23b8eed9c90\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-clwkt" Oct 30 01:26:05.114672 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:05.114637 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:05.279523 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:05.279431 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-clwkt" Oct 30 01:26:05.577894 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:05.577859 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-clwkt"] Oct 30 01:26:05.580678 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:26:05.580649 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc92a422d_2da3_4ce1_9155_a23b8eed9c90.slice/crio-282ed3508c84d3b9d864dba428c801a0ade2eb8408f87197fc7365ffd190d136 WatchSource:0}: Error finding container 282ed3508c84d3b9d864dba428c801a0ade2eb8408f87197fc7365ffd190d136: Status 404 returned error can't find the container with id 282ed3508c84d3b9d864dba428c801a0ade2eb8408f87197fc7365ffd190d136 Oct 30 01:26:06.328372 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:06.328332 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-clwkt" event={"ID":"c92a422d-2da3-4ce1-9155-a23b8eed9c90","Type":"ContainerStarted","Data":"282ed3508c84d3b9d864dba428c801a0ade2eb8408f87197fc7365ffd190d136"} Oct 30 01:26:06.340885 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:06.340850 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:07.335481 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:07.335435 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-clwkt" event={"ID":"c92a422d-2da3-4ce1-9155-a23b8eed9c90","Type":"ContainerStarted","Data":"76581c5b4bd066c891e147b113ceae81ba0cacc2ea7cd2fb70257ef253eca996"} Oct 30 01:26:07.371861 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:07.371799 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-clwkt" podStartSLOduration=2.251837476 podStartE2EDuration="3.371778524s" totalImagesPullingTime="1.119941048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:07.371643114 +0000 UTC m=+492.948068529" watchObservedRunningTime="2026-10-30 01:26:07.371778524 +0000 UTC m=+492.948203940" Oct 30 01:26:08.159716 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:08.159678 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:08.279925 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:08.279886 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-clwkt" Oct 30 01:26:11.646607 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:11.646566 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-dnvm5"] Oct 30 01:26:11.650044 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:11.650020 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-dnvm5" Oct 30 01:26:11.673364 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:11.673339 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 30 01:26:11.676421 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:11.676389 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-dnvm5"] Oct 30 01:26:11.753488 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:11.753441 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24fa6193-db5f-447d-9c2d-3e62c06e59de-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-dnvm5\" (UID: \"24fa6193-db5f-447d-9c2d-3e62c06e59de\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dnvm5" Oct 30 01:26:11.753661 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:11.753499 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24fa6193-db5f-447d-9c2d-3e62c06e59de-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-dnvm5\" (UID: \"24fa6193-db5f-447d-9c2d-3e62c06e59de\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dnvm5" Oct 30 01:26:11.753661 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:11.753600 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24fa6193-db5f-447d-9c2d-3e62c06e59de-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-dnvm5\" (UID: \"24fa6193-db5f-447d-9c2d-3e62c06e59de\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dnvm5" Oct 30 01:26:11.753661 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:11.753652 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24fa6193-db5f-447d-9c2d-3e62c06e59de-home\") pod \"echo-simulator-kserve-64b6f8f96c-dnvm5\" (UID: \"24fa6193-db5f-447d-9c2d-3e62c06e59de\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dnvm5" Oct 30 01:26:11.753815 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:11.753760 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24fa6193-db5f-447d-9c2d-3e62c06e59de-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-dnvm5\" (UID: \"24fa6193-db5f-447d-9c2d-3e62c06e59de\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dnvm5" Oct 30 01:26:11.753815 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:11.753798 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5xn6\" (UniqueName: \"kubernetes.io/projected/24fa6193-db5f-447d-9c2d-3e62c06e59de-kube-api-access-k5xn6\") pod \"echo-simulator-kserve-64b6f8f96c-dnvm5\" (UID: \"24fa6193-db5f-447d-9c2d-3e62c06e59de\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dnvm5" Oct 30 01:26:11.798793 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:11.798730 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:11.854542 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:11.854503 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24fa6193-db5f-447d-9c2d-3e62c06e59de-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-dnvm5\" (UID: \"24fa6193-db5f-447d-9c2d-3e62c06e59de\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dnvm5" Oct 30 01:26:11.854731 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:11.854555 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24fa6193-db5f-447d-9c2d-3e62c06e59de-home\") pod \"echo-simulator-kserve-64b6f8f96c-dnvm5\" (UID: \"24fa6193-db5f-447d-9c2d-3e62c06e59de\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dnvm5" Oct 30 01:26:11.854731 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:11.854593 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24fa6193-db5f-447d-9c2d-3e62c06e59de-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-dnvm5\" (UID: \"24fa6193-db5f-447d-9c2d-3e62c06e59de\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dnvm5" Oct 30 01:26:11.854731 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:11.854620 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5xn6\" (UniqueName: \"kubernetes.io/projected/24fa6193-db5f-447d-9c2d-3e62c06e59de-kube-api-access-k5xn6\") pod \"echo-simulator-kserve-64b6f8f96c-dnvm5\" (UID: \"24fa6193-db5f-447d-9c2d-3e62c06e59de\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dnvm5" Oct 30 01:26:11.854731 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:11.854669 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24fa6193-db5f-447d-9c2d-3e62c06e59de-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-dnvm5\" (UID: \"24fa6193-db5f-447d-9c2d-3e62c06e59de\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dnvm5" Oct 30 01:26:11.854731 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:11.854717 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24fa6193-db5f-447d-9c2d-3e62c06e59de-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-dnvm5\" (UID: \"24fa6193-db5f-447d-9c2d-3e62c06e59de\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dnvm5" Oct 30 01:26:11.855083 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:11.854991 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24fa6193-db5f-447d-9c2d-3e62c06e59de-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-dnvm5\" (UID: \"24fa6193-db5f-447d-9c2d-3e62c06e59de\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dnvm5" Oct 30 01:26:11.855176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:11.855157 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24fa6193-db5f-447d-9c2d-3e62c06e59de-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-dnvm5\" (UID: \"24fa6193-db5f-447d-9c2d-3e62c06e59de\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dnvm5" Oct 30 01:26:11.855312 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:11.855284 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24fa6193-db5f-447d-9c2d-3e62c06e59de-home\") pod \"echo-simulator-kserve-64b6f8f96c-dnvm5\" (UID: \"24fa6193-db5f-447d-9c2d-3e62c06e59de\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dnvm5" Oct 30 01:26:11.857055 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:11.857029 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24fa6193-db5f-447d-9c2d-3e62c06e59de-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-dnvm5\" (UID: \"24fa6193-db5f-447d-9c2d-3e62c06e59de\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dnvm5" Oct 30 01:26:11.857364 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:11.857346 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24fa6193-db5f-447d-9c2d-3e62c06e59de-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-dnvm5\" (UID: \"24fa6193-db5f-447d-9c2d-3e62c06e59de\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dnvm5" Oct 30 01:26:11.871373 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:11.871339 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5xn6\" (UniqueName: \"kubernetes.io/projected/24fa6193-db5f-447d-9c2d-3e62c06e59de-kube-api-access-k5xn6\") pod \"echo-simulator-kserve-64b6f8f96c-dnvm5\" (UID: \"24fa6193-db5f-447d-9c2d-3e62c06e59de\") " pod="llm/echo-simulator-kserve-64b6f8f96c-dnvm5" Oct 30 01:26:11.964641 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:11.964603 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-dnvm5" Oct 30 01:26:12.257425 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:12.257394 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-dnvm5"] Oct 30 01:26:12.259237 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:26:12.259206 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24fa6193_db5f_447d_9c2d_3e62c06e59de.slice/crio-9afcca1b8a60ea3fa51b9b2c61b1fd0d14505ec5417b82f065612ff452806941 WatchSource:0}: Error finding container 9afcca1b8a60ea3fa51b9b2c61b1fd0d14505ec5417b82f065612ff452806941: Status 404 returned error can't find the container with id 9afcca1b8a60ea3fa51b9b2c61b1fd0d14505ec5417b82f065612ff452806941 Oct 30 01:26:12.362927 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:12.362890 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-dnvm5" event={"ID":"24fa6193-db5f-447d-9c2d-3e62c06e59de","Type":"ContainerStarted","Data":"9afcca1b8a60ea3fa51b9b2c61b1fd0d14505ec5417b82f065612ff452806941"} Oct 30 01:26:13.439889 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:13.439851 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f"] Oct 30 01:26:13.444540 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:13.444507 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f" Oct 30 01:26:13.459341 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:13.459269 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f"] Oct 30 01:26:13.465924 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:13.465896 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 01:26:13.470367 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:13.470321 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02ae87f3-5a71-4a60-af7b-98965dfd5862-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f\" (UID: \"02ae87f3-5a71-4a60-af7b-98965dfd5862\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f" Oct 30 01:26:13.470543 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:13.470375 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02ae87f3-5a71-4a60-af7b-98965dfd5862-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f\" (UID: \"02ae87f3-5a71-4a60-af7b-98965dfd5862\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f" Oct 30 01:26:13.470543 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:13.470432 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4vts\" (UniqueName: \"kubernetes.io/projected/02ae87f3-5a71-4a60-af7b-98965dfd5862-kube-api-access-k4vts\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f\" (UID: \"02ae87f3-5a71-4a60-af7b-98965dfd5862\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f" Oct 30 01:26:13.470543 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:13.470505 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02ae87f3-5a71-4a60-af7b-98965dfd5862-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f\" (UID: \"02ae87f3-5a71-4a60-af7b-98965dfd5862\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f" Oct 30 01:26:13.470708 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:13.470550 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02ae87f3-5a71-4a60-af7b-98965dfd5862-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f\" (UID: \"02ae87f3-5a71-4a60-af7b-98965dfd5862\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f" Oct 30 01:26:13.470708 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:13.470586 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02ae87f3-5a71-4a60-af7b-98965dfd5862-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f\" (UID: \"02ae87f3-5a71-4a60-af7b-98965dfd5862\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f" Oct 30 01:26:13.571637 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:13.571594 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02ae87f3-5a71-4a60-af7b-98965dfd5862-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f\" (UID: \"02ae87f3-5a71-4a60-af7b-98965dfd5862\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f" Oct 30 01:26:13.571855 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:13.571665 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02ae87f3-5a71-4a60-af7b-98965dfd5862-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f\" (UID: \"02ae87f3-5a71-4a60-af7b-98965dfd5862\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f" Oct 30 01:26:13.571855 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:13.571704 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02ae87f3-5a71-4a60-af7b-98965dfd5862-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f\" (UID: \"02ae87f3-5a71-4a60-af7b-98965dfd5862\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f" Oct 30 01:26:13.571855 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:13.571769 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02ae87f3-5a71-4a60-af7b-98965dfd5862-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f\" (UID: \"02ae87f3-5a71-4a60-af7b-98965dfd5862\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f" Oct 30 01:26:13.571855 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:13.571804 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02ae87f3-5a71-4a60-af7b-98965dfd5862-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f\" (UID: \"02ae87f3-5a71-4a60-af7b-98965dfd5862\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f" Oct 30 01:26:13.572053 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:13.571859 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4vts\" (UniqueName: \"kubernetes.io/projected/02ae87f3-5a71-4a60-af7b-98965dfd5862-kube-api-access-k4vts\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f\" (UID: \"02ae87f3-5a71-4a60-af7b-98965dfd5862\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f" Oct 30 01:26:13.572163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:13.572142 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02ae87f3-5a71-4a60-af7b-98965dfd5862-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f\" (UID: \"02ae87f3-5a71-4a60-af7b-98965dfd5862\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f" Oct 30 01:26:13.572234 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:13.572209 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02ae87f3-5a71-4a60-af7b-98965dfd5862-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f\" (UID: \"02ae87f3-5a71-4a60-af7b-98965dfd5862\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f" Oct 30 01:26:13.572493 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:13.572471 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02ae87f3-5a71-4a60-af7b-98965dfd5862-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f\" (UID: \"02ae87f3-5a71-4a60-af7b-98965dfd5862\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f" Oct 30 01:26:13.574818 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:13.574785 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02ae87f3-5a71-4a60-af7b-98965dfd5862-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f\" (UID: \"02ae87f3-5a71-4a60-af7b-98965dfd5862\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f" Oct 30 01:26:13.574999 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:13.574973 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02ae87f3-5a71-4a60-af7b-98965dfd5862-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f\" (UID: \"02ae87f3-5a71-4a60-af7b-98965dfd5862\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f" Oct 30 01:26:13.593316 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:13.593255 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:13.595066 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:13.595037 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4vts\" (UniqueName: \"kubernetes.io/projected/02ae87f3-5a71-4a60-af7b-98965dfd5862-kube-api-access-k4vts\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f\" (UID: \"02ae87f3-5a71-4a60-af7b-98965dfd5862\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f" Oct 30 01:26:13.784820 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:13.782948 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f" Oct 30 01:26:14.081906 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:14.081873 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f"] Oct 30 01:26:14.084255 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:26:14.084223 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02ae87f3_5a71_4a60_af7b_98965dfd5862.slice/crio-df6a67bbeba6b795ec26edbf3c5aebea8e684bf3d78e9c6ce209ef3ac9819852 WatchSource:0}: Error finding container df6a67bbeba6b795ec26edbf3c5aebea8e684bf3d78e9c6ce209ef3ac9819852: Status 404 returned error can't find the container with id df6a67bbeba6b795ec26edbf3c5aebea8e684bf3d78e9c6ce209ef3ac9819852 Oct 30 01:26:14.377768 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:14.377662 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-dnvm5" event={"ID":"24fa6193-db5f-447d-9c2d-3e62c06e59de","Type":"ContainerStarted","Data":"9fc8354935cb2946d1785c59df345b3d665507584c50127f93e83b97e7a6f016"} Oct 30 01:26:14.379423 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:14.379395 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f" event={"ID":"02ae87f3-5a71-4a60-af7b-98965dfd5862","Type":"ContainerStarted","Data":"033adffb00fae7ba6090cf8334736fac00cb4d33c8699c47c546b0692e8aeba8"} Oct 30 01:26:14.379570 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:14.379447 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f" event={"ID":"02ae87f3-5a71-4a60-af7b-98965dfd5862","Type":"ContainerStarted","Data":"df6a67bbeba6b795ec26edbf3c5aebea8e684bf3d78e9c6ce209ef3ac9819852"} Oct 30 01:26:14.413126 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:14.413070 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-dnvm5" podStartSLOduration=1.762075793 podStartE2EDuration="3.413052925s" totalImagesPullingTime="1.650977132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:14.411159458 +0000 UTC m=+499.987584860" watchObservedRunningTime="2026-10-30 01:26:14.413052925 +0000 UTC m=+499.989478327" Oct 30 01:26:14.445622 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:14.445564 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f" podStartSLOduration=1.264384193 podStartE2EDuration="1.445549151s" totalImagesPullingTime="181.164958ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:14.44440212 +0000 UTC m=+500.020827525" watchObservedRunningTime="2026-10-30 01:26:14.445549151 +0000 UTC m=+500.021974553" Oct 30 01:26:14.688100 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:26:14.688061 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:26:14.784088 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:14.784056 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f" Oct 30 01:26:14.965737 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:14.965641 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-dnvm5" Oct 30 01:26:15.280299 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:15.280215 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-clwkt" Oct 30 01:26:15.293412 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:15.293383 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-clwkt" Oct 30 01:26:15.395965 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:15.395924 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-clwkt" Oct 30 01:26:16.207828 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:16.207789 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:21.965047 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:21.964999 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-dnvm5" Oct 30 01:26:21.977895 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:21.977855 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-dnvm5" Oct 30 01:26:22.442869 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:22.442827 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-dnvm5" Oct 30 01:26:23.784805 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:23.784734 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f" Oct 30 01:26:23.797338 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:23.797308 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f" Oct 30 01:26:24.454121 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:24.454083 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f" Oct 30 01:26:31.295453 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:31.295401 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-f9c56f786-455r7"] Oct 30 01:26:31.301798 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:31.301773 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f9c56f786-455r7" Oct 30 01:26:31.314608 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:31.314575 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f9c56f786-455r7"] Oct 30 01:26:31.460399 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:31.460358 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3d30861b-c70b-4c15-ac36-8ff66d801f09-tls-cert\") pod \"authorino-f9c56f786-455r7\" (UID: \"3d30861b-c70b-4c15-ac36-8ff66d801f09\") " pod="kuadrant-system/authorino-f9c56f786-455r7" Oct 30 01:26:31.460399 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:31.460404 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6pjg\" (UniqueName: \"kubernetes.io/projected/3d30861b-c70b-4c15-ac36-8ff66d801f09-kube-api-access-x6pjg\") pod \"authorino-f9c56f786-455r7\" (UID: \"3d30861b-c70b-4c15-ac36-8ff66d801f09\") " pod="kuadrant-system/authorino-f9c56f786-455r7" Oct 30 01:26:31.561428 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:31.561330 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3d30861b-c70b-4c15-ac36-8ff66d801f09-tls-cert\") pod \"authorino-f9c56f786-455r7\" (UID: \"3d30861b-c70b-4c15-ac36-8ff66d801f09\") " pod="kuadrant-system/authorino-f9c56f786-455r7" Oct 30 01:26:31.561428 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:31.561379 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6pjg\" (UniqueName: \"kubernetes.io/projected/3d30861b-c70b-4c15-ac36-8ff66d801f09-kube-api-access-x6pjg\") pod \"authorino-f9c56f786-455r7\" (UID: \"3d30861b-c70b-4c15-ac36-8ff66d801f09\") " pod="kuadrant-system/authorino-f9c56f786-455r7" Oct 30 01:26:31.564034 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:31.564006 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3d30861b-c70b-4c15-ac36-8ff66d801f09-tls-cert\") pod \"authorino-f9c56f786-455r7\" (UID: \"3d30861b-c70b-4c15-ac36-8ff66d801f09\") " pod="kuadrant-system/authorino-f9c56f786-455r7" Oct 30 01:26:31.578364 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:31.578324 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6pjg\" (UniqueName: \"kubernetes.io/projected/3d30861b-c70b-4c15-ac36-8ff66d801f09-kube-api-access-x6pjg\") pod \"authorino-f9c56f786-455r7\" (UID: \"3d30861b-c70b-4c15-ac36-8ff66d801f09\") " pod="kuadrant-system/authorino-f9c56f786-455r7" Oct 30 01:26:31.615152 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:31.615105 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f9c56f786-455r7" Oct 30 01:26:31.906852 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:31.906811 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f9c56f786-455r7"] Oct 30 01:26:31.909350 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:26:31.909305 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d30861b_c70b_4c15_ac36_8ff66d801f09.slice/crio-7e3c6e9d15f52c3e112429d6e42257bf7b4ec209ea3797a61387ac6054ef408f WatchSource:0}: Error finding container 7e3c6e9d15f52c3e112429d6e42257bf7b4ec209ea3797a61387ac6054ef408f: Status 404 returned error can't find the container with id 7e3c6e9d15f52c3e112429d6e42257bf7b4ec209ea3797a61387ac6054ef408f Oct 30 01:26:31.910860 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:31.910838 6572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:26:32.487845 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:32.487804 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f9c56f786-455r7" event={"ID":"3d30861b-c70b-4c15-ac36-8ff66d801f09","Type":"ContainerStarted","Data":"7e3c6e9d15f52c3e112429d6e42257bf7b4ec209ea3797a61387ac6054ef408f"} Oct 30 01:26:33.495106 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:33.495053 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f9c56f786-455r7" event={"ID":"3d30861b-c70b-4c15-ac36-8ff66d801f09","Type":"ContainerStarted","Data":"ff85962880afe8810485471b00727716b48242c8d7c2eddc58267761239b75e5"} Oct 30 01:26:33.530897 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:33.530831 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-f9c56f786-455r7" podStartSLOduration=1.7599329689999998 podStartE2EDuration="2.530813667s" totalImagesPullingTime="770.880698ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:33.528782786 +0000 UTC m=+519.105208190" watchObservedRunningTime="2026-10-30 01:26:33.530813667 +0000 UTC m=+519.107239070" Oct 30 01:26:33.565512 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:33.565465 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59bfd57dc7-xnhkc"] Oct 30 01:26:33.565770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:33.565727 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-59bfd57dc7-xnhkc" podUID="407180e4-1f73-42c3-b39c-d92378649332" containerName="authorino" containerID="cri-o://d19189bbdf2a700bf7fdd9aed2220aaa969c153ed99a3b4f7dbc3555a0da05d9" gracePeriod=30 Oct 30 01:26:33.834161 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:33.834129 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59bfd57dc7-xnhkc" Oct 30 01:26:33.989850 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:33.989802 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/407180e4-1f73-42c3-b39c-d92378649332-kube-api-access-ftr99\" (UniqueName: \"kubernetes.io/projected/407180e4-1f73-42c3-b39c-d92378649332-kube-api-access-ftr99\") pod \"407180e4-1f73-42c3-b39c-d92378649332\" (UID: \"407180e4-1f73-42c3-b39c-d92378649332\") " Oct 30 01:26:33.990033 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:33.989896 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/407180e4-1f73-42c3-b39c-d92378649332-tls-cert\" (UniqueName: \"kubernetes.io/secret/407180e4-1f73-42c3-b39c-d92378649332-tls-cert\") pod \"407180e4-1f73-42c3-b39c-d92378649332\" (UID: \"407180e4-1f73-42c3-b39c-d92378649332\") " Oct 30 01:26:33.992096 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:33.992055 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/407180e4-1f73-42c3-b39c-d92378649332-kube-api-access-ftr99" pod "407180e4-1f73-42c3-b39c-d92378649332" (UID: "407180e4-1f73-42c3-b39c-d92378649332"). InnerVolumeSpecName "kube-api-access-ftr99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:26:34.001094 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:34.001050 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/407180e4-1f73-42c3-b39c-d92378649332-tls-cert" pod "407180e4-1f73-42c3-b39c-d92378649332" (UID: "407180e4-1f73-42c3-b39c-d92378649332"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:26:34.091408 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:34.091313 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftr99\" (UniqueName: \"kubernetes.io/projected/407180e4-1f73-42c3-b39c-d92378649332-kube-api-access-ftr99\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:26:34.091408 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:34.091352 6572 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/407180e4-1f73-42c3-b39c-d92378649332-tls-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:26:34.502222 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:34.502187 6572 generic.go:356] "Generic (PLEG): container finished" podID="407180e4-1f73-42c3-b39c-d92378649332" containerID="d19189bbdf2a700bf7fdd9aed2220aaa969c153ed99a3b4f7dbc3555a0da05d9" exitCode=0 Oct 30 01:26:34.502695 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:34.502238 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59bfd57dc7-xnhkc" Oct 30 01:26:34.502695 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:34.502272 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59bfd57dc7-xnhkc" event={"ID":"407180e4-1f73-42c3-b39c-d92378649332","Type":"ContainerDied","Data":"d19189bbdf2a700bf7fdd9aed2220aaa969c153ed99a3b4f7dbc3555a0da05d9"} Oct 30 01:26:34.502695 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:34.502319 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59bfd57dc7-xnhkc" event={"ID":"407180e4-1f73-42c3-b39c-d92378649332","Type":"ContainerDied","Data":"81616d1f66e7e449753dada95fc010e3c49d0dd60ec8a6ae7135a68fa143f276"} Oct 30 01:26:34.502695 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:34.502342 6572 scope.go:122] "RemoveContainer" containerID="d19189bbdf2a700bf7fdd9aed2220aaa969c153ed99a3b4f7dbc3555a0da05d9" Oct 30 01:26:34.514710 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:34.514682 6572 scope.go:122] "RemoveContainer" containerID="d19189bbdf2a700bf7fdd9aed2220aaa969c153ed99a3b4f7dbc3555a0da05d9" Oct 30 01:26:34.515078 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:26:34.515056 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d19189bbdf2a700bf7fdd9aed2220aaa969c153ed99a3b4f7dbc3555a0da05d9\": container with ID starting with d19189bbdf2a700bf7fdd9aed2220aaa969c153ed99a3b4f7dbc3555a0da05d9 not found: ID does not exist" containerID="d19189bbdf2a700bf7fdd9aed2220aaa969c153ed99a3b4f7dbc3555a0da05d9" Oct 30 01:26:34.515132 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:34.515090 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d19189bbdf2a700bf7fdd9aed2220aaa969c153ed99a3b4f7dbc3555a0da05d9"} err="failed to get container status \"d19189bbdf2a700bf7fdd9aed2220aaa969c153ed99a3b4f7dbc3555a0da05d9\": rpc error: code = NotFound desc = could not find container \"d19189bbdf2a700bf7fdd9aed2220aaa969c153ed99a3b4f7dbc3555a0da05d9\": container with ID starting with d19189bbdf2a700bf7fdd9aed2220aaa969c153ed99a3b4f7dbc3555a0da05d9 not found: ID does not exist" Oct 30 01:26:34.553633 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:34.553596 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59bfd57dc7-xnhkc"] Oct 30 01:26:34.562185 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:34.562152 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59bfd57dc7-xnhkc"] Oct 30 01:26:34.615994 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:34.615951 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:34.811889 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:34.811804 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:36.144962 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:36.144905 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:36.351770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:36.351358 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:36.541243 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:36.541206 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="407180e4-1f73-42c3-b39c-d92378649332" path="/var/lib/kubelet/pods/407180e4-1f73-42c3-b39c-d92378649332/volumes" Oct 30 01:26:43.377939 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:43.377501 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db"] Oct 30 01:26:43.380197 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:43.379206 6572 cpu_manager.go:414] "containerMap: removing container" podUID="407180e4-1f73-42c3-b39c-d92378649332" containerName="authorino" Oct 30 01:26:43.380197 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:43.379235 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="407180e4-1f73-42c3-b39c-d92378649332" containerName="authorino" Oct 30 01:26:43.380197 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:43.379373 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="407180e4-1f73-42c3-b39c-d92378649332" containerName="authorino" Oct 30 01:26:43.383059 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:43.383028 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" Oct 30 01:26:43.386880 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:43.384497 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cf2db\" (UID: \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" Oct 30 01:26:43.386880 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:43.384582 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cf2db\" (UID: \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" Oct 30 01:26:43.386880 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:43.384624 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cf2db\" (UID: \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" Oct 30 01:26:43.386880 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:43.384670 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cf2db\" (UID: \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" Oct 30 01:26:43.386880 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:43.384716 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cf2db\" (UID: \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" Oct 30 01:26:43.386880 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:43.384902 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cf2db\" (UID: \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" Oct 30 01:26:43.386880 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:43.385003 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cf2db\" (UID: \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" Oct 30 01:26:43.386880 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:43.385133 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbdr4\" (UniqueName: \"kubernetes.io/projected/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-kube-api-access-xbdr4\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cf2db\" (UID: \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" Oct 30 01:26:43.386880 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:43.385188 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cf2db\" (UID: \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" Oct 30 01:26:43.396866 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:43.396835 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 01:26:43.406321 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:43.406266 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-s9nfd\"" Oct 30 01:26:43.408918 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:43.408875 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db"] Oct 30 01:26:43.485920 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:43.485881 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cf2db\" (UID: \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" Oct 30 01:26:43.485920 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:43.485924 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cf2db\" (UID: \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" Oct 30 01:26:43.486184 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:43.485990 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cf2db\" (UID: \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" Oct 30 01:26:43.486184 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:43.486022 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cf2db\" (UID: \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" Oct 30 01:26:43.486184 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:43.486067 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cf2db\" (UID: \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" Oct 30 01:26:43.486184 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:43.486091 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cf2db\" (UID: \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" Oct 30 01:26:43.486184 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:43.486114 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cf2db\" (UID: \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" Oct 30 01:26:43.486184 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:43.486160 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cf2db\" (UID: \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" Oct 30 01:26:43.486507 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:43.486196 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbdr4\" (UniqueName: \"kubernetes.io/projected/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-kube-api-access-xbdr4\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cf2db\" (UID: \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" Oct 30 01:26:43.486507 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:43.486389 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cf2db\" (UID: \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" Oct 30 01:26:43.486621 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:43.486492 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cf2db\" (UID: \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" Oct 30 01:26:43.486678 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:43.486641 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cf2db\" (UID: \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" Oct 30 01:26:43.486944 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:43.486915 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cf2db\" (UID: \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" Oct 30 01:26:43.487153 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:43.487132 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cf2db\" (UID: \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" Oct 30 01:26:43.488992 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:43.488963 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cf2db\" (UID: \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" Oct 30 01:26:43.489117 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:43.489045 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cf2db\" (UID: \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" Oct 30 01:26:43.505537 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:43.505504 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cf2db\" (UID: \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" Oct 30 01:26:43.506546 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:43.506527 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbdr4\" (UniqueName: \"kubernetes.io/projected/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-kube-api-access-xbdr4\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cf2db\" (UID: \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" Oct 30 01:26:43.702946 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:43.702892 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" Oct 30 01:26:44.020557 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:44.020530 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db"] Oct 30 01:26:44.022882 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:26:44.022856 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod432d7644_c50a_49bb_96d5_5f6f14aa1c6e.slice/crio-bfe612738d604fe453162e9aab01ae1f41430359b9d8936dbce9a6e5a2c9aa38 WatchSource:0}: Error finding container bfe612738d604fe453162e9aab01ae1f41430359b9d8936dbce9a6e5a2c9aa38: Status 404 returned error can't find the container with id bfe612738d604fe453162e9aab01ae1f41430359b9d8936dbce9a6e5a2c9aa38 Oct 30 01:26:44.562552 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:44.562517 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" event={"ID":"432d7644-c50a-49bb-96d5-5f6f14aa1c6e","Type":"ContainerStarted","Data":"bfe612738d604fe453162e9aab01ae1f41430359b9d8936dbce9a6e5a2c9aa38"} Oct 30 01:26:44.700387 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:26:44.700323 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:26:46.220287 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:46.220244 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:46.614568 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:46.614526 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:26:46.614713 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:46.614613 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:26:46.614713 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:46.614659 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:26:47.581828 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:47.581783 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" event={"ID":"432d7644-c50a-49bb-96d5-5f6f14aa1c6e","Type":"ContainerStarted","Data":"8dd1174b950c9b3bdd11c3d61eb81d0b38513b2be0dc776a8ef5c52dbb25ed66"} Oct 30 01:26:47.641455 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:47.641398 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" podStartSLOduration=2.052654173 podStartE2EDuration="4.641382273s" totalImagesPullingTime="2.5887281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:47.6312807 +0000 UTC m=+533.207706103" watchObservedRunningTime="2026-10-30 01:26:47.641382273 +0000 UTC m=+533.217807675" Oct 30 01:26:47.703108 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:47.703063 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" Oct 30 01:26:47.704718 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:47.704671 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-cf2db container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 01:26:47.704846 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:47.704742 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" podUID="432d7644-c50a-49bb-96d5-5f6f14aa1c6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 01:26:48.704283 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:48.704241 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-cf2db container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 01:26:48.704712 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:48.704303 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" podUID="432d7644-c50a-49bb-96d5-5f6f14aa1c6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 01:26:48.842758 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:48.842718 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:49.032289 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:49.032185 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:49.703641 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:49.703602 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-cf2db container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 01:26:49.703855 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:49.703669 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" podUID="432d7644-c50a-49bb-96d5-5f6f14aa1c6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 01:26:50.704006 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:50.703963 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-cf2db container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 01:26:50.704527 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:50.704035 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" podUID="432d7644-c50a-49bb-96d5-5f6f14aa1c6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 01:26:51.703331 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:51.703289 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-cf2db container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 01:26:51.703519 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:51.703350 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" podUID="432d7644-c50a-49bb-96d5-5f6f14aa1c6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 01:26:52.704131 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:52.704088 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-cf2db container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 01:26:52.704576 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:52.704153 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" podUID="432d7644-c50a-49bb-96d5-5f6f14aa1c6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 01:26:53.703705 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:53.703662 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-cf2db container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 01:26:53.704047 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:53.703730 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" podUID="432d7644-c50a-49bb-96d5-5f6f14aa1c6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 01:26:55.050774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:55.050715 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" Oct 30 01:26:55.051488 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:55.051464 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" Oct 30 01:26:55.052086 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:55.052065 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" Oct 30 01:26:58.419650 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.419608 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6"] Oct 30 01:26:58.424937 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.424912 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" Oct 30 01:26:58.435761 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.435707 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct"] Oct 30 01:26:58.439971 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.439947 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" Oct 30 01:26:58.446560 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.446521 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6"] Oct 30 01:26:58.461358 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.461322 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct"] Oct 30 01:26:58.540820 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.540789 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcda177c-9d37-4f05-ac01-d99f9139f87e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tznct\" (UID: \"dcda177c-9d37-4f05-ac01-d99f9139f87e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" Oct 30 01:26:58.540820 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.540823 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f0afe7-d54e-4577-857a-da30b84237de-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zsgv6\" (UID: \"e1f0afe7-d54e-4577-857a-da30b84237de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" Oct 30 01:26:58.541028 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.540943 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wg7h\" (UniqueName: \"kubernetes.io/projected/dcda177c-9d37-4f05-ac01-d99f9139f87e-kube-api-access-9wg7h\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tznct\" (UID: \"dcda177c-9d37-4f05-ac01-d99f9139f87e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" Oct 30 01:26:58.541028 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.540987 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcda177c-9d37-4f05-ac01-d99f9139f87e-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tznct\" (UID: \"dcda177c-9d37-4f05-ac01-d99f9139f87e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" Oct 30 01:26:58.541028 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.541008 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcda177c-9d37-4f05-ac01-d99f9139f87e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tznct\" (UID: \"dcda177c-9d37-4f05-ac01-d99f9139f87e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" Oct 30 01:26:58.541129 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.541076 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcda177c-9d37-4f05-ac01-d99f9139f87e-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tznct\" (UID: \"dcda177c-9d37-4f05-ac01-d99f9139f87e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" Oct 30 01:26:58.541129 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.541094 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcda177c-9d37-4f05-ac01-d99f9139f87e-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tznct\" (UID: \"dcda177c-9d37-4f05-ac01-d99f9139f87e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" Oct 30 01:26:58.541129 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.541109 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1f0afe7-d54e-4577-857a-da30b84237de-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zsgv6\" (UID: \"e1f0afe7-d54e-4577-857a-da30b84237de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" Oct 30 01:26:58.541229 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.541137 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1f0afe7-d54e-4577-857a-da30b84237de-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zsgv6\" (UID: \"e1f0afe7-d54e-4577-857a-da30b84237de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" Oct 30 01:26:58.541229 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.541175 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1f0afe7-d54e-4577-857a-da30b84237de-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zsgv6\" (UID: \"e1f0afe7-d54e-4577-857a-da30b84237de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" Oct 30 01:26:58.541229 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.541209 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1f0afe7-d54e-4577-857a-da30b84237de-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zsgv6\" (UID: \"e1f0afe7-d54e-4577-857a-da30b84237de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" Oct 30 01:26:58.541322 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.541228 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1f0afe7-d54e-4577-857a-da30b84237de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zsgv6\" (UID: \"e1f0afe7-d54e-4577-857a-da30b84237de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" Oct 30 01:26:58.541356 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.541320 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcda177c-9d37-4f05-ac01-d99f9139f87e-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tznct\" (UID: \"dcda177c-9d37-4f05-ac01-d99f9139f87e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" Oct 30 01:26:58.541356 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.541342 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1f0afe7-d54e-4577-857a-da30b84237de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zsgv6\" (UID: \"e1f0afe7-d54e-4577-857a-da30b84237de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" Oct 30 01:26:58.541421 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.541367 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnptz\" (UniqueName: \"kubernetes.io/projected/e1f0afe7-d54e-4577-857a-da30b84237de-kube-api-access-gnptz\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zsgv6\" (UID: \"e1f0afe7-d54e-4577-857a-da30b84237de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" Oct 30 01:26:58.541457 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.541432 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcda177c-9d37-4f05-ac01-d99f9139f87e-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tznct\" (UID: \"dcda177c-9d37-4f05-ac01-d99f9139f87e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" Oct 30 01:26:58.541492 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.541464 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcda177c-9d37-4f05-ac01-d99f9139f87e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tznct\" (UID: \"dcda177c-9d37-4f05-ac01-d99f9139f87e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" Oct 30 01:26:58.541492 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.541487 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f0afe7-d54e-4577-857a-da30b84237de-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zsgv6\" (UID: \"e1f0afe7-d54e-4577-857a-da30b84237de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" Oct 30 01:26:58.642023 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.641976 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcda177c-9d37-4f05-ac01-d99f9139f87e-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tznct\" (UID: \"dcda177c-9d37-4f05-ac01-d99f9139f87e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" Oct 30 01:26:58.642023 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.642022 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcda177c-9d37-4f05-ac01-d99f9139f87e-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tznct\" (UID: \"dcda177c-9d37-4f05-ac01-d99f9139f87e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" Oct 30 01:26:58.642302 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.642046 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1f0afe7-d54e-4577-857a-da30b84237de-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zsgv6\" (UID: \"e1f0afe7-d54e-4577-857a-da30b84237de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" Oct 30 01:26:58.642302 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.642159 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1f0afe7-d54e-4577-857a-da30b84237de-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zsgv6\" (UID: \"e1f0afe7-d54e-4577-857a-da30b84237de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" Oct 30 01:26:58.642302 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.642197 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1f0afe7-d54e-4577-857a-da30b84237de-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zsgv6\" (UID: \"e1f0afe7-d54e-4577-857a-da30b84237de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" Oct 30 01:26:58.642302 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.642242 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1f0afe7-d54e-4577-857a-da30b84237de-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zsgv6\" (UID: \"e1f0afe7-d54e-4577-857a-da30b84237de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" Oct 30 01:26:58.642302 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.642261 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1f0afe7-d54e-4577-857a-da30b84237de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zsgv6\" (UID: \"e1f0afe7-d54e-4577-857a-da30b84237de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" Oct 30 01:26:58.642565 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.642308 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcda177c-9d37-4f05-ac01-d99f9139f87e-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tznct\" (UID: \"dcda177c-9d37-4f05-ac01-d99f9139f87e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" Oct 30 01:26:58.642565 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.642337 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1f0afe7-d54e-4577-857a-da30b84237de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zsgv6\" (UID: \"e1f0afe7-d54e-4577-857a-da30b84237de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" Oct 30 01:26:58.642565 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.642366 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnptz\" (UniqueName: \"kubernetes.io/projected/e1f0afe7-d54e-4577-857a-da30b84237de-kube-api-access-gnptz\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zsgv6\" (UID: \"e1f0afe7-d54e-4577-857a-da30b84237de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" Oct 30 01:26:58.642565 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.642413 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcda177c-9d37-4f05-ac01-d99f9139f87e-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tznct\" (UID: \"dcda177c-9d37-4f05-ac01-d99f9139f87e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" Oct 30 01:26:58.642565 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.642466 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcda177c-9d37-4f05-ac01-d99f9139f87e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tznct\" (UID: \"dcda177c-9d37-4f05-ac01-d99f9139f87e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" Oct 30 01:26:58.642565 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.642504 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f0afe7-d54e-4577-857a-da30b84237de-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zsgv6\" (UID: \"e1f0afe7-d54e-4577-857a-da30b84237de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" Oct 30 01:26:58.642565 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.642537 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcda177c-9d37-4f05-ac01-d99f9139f87e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tznct\" (UID: \"dcda177c-9d37-4f05-ac01-d99f9139f87e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" Oct 30 01:26:58.642565 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.642557 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1f0afe7-d54e-4577-857a-da30b84237de-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zsgv6\" (UID: \"e1f0afe7-d54e-4577-857a-da30b84237de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" Oct 30 01:26:58.642565 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.642563 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f0afe7-d54e-4577-857a-da30b84237de-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zsgv6\" (UID: \"e1f0afe7-d54e-4577-857a-da30b84237de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" Oct 30 01:26:58.643059 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.642626 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcda177c-9d37-4f05-ac01-d99f9139f87e-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tznct\" (UID: \"dcda177c-9d37-4f05-ac01-d99f9139f87e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" Oct 30 01:26:58.643059 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.642627 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1f0afe7-d54e-4577-857a-da30b84237de-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zsgv6\" (UID: \"e1f0afe7-d54e-4577-857a-da30b84237de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" Oct 30 01:26:58.643059 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.642792 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wg7h\" (UniqueName: \"kubernetes.io/projected/dcda177c-9d37-4f05-ac01-d99f9139f87e-kube-api-access-9wg7h\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tznct\" (UID: \"dcda177c-9d37-4f05-ac01-d99f9139f87e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" Oct 30 01:26:58.643059 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.642840 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcda177c-9d37-4f05-ac01-d99f9139f87e-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tznct\" (UID: \"dcda177c-9d37-4f05-ac01-d99f9139f87e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" Oct 30 01:26:58.643059 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.642872 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcda177c-9d37-4f05-ac01-d99f9139f87e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tznct\" (UID: \"dcda177c-9d37-4f05-ac01-d99f9139f87e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" Oct 30 01:26:58.643059 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.642895 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcda177c-9d37-4f05-ac01-d99f9139f87e-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tznct\" (UID: \"dcda177c-9d37-4f05-ac01-d99f9139f87e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" Oct 30 01:26:58.643059 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.642939 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f0afe7-d54e-4577-857a-da30b84237de-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zsgv6\" (UID: \"e1f0afe7-d54e-4577-857a-da30b84237de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" Oct 30 01:26:58.643059 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.642463 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcda177c-9d37-4f05-ac01-d99f9139f87e-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tznct\" (UID: \"dcda177c-9d37-4f05-ac01-d99f9139f87e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" Oct 30 01:26:58.643474 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.643228 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcda177c-9d37-4f05-ac01-d99f9139f87e-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tznct\" (UID: \"dcda177c-9d37-4f05-ac01-d99f9139f87e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" Oct 30 01:26:58.643474 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.643364 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1f0afe7-d54e-4577-857a-da30b84237de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zsgv6\" (UID: \"e1f0afe7-d54e-4577-857a-da30b84237de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" Oct 30 01:26:58.643596 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.643477 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f0afe7-d54e-4577-857a-da30b84237de-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zsgv6\" (UID: \"e1f0afe7-d54e-4577-857a-da30b84237de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" Oct 30 01:26:58.643730 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.643705 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcda177c-9d37-4f05-ac01-d99f9139f87e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tznct\" (UID: \"dcda177c-9d37-4f05-ac01-d99f9139f87e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" Oct 30 01:26:58.644655 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.644622 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1f0afe7-d54e-4577-857a-da30b84237de-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zsgv6\" (UID: \"e1f0afe7-d54e-4577-857a-da30b84237de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" Oct 30 01:26:58.645058 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.645038 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcda177c-9d37-4f05-ac01-d99f9139f87e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tznct\" (UID: \"dcda177c-9d37-4f05-ac01-d99f9139f87e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" Oct 30 01:26:58.645268 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.645247 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcda177c-9d37-4f05-ac01-d99f9139f87e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tznct\" (UID: \"dcda177c-9d37-4f05-ac01-d99f9139f87e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" Oct 30 01:26:58.645401 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.645381 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1f0afe7-d54e-4577-857a-da30b84237de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zsgv6\" (UID: \"e1f0afe7-d54e-4577-857a-da30b84237de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" Oct 30 01:26:58.658997 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.658964 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1f0afe7-d54e-4577-857a-da30b84237de-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zsgv6\" (UID: \"e1f0afe7-d54e-4577-857a-da30b84237de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" Oct 30 01:26:58.658997 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.658989 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcda177c-9d37-4f05-ac01-d99f9139f87e-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tznct\" (UID: \"dcda177c-9d37-4f05-ac01-d99f9139f87e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" Oct 30 01:26:58.659184 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.659052 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnptz\" (UniqueName: \"kubernetes.io/projected/e1f0afe7-d54e-4577-857a-da30b84237de-kube-api-access-gnptz\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zsgv6\" (UID: \"e1f0afe7-d54e-4577-857a-da30b84237de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" Oct 30 01:26:58.659461 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.659441 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wg7h\" (UniqueName: \"kubernetes.io/projected/dcda177c-9d37-4f05-ac01-d99f9139f87e-kube-api-access-9wg7h\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tznct\" (UID: \"dcda177c-9d37-4f05-ac01-d99f9139f87e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" Oct 30 01:26:58.743033 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.742933 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" Oct 30 01:26:58.760493 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:58.760457 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" Oct 30 01:26:59.058359 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:59.058315 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6"] Oct 30 01:26:59.061246 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:26:59.061202 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1f0afe7_d54e_4577_857a_da30b84237de.slice/crio-4aeebf8a79ab6e8b11bf735eda4a8a7be01fac1a60a06d792a73608a3b1b41e9 WatchSource:0}: Error finding container 4aeebf8a79ab6e8b11bf735eda4a8a7be01fac1a60a06d792a73608a3b1b41e9: Status 404 returned error can't find the container with id 4aeebf8a79ab6e8b11bf735eda4a8a7be01fac1a60a06d792a73608a3b1b41e9 Oct 30 01:26:59.063674 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:59.063639 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:26:59.063725 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:59.063717 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:26:59.063792 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:59.063775 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:26:59.090793 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:59.090489 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct"] Oct 30 01:26:59.092912 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:26:59.092878 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcda177c_9d37_4f05_ac01_d99f9139f87e.slice/crio-1a7d136a8accebbad4e9a549da9791ced4e2d574b2880967b013f70ada82efa2 WatchSource:0}: Error finding container 1a7d136a8accebbad4e9a549da9791ced4e2d574b2880967b013f70ada82efa2: Status 404 returned error can't find the container with id 1a7d136a8accebbad4e9a549da9791ced4e2d574b2880967b013f70ada82efa2 Oct 30 01:26:59.095431 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:59.095392 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:26:59.095531 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:59.095479 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:26:59.095531 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:59.095523 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:26:59.666502 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:59.666458 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" event={"ID":"dcda177c-9d37-4f05-ac01-d99f9139f87e","Type":"ContainerStarted","Data":"9ebd16028c44463f7b7e12509b4614b5d1d14125135eb71a7f73a677aa136a73"} Oct 30 01:26:59.667038 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:59.666558 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" event={"ID":"dcda177c-9d37-4f05-ac01-d99f9139f87e","Type":"ContainerStarted","Data":"1a7d136a8accebbad4e9a549da9791ced4e2d574b2880967b013f70ada82efa2"} Oct 30 01:26:59.668106 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:59.668081 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" event={"ID":"e1f0afe7-d54e-4577-857a-da30b84237de","Type":"ContainerStarted","Data":"9dd72b963ccf9883ba203df86886469c88ff25bdd02ee0930d07e87428914ba1"} Oct 30 01:26:59.668212 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:59.668123 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" event={"ID":"e1f0afe7-d54e-4577-857a-da30b84237de","Type":"ContainerStarted","Data":"4aeebf8a79ab6e8b11bf735eda4a8a7be01fac1a60a06d792a73608a3b1b41e9"} Oct 30 01:26:59.733771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:59.731892 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" podStartSLOduration=1.73186749 podStartE2EDuration="1.73186749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:59.714292149 +0000 UTC m=+545.290717587" watchObservedRunningTime="2026-10-30 01:26:59.73186749 +0000 UTC m=+545.308292893" Oct 30 01:26:59.748774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:59.745879 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" Oct 30 01:26:59.760564 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:26:59.760522 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" Oct 30 01:27:00.745876 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:00.745832 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-zsgv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 01:27:00.746516 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:00.746477 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" podUID="e1f0afe7-d54e-4577-857a-da30b84237de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 01:27:00.762114 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:00.762079 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-tznct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 01:27:00.762289 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:00.762134 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" podUID="dcda177c-9d37-4f05-ac01-d99f9139f87e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 01:27:01.746796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:01.746169 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-zsgv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 01:27:01.746796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:01.746239 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" podUID="e1f0afe7-d54e-4577-857a-da30b84237de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 01:27:01.761198 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:01.761151 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-tznct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 01:27:01.761385 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:01.761223 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" podUID="dcda177c-9d37-4f05-ac01-d99f9139f87e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 01:27:02.749775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:02.744200 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-zsgv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 01:27:02.749775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:02.744268 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" podUID="e1f0afe7-d54e-4577-857a-da30b84237de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 01:27:02.761102 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:02.761062 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-tznct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 01:27:02.761323 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:02.761219 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" podUID="dcda177c-9d37-4f05-ac01-d99f9139f87e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 01:27:02.832419 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:02.832381 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:03.744273 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:03.744177 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-zsgv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 01:27:03.744672 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:03.744636 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" podUID="e1f0afe7-d54e-4577-857a-da30b84237de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 01:27:03.761042 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:03.761001 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-tznct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 01:27:03.761460 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:03.761071 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" podUID="dcda177c-9d37-4f05-ac01-d99f9139f87e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 01:27:04.746771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:04.745850 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-zsgv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 01:27:04.746771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:04.745915 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" podUID="e1f0afe7-d54e-4577-857a-da30b84237de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 01:27:04.760987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:04.760947 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-tznct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 01:27:04.761284 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:04.761252 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" podUID="dcda177c-9d37-4f05-ac01-d99f9139f87e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 01:27:05.744265 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:05.744226 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-zsgv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 01:27:05.744453 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:05.744299 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" podUID="e1f0afe7-d54e-4577-857a-da30b84237de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 01:27:05.761651 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:05.761618 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-tznct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 01:27:05.762056 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:05.761673 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" podUID="dcda177c-9d37-4f05-ac01-d99f9139f87e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 01:27:07.505168 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:07.505124 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" Oct 30 01:27:07.505870 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:07.505844 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" Oct 30 01:27:07.506689 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:07.506664 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" Oct 30 01:27:07.539093 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:07.539061 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" Oct 30 01:27:07.539374 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:07.539347 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" Oct 30 01:27:07.540252 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:07.540229 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" Oct 30 01:27:07.552015 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:07.551957 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" podStartSLOduration=9.551943565 podStartE2EDuration="9.551943565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:59.759167212 +0000 UTC m=+545.335592615" watchObservedRunningTime="2026-10-30 01:27:07.551943565 +0000 UTC m=+553.128368966" Oct 30 01:27:13.487773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:13.487703 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr"] Oct 30 01:27:13.491929 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:13.491904 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" Oct 30 01:27:13.518731 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:13.518692 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr"] Oct 30 01:27:13.624304 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:13.624261 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51505b47-e483-40b3-bbac-a8aedafc4037-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-57zjr\" (UID: \"51505b47-e483-40b3-bbac-a8aedafc4037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" Oct 30 01:27:13.624304 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:13.624309 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51505b47-e483-40b3-bbac-a8aedafc4037-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-57zjr\" (UID: \"51505b47-e483-40b3-bbac-a8aedafc4037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" Oct 30 01:27:13.624544 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:13.624456 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwtxt\" (UniqueName: \"kubernetes.io/projected/51505b47-e483-40b3-bbac-a8aedafc4037-kube-api-access-hwtxt\") pod \"maas-default-gateway-openshift-default-5bc884dd87-57zjr\" (UID: \"51505b47-e483-40b3-bbac-a8aedafc4037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" Oct 30 01:27:13.624544 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:13.624508 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51505b47-e483-40b3-bbac-a8aedafc4037-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-57zjr\" (UID: \"51505b47-e483-40b3-bbac-a8aedafc4037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" Oct 30 01:27:13.624611 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:13.624547 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51505b47-e483-40b3-bbac-a8aedafc4037-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-57zjr\" (UID: \"51505b47-e483-40b3-bbac-a8aedafc4037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" Oct 30 01:27:13.624661 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:13.624634 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51505b47-e483-40b3-bbac-a8aedafc4037-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-57zjr\" (UID: \"51505b47-e483-40b3-bbac-a8aedafc4037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" Oct 30 01:27:13.624713 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:13.624693 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51505b47-e483-40b3-bbac-a8aedafc4037-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-57zjr\" (UID: \"51505b47-e483-40b3-bbac-a8aedafc4037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" Oct 30 01:27:13.624789 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:13.624775 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51505b47-e483-40b3-bbac-a8aedafc4037-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-57zjr\" (UID: \"51505b47-e483-40b3-bbac-a8aedafc4037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" Oct 30 01:27:13.624836 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:13.624810 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51505b47-e483-40b3-bbac-a8aedafc4037-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-57zjr\" (UID: \"51505b47-e483-40b3-bbac-a8aedafc4037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" Oct 30 01:27:13.725793 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:13.725705 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51505b47-e483-40b3-bbac-a8aedafc4037-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-57zjr\" (UID: \"51505b47-e483-40b3-bbac-a8aedafc4037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" Oct 30 01:27:13.725793 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:13.725812 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51505b47-e483-40b3-bbac-a8aedafc4037-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-57zjr\" (UID: \"51505b47-e483-40b3-bbac-a8aedafc4037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" Oct 30 01:27:13.726082 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:13.725832 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51505b47-e483-40b3-bbac-a8aedafc4037-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-57zjr\" (UID: \"51505b47-e483-40b3-bbac-a8aedafc4037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" Oct 30 01:27:13.726082 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:13.725869 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwtxt\" (UniqueName: \"kubernetes.io/projected/51505b47-e483-40b3-bbac-a8aedafc4037-kube-api-access-hwtxt\") pod \"maas-default-gateway-openshift-default-5bc884dd87-57zjr\" (UID: \"51505b47-e483-40b3-bbac-a8aedafc4037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" Oct 30 01:27:13.726082 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:13.725912 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51505b47-e483-40b3-bbac-a8aedafc4037-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-57zjr\" (UID: \"51505b47-e483-40b3-bbac-a8aedafc4037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" Oct 30 01:27:13.726082 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:13.725960 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51505b47-e483-40b3-bbac-a8aedafc4037-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-57zjr\" (UID: \"51505b47-e483-40b3-bbac-a8aedafc4037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" Oct 30 01:27:13.726082 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:13.726014 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51505b47-e483-40b3-bbac-a8aedafc4037-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-57zjr\" (UID: \"51505b47-e483-40b3-bbac-a8aedafc4037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" Oct 30 01:27:13.726082 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:13.726038 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51505b47-e483-40b3-bbac-a8aedafc4037-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-57zjr\" (UID: \"51505b47-e483-40b3-bbac-a8aedafc4037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" Oct 30 01:27:13.726082 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:13.726061 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51505b47-e483-40b3-bbac-a8aedafc4037-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-57zjr\" (UID: \"51505b47-e483-40b3-bbac-a8aedafc4037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" Oct 30 01:27:13.726430 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:13.726310 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51505b47-e483-40b3-bbac-a8aedafc4037-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-57zjr\" (UID: \"51505b47-e483-40b3-bbac-a8aedafc4037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" Oct 30 01:27:13.726514 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:13.726485 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51505b47-e483-40b3-bbac-a8aedafc4037-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-57zjr\" (UID: \"51505b47-e483-40b3-bbac-a8aedafc4037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" Oct 30 01:27:13.726577 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:13.726514 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51505b47-e483-40b3-bbac-a8aedafc4037-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-57zjr\" (UID: \"51505b47-e483-40b3-bbac-a8aedafc4037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" Oct 30 01:27:13.726632 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:13.726593 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51505b47-e483-40b3-bbac-a8aedafc4037-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-57zjr\" (UID: \"51505b47-e483-40b3-bbac-a8aedafc4037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" Oct 30 01:27:13.726786 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:13.726765 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51505b47-e483-40b3-bbac-a8aedafc4037-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-57zjr\" (UID: \"51505b47-e483-40b3-bbac-a8aedafc4037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" Oct 30 01:27:13.728200 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:13.728174 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51505b47-e483-40b3-bbac-a8aedafc4037-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-57zjr\" (UID: \"51505b47-e483-40b3-bbac-a8aedafc4037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" Oct 30 01:27:13.728569 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:13.728551 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51505b47-e483-40b3-bbac-a8aedafc4037-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-57zjr\" (UID: \"51505b47-e483-40b3-bbac-a8aedafc4037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" Oct 30 01:27:13.743253 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:13.743184 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51505b47-e483-40b3-bbac-a8aedafc4037-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-57zjr\" (UID: \"51505b47-e483-40b3-bbac-a8aedafc4037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" Oct 30 01:27:13.743253 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:13.743215 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwtxt\" (UniqueName: \"kubernetes.io/projected/51505b47-e483-40b3-bbac-a8aedafc4037-kube-api-access-hwtxt\") pod \"maas-default-gateway-openshift-default-5bc884dd87-57zjr\" (UID: \"51505b47-e483-40b3-bbac-a8aedafc4037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" Oct 30 01:27:13.806573 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:13.806533 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" Oct 30 01:27:14.116875 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:14.116846 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr"] Oct 30 01:27:14.118590 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:27:14.118556 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51505b47_e483_40b3_bbac_a8aedafc4037.slice/crio-88a4e01ae82a3232ce40384da8366d070f84d705b7b560ab6b79071e0f673d42 WatchSource:0}: Error finding container 88a4e01ae82a3232ce40384da8366d070f84d705b7b560ab6b79071e0f673d42: Status 404 returned error can't find the container with id 88a4e01ae82a3232ce40384da8366d070f84d705b7b560ab6b79071e0f673d42 Oct 30 01:27:14.120970 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:14.120938 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:27:14.121061 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:14.121007 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:27:14.121061 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:14.121037 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:27:14.692364 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:27:14.692323 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:27:14.756119 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:14.756033 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" event={"ID":"51505b47-e483-40b3-bbac-a8aedafc4037","Type":"ContainerStarted","Data":"57103f7121857b7de41efb8727b9b01bad751fd002eb4fc8f4e3af12efaf7849"} Oct 30 01:27:14.756119 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:14.756093 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" event={"ID":"51505b47-e483-40b3-bbac-a8aedafc4037","Type":"ContainerStarted","Data":"88a4e01ae82a3232ce40384da8366d070f84d705b7b560ab6b79071e0f673d42"} Oct 30 01:27:14.815773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:14.811858 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" Oct 30 01:27:14.823771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:14.817055 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" podStartSLOduration=1.817035832 podStartE2EDuration="1.817035832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:14.816261976 +0000 UTC m=+560.392687383" watchObservedRunningTime="2026-10-30 01:27:14.817035832 +0000 UTC m=+560.393461238" Oct 30 01:27:15.826781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:15.826431 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-57zjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 01:27:15.826781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:15.826510 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" podUID="51505b47-e483-40b3-bbac-a8aedafc4037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 01:27:16.825772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:16.825163 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-57zjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 01:27:16.825772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:16.825226 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" podUID="51505b47-e483-40b3-bbac-a8aedafc4037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 01:27:17.731566 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:17.731512 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:17.823205 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:17.823162 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-57zjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 01:27:17.823493 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:17.823455 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" podUID="51505b47-e483-40b3-bbac-a8aedafc4037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 01:27:17.937774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:17.937706 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:18.246220 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:18.246176 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:18.531825 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:18.531706 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:18.701068 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:18.701020 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g"] Oct 30 01:27:18.712778 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:18.710851 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:27:18.741676 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:18.740794 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g"] Oct 30 01:27:18.743209 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:18.743186 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:18.747458 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:18.747438 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-d9801220-gw-openshift-default-dockercfg-ndc5m\"" Oct 30 01:27:18.777589 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:18.777555 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ndtg\" (UniqueName: \"kubernetes.io/projected/2b8b0408-785a-4e6f-b6d9-5c80932666ff-kube-api-access-6ndtg\") pod \"e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g\" (UID: \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\") " pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:27:18.777589 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:18.777601 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b8b0408-785a-4e6f-b6d9-5c80932666ff-istio-data\") pod \"e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g\" (UID: \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\") " pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:27:18.777826 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:18.777662 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b8b0408-785a-4e6f-b6d9-5c80932666ff-workload-certs\") pod \"e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g\" (UID: \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\") " pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:27:18.777826 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:18.777692 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b8b0408-785a-4e6f-b6d9-5c80932666ff-credential-socket\") pod \"e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g\" (UID: \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\") " pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:27:18.777826 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:18.777714 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b8b0408-785a-4e6f-b6d9-5c80932666ff-istio-token\") pod \"e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g\" (UID: \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\") " pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:27:18.777826 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:18.777810 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b8b0408-785a-4e6f-b6d9-5c80932666ff-istio-envoy\") pod \"e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g\" (UID: \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\") " pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:27:18.777982 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:18.777853 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b8b0408-785a-4e6f-b6d9-5c80932666ff-istio-podinfo\") pod \"e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g\" (UID: \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\") " pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:27:18.777982 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:18.777873 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b8b0408-785a-4e6f-b6d9-5c80932666ff-istiod-ca-cert\") pod \"e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g\" (UID: \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\") " pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:27:18.777982 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:18.777918 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b8b0408-785a-4e6f-b6d9-5c80932666ff-workload-socket\") pod \"e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g\" (UID: \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\") " pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:27:18.829842 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:18.829146 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-57zjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 01:27:18.829842 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:18.829228 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" podUID="51505b47-e483-40b3-bbac-a8aedafc4037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 01:27:18.879535 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:18.879491 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ndtg\" (UniqueName: \"kubernetes.io/projected/2b8b0408-785a-4e6f-b6d9-5c80932666ff-kube-api-access-6ndtg\") pod \"e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g\" (UID: \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\") " pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:27:18.879791 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:18.879565 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b8b0408-785a-4e6f-b6d9-5c80932666ff-istio-data\") pod \"e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g\" (UID: \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\") " pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:27:18.879791 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:18.879615 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b8b0408-785a-4e6f-b6d9-5c80932666ff-workload-certs\") pod \"e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g\" (UID: \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\") " pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:27:18.879791 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:18.879668 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b8b0408-785a-4e6f-b6d9-5c80932666ff-credential-socket\") pod \"e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g\" (UID: \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\") " pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:27:18.879791 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:18.879703 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b8b0408-785a-4e6f-b6d9-5c80932666ff-istio-token\") pod \"e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g\" (UID: \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\") " pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:27:18.879791 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:18.879778 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b8b0408-785a-4e6f-b6d9-5c80932666ff-istio-envoy\") pod \"e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g\" (UID: \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\") " pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:27:18.880070 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:18.879814 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b8b0408-785a-4e6f-b6d9-5c80932666ff-istio-podinfo\") pod \"e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g\" (UID: \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\") " pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:27:18.880070 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:18.879839 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b8b0408-785a-4e6f-b6d9-5c80932666ff-istiod-ca-cert\") pod \"e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g\" (UID: \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\") " pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:27:18.880070 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:18.879865 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b8b0408-785a-4e6f-b6d9-5c80932666ff-workload-socket\") pod \"e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g\" (UID: \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\") " pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:27:18.880212 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:18.880134 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b8b0408-785a-4e6f-b6d9-5c80932666ff-istio-data\") pod \"e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g\" (UID: \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\") " pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:27:18.880274 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:18.880209 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b8b0408-785a-4e6f-b6d9-5c80932666ff-credential-socket\") pod \"e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g\" (UID: \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\") " pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:27:18.880327 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:18.880283 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b8b0408-785a-4e6f-b6d9-5c80932666ff-workload-certs\") pod \"e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g\" (UID: \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\") " pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:27:18.880366 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:18.880347 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b8b0408-785a-4e6f-b6d9-5c80932666ff-workload-socket\") pod \"e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g\" (UID: \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\") " pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:27:18.880704 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:18.880676 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b8b0408-785a-4e6f-b6d9-5c80932666ff-istiod-ca-cert\") pod \"e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g\" (UID: \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\") " pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:27:18.882265 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:18.882240 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b8b0408-785a-4e6f-b6d9-5c80932666ff-istio-envoy\") pod \"e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g\" (UID: \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\") " pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:27:18.882513 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:18.882493 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b8b0408-785a-4e6f-b6d9-5c80932666ff-istio-podinfo\") pod \"e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g\" (UID: \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\") " pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:27:18.895586 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:18.895563 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b8b0408-785a-4e6f-b6d9-5c80932666ff-istio-token\") pod \"e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g\" (UID: \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\") " pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:27:18.895911 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:18.895893 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ndtg\" (UniqueName: \"kubernetes.io/projected/2b8b0408-785a-4e6f-b6d9-5c80932666ff-kube-api-access-6ndtg\") pod \"e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g\" (UID: \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\") " pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:27:18.935299 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:18.935260 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:19.055429 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:19.055392 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:27:19.176923 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:19.176879 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:19.382628 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:19.382597 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g"] Oct 30 01:27:19.384560 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:27:19.384529 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b8b0408_785a_4e6f_b6d9_5c80932666ff.slice/crio-e00c6866ba07745034f913b3c2d862c5e58dd379d6186a4f0d706606e6c7eda6 WatchSource:0}: Error finding container e00c6866ba07745034f913b3c2d862c5e58dd379d6186a4f0d706606e6c7eda6: Status 404 returned error can't find the container with id e00c6866ba07745034f913b3c2d862c5e58dd379d6186a4f0d706606e6c7eda6 Oct 30 01:27:19.387097 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:19.387060 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:27:19.387176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:19.387133 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:27:19.387219 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:19.387172 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:27:19.727768 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:19.727395 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:19.786950 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:19.786906 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" event={"ID":"2b8b0408-785a-4e6f-b6d9-5c80932666ff","Type":"ContainerStarted","Data":"0d183455f54b4d6ff67c008e3f27844d98a7428b92238d5930f604d4bf662765"} Oct 30 01:27:19.787320 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:19.786976 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" event={"ID":"2b8b0408-785a-4e6f-b6d9-5c80932666ff","Type":"ContainerStarted","Data":"e00c6866ba07745034f913b3c2d862c5e58dd379d6186a4f0d706606e6c7eda6"} Oct 30 01:27:19.811732 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:19.811698 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-57zjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 01:27:19.812043 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:19.812010 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" podUID="51505b47-e483-40b3-bbac-a8aedafc4037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 01:27:19.842307 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:19.842242 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" podStartSLOduration=1.842226486 podStartE2EDuration="1.842226486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:19.834319532 +0000 UTC m=+565.410744937" watchObservedRunningTime="2026-10-30 01:27:19.842226486 +0000 UTC m=+565.418651888" Oct 30 01:27:20.032821 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.029652 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp"] Oct 30 01:27:20.038825 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.038639 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:27:20.053124 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.052793 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f19ba569-openshift-default-dockercfg-8wf9v\"" Oct 30 01:27:20.055928 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.055871 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:27:20.056771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.056733 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp"] Oct 30 01:27:20.061861 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.061838 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:27:20.092725 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.092692 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab69c192-5a88-4a77-844c-4306956a9d82-istio-podinfo\") pod \"e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp\" (UID: \"ab69c192-5a88-4a77-844c-4306956a9d82\") " pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:27:20.092961 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.092779 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab69c192-5a88-4a77-844c-4306956a9d82-istio-envoy\") pod \"e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp\" (UID: \"ab69c192-5a88-4a77-844c-4306956a9d82\") " pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:27:20.092961 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.092827 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab69c192-5a88-4a77-844c-4306956a9d82-istio-token\") pod \"e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp\" (UID: \"ab69c192-5a88-4a77-844c-4306956a9d82\") " pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:27:20.092961 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.092935 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab69c192-5a88-4a77-844c-4306956a9d82-credential-socket\") pod \"e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp\" (UID: \"ab69c192-5a88-4a77-844c-4306956a9d82\") " pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:27:20.093095 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.092973 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2t9d\" (UniqueName: \"kubernetes.io/projected/ab69c192-5a88-4a77-844c-4306956a9d82-kube-api-access-r2t9d\") pod \"e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp\" (UID: \"ab69c192-5a88-4a77-844c-4306956a9d82\") " pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:27:20.093095 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.093062 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab69c192-5a88-4a77-844c-4306956a9d82-workload-socket\") pod \"e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp\" (UID: \"ab69c192-5a88-4a77-844c-4306956a9d82\") " pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:27:20.093181 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.093095 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab69c192-5a88-4a77-844c-4306956a9d82-workload-certs\") pod \"e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp\" (UID: \"ab69c192-5a88-4a77-844c-4306956a9d82\") " pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:27:20.093181 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.093154 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab69c192-5a88-4a77-844c-4306956a9d82-istio-data\") pod \"e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp\" (UID: \"ab69c192-5a88-4a77-844c-4306956a9d82\") " pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:27:20.093282 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.093200 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab69c192-5a88-4a77-844c-4306956a9d82-istiod-ca-cert\") pod \"e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp\" (UID: \"ab69c192-5a88-4a77-844c-4306956a9d82\") " pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:27:20.176332 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.176297 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:20.194796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.194734 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab69c192-5a88-4a77-844c-4306956a9d82-credential-socket\") pod \"e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp\" (UID: \"ab69c192-5a88-4a77-844c-4306956a9d82\") " pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:27:20.195011 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.194827 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2t9d\" (UniqueName: \"kubernetes.io/projected/ab69c192-5a88-4a77-844c-4306956a9d82-kube-api-access-r2t9d\") pod \"e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp\" (UID: \"ab69c192-5a88-4a77-844c-4306956a9d82\") " pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:27:20.195011 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.194874 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab69c192-5a88-4a77-844c-4306956a9d82-workload-socket\") pod \"e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp\" (UID: \"ab69c192-5a88-4a77-844c-4306956a9d82\") " pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:27:20.195011 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.194920 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab69c192-5a88-4a77-844c-4306956a9d82-workload-certs\") pod \"e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp\" (UID: \"ab69c192-5a88-4a77-844c-4306956a9d82\") " pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:27:20.195011 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.194968 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab69c192-5a88-4a77-844c-4306956a9d82-istio-data\") pod \"e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp\" (UID: \"ab69c192-5a88-4a77-844c-4306956a9d82\") " pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:27:20.195233 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.195023 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab69c192-5a88-4a77-844c-4306956a9d82-istiod-ca-cert\") pod \"e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp\" (UID: \"ab69c192-5a88-4a77-844c-4306956a9d82\") " pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:27:20.195233 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.195082 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab69c192-5a88-4a77-844c-4306956a9d82-istio-podinfo\") pod \"e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp\" (UID: \"ab69c192-5a88-4a77-844c-4306956a9d82\") " pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:27:20.195233 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.195139 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab69c192-5a88-4a77-844c-4306956a9d82-istio-envoy\") pod \"e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp\" (UID: \"ab69c192-5a88-4a77-844c-4306956a9d82\") " pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:27:20.195233 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.195174 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab69c192-5a88-4a77-844c-4306956a9d82-credential-socket\") pod \"e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp\" (UID: \"ab69c192-5a88-4a77-844c-4306956a9d82\") " pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:27:20.195233 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.195192 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab69c192-5a88-4a77-844c-4306956a9d82-istio-token\") pod \"e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp\" (UID: \"ab69c192-5a88-4a77-844c-4306956a9d82\") " pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:27:20.195853 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.195582 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab69c192-5a88-4a77-844c-4306956a9d82-workload-socket\") pod \"e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp\" (UID: \"ab69c192-5a88-4a77-844c-4306956a9d82\") " pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:27:20.195853 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.195647 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab69c192-5a88-4a77-844c-4306956a9d82-istio-data\") pod \"e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp\" (UID: \"ab69c192-5a88-4a77-844c-4306956a9d82\") " pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:27:20.196137 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.196116 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab69c192-5a88-4a77-844c-4306956a9d82-workload-certs\") pod \"e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp\" (UID: \"ab69c192-5a88-4a77-844c-4306956a9d82\") " pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:27:20.196818 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.196793 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab69c192-5a88-4a77-844c-4306956a9d82-istiod-ca-cert\") pod \"e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp\" (UID: \"ab69c192-5a88-4a77-844c-4306956a9d82\") " pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:27:20.197966 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.197935 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab69c192-5a88-4a77-844c-4306956a9d82-istio-envoy\") pod \"e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp\" (UID: \"ab69c192-5a88-4a77-844c-4306956a9d82\") " pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:27:20.198113 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.198094 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab69c192-5a88-4a77-844c-4306956a9d82-istio-podinfo\") pod \"e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp\" (UID: \"ab69c192-5a88-4a77-844c-4306956a9d82\") " pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:27:20.211684 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.211655 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab69c192-5a88-4a77-844c-4306956a9d82-istio-token\") pod \"e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp\" (UID: \"ab69c192-5a88-4a77-844c-4306956a9d82\") " pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:27:20.211934 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.211912 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2t9d\" (UniqueName: \"kubernetes.io/projected/ab69c192-5a88-4a77-844c-4306956a9d82-kube-api-access-r2t9d\") pod \"e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp\" (UID: \"ab69c192-5a88-4a77-844c-4306956a9d82\") " pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:27:20.355822 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.355722 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:20.363512 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.363482 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:27:20.678813 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.671679 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp"] Oct 30 01:27:20.678813 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:27:20.674997 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab69c192_5a88_4a77_844c_4306956a9d82.slice/crio-03119e6fa9d1b75a9ef272d4b9426c0a207911a42db91ba36e8b51077aea335b WatchSource:0}: Error finding container 03119e6fa9d1b75a9ef272d4b9426c0a207911a42db91ba36e8b51077aea335b: Status 404 returned error can't find the container with id 03119e6fa9d1b75a9ef272d4b9426c0a207911a42db91ba36e8b51077aea335b Oct 30 01:27:20.681094 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.681051 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:27:20.681218 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.681169 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:27:20.681290 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.681214 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:27:20.800185 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.799458 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" event={"ID":"ab69c192-5a88-4a77-844c-4306956a9d82","Type":"ContainerStarted","Data":"9e02727aa04335839064a9062ddc93357d064ed6b9683fd2601d62b5c4f2a1bc"} Oct 30 01:27:20.800185 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.799532 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" event={"ID":"ab69c192-5a88-4a77-844c-4306956a9d82","Type":"ContainerStarted","Data":"03119e6fa9d1b75a9ef272d4b9426c0a207911a42db91ba36e8b51077aea335b"} Oct 30 01:27:20.801662 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.800999 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:27:20.801662 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.801081 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:27:20.807595 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.807566 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-57zjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 01:27:20.807778 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.807622 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" podUID="51505b47-e483-40b3-bbac-a8aedafc4037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 01:27:20.837473 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:20.837406 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" podStartSLOduration=0.837387903 podStartE2EDuration="837.387903ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:20.835533138 +0000 UTC m=+566.411958541" watchObservedRunningTime="2026-10-30 01:27:20.837387903 +0000 UTC m=+566.413813304" Oct 30 01:27:21.364504 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:21.364451 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:27:21.369658 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:21.369622 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:27:21.718897 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:21.718863 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:21.806259 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:21.806220 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:27:21.807192 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:21.807161 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:27:21.959522 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:21.959481 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:22.230987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:22.230948 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:22.416995 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:22.416960 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:22.609304 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:22.609210 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" Oct 30 01:27:22.609662 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:22.609632 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" Oct 30 01:27:22.610572 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:22.610547 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" Oct 30 01:27:23.075413 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:23.075376 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:23.289760 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:23.289710 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:24.796205 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:24.796169 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:25.017117 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:25.017082 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:25.695512 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:25.695475 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:25.727157 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:25.727115 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f29772b9-5b5fd8b8b6-6q426"] Oct 30 01:27:25.730980 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:25.730957 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f29772b9-5b5fd8b8b6-6q426" Oct 30 01:27:25.748035 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:25.748006 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f29772b9-5b5fd8b8b6-6q426"] Oct 30 01:27:25.755121 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:25.755062 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f29772b9\"" Oct 30 01:27:25.755338 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:25.755317 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f29772b9-dockercfg-vbhvc\"" Oct 30 01:27:25.862500 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:25.862448 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef38b4fb-5efb-440a-b9b1-a5848fc5a93b-plugins-config-volume\") pod \"payload-processing-e2e-ait-f29772b9-5b5fd8b8b6-6q426\" (UID: \"ef38b4fb-5efb-440a-b9b1-a5848fc5a93b\") " pod="openshift-ingress/payload-processing-e2e-ait-f29772b9-5b5fd8b8b6-6q426" Oct 30 01:27:25.863049 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:25.862629 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcglw\" (UniqueName: \"kubernetes.io/projected/ef38b4fb-5efb-440a-b9b1-a5848fc5a93b-kube-api-access-tcglw\") pod \"payload-processing-e2e-ait-f29772b9-5b5fd8b8b6-6q426\" (UID: \"ef38b4fb-5efb-440a-b9b1-a5848fc5a93b\") " pod="openshift-ingress/payload-processing-e2e-ait-f29772b9-5b5fd8b8b6-6q426" Oct 30 01:27:25.893934 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:25.893899 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:25.964271 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:25.964186 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcglw\" (UniqueName: \"kubernetes.io/projected/ef38b4fb-5efb-440a-b9b1-a5848fc5a93b-kube-api-access-tcglw\") pod \"payload-processing-e2e-ait-f29772b9-5b5fd8b8b6-6q426\" (UID: \"ef38b4fb-5efb-440a-b9b1-a5848fc5a93b\") " pod="openshift-ingress/payload-processing-e2e-ait-f29772b9-5b5fd8b8b6-6q426" Oct 30 01:27:25.964419 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:25.964299 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef38b4fb-5efb-440a-b9b1-a5848fc5a93b-plugins-config-volume\") pod \"payload-processing-e2e-ait-f29772b9-5b5fd8b8b6-6q426\" (UID: \"ef38b4fb-5efb-440a-b9b1-a5848fc5a93b\") " pod="openshift-ingress/payload-processing-e2e-ait-f29772b9-5b5fd8b8b6-6q426" Oct 30 01:27:25.965073 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:25.965048 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef38b4fb-5efb-440a-b9b1-a5848fc5a93b-plugins-config-volume\") pod \"payload-processing-e2e-ait-f29772b9-5b5fd8b8b6-6q426\" (UID: \"ef38b4fb-5efb-440a-b9b1-a5848fc5a93b\") " pod="openshift-ingress/payload-processing-e2e-ait-f29772b9-5b5fd8b8b6-6q426" Oct 30 01:27:25.983168 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:25.983136 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcglw\" (UniqueName: \"kubernetes.io/projected/ef38b4fb-5efb-440a-b9b1-a5848fc5a93b-kube-api-access-tcglw\") pod \"payload-processing-e2e-ait-f29772b9-5b5fd8b8b6-6q426\" (UID: \"ef38b4fb-5efb-440a-b9b1-a5848fc5a93b\") " pod="openshift-ingress/payload-processing-e2e-ait-f29772b9-5b5fd8b8b6-6q426" Oct 30 01:27:26.046199 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:26.046170 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f29772b9-5b5fd8b8b6-6q426" Oct 30 01:27:26.115941 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:26.115906 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:26.305423 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:26.305344 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:26.335977 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:26.335949 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f29772b9-5b5fd8b8b6-6q426"] Oct 30 01:27:26.337961 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:27:26.337937 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef38b4fb_5efb_440a_b9b1_a5848fc5a93b.slice/crio-abaa97620def1269373a337db98008cb90b7e53f7251e61fb21947ee875cf3b4 WatchSource:0}: Error finding container abaa97620def1269373a337db98008cb90b7e53f7251e61fb21947ee875cf3b4: Status 404 returned error can't find the container with id abaa97620def1269373a337db98008cb90b7e53f7251e61fb21947ee875cf3b4 Oct 30 01:27:26.501016 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:26.500973 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:26.701657 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:26.701626 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:26.841800 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:26.841737 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f29772b9-5b5fd8b8b6-6q426" event={"ID":"ef38b4fb-5efb-440a-b9b1-a5848fc5a93b","Type":"ContainerStarted","Data":"abaa97620def1269373a337db98008cb90b7e53f7251e61fb21947ee875cf3b4"} Oct 30 01:27:26.888483 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:26.888453 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:27.085015 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:27.084925 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:27.280607 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:27.280564 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:27.904171 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:27.904110 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:27.988915 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:27.988880 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f19ba569-67cb76cc8c-wsn9r"] Oct 30 01:27:27.999640 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:27.999613 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f19ba569-67cb76cc8c-wsn9r" Oct 30 01:27:28.008104 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:28.007804 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f19ba569-67cb76cc8c-wsn9r"] Oct 30 01:27:28.040302 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:28.040166 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f19ba569\"" Oct 30 01:27:28.051189 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:28.049992 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f19ba569-dockercfg-xk4zx\"" Oct 30 01:27:28.084438 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:28.084394 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/167e5371-3c5f-4055-b7f4-3e926585a77e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f19ba569-67cb76cc8c-wsn9r\" (UID: \"167e5371-3c5f-4055-b7f4-3e926585a77e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f19ba569-67cb76cc8c-wsn9r" Oct 30 01:27:28.084638 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:28.084468 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fzvl\" (UniqueName: \"kubernetes.io/projected/167e5371-3c5f-4055-b7f4-3e926585a77e-kube-api-access-4fzvl\") pod \"payload-pre-processing-e2e-ait-f19ba569-67cb76cc8c-wsn9r\" (UID: \"167e5371-3c5f-4055-b7f4-3e926585a77e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f19ba569-67cb76cc8c-wsn9r" Oct 30 01:27:28.095854 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:28.095814 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:28.186288 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:28.186254 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/167e5371-3c5f-4055-b7f4-3e926585a77e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f19ba569-67cb76cc8c-wsn9r\" (UID: \"167e5371-3c5f-4055-b7f4-3e926585a77e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f19ba569-67cb76cc8c-wsn9r" Oct 30 01:27:28.186481 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:28.186303 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fzvl\" (UniqueName: \"kubernetes.io/projected/167e5371-3c5f-4055-b7f4-3e926585a77e-kube-api-access-4fzvl\") pod \"payload-pre-processing-e2e-ait-f19ba569-67cb76cc8c-wsn9r\" (UID: \"167e5371-3c5f-4055-b7f4-3e926585a77e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f19ba569-67cb76cc8c-wsn9r" Oct 30 01:27:28.186988 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:28.186960 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/167e5371-3c5f-4055-b7f4-3e926585a77e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f19ba569-67cb76cc8c-wsn9r\" (UID: \"167e5371-3c5f-4055-b7f4-3e926585a77e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f19ba569-67cb76cc8c-wsn9r" Oct 30 01:27:28.208256 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:28.208220 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fzvl\" (UniqueName: \"kubernetes.io/projected/167e5371-3c5f-4055-b7f4-3e926585a77e-kube-api-access-4fzvl\") pod \"payload-pre-processing-e2e-ait-f19ba569-67cb76cc8c-wsn9r\" (UID: \"167e5371-3c5f-4055-b7f4-3e926585a77e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f19ba569-67cb76cc8c-wsn9r" Oct 30 01:27:28.323657 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:28.323615 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f19ba569-67cb76cc8c-wsn9r" Oct 30 01:27:28.488217 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:28.488181 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:28.623508 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:28.623477 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f19ba569-67cb76cc8c-wsn9r"] Oct 30 01:27:28.625376 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:27:28.625348 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod167e5371_3c5f_4055_b7f4_3e926585a77e.slice/crio-9b807b309eeda54fda9565b7f182bcee1a70ed0cbbf5964fab50f37c91ca8898 WatchSource:0}: Error finding container 9b807b309eeda54fda9565b7f182bcee1a70ed0cbbf5964fab50f37c91ca8898: Status 404 returned error can't find the container with id 9b807b309eeda54fda9565b7f182bcee1a70ed0cbbf5964fab50f37c91ca8898 Oct 30 01:27:28.858041 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:28.857939 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f29772b9-5b5fd8b8b6-6q426" event={"ID":"ef38b4fb-5efb-440a-b9b1-a5848fc5a93b","Type":"ContainerStarted","Data":"1edc53533e2e70d6e5e9bd1a0edbfc8347b1d1e3d9049d54c29251f6e430f475"} Oct 30 01:27:28.858041 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:28.858033 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f29772b9-5b5fd8b8b6-6q426" Oct 30 01:27:28.859454 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:28.859421 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f19ba569-67cb76cc8c-wsn9r" event={"ID":"167e5371-3c5f-4055-b7f4-3e926585a77e","Type":"ContainerStarted","Data":"1adfd64134181ed41656ccdbd97bbc1763487ae55b004fce8bbf2382dd2438c8"} Oct 30 01:27:28.859570 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:28.859461 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f19ba569-67cb76cc8c-wsn9r" event={"ID":"167e5371-3c5f-4055-b7f4-3e926585a77e","Type":"ContainerStarted","Data":"9b807b309eeda54fda9565b7f182bcee1a70ed0cbbf5964fab50f37c91ca8898"} Oct 30 01:27:28.859616 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:28.859571 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f19ba569-67cb76cc8c-wsn9r" Oct 30 01:27:28.893288 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:28.893229 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f29772b9-5b5fd8b8b6-6q426" podStartSLOduration=1.8426223400000001 podStartE2EDuration="3.893213753s" totalImagesPullingTime="2.050591413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:28.891071271 +0000 UTC m=+574.467496685" watchObservedRunningTime="2026-10-30 01:27:28.893213753 +0000 UTC m=+574.469639154" Oct 30 01:27:28.931633 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:28.931577 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f19ba569-67cb76cc8c-wsn9r" podStartSLOduration=1.931559429 podStartE2EDuration="1.931559429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:28.928807071 +0000 UTC m=+574.505232474" watchObservedRunningTime="2026-10-30 01:27:28.931559429 +0000 UTC m=+574.507984831" Oct 30 01:27:29.781397 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:29.781357 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:29.976982 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:29.976942 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:30.331724 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:30.331689 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:30.528360 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:30.528274 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:30.736769 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:30.736522 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:31.666336 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:31.666090 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:31.679475 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:31.679444 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-18c05df8-dfcbccf8-vvssk"] Oct 30 01:27:31.683503 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:31.683482 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-18c05df8-dfcbccf8-vvssk" Oct 30 01:27:31.699460 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:31.699419 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-18c05df8\"" Oct 30 01:27:31.769771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:31.768180 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-18c05df8-dfcbccf8-vvssk"] Oct 30 01:27:31.773247 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:31.773205 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e77a029-818a-4930-8ad5-6f341fa0a3b3-plugins-config-volume\") pod \"payload-processing-e2e-derive-18c05df8-dfcbccf8-vvssk\" (UID: \"7e77a029-818a-4930-8ad5-6f341fa0a3b3\") " pod="openshift-ingress/payload-processing-e2e-derive-18c05df8-dfcbccf8-vvssk" Oct 30 01:27:31.773584 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:31.773557 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkf9s\" (UniqueName: \"kubernetes.io/projected/7e77a029-818a-4930-8ad5-6f341fa0a3b3-kube-api-access-pkf9s\") pod \"payload-processing-e2e-derive-18c05df8-dfcbccf8-vvssk\" (UID: \"7e77a029-818a-4930-8ad5-6f341fa0a3b3\") " pod="openshift-ingress/payload-processing-e2e-derive-18c05df8-dfcbccf8-vvssk" Oct 30 01:27:31.775471 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:31.775448 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-18c05df8-679684cfd7-dvp7m"] Oct 30 01:27:31.780662 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:31.779923 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-18c05df8-679684cfd7-dvp7m"] Oct 30 01:27:31.780662 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:31.780008 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-18c05df8-679684cfd7-dvp7m" Oct 30 01:27:31.884870 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:31.868242 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:31.889208 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:31.886995 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-18c05df8-dockercfg-9mk9n\"" Oct 30 01:27:31.889208 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:31.887957 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85045c0a-97ae-432c-a346-d60115100fad-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-18c05df8-679684cfd7-dvp7m\" (UID: \"85045c0a-97ae-432c-a346-d60115100fad\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-18c05df8-679684cfd7-dvp7m" Oct 30 01:27:31.889208 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:31.888000 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj9cd\" (UniqueName: \"kubernetes.io/projected/85045c0a-97ae-432c-a346-d60115100fad-kube-api-access-gj9cd\") pod \"payload-pre-processing-e2e-derive-18c05df8-679684cfd7-dvp7m\" (UID: \"85045c0a-97ae-432c-a346-d60115100fad\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-18c05df8-679684cfd7-dvp7m" Oct 30 01:27:31.889208 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:31.888048 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e77a029-818a-4930-8ad5-6f341fa0a3b3-plugins-config-volume\") pod \"payload-processing-e2e-derive-18c05df8-dfcbccf8-vvssk\" (UID: \"7e77a029-818a-4930-8ad5-6f341fa0a3b3\") " pod="openshift-ingress/payload-processing-e2e-derive-18c05df8-dfcbccf8-vvssk" Oct 30 01:27:31.889208 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:31.888103 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkf9s\" (UniqueName: \"kubernetes.io/projected/7e77a029-818a-4930-8ad5-6f341fa0a3b3-kube-api-access-pkf9s\") pod \"payload-processing-e2e-derive-18c05df8-dfcbccf8-vvssk\" (UID: \"7e77a029-818a-4930-8ad5-6f341fa0a3b3\") " pod="openshift-ingress/payload-processing-e2e-derive-18c05df8-dfcbccf8-vvssk" Oct 30 01:27:31.889967 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:31.889930 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e77a029-818a-4930-8ad5-6f341fa0a3b3-plugins-config-volume\") pod \"payload-processing-e2e-derive-18c05df8-dfcbccf8-vvssk\" (UID: \"7e77a029-818a-4930-8ad5-6f341fa0a3b3\") " pod="openshift-ingress/payload-processing-e2e-derive-18c05df8-dfcbccf8-vvssk" Oct 30 01:27:31.904705 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:31.904667 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkf9s\" (UniqueName: \"kubernetes.io/projected/7e77a029-818a-4930-8ad5-6f341fa0a3b3-kube-api-access-pkf9s\") pod \"payload-processing-e2e-derive-18c05df8-dfcbccf8-vvssk\" (UID: \"7e77a029-818a-4930-8ad5-6f341fa0a3b3\") " pod="openshift-ingress/payload-processing-e2e-derive-18c05df8-dfcbccf8-vvssk" Oct 30 01:27:31.988826 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:31.988715 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85045c0a-97ae-432c-a346-d60115100fad-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-18c05df8-679684cfd7-dvp7m\" (UID: \"85045c0a-97ae-432c-a346-d60115100fad\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-18c05df8-679684cfd7-dvp7m" Oct 30 01:27:31.988826 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:31.988785 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj9cd\" (UniqueName: \"kubernetes.io/projected/85045c0a-97ae-432c-a346-d60115100fad-kube-api-access-gj9cd\") pod \"payload-pre-processing-e2e-derive-18c05df8-679684cfd7-dvp7m\" (UID: \"85045c0a-97ae-432c-a346-d60115100fad\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-18c05df8-679684cfd7-dvp7m" Oct 30 01:27:31.989503 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:31.989483 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85045c0a-97ae-432c-a346-d60115100fad-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-18c05df8-679684cfd7-dvp7m\" (UID: \"85045c0a-97ae-432c-a346-d60115100fad\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-18c05df8-679684cfd7-dvp7m" Oct 30 01:27:31.997674 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:31.997651 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-18c05df8-dfcbccf8-vvssk" Oct 30 01:27:32.016768 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:32.006073 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj9cd\" (UniqueName: \"kubernetes.io/projected/85045c0a-97ae-432c-a346-d60115100fad-kube-api-access-gj9cd\") pod \"payload-pre-processing-e2e-derive-18c05df8-679684cfd7-dvp7m\" (UID: \"85045c0a-97ae-432c-a346-d60115100fad\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-18c05df8-679684cfd7-dvp7m" Oct 30 01:27:32.096312 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:32.096073 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-18c05df8-679684cfd7-dvp7m" Oct 30 01:27:32.348084 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:32.348041 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:32.461305 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:32.461265 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-18c05df8-dfcbccf8-vvssk"] Oct 30 01:27:32.465972 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:27:32.465940 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e77a029_818a_4930_8ad5_6f341fa0a3b3.slice/crio-9599170eb0656e4b2b8bd6413ec72e68bb73e313e173161937c24481b64809bb WatchSource:0}: Error finding container 9599170eb0656e4b2b8bd6413ec72e68bb73e313e173161937c24481b64809bb: Status 404 returned error can't find the container with id 9599170eb0656e4b2b8bd6413ec72e68bb73e313e173161937c24481b64809bb Oct 30 01:27:32.534272 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:32.534199 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:32.571696 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:32.571669 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-18c05df8-679684cfd7-dvp7m"] Oct 30 01:27:32.900014 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:32.899902 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-18c05df8-dfcbccf8-vvssk" event={"ID":"7e77a029-818a-4930-8ad5-6f341fa0a3b3","Type":"ContainerStarted","Data":"03ffa576a14e105aadda91ae774a38891a40d8785ba7c2e32c21afa38bd080f1"} Oct 30 01:27:32.900014 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:32.899966 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-18c05df8-dfcbccf8-vvssk" event={"ID":"7e77a029-818a-4930-8ad5-6f341fa0a3b3","Type":"ContainerStarted","Data":"9599170eb0656e4b2b8bd6413ec72e68bb73e313e173161937c24481b64809bb"} Oct 30 01:27:32.900636 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:32.900030 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-18c05df8-dfcbccf8-vvssk" Oct 30 01:27:32.901955 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:32.901928 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-18c05df8-679684cfd7-dvp7m" event={"ID":"85045c0a-97ae-432c-a346-d60115100fad","Type":"ContainerStarted","Data":"9877c2a88a28bd1bc3a6c9c679f05c3a1fd4eedec7b1688fb73e28f6b3799816"} Oct 30 01:27:32.902025 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:32.901981 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-18c05df8-679684cfd7-dvp7m" event={"ID":"85045c0a-97ae-432c-a346-d60115100fad","Type":"ContainerStarted","Data":"2a2ca160d95b159458a02a7d9b3c47c7dde40193ccb86b3f3159d300bf3eb961"} Oct 30 01:27:32.902079 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:32.902057 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-18c05df8-679684cfd7-dvp7m" Oct 30 01:27:32.957773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:32.946347 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-18c05df8-dfcbccf8-vvssk" podStartSLOduration=1.946326595 podStartE2EDuration="1.946326595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:32.945118638 +0000 UTC m=+578.521544041" watchObservedRunningTime="2026-10-30 01:27:32.946326595 +0000 UTC m=+578.522751995" Oct 30 01:27:32.982560 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:32.982490 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-18c05df8-679684cfd7-dvp7m" podStartSLOduration=1.982471672 podStartE2EDuration="1.982471672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:32.981930499 +0000 UTC m=+578.558355902" watchObservedRunningTime="2026-10-30 01:27:32.982471672 +0000 UTC m=+578.558897074" Oct 30 01:27:34.079733 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:34.079696 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:35.573167 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:35.573072 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:36.841979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:36.841920 6572 patch_prober.go:29] interesting pod/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:27:36.842387 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:36.842026 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" podUID="2b8b0408-785a-4e6f-b6d9-5c80932666ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:27:37.323159 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:37.323124 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:37.847889 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:37.847857 6572 patch_prober.go:29] interesting pod/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:27:37.848257 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:37.847909 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" podUID="ab69c192-5a88-4a77-844c-4306956a9d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:27:38.996390 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:38.996362 6572 generic.go:356] "Generic (PLEG): container finished" podID="2b8b0408-785a-4e6f-b6d9-5c80932666ff" containerID="0d183455f54b4d6ff67c008e3f27844d98a7428b92238d5930f604d4bf662765" exitCode=0 Oct 30 01:27:39.002011 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:38.996445 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" event={"ID":"2b8b0408-785a-4e6f-b6d9-5c80932666ff","Type":"ContainerDied","Data":"0d183455f54b4d6ff67c008e3f27844d98a7428b92238d5930f604d4bf662765"} Oct 30 01:27:39.002011 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:38.996945 6572 scope.go:122] "RemoveContainer" containerID="0d183455f54b4d6ff67c008e3f27844d98a7428b92238d5930f604d4bf662765" Oct 30 01:27:39.002011 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:38.998275 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:27:39.002011 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:38.998340 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:27:39.002011 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:38.998380 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:27:39.068737 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:39.068710 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:39.349720 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:39.349618 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:39.532443 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:39.532404 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:39.870072 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:39.870045 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f29772b9-5b5fd8b8b6-6q426" Oct 30 01:27:39.870231 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:39.870108 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f19ba569-67cb76cc8c-wsn9r" Oct 30 01:27:40.003917 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:40.003885 6572 generic.go:356] "Generic (PLEG): container finished" podID="ab69c192-5a88-4a77-844c-4306956a9d82" containerID="9e02727aa04335839064a9062ddc93357d064ed6b9683fd2601d62b5c4f2a1bc" exitCode=0 Oct 30 01:27:40.004319 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:40.003939 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" event={"ID":"ab69c192-5a88-4a77-844c-4306956a9d82","Type":"ContainerDied","Data":"9e02727aa04335839064a9062ddc93357d064ed6b9683fd2601d62b5c4f2a1bc"} Oct 30 01:27:40.004415 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:40.004393 6572 scope.go:122] "RemoveContainer" containerID="9e02727aa04335839064a9062ddc93357d064ed6b9683fd2601d62b5c4f2a1bc" Oct 30 01:27:40.011775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:40.005735 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:27:40.012024 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:40.011993 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:27:40.017774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:40.012159 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:27:40.026789 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:40.021688 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" event={"ID":"2b8b0408-785a-4e6f-b6d9-5c80932666ff","Type":"ContainerStarted","Data":"fcfbeba47b062052350d41d8cdadd999742231129b3d87c1ea651f317024412d"} Oct 30 01:27:40.056035 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:40.056007 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:27:40.056200 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:40.056180 6572 patch_prober.go:29] interesting pod/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 30 01:27:40.056251 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:40.056232 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" podUID="2b8b0408-785a-4e6f-b6d9-5c80932666ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 30 01:27:40.878313 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:40.878280 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:41.058773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:41.050106 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" event={"ID":"ab69c192-5a88-4a77-844c-4306956a9d82","Type":"ContainerStarted","Data":"ab76f192ce2b103fd56322847ebeb09ce27ce49540f5981a42c889b3fabde864"} Oct 30 01:27:41.058773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:41.057103 6572 patch_prober.go:29] interesting pod/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 30 01:27:41.058773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:41.057170 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" podUID="2b8b0408-785a-4e6f-b6d9-5c80932666ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 30 01:27:41.364551 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:41.364504 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:27:41.364786 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:41.364569 6572 patch_prober.go:29] interesting pod/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 01:27:41.364786 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:41.364612 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" podUID="ab69c192-5a88-4a77-844c-4306956a9d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 01:27:42.056528 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:42.056110 6572 patch_prober.go:29] interesting pod/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 30 01:27:42.056528 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:42.056185 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" podUID="2b8b0408-785a-4e6f-b6d9-5c80932666ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 30 01:27:42.364380 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:42.364267 6572 patch_prober.go:29] interesting pod/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 01:27:42.364380 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:42.364351 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" podUID="ab69c192-5a88-4a77-844c-4306956a9d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 01:27:42.489351 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:42.489202 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:43.056217 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:43.056175 6572 patch_prober.go:29] interesting pod/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 30 01:27:43.056510 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:43.056479 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" podUID="2b8b0408-785a-4e6f-b6d9-5c80932666ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 30 01:27:43.364460 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:43.364197 6572 patch_prober.go:29] interesting pod/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 01:27:43.364460 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:43.364250 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" podUID="ab69c192-5a88-4a77-844c-4306956a9d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 01:27:43.965974 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:43.965938 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-18c05df8-dfcbccf8-vvssk" Oct 30 01:27:43.967872 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:43.967843 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-18c05df8-679684cfd7-dvp7m" Oct 30 01:27:44.068076 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:44.068023 6572 patch_prober.go:29] interesting pod/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 30 01:27:44.068290 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:44.068090 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" podUID="2b8b0408-785a-4e6f-b6d9-5c80932666ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 30 01:27:44.315875 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:44.315783 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:44.364354 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:44.364301 6572 patch_prober.go:29] interesting pod/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 01:27:44.364554 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:44.364364 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" podUID="ab69c192-5a88-4a77-844c-4306956a9d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 01:27:44.687700 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:27:44.687657 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:27:45.056498 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:45.056401 6572 patch_prober.go:29] interesting pod/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 30 01:27:45.056498 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:45.056461 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" podUID="2b8b0408-785a-4e6f-b6d9-5c80932666ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 30 01:27:45.364280 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:45.364190 6572 patch_prober.go:29] interesting pod/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 01:27:45.364280 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:45.364251 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" podUID="ab69c192-5a88-4a77-844c-4306956a9d82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 01:27:46.030855 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:46.029808 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:46.060341 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:46.060305 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:27:46.060647 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:46.060622 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:27:46.061471 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:46.061452 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:27:46.389141 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:46.389053 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:27:46.389716 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:46.389687 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:27:46.390874 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:46.390851 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:27:47.422454 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:47.422421 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:47.616593 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:47.616559 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:47.885473 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:47.885371 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:48.007120 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:48.007072 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f29772b9-bwtgx"] Oct 30 01:27:48.014077 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:48.014039 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f29772b9-bwtgx" Oct 30 01:27:48.031733 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:48.031609 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f29772b9-bwtgx"] Oct 30 01:27:48.038089 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:48.038057 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:27:48.038293 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:48.038071 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:27:48.038468 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:48.038203 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:27:48.038565 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:48.038336 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:27:48.107619 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:48.107580 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-528p8\" (UniqueName: \"kubernetes.io/projected/8e639be8-9b7f-4cbc-a246-6e2b53275009-kube-api-access-528p8\") pod \"maas-api-revoke-keys-e2e-ait-f29772b9-bwtgx\" (UID: \"8e639be8-9b7f-4cbc-a246-6e2b53275009\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f29772b9-bwtgx" Oct 30 01:27:48.107845 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:48.107632 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e639be8-9b7f-4cbc-a246-6e2b53275009-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f29772b9-bwtgx\" (UID: \"8e639be8-9b7f-4cbc-a246-6e2b53275009\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f29772b9-bwtgx" Oct 30 01:27:48.107845 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:48.107782 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e639be8-9b7f-4cbc-a246-6e2b53275009-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f29772b9-bwtgx\" (UID: \"8e639be8-9b7f-4cbc-a246-6e2b53275009\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f29772b9-bwtgx" Oct 30 01:27:48.209410 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:48.209368 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-528p8\" (UniqueName: \"kubernetes.io/projected/8e639be8-9b7f-4cbc-a246-6e2b53275009-kube-api-access-528p8\") pod \"maas-api-revoke-keys-e2e-ait-f29772b9-bwtgx\" (UID: \"8e639be8-9b7f-4cbc-a246-6e2b53275009\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f29772b9-bwtgx" Oct 30 01:27:48.209624 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:48.209429 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e639be8-9b7f-4cbc-a246-6e2b53275009-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f29772b9-bwtgx\" (UID: \"8e639be8-9b7f-4cbc-a246-6e2b53275009\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f29772b9-bwtgx" Oct 30 01:27:48.209624 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:48.209496 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e639be8-9b7f-4cbc-a246-6e2b53275009-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f29772b9-bwtgx\" (UID: \"8e639be8-9b7f-4cbc-a246-6e2b53275009\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f29772b9-bwtgx" Oct 30 01:27:48.210900 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:48.210868 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e639be8-9b7f-4cbc-a246-6e2b53275009-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f29772b9-bwtgx\" (UID: \"8e639be8-9b7f-4cbc-a246-6e2b53275009\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f29772b9-bwtgx" Oct 30 01:27:48.214198 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:48.214169 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e639be8-9b7f-4cbc-a246-6e2b53275009-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f29772b9-bwtgx\" (UID: \"8e639be8-9b7f-4cbc-a246-6e2b53275009\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f29772b9-bwtgx" Oct 30 01:27:48.225758 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:48.225727 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-528p8\" (UniqueName: \"kubernetes.io/projected/8e639be8-9b7f-4cbc-a246-6e2b53275009-kube-api-access-528p8\") pod \"maas-api-revoke-keys-e2e-ait-f29772b9-bwtgx\" (UID: \"8e639be8-9b7f-4cbc-a246-6e2b53275009\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f29772b9-bwtgx" Oct 30 01:27:48.336174 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:48.336135 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f29772b9-bwtgx" Oct 30 01:27:48.632339 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:48.632313 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f29772b9-bwtgx"] Oct 30 01:27:48.634480 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:27:48.634448 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e639be8_9b7f_4cbc_a246_6e2b53275009.slice/crio-91860e0f665b7cee38bac238bd661ec2e76fc0f19a2cf8db699f5d54d6247de2 WatchSource:0}: Error finding container 91860e0f665b7cee38bac238bd661ec2e76fc0f19a2cf8db699f5d54d6247de2: Status 404 returned error can't find the container with id 91860e0f665b7cee38bac238bd661ec2e76fc0f19a2cf8db699f5d54d6247de2 Oct 30 01:27:49.105535 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:49.105499 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f29772b9-bwtgx" event={"ID":"8e639be8-9b7f-4cbc-a246-6e2b53275009","Type":"ContainerStarted","Data":"91860e0f665b7cee38bac238bd661ec2e76fc0f19a2cf8db699f5d54d6247de2"} Oct 30 01:27:49.702136 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:49.702093 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:51.180002 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:51.179967 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:52.892643 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:52.892551 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:53.191463 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:53.191432 6572 generic.go:356] "Generic (PLEG): container finished" podID="8e639be8-9b7f-4cbc-a246-6e2b53275009" containerID="8c4818b9baed1460a0445ab6ec68d4fa13de58ada6979c03276a778513a5a175" exitCode=0 Oct 30 01:27:53.191659 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:53.191513 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f29772b9-bwtgx" event={"ID":"8e639be8-9b7f-4cbc-a246-6e2b53275009","Type":"ContainerDied","Data":"8c4818b9baed1460a0445ab6ec68d4fa13de58ada6979c03276a778513a5a175"} Oct 30 01:27:53.955258 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:53.955219 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f19ba569-67cb76cc8c-wsn9r"] Oct 30 01:27:54.146062 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:54.146023 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:54.345593 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:54.345562 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:54.353602 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:54.353574 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f29772b9-bwtgx" Oct 30 01:27:54.378007 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:54.377981 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e639be8-9b7f-4cbc-a246-6e2b53275009-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e639be8-9b7f-4cbc-a246-6e2b53275009-maas-api-openshift-ca\") pod \"8e639be8-9b7f-4cbc-a246-6e2b53275009\" (UID: \"8e639be8-9b7f-4cbc-a246-6e2b53275009\") " Oct 30 01:27:54.378141 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:54.378029 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e639be8-9b7f-4cbc-a246-6e2b53275009-kube-api-access-528p8\" (UniqueName: \"kubernetes.io/projected/8e639be8-9b7f-4cbc-a246-6e2b53275009-kube-api-access-528p8\") pod \"8e639be8-9b7f-4cbc-a246-6e2b53275009\" (UID: \"8e639be8-9b7f-4cbc-a246-6e2b53275009\") " Oct 30 01:27:54.378141 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:54.378094 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e639be8-9b7f-4cbc-a246-6e2b53275009-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e639be8-9b7f-4cbc-a246-6e2b53275009-maas-api-xks-ca\") pod \"8e639be8-9b7f-4cbc-a246-6e2b53275009\" (UID: \"8e639be8-9b7f-4cbc-a246-6e2b53275009\") " Oct 30 01:27:54.378341 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:54.378317 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e639be8-9b7f-4cbc-a246-6e2b53275009-maas-api-openshift-ca" pod "8e639be8-9b7f-4cbc-a246-6e2b53275009" (UID: "8e639be8-9b7f-4cbc-a246-6e2b53275009"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:54.378495 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:54.378479 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e639be8-9b7f-4cbc-a246-6e2b53275009-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:27:54.380186 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:54.380163 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e639be8-9b7f-4cbc-a246-6e2b53275009-maas-api-xks-ca" pod "8e639be8-9b7f-4cbc-a246-6e2b53275009" (UID: "8e639be8-9b7f-4cbc-a246-6e2b53275009"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:27:54.380290 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:54.380223 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e639be8-9b7f-4cbc-a246-6e2b53275009-kube-api-access-528p8" pod "8e639be8-9b7f-4cbc-a246-6e2b53275009" (UID: "8e639be8-9b7f-4cbc-a246-6e2b53275009"). InnerVolumeSpecName "kube-api-access-528p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:54.479478 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:54.479446 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-528p8\" (UniqueName: \"kubernetes.io/projected/8e639be8-9b7f-4cbc-a246-6e2b53275009-kube-api-access-528p8\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:27:54.479478 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:54.479472 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e639be8-9b7f-4cbc-a246-6e2b53275009-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:27:54.536389 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:54.536270 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:54.777545 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:54.777499 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:54.981999 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:54.981967 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:55.056835 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:55.056803 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g"] Oct 30 01:27:55.057057 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:55.057035 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" podUID="2b8b0408-785a-4e6f-b6d9-5c80932666ff" containerName="istio-proxy" containerID="cri-o://fcfbeba47b062052350d41d8cdadd999742231129b3d87c1ea651f317024412d" gracePeriod=30 Oct 30 01:27:55.193061 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:55.193002 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:55.206404 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:55.206373 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f29772b9-bwtgx" Oct 30 01:27:55.206578 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:55.206373 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f29772b9-bwtgx" event={"ID":"8e639be8-9b7f-4cbc-a246-6e2b53275009","Type":"ContainerDied","Data":"91860e0f665b7cee38bac238bd661ec2e76fc0f19a2cf8db699f5d54d6247de2"} Oct 30 01:27:55.206578 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:55.206503 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91860e0f665b7cee38bac238bd661ec2e76fc0f19a2cf8db699f5d54d6247de2" Oct 30 01:27:55.632543 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:55.632506 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:56.043304 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:56.043271 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:56.523978 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:56.523935 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:56.771672 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:56.771633 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:57.509828 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:57.509790 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:58.265349 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:58.265301 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:58.454979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:58.454941 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:58.750083 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:58.750043 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f29772b9-5b5fd8b8b6-6q426"] Oct 30 01:27:58.915722 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:58.915680 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:58.955791 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:58.955717 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f19ba569-67cb76cc8c-wsn9r" podUID="167e5371-3c5f-4055-b7f4-3e926585a77e" containerName="payload-pre-processing" containerID="cri-o://1adfd64134181ed41656ccdbd97bbc1763487ae55b004fce8bbf2382dd2438c8" gracePeriod=25 Oct 30 01:27:59.126758 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:59.126558 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:59.234322 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:59.234293 6572 generic.go:356] "Generic (PLEG): container finished" podID="167e5371-3c5f-4055-b7f4-3e926585a77e" containerID="1adfd64134181ed41656ccdbd97bbc1763487ae55b004fce8bbf2382dd2438c8" exitCode=0 Oct 30 01:27:59.234505 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:59.234367 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f19ba569-67cb76cc8c-wsn9r" event={"ID":"167e5371-3c5f-4055-b7f4-3e926585a77e","Type":"ContainerDied","Data":"1adfd64134181ed41656ccdbd97bbc1763487ae55b004fce8bbf2382dd2438c8"} Oct 30 01:27:59.234505 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:59.234429 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f19ba569-67cb76cc8c-wsn9r" event={"ID":"167e5371-3c5f-4055-b7f4-3e926585a77e","Type":"ContainerDied","Data":"9b807b309eeda54fda9565b7f182bcee1a70ed0cbbf5964fab50f37c91ca8898"} Oct 30 01:27:59.234505 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:59.234451 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b807b309eeda54fda9565b7f182bcee1a70ed0cbbf5964fab50f37c91ca8898" Oct 30 01:27:59.239657 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:59.239630 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f19ba569-67cb76cc8c-wsn9r" Oct 30 01:27:59.337594 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:59.337491 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:59.342441 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:59.342410 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/167e5371-3c5f-4055-b7f4-3e926585a77e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/167e5371-3c5f-4055-b7f4-3e926585a77e-plugins-config-volume\") pod \"167e5371-3c5f-4055-b7f4-3e926585a77e\" (UID: \"167e5371-3c5f-4055-b7f4-3e926585a77e\") " Oct 30 01:27:59.342627 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:59.342604 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/167e5371-3c5f-4055-b7f4-3e926585a77e-kube-api-access-4fzvl\" (UniqueName: \"kubernetes.io/projected/167e5371-3c5f-4055-b7f4-3e926585a77e-kube-api-access-4fzvl\") pod \"167e5371-3c5f-4055-b7f4-3e926585a77e\" (UID: \"167e5371-3c5f-4055-b7f4-3e926585a77e\") " Oct 30 01:27:59.342861 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:59.342839 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/167e5371-3c5f-4055-b7f4-3e926585a77e-plugins-config-volume" pod "167e5371-3c5f-4055-b7f4-3e926585a77e" (UID: "167e5371-3c5f-4055-b7f4-3e926585a77e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:59.344900 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:59.344879 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/167e5371-3c5f-4055-b7f4-3e926585a77e-kube-api-access-4fzvl" pod "167e5371-3c5f-4055-b7f4-3e926585a77e" (UID: "167e5371-3c5f-4055-b7f4-3e926585a77e"). InnerVolumeSpecName "kube-api-access-4fzvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:59.444288 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:59.444254 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fzvl\" (UniqueName: \"kubernetes.io/projected/167e5371-3c5f-4055-b7f4-3e926585a77e-kube-api-access-4fzvl\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:27:59.444288 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:59.444289 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/167e5371-3c5f-4055-b7f4-3e926585a77e-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:27:59.522936 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:59.522900 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:59.723908 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:27:59.723866 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:00.079347 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.079313 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:00.245761 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.245723 6572 generic.go:356] "Generic (PLEG): container finished" podID="2b8b0408-785a-4e6f-b6d9-5c80932666ff" containerID="fcfbeba47b062052350d41d8cdadd999742231129b3d87c1ea651f317024412d" exitCode=0 Oct 30 01:28:00.245933 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.245781 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" event={"ID":"2b8b0408-785a-4e6f-b6d9-5c80932666ff","Type":"ContainerDied","Data":"fcfbeba47b062052350d41d8cdadd999742231129b3d87c1ea651f317024412d"} Oct 30 01:28:00.245933 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.245852 6572 scope.go:122] "RemoveContainer" containerID="0d183455f54b4d6ff67c008e3f27844d98a7428b92238d5930f604d4bf662765" Oct 30 01:28:00.245933 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.245865 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f19ba569-67cb76cc8c-wsn9r" Oct 30 01:28:00.272553 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.272494 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:00.292308 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.292271 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f19ba569-67cb76cc8c-wsn9r"] Oct 30 01:28:00.301417 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.301377 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f19ba569-67cb76cc8c-wsn9r"] Oct 30 01:28:00.321408 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.321373 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:28:00.454443 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.454402 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b8b0408-785a-4e6f-b6d9-5c80932666ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b8b0408-785a-4e6f-b6d9-5c80932666ff-istiod-ca-cert\") pod \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\" (UID: \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\") " Oct 30 01:28:00.454443 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.454445 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b8b0408-785a-4e6f-b6d9-5c80932666ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b8b0408-785a-4e6f-b6d9-5c80932666ff-istio-data\") pod \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\" (UID: \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\") " Oct 30 01:28:00.454701 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.454464 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b8b0408-785a-4e6f-b6d9-5c80932666ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b8b0408-785a-4e6f-b6d9-5c80932666ff-workload-socket\") pod \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\" (UID: \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\") " Oct 30 01:28:00.454701 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.454494 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b8b0408-785a-4e6f-b6d9-5c80932666ff-kube-api-access-6ndtg\" (UniqueName: \"kubernetes.io/projected/2b8b0408-785a-4e6f-b6d9-5c80932666ff-kube-api-access-6ndtg\") pod \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\" (UID: \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\") " Oct 30 01:28:00.454701 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.454604 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b8b0408-785a-4e6f-b6d9-5c80932666ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b8b0408-785a-4e6f-b6d9-5c80932666ff-istio-podinfo\") pod \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\" (UID: \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\") " Oct 30 01:28:00.454701 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.454639 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b8b0408-785a-4e6f-b6d9-5c80932666ff-istio-token\" (UniqueName: \"kubernetes.io/projected/2b8b0408-785a-4e6f-b6d9-5c80932666ff-istio-token\") pod \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\" (UID: \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\") " Oct 30 01:28:00.454701 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.454692 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b8b0408-785a-4e6f-b6d9-5c80932666ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b8b0408-785a-4e6f-b6d9-5c80932666ff-istio-envoy\") pod \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\" (UID: \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\") " Oct 30 01:28:00.454976 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.454728 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b8b0408-785a-4e6f-b6d9-5c80932666ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b8b0408-785a-4e6f-b6d9-5c80932666ff-credential-socket\") pod \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\" (UID: \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\") " Oct 30 01:28:00.454976 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.454797 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b8b0408-785a-4e6f-b6d9-5c80932666ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b8b0408-785a-4e6f-b6d9-5c80932666ff-workload-certs\") pod \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\" (UID: \"2b8b0408-785a-4e6f-b6d9-5c80932666ff\") " Oct 30 01:28:00.454976 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.454820 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b8b0408-785a-4e6f-b6d9-5c80932666ff-istio-data" pod "2b8b0408-785a-4e6f-b6d9-5c80932666ff" (UID: "2b8b0408-785a-4e6f-b6d9-5c80932666ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:00.454976 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.454855 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b8b0408-785a-4e6f-b6d9-5c80932666ff-workload-socket" pod "2b8b0408-785a-4e6f-b6d9-5c80932666ff" (UID: "2b8b0408-785a-4e6f-b6d9-5c80932666ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:00.455177 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.454986 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b8b0408-785a-4e6f-b6d9-5c80932666ff-istiod-ca-cert" pod "2b8b0408-785a-4e6f-b6d9-5c80932666ff" (UID: "2b8b0408-785a-4e6f-b6d9-5c80932666ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:00.455177 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.455090 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b8b0408-785a-4e6f-b6d9-5c80932666ff-credential-socket" pod "2b8b0408-785a-4e6f-b6d9-5c80932666ff" (UID: "2b8b0408-785a-4e6f-b6d9-5c80932666ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:00.455253 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.455192 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b8b0408-785a-4e6f-b6d9-5c80932666ff-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.455253 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.455212 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b8b0408-785a-4e6f-b6d9-5c80932666ff-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.455253 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.455226 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b8b0408-785a-4e6f-b6d9-5c80932666ff-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.455253 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.455238 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b8b0408-785a-4e6f-b6d9-5c80932666ff-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.455392 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.455294 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b8b0408-785a-4e6f-b6d9-5c80932666ff-workload-certs" pod "2b8b0408-785a-4e6f-b6d9-5c80932666ff" (UID: "2b8b0408-785a-4e6f-b6d9-5c80932666ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:00.456986 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.456952 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b8b0408-785a-4e6f-b6d9-5c80932666ff-kube-api-access-6ndtg" pod "2b8b0408-785a-4e6f-b6d9-5c80932666ff" (UID: "2b8b0408-785a-4e6f-b6d9-5c80932666ff"). InnerVolumeSpecName "kube-api-access-6ndtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:00.457098 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.457044 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b8b0408-785a-4e6f-b6d9-5c80932666ff-istio-envoy" pod "2b8b0408-785a-4e6f-b6d9-5c80932666ff" (UID: "2b8b0408-785a-4e6f-b6d9-5c80932666ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:00.457098 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.457058 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b8b0408-785a-4e6f-b6d9-5c80932666ff-istio-podinfo" pod "2b8b0408-785a-4e6f-b6d9-5c80932666ff" (UID: "2b8b0408-785a-4e6f-b6d9-5c80932666ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:28:00.457098 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.457064 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b8b0408-785a-4e6f-b6d9-5c80932666ff-istio-token" pod "2b8b0408-785a-4e6f-b6d9-5c80932666ff" (UID: "2b8b0408-785a-4e6f-b6d9-5c80932666ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:00.542487 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.542451 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="167e5371-3c5f-4055-b7f4-3e926585a77e" path="/var/lib/kubelet/pods/167e5371-3c5f-4055-b7f4-3e926585a77e/volumes" Oct 30 01:28:00.556022 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.555982 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b8b0408-785a-4e6f-b6d9-5c80932666ff-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.556022 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.556018 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b8b0408-785a-4e6f-b6d9-5c80932666ff-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.556022 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.556027 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b8b0408-785a-4e6f-b6d9-5c80932666ff-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.556271 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.556036 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b8b0408-785a-4e6f-b6d9-5c80932666ff-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.556271 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.556045 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ndtg\" (UniqueName: \"kubernetes.io/projected/2b8b0408-785a-4e6f-b6d9-5c80932666ff-kube-api-access-6ndtg\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.560076 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.560048 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:00.762148 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.762065 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp"] Oct 30 01:28:00.762318 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.762290 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" podUID="ab69c192-5a88-4a77-844c-4306956a9d82" containerName="istio-proxy" containerID="cri-o://ab76f192ce2b103fd56322847ebeb09ce27ce49540f5981a42c889b3fabde864" gracePeriod=30 Oct 30 01:28:00.796929 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.796893 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:00.948199 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.948160 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm"] Oct 30 01:28:00.948664 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.948649 6572 cpu_manager.go:414] "containerMap: removing container" podUID="8e639be8-9b7f-4cbc-a246-6e2b53275009" containerName="revoke-keys" Oct 30 01:28:00.948726 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.948665 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e639be8-9b7f-4cbc-a246-6e2b53275009" containerName="revoke-keys" Oct 30 01:28:00.948726 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.948706 6572 cpu_manager.go:414] "containerMap: removing container" podUID="167e5371-3c5f-4055-b7f4-3e926585a77e" containerName="payload-pre-processing" Oct 30 01:28:00.948726 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.948712 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="167e5371-3c5f-4055-b7f4-3e926585a77e" containerName="payload-pre-processing" Oct 30 01:28:00.948726 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.948720 6572 cpu_manager.go:414] "containerMap: removing container" podUID="2b8b0408-785a-4e6f-b6d9-5c80932666ff" containerName="istio-proxy" Oct 30 01:28:00.948726 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.948725 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b8b0408-785a-4e6f-b6d9-5c80932666ff" containerName="istio-proxy" Oct 30 01:28:00.948910 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.948760 6572 cpu_manager.go:414] "containerMap: removing container" podUID="2b8b0408-785a-4e6f-b6d9-5c80932666ff" containerName="istio-proxy" Oct 30 01:28:00.948910 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.948768 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b8b0408-785a-4e6f-b6d9-5c80932666ff" containerName="istio-proxy" Oct 30 01:28:00.948910 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.948851 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="167e5371-3c5f-4055-b7f4-3e926585a77e" containerName="payload-pre-processing" Oct 30 01:28:00.948910 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.948858 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b8b0408-785a-4e6f-b6d9-5c80932666ff" containerName="istio-proxy" Oct 30 01:28:00.948910 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.948867 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b8b0408-785a-4e6f-b6d9-5c80932666ff" containerName="istio-proxy" Oct 30 01:28:00.948910 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.948874 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e639be8-9b7f-4cbc-a246-6e2b53275009" containerName="revoke-keys" Oct 30 01:28:00.953835 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.953811 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:28:00.973288 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.970326 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-eda881-openshift-default-dockercfg-jwbsd\"" Oct 30 01:28:00.975853 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:00.975816 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm"] Oct 30 01:28:01.050604 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.050516 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:01.059865 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.059826 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb45f4d6-fea9-4316-97c7-0fa7911073cc-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm\" (UID: \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:28:01.060054 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.059876 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49dlf\" (UniqueName: \"kubernetes.io/projected/eb45f4d6-fea9-4316-97c7-0fa7911073cc-kube-api-access-49dlf\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm\" (UID: \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:28:01.060054 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.059949 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb45f4d6-fea9-4316-97c7-0fa7911073cc-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm\" (UID: \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:28:01.060054 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.059982 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb45f4d6-fea9-4316-97c7-0fa7911073cc-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm\" (UID: \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:28:01.060163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.060089 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb45f4d6-fea9-4316-97c7-0fa7911073cc-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm\" (UID: \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:28:01.060163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.060123 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb45f4d6-fea9-4316-97c7-0fa7911073cc-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm\" (UID: \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:28:01.060318 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.060303 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb45f4d6-fea9-4316-97c7-0fa7911073cc-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm\" (UID: \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:28:01.060362 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.060349 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb45f4d6-fea9-4316-97c7-0fa7911073cc-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm\" (UID: \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:28:01.060398 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.060371 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb45f4d6-fea9-4316-97c7-0fa7911073cc-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm\" (UID: \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:28:01.161488 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.161440 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb45f4d6-fea9-4316-97c7-0fa7911073cc-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm\" (UID: \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:28:01.162009 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.161508 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb45f4d6-fea9-4316-97c7-0fa7911073cc-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm\" (UID: \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:28:01.162009 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.161526 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb45f4d6-fea9-4316-97c7-0fa7911073cc-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm\" (UID: \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:28:01.162009 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.161553 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb45f4d6-fea9-4316-97c7-0fa7911073cc-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm\" (UID: \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:28:01.162009 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.161576 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49dlf\" (UniqueName: \"kubernetes.io/projected/eb45f4d6-fea9-4316-97c7-0fa7911073cc-kube-api-access-49dlf\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm\" (UID: \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:28:01.162009 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.161599 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb45f4d6-fea9-4316-97c7-0fa7911073cc-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm\" (UID: \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:28:01.162009 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.161630 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb45f4d6-fea9-4316-97c7-0fa7911073cc-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm\" (UID: \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:28:01.162009 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.161706 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb45f4d6-fea9-4316-97c7-0fa7911073cc-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm\" (UID: \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:28:01.162009 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.161732 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb45f4d6-fea9-4316-97c7-0fa7911073cc-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm\" (UID: \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:28:01.162443 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.162023 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb45f4d6-fea9-4316-97c7-0fa7911073cc-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm\" (UID: \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:28:01.162443 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.162071 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb45f4d6-fea9-4316-97c7-0fa7911073cc-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm\" (UID: \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:28:01.162443 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.162291 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb45f4d6-fea9-4316-97c7-0fa7911073cc-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm\" (UID: \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:28:01.162770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.162726 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb45f4d6-fea9-4316-97c7-0fa7911073cc-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm\" (UID: \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:28:01.162846 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.162737 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb45f4d6-fea9-4316-97c7-0fa7911073cc-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm\" (UID: \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:28:01.163821 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.163801 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb45f4d6-fea9-4316-97c7-0fa7911073cc-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm\" (UID: \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:28:01.164136 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.164114 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb45f4d6-fea9-4316-97c7-0fa7911073cc-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm\" (UID: \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:28:01.178615 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.178575 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb45f4d6-fea9-4316-97c7-0fa7911073cc-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm\" (UID: \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:28:01.178733 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.178691 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49dlf\" (UniqueName: \"kubernetes.io/projected/eb45f4d6-fea9-4316-97c7-0fa7911073cc-kube-api-access-49dlf\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm\" (UID: \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:28:01.251585 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.251542 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:01.253396 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.253370 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" Oct 30 01:28:01.253554 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.253389 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g" event={"ID":"2b8b0408-785a-4e6f-b6d9-5c80932666ff","Type":"ContainerDied","Data":"e00c6866ba07745034f913b3c2d862c5e58dd379d6186a4f0d706606e6c7eda6"} Oct 30 01:28:01.253554 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.253456 6572 scope.go:122] "RemoveContainer" containerID="fcfbeba47b062052350d41d8cdadd999742231129b3d87c1ea651f317024412d" Oct 30 01:28:01.276950 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.276915 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:28:01.301419 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.301336 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g"] Oct 30 01:28:01.309379 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.309339 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-d9801220-gw-openshift-default-678c79997c-hcb9g"] Oct 30 01:28:01.392406 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.392369 6572 patch_prober.go:29] interesting pod/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:28:01.392574 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.392428 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" podUID="ab69c192-5a88-4a77-844c-4306956a9d82" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:28:01.510905 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.510858 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:01.588635 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.588605 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm"] Oct 30 01:28:01.590650 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:28:01.590608 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb45f4d6_fea9_4316_97c7_0fa7911073cc.slice/crio-d9c8ab09ce7e0b547369f5ec92eccca00ac04949985a372d63646696e654dd1b WatchSource:0}: Error finding container d9c8ab09ce7e0b547369f5ec92eccca00ac04949985a372d63646696e654dd1b: Status 404 returned error can't find the container with id d9c8ab09ce7e0b547369f5ec92eccca00ac04949985a372d63646696e654dd1b Oct 30 01:28:01.593565 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.593524 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:28:01.593683 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.593615 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:28:01.593683 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.593668 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:28:01.733348 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:01.733315 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:02.184306 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:02.184263 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:02.261898 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:02.261862 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" event={"ID":"eb45f4d6-fea9-4316-97c7-0fa7911073cc","Type":"ContainerStarted","Data":"44f47ba8ff23967e0342eae0a84955d77338ce62231d4e898a68f1dd83c74a9a"} Oct 30 01:28:02.262092 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:02.261922 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" event={"ID":"eb45f4d6-fea9-4316-97c7-0fa7911073cc","Type":"ContainerStarted","Data":"d9c8ab09ce7e0b547369f5ec92eccca00ac04949985a372d63646696e654dd1b"} Oct 30 01:28:02.277866 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:02.277817 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:28:02.284036 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:02.284004 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:28:02.300628 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:02.300563 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" podStartSLOduration=2.300547967 podStartE2EDuration="2.300547967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:02.299265541 +0000 UTC m=+607.875690947" watchObservedRunningTime="2026-10-30 01:28:02.300547967 +0000 UTC m=+607.876973370" Oct 30 01:28:02.388108 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:02.388072 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:02.542431 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:02.542345 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b8b0408-785a-4e6f-b6d9-5c80932666ff" path="/var/lib/kubelet/pods/2b8b0408-785a-4e6f-b6d9-5c80932666ff/volumes" Oct 30 01:28:02.575362 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:02.575327 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-558327-6bc8566f5f-gbqkb"] Oct 30 01:28:02.579627 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:02.579597 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-558327-6bc8566f5f-gbqkb" Oct 30 01:28:02.585325 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:02.585295 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:02.596154 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:02.595638 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-558327-dockercfg-6pfml\"" Oct 30 01:28:02.599771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:02.599239 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-558327-6bc8566f5f-gbqkb"] Oct 30 01:28:02.651077 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:02.651041 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-558327\"" Oct 30 01:28:02.678665 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:02.678622 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e07b1c7e-fe5d-4dff-b4aa-6d03ce285792-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-558327-6bc8566f5f-gbqkb\" (UID: \"e07b1c7e-fe5d-4dff-b4aa-6d03ce285792\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-558327-6bc8566f5f-gbqkb" Oct 30 01:28:02.678863 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:02.678686 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-229lv\" (UniqueName: \"kubernetes.io/projected/e07b1c7e-fe5d-4dff-b4aa-6d03ce285792-kube-api-access-229lv\") pod \"payload-pre-processing-e2e-worker-w3-558327-6bc8566f5f-gbqkb\" (UID: \"e07b1c7e-fe5d-4dff-b4aa-6d03ce285792\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-558327-6bc8566f5f-gbqkb" Oct 30 01:28:02.779530 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:02.779493 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e07b1c7e-fe5d-4dff-b4aa-6d03ce285792-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-558327-6bc8566f5f-gbqkb\" (UID: \"e07b1c7e-fe5d-4dff-b4aa-6d03ce285792\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-558327-6bc8566f5f-gbqkb" Oct 30 01:28:02.779726 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:02.779556 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-229lv\" (UniqueName: \"kubernetes.io/projected/e07b1c7e-fe5d-4dff-b4aa-6d03ce285792-kube-api-access-229lv\") pod \"payload-pre-processing-e2e-worker-w3-558327-6bc8566f5f-gbqkb\" (UID: \"e07b1c7e-fe5d-4dff-b4aa-6d03ce285792\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-558327-6bc8566f5f-gbqkb" Oct 30 01:28:02.780330 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:02.780305 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e07b1c7e-fe5d-4dff-b4aa-6d03ce285792-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-558327-6bc8566f5f-gbqkb\" (UID: \"e07b1c7e-fe5d-4dff-b4aa-6d03ce285792\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-558327-6bc8566f5f-gbqkb" Oct 30 01:28:02.798449 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:02.798372 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-229lv\" (UniqueName: \"kubernetes.io/projected/e07b1c7e-fe5d-4dff-b4aa-6d03ce285792-kube-api-access-229lv\") pod \"payload-pre-processing-e2e-worker-w3-558327-6bc8566f5f-gbqkb\" (UID: \"e07b1c7e-fe5d-4dff-b4aa-6d03ce285792\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-558327-6bc8566f5f-gbqkb" Oct 30 01:28:02.894529 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:02.894484 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-558327-6bc8566f5f-gbqkb" Oct 30 01:28:02.955287 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:02.955254 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:03.190896 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:03.190871 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-558327-6bc8566f5f-gbqkb"] Oct 30 01:28:03.192846 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:28:03.192818 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode07b1c7e_fe5d_4dff_b4aa_6d03ce285792.slice/crio-e6f858eba8f546bea95f603f9fa04551722e085ade6d7431a151f85aaadf5777 WatchSource:0}: Error finding container e6f858eba8f546bea95f603f9fa04551722e085ade6d7431a151f85aaadf5777: Status 404 returned error can't find the container with id e6f858eba8f546bea95f603f9fa04551722e085ade6d7431a151f85aaadf5777 Oct 30 01:28:03.270100 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:03.270066 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-558327-6bc8566f5f-gbqkb" event={"ID":"e07b1c7e-fe5d-4dff-b4aa-6d03ce285792","Type":"ContainerStarted","Data":"1bfa6bea1affb9adc3567789ae6772fa2789b92b5c8dc869859208be9cf3a785"} Oct 30 01:28:03.270228 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:03.270132 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-558327-6bc8566f5f-gbqkb" event={"ID":"e07b1c7e-fe5d-4dff-b4aa-6d03ce285792","Type":"ContainerStarted","Data":"e6f858eba8f546bea95f603f9fa04551722e085ade6d7431a151f85aaadf5777"} Oct 30 01:28:03.270319 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:03.270301 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-558327-6bc8566f5f-gbqkb" Oct 30 01:28:03.270372 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:03.270352 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:28:03.271253 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:03.271232 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:28:03.306967 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:03.306903 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-558327-6bc8566f5f-gbqkb" podStartSLOduration=1.306885085 podStartE2EDuration="1.306885085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:03.304350591 +0000 UTC m=+608.880775992" watchObservedRunningTime="2026-10-30 01:28:03.306885085 +0000 UTC m=+608.883310486" Oct 30 01:28:03.751154 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:03.751099 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f29772b9-5b5fd8b8b6-6q426" podUID="ef38b4fb-5efb-440a-b9b1-a5848fc5a93b" containerName="payload-processing" containerID="cri-o://1edc53533e2e70d6e5e9bd1a0edbfc8347b1d1e3d9049d54c29251f6e430f475" gracePeriod=25 Oct 30 01:28:03.767973 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:03.767937 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:03.972293 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:03.972257 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:04.022420 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:04.022390 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f29772b9-5b5fd8b8b6-6q426" Oct 30 01:28:04.096138 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:04.096100 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef38b4fb-5efb-440a-b9b1-a5848fc5a93b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef38b4fb-5efb-440a-b9b1-a5848fc5a93b-plugins-config-volume\") pod \"ef38b4fb-5efb-440a-b9b1-a5848fc5a93b\" (UID: \"ef38b4fb-5efb-440a-b9b1-a5848fc5a93b\") " Oct 30 01:28:04.096329 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:04.096176 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef38b4fb-5efb-440a-b9b1-a5848fc5a93b-kube-api-access-tcglw\" (UniqueName: \"kubernetes.io/projected/ef38b4fb-5efb-440a-b9b1-a5848fc5a93b-kube-api-access-tcglw\") pod \"ef38b4fb-5efb-440a-b9b1-a5848fc5a93b\" (UID: \"ef38b4fb-5efb-440a-b9b1-a5848fc5a93b\") " Oct 30 01:28:04.096522 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:04.096500 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef38b4fb-5efb-440a-b9b1-a5848fc5a93b-plugins-config-volume" pod "ef38b4fb-5efb-440a-b9b1-a5848fc5a93b" (UID: "ef38b4fb-5efb-440a-b9b1-a5848fc5a93b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:04.098297 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:04.098270 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef38b4fb-5efb-440a-b9b1-a5848fc5a93b-kube-api-access-tcglw" pod "ef38b4fb-5efb-440a-b9b1-a5848fc5a93b" (UID: "ef38b4fb-5efb-440a-b9b1-a5848fc5a93b"). InnerVolumeSpecName "kube-api-access-tcglw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:04.168046 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:04.168011 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:04.197253 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:04.197213 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef38b4fb-5efb-440a-b9b1-a5848fc5a93b-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:28:04.197253 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:04.197245 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcglw\" (UniqueName: \"kubernetes.io/projected/ef38b4fb-5efb-440a-b9b1-a5848fc5a93b-kube-api-access-tcglw\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:28:04.287709 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:04.287604 6572 generic.go:356] "Generic (PLEG): container finished" podID="ef38b4fb-5efb-440a-b9b1-a5848fc5a93b" containerID="1edc53533e2e70d6e5e9bd1a0edbfc8347b1d1e3d9049d54c29251f6e430f475" exitCode=0 Oct 30 01:28:04.287709 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:04.287677 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f29772b9-5b5fd8b8b6-6q426" Oct 30 01:28:04.287709 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:04.287686 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f29772b9-5b5fd8b8b6-6q426" event={"ID":"ef38b4fb-5efb-440a-b9b1-a5848fc5a93b","Type":"ContainerDied","Data":"1edc53533e2e70d6e5e9bd1a0edbfc8347b1d1e3d9049d54c29251f6e430f475"} Oct 30 01:28:04.288040 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:04.287798 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f29772b9-5b5fd8b8b6-6q426" event={"ID":"ef38b4fb-5efb-440a-b9b1-a5848fc5a93b","Type":"ContainerDied","Data":"abaa97620def1269373a337db98008cb90b7e53f7251e61fb21947ee875cf3b4"} Oct 30 01:28:04.288040 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:04.287824 6572 scope.go:122] "RemoveContainer" containerID="1edc53533e2e70d6e5e9bd1a0edbfc8347b1d1e3d9049d54c29251f6e430f475" Oct 30 01:28:04.299322 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:04.299295 6572 scope.go:122] "RemoveContainer" containerID="1edc53533e2e70d6e5e9bd1a0edbfc8347b1d1e3d9049d54c29251f6e430f475" Oct 30 01:28:04.299787 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:28:04.299719 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1edc53533e2e70d6e5e9bd1a0edbfc8347b1d1e3d9049d54c29251f6e430f475\": container with ID starting with 1edc53533e2e70d6e5e9bd1a0edbfc8347b1d1e3d9049d54c29251f6e430f475 not found: ID does not exist" containerID="1edc53533e2e70d6e5e9bd1a0edbfc8347b1d1e3d9049d54c29251f6e430f475" Oct 30 01:28:04.299880 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:04.299816 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1edc53533e2e70d6e5e9bd1a0edbfc8347b1d1e3d9049d54c29251f6e430f475"} err="failed to get container status \"1edc53533e2e70d6e5e9bd1a0edbfc8347b1d1e3d9049d54c29251f6e430f475\": rpc error: code = NotFound desc = could not find container \"1edc53533e2e70d6e5e9bd1a0edbfc8347b1d1e3d9049d54c29251f6e430f475\": container with ID starting with 1edc53533e2e70d6e5e9bd1a0edbfc8347b1d1e3d9049d54c29251f6e430f475 not found: ID does not exist" Oct 30 01:28:04.334680 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:04.334641 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f29772b9-5b5fd8b8b6-6q426"] Oct 30 01:28:04.344021 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:04.343987 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f29772b9-5b5fd8b8b6-6q426"] Oct 30 01:28:04.543720 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:04.543637 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef38b4fb-5efb-440a-b9b1-a5848fc5a93b" path="/var/lib/kubelet/pods/ef38b4fb-5efb-440a-b9b1-a5848fc5a93b/volumes" Oct 30 01:28:05.005635 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:05.005590 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:05.212197 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:05.212161 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:05.798032 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:05.797996 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:05.837314 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:05.837274 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f29772b9-bwtgx"] Oct 30 01:28:05.846391 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:05.846362 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f29772b9-bwtgx"] Oct 30 01:28:05.878414 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:05.878161 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-18c05df8-dfcbccf8-vvssk"] Oct 30 01:28:05.910598 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:05.910562 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-18c05df8-679684cfd7-dvp7m"] Oct 30 01:28:05.981477 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:05.981430 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:06.057465 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.057436 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:28:06.116719 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.116676 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab69c192-5a88-4a77-844c-4306956a9d82-kube-api-access-r2t9d\" (UniqueName: \"kubernetes.io/projected/ab69c192-5a88-4a77-844c-4306956a9d82-kube-api-access-r2t9d\") pod \"ab69c192-5a88-4a77-844c-4306956a9d82\" (UID: \"ab69c192-5a88-4a77-844c-4306956a9d82\") " Oct 30 01:28:06.116924 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.116744 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab69c192-5a88-4a77-844c-4306956a9d82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab69c192-5a88-4a77-844c-4306956a9d82-istiod-ca-cert\") pod \"ab69c192-5a88-4a77-844c-4306956a9d82\" (UID: \"ab69c192-5a88-4a77-844c-4306956a9d82\") " Oct 30 01:28:06.116924 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.116820 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab69c192-5a88-4a77-844c-4306956a9d82-istio-token\" (UniqueName: \"kubernetes.io/projected/ab69c192-5a88-4a77-844c-4306956a9d82-istio-token\") pod \"ab69c192-5a88-4a77-844c-4306956a9d82\" (UID: \"ab69c192-5a88-4a77-844c-4306956a9d82\") " Oct 30 01:28:06.116924 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.116857 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab69c192-5a88-4a77-844c-4306956a9d82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab69c192-5a88-4a77-844c-4306956a9d82-workload-socket\") pod \"ab69c192-5a88-4a77-844c-4306956a9d82\" (UID: \"ab69c192-5a88-4a77-844c-4306956a9d82\") " Oct 30 01:28:06.116924 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.116894 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab69c192-5a88-4a77-844c-4306956a9d82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab69c192-5a88-4a77-844c-4306956a9d82-credential-socket\") pod \"ab69c192-5a88-4a77-844c-4306956a9d82\" (UID: \"ab69c192-5a88-4a77-844c-4306956a9d82\") " Oct 30 01:28:06.117137 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.116942 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab69c192-5a88-4a77-844c-4306956a9d82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab69c192-5a88-4a77-844c-4306956a9d82-istio-envoy\") pod \"ab69c192-5a88-4a77-844c-4306956a9d82\" (UID: \"ab69c192-5a88-4a77-844c-4306956a9d82\") " Oct 30 01:28:06.117137 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.116986 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab69c192-5a88-4a77-844c-4306956a9d82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab69c192-5a88-4a77-844c-4306956a9d82-istio-data\") pod \"ab69c192-5a88-4a77-844c-4306956a9d82\" (UID: \"ab69c192-5a88-4a77-844c-4306956a9d82\") " Oct 30 01:28:06.117137 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.117024 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab69c192-5a88-4a77-844c-4306956a9d82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab69c192-5a88-4a77-844c-4306956a9d82-istio-podinfo\") pod \"ab69c192-5a88-4a77-844c-4306956a9d82\" (UID: \"ab69c192-5a88-4a77-844c-4306956a9d82\") " Oct 30 01:28:06.117290 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.117137 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab69c192-5a88-4a77-844c-4306956a9d82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab69c192-5a88-4a77-844c-4306956a9d82-workload-certs\") pod \"ab69c192-5a88-4a77-844c-4306956a9d82\" (UID: \"ab69c192-5a88-4a77-844c-4306956a9d82\") " Oct 30 01:28:06.117361 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.117242 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab69c192-5a88-4a77-844c-4306956a9d82-istiod-ca-cert" pod "ab69c192-5a88-4a77-844c-4306956a9d82" (UID: "ab69c192-5a88-4a77-844c-4306956a9d82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:06.117429 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.117252 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab69c192-5a88-4a77-844c-4306956a9d82-workload-socket" pod "ab69c192-5a88-4a77-844c-4306956a9d82" (UID: "ab69c192-5a88-4a77-844c-4306956a9d82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:06.117480 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.117328 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab69c192-5a88-4a77-844c-4306956a9d82-istio-data" pod "ab69c192-5a88-4a77-844c-4306956a9d82" (UID: "ab69c192-5a88-4a77-844c-4306956a9d82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:06.117592 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.117573 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab69c192-5a88-4a77-844c-4306956a9d82-workload-certs" pod "ab69c192-5a88-4a77-844c-4306956a9d82" (UID: "ab69c192-5a88-4a77-844c-4306956a9d82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:06.117673 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.117653 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab69c192-5a88-4a77-844c-4306956a9d82-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:28:06.117727 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.117678 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab69c192-5a88-4a77-844c-4306956a9d82-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:28:06.117727 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.117689 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab69c192-5a88-4a77-844c-4306956a9d82-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:28:06.117727 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.117683 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab69c192-5a88-4a77-844c-4306956a9d82-credential-socket" pod "ab69c192-5a88-4a77-844c-4306956a9d82" (UID: "ab69c192-5a88-4a77-844c-4306956a9d82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:06.117727 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.117700 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab69c192-5a88-4a77-844c-4306956a9d82-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:28:06.119288 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.119259 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab69c192-5a88-4a77-844c-4306956a9d82-kube-api-access-r2t9d" pod "ab69c192-5a88-4a77-844c-4306956a9d82" (UID: "ab69c192-5a88-4a77-844c-4306956a9d82"). InnerVolumeSpecName "kube-api-access-r2t9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:06.119430 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.119407 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab69c192-5a88-4a77-844c-4306956a9d82-istio-token" pod "ab69c192-5a88-4a77-844c-4306956a9d82" (UID: "ab69c192-5a88-4a77-844c-4306956a9d82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:06.119853 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.119828 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab69c192-5a88-4a77-844c-4306956a9d82-istio-podinfo" pod "ab69c192-5a88-4a77-844c-4306956a9d82" (UID: "ab69c192-5a88-4a77-844c-4306956a9d82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:28:06.119853 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.119840 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab69c192-5a88-4a77-844c-4306956a9d82-istio-envoy" pod "ab69c192-5a88-4a77-844c-4306956a9d82" (UID: "ab69c192-5a88-4a77-844c-4306956a9d82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:06.195823 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.195734 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:06.219143 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.219098 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2t9d\" (UniqueName: \"kubernetes.io/projected/ab69c192-5a88-4a77-844c-4306956a9d82-kube-api-access-r2t9d\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:28:06.219143 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.219135 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab69c192-5a88-4a77-844c-4306956a9d82-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:28:06.219552 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.219155 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab69c192-5a88-4a77-844c-4306956a9d82-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:28:06.219552 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.219167 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab69c192-5a88-4a77-844c-4306956a9d82-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:28:06.219552 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.219179 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab69c192-5a88-4a77-844c-4306956a9d82-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:28:06.304118 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.304024 6572 generic.go:356] "Generic (PLEG): container finished" podID="ab69c192-5a88-4a77-844c-4306956a9d82" containerID="ab76f192ce2b103fd56322847ebeb09ce27ce49540f5981a42c889b3fabde864" exitCode=0 Oct 30 01:28:06.304118 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.304080 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" Oct 30 01:28:06.304118 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.304107 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" event={"ID":"ab69c192-5a88-4a77-844c-4306956a9d82","Type":"ContainerDied","Data":"ab76f192ce2b103fd56322847ebeb09ce27ce49540f5981a42c889b3fabde864"} Oct 30 01:28:06.304374 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.304166 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp" event={"ID":"ab69c192-5a88-4a77-844c-4306956a9d82","Type":"ContainerDied","Data":"03119e6fa9d1b75a9ef272d4b9426c0a207911a42db91ba36e8b51077aea335b"} Oct 30 01:28:06.304374 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.304192 6572 scope.go:122] "RemoveContainer" containerID="ab76f192ce2b103fd56322847ebeb09ce27ce49540f5981a42c889b3fabde864" Oct 30 01:28:06.314474 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.314455 6572 scope.go:122] "RemoveContainer" containerID="9e02727aa04335839064a9062ddc93357d064ed6b9683fd2601d62b5c4f2a1bc" Oct 30 01:28:06.324373 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.324352 6572 scope.go:122] "RemoveContainer" containerID="ab76f192ce2b103fd56322847ebeb09ce27ce49540f5981a42c889b3fabde864" Oct 30 01:28:06.324681 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:28:06.324663 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab76f192ce2b103fd56322847ebeb09ce27ce49540f5981a42c889b3fabde864\": container with ID starting with ab76f192ce2b103fd56322847ebeb09ce27ce49540f5981a42c889b3fabde864 not found: ID does not exist" containerID="ab76f192ce2b103fd56322847ebeb09ce27ce49540f5981a42c889b3fabde864" Oct 30 01:28:06.324784 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.324703 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab76f192ce2b103fd56322847ebeb09ce27ce49540f5981a42c889b3fabde864"} err="failed to get container status \"ab76f192ce2b103fd56322847ebeb09ce27ce49540f5981a42c889b3fabde864\": rpc error: code = NotFound desc = could not find container \"ab76f192ce2b103fd56322847ebeb09ce27ce49540f5981a42c889b3fabde864\": container with ID starting with ab76f192ce2b103fd56322847ebeb09ce27ce49540f5981a42c889b3fabde864 not found: ID does not exist" Oct 30 01:28:06.324784 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.324727 6572 scope.go:122] "RemoveContainer" containerID="9e02727aa04335839064a9062ddc93357d064ed6b9683fd2601d62b5c4f2a1bc" Oct 30 01:28:06.325016 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:28:06.324995 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e02727aa04335839064a9062ddc93357d064ed6b9683fd2601d62b5c4f2a1bc\": container with ID starting with 9e02727aa04335839064a9062ddc93357d064ed6b9683fd2601d62b5c4f2a1bc not found: ID does not exist" containerID="9e02727aa04335839064a9062ddc93357d064ed6b9683fd2601d62b5c4f2a1bc" Oct 30 01:28:06.325055 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.325025 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e02727aa04335839064a9062ddc93357d064ed6b9683fd2601d62b5c4f2a1bc"} err="failed to get container status \"9e02727aa04335839064a9062ddc93357d064ed6b9683fd2601d62b5c4f2a1bc\": rpc error: code = NotFound desc = could not find container \"9e02727aa04335839064a9062ddc93357d064ed6b9683fd2601d62b5c4f2a1bc\": container with ID starting with 9e02727aa04335839064a9062ddc93357d064ed6b9683fd2601d62b5c4f2a1bc not found: ID does not exist" Oct 30 01:28:06.365843 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.365802 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp"] Oct 30 01:28:06.372190 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.372160 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f19ba569-openshift-default-d57fd765f-5rvxp"] Oct 30 01:28:06.542175 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.542141 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e639be8-9b7f-4cbc-a246-6e2b53275009" path="/var/lib/kubelet/pods/8e639be8-9b7f-4cbc-a246-6e2b53275009/volumes" Oct 30 01:28:06.542577 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:06.542564 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab69c192-5a88-4a77-844c-4306956a9d82" path="/var/lib/kubelet/pods/ab69c192-5a88-4a77-844c-4306956a9d82/volumes" Oct 30 01:28:07.290536 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:07.290499 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-eda881-5c7cd4d9b-hqnjh"] Oct 30 01:28:07.291191 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:07.290566 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:07.291261 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:07.291241 6572 cpu_manager.go:414] "containerMap: removing container" podUID="ab69c192-5a88-4a77-844c-4306956a9d82" containerName="istio-proxy" Oct 30 01:28:07.291324 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:07.291295 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab69c192-5a88-4a77-844c-4306956a9d82" containerName="istio-proxy" Oct 30 01:28:07.291324 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:07.291316 6572 cpu_manager.go:414] "containerMap: removing container" podUID="ab69c192-5a88-4a77-844c-4306956a9d82" containerName="istio-proxy" Oct 30 01:28:07.291437 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:07.291325 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab69c192-5a88-4a77-844c-4306956a9d82" containerName="istio-proxy" Oct 30 01:28:07.291437 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:07.291361 6572 cpu_manager.go:414] "containerMap: removing container" podUID="ef38b4fb-5efb-440a-b9b1-a5848fc5a93b" containerName="payload-processing" Oct 30 01:28:07.291437 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:07.291370 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef38b4fb-5efb-440a-b9b1-a5848fc5a93b" containerName="payload-processing" Oct 30 01:28:07.291584 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:07.291503 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab69c192-5a88-4a77-844c-4306956a9d82" containerName="istio-proxy" Oct 30 01:28:07.291584 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:07.291516 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab69c192-5a88-4a77-844c-4306956a9d82" containerName="istio-proxy" Oct 30 01:28:07.291584 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:07.291530 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef38b4fb-5efb-440a-b9b1-a5848fc5a93b" containerName="payload-processing" Oct 30 01:28:07.296353 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:07.296325 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-eda881-5c7cd4d9b-hqnjh" Oct 30 01:28:07.313565 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:07.313519 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-eda881-5c7cd4d9b-hqnjh"] Oct 30 01:28:07.319602 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:07.319568 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-eda881\"" Oct 30 01:28:07.359026 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:07.358986 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-eda881-67d94b78bc-6n9dq"] Oct 30 01:28:07.362972 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:07.362944 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eda881-67d94b78bc-6n9dq" Oct 30 01:28:07.387949 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:07.387916 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-eda881-67d94b78bc-6n9dq"] Oct 30 01:28:07.396241 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:07.396206 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-eda881-dockercfg-mclzw\"" Oct 30 01:28:07.436773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:07.436542 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de76834-8ea2-4e53-8e91-ebd04ae69886-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-eda881-5c7cd4d9b-hqnjh\" (UID: \"2de76834-8ea2-4e53-8e91-ebd04ae69886\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-eda881-5c7cd4d9b-hqnjh" Oct 30 01:28:07.436773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:07.436667 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d6c050f-1131-4874-b064-3ef68d3b990b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-eda881-67d94b78bc-6n9dq\" (UID: \"6d6c050f-1131-4874-b064-3ef68d3b990b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eda881-67d94b78bc-6n9dq" Oct 30 01:28:07.436773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:07.436705 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x5mn\" (UniqueName: \"kubernetes.io/projected/6d6c050f-1131-4874-b064-3ef68d3b990b-kube-api-access-7x5mn\") pod \"payload-pre-processing-e2e-worker-w2-eda881-67d94b78bc-6n9dq\" (UID: \"6d6c050f-1131-4874-b064-3ef68d3b990b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eda881-67d94b78bc-6n9dq" Oct 30 01:28:07.437099 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:07.436796 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw7qr\" (UniqueName: \"kubernetes.io/projected/2de76834-8ea2-4e53-8e91-ebd04ae69886-kube-api-access-qw7qr\") pod \"payload-processing-e2e-worker-w2-eda881-5c7cd4d9b-hqnjh\" (UID: \"2de76834-8ea2-4e53-8e91-ebd04ae69886\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-eda881-5c7cd4d9b-hqnjh" Oct 30 01:28:07.501134 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:07.501100 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:07.537988 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:07.537948 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d6c050f-1131-4874-b064-3ef68d3b990b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-eda881-67d94b78bc-6n9dq\" (UID: \"6d6c050f-1131-4874-b064-3ef68d3b990b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eda881-67d94b78bc-6n9dq" Oct 30 01:28:07.538172 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:07.537992 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x5mn\" (UniqueName: \"kubernetes.io/projected/6d6c050f-1131-4874-b064-3ef68d3b990b-kube-api-access-7x5mn\") pod \"payload-pre-processing-e2e-worker-w2-eda881-67d94b78bc-6n9dq\" (UID: \"6d6c050f-1131-4874-b064-3ef68d3b990b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eda881-67d94b78bc-6n9dq" Oct 30 01:28:07.538233 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:07.538171 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw7qr\" (UniqueName: \"kubernetes.io/projected/2de76834-8ea2-4e53-8e91-ebd04ae69886-kube-api-access-qw7qr\") pod \"payload-processing-e2e-worker-w2-eda881-5c7cd4d9b-hqnjh\" (UID: \"2de76834-8ea2-4e53-8e91-ebd04ae69886\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-eda881-5c7cd4d9b-hqnjh" Oct 30 01:28:07.538314 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:07.538280 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de76834-8ea2-4e53-8e91-ebd04ae69886-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-eda881-5c7cd4d9b-hqnjh\" (UID: \"2de76834-8ea2-4e53-8e91-ebd04ae69886\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-eda881-5c7cd4d9b-hqnjh" Oct 30 01:28:07.538871 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:07.538848 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d6c050f-1131-4874-b064-3ef68d3b990b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-eda881-67d94b78bc-6n9dq\" (UID: \"6d6c050f-1131-4874-b064-3ef68d3b990b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eda881-67d94b78bc-6n9dq" Oct 30 01:28:07.539232 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:07.539204 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de76834-8ea2-4e53-8e91-ebd04ae69886-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-eda881-5c7cd4d9b-hqnjh\" (UID: \"2de76834-8ea2-4e53-8e91-ebd04ae69886\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-eda881-5c7cd4d9b-hqnjh" Oct 30 01:28:07.563459 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:07.563372 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw7qr\" (UniqueName: \"kubernetes.io/projected/2de76834-8ea2-4e53-8e91-ebd04ae69886-kube-api-access-qw7qr\") pod \"payload-processing-e2e-worker-w2-eda881-5c7cd4d9b-hqnjh\" (UID: \"2de76834-8ea2-4e53-8e91-ebd04ae69886\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-eda881-5c7cd4d9b-hqnjh" Oct 30 01:28:07.564326 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:07.564300 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x5mn\" (UniqueName: \"kubernetes.io/projected/6d6c050f-1131-4874-b064-3ef68d3b990b-kube-api-access-7x5mn\") pod \"payload-pre-processing-e2e-worker-w2-eda881-67d94b78bc-6n9dq\" (UID: \"6d6c050f-1131-4874-b064-3ef68d3b990b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eda881-67d94b78bc-6n9dq" Oct 30 01:28:07.612164 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:07.612123 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-eda881-5c7cd4d9b-hqnjh" Oct 30 01:28:07.677370 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:07.677097 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eda881-67d94b78bc-6n9dq" Oct 30 01:28:07.718988 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:07.718829 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:07.918240 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:07.918198 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-eda881-5c7cd4d9b-hqnjh"] Oct 30 01:28:07.922090 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:28:07.922047 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2de76834_8ea2_4e53_8e91_ebd04ae69886.slice/crio-5b3eb5123329e0cbf0385683507ad72f0f152da6da1b1753259c5776ce9538a8 WatchSource:0}: Error finding container 5b3eb5123329e0cbf0385683507ad72f0f152da6da1b1753259c5776ce9538a8: Status 404 returned error can't find the container with id 5b3eb5123329e0cbf0385683507ad72f0f152da6da1b1753259c5776ce9538a8 Oct 30 01:28:07.931139 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:07.931104 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:08.036432 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:08.030301 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-eda881-67d94b78bc-6n9dq"] Oct 30 01:28:08.036432 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:28:08.033311 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d6c050f_1131_4874_b064_3ef68d3b990b.slice/crio-d97118b851aaef57b838be4ec92d930b3ee59a1b90c372278b32a8d7114ff9b4 WatchSource:0}: Error finding container d97118b851aaef57b838be4ec92d930b3ee59a1b90c372278b32a8d7114ff9b4: Status 404 returned error can't find the container with id d97118b851aaef57b838be4ec92d930b3ee59a1b90c372278b32a8d7114ff9b4 Oct 30 01:28:08.241561 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:08.241458 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:08.327666 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:08.327626 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-eda881-5c7cd4d9b-hqnjh" event={"ID":"2de76834-8ea2-4e53-8e91-ebd04ae69886","Type":"ContainerStarted","Data":"ef418e27639b8c25bc243d572d9415a6b4434f39f26edd2ce9aabbf9af948479"} Oct 30 01:28:08.328186 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:08.327695 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-eda881-5c7cd4d9b-hqnjh" Oct 30 01:28:08.328186 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:08.327718 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-eda881-5c7cd4d9b-hqnjh" event={"ID":"2de76834-8ea2-4e53-8e91-ebd04ae69886","Type":"ContainerStarted","Data":"5b3eb5123329e0cbf0385683507ad72f0f152da6da1b1753259c5776ce9538a8"} Oct 30 01:28:08.329583 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:08.329550 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eda881-67d94b78bc-6n9dq" event={"ID":"6d6c050f-1131-4874-b064-3ef68d3b990b","Type":"ContainerStarted","Data":"6737b3a464cecce2a8a42b64ad22a0549c0a53b74b0a127f19e40fb215c6e941"} Oct 30 01:28:08.329673 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:08.329602 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eda881-67d94b78bc-6n9dq" event={"ID":"6d6c050f-1131-4874-b064-3ef68d3b990b","Type":"ContainerStarted","Data":"d97118b851aaef57b838be4ec92d930b3ee59a1b90c372278b32a8d7114ff9b4"} Oct 30 01:28:08.329673 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:08.329626 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eda881-67d94b78bc-6n9dq" Oct 30 01:28:08.362884 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:08.362823 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-eda881-5c7cd4d9b-hqnjh" podStartSLOduration=1.36280623 podStartE2EDuration="1.36280623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:08.361180813 +0000 UTC m=+613.937606214" watchObservedRunningTime="2026-10-30 01:28:08.36280623 +0000 UTC m=+613.939231632" Oct 30 01:28:08.396575 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:08.396525 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eda881-67d94b78bc-6n9dq" podStartSLOduration=1.396508681 podStartE2EDuration="1.396508681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:08.394910499 +0000 UTC m=+613.971335900" watchObservedRunningTime="2026-10-30 01:28:08.396508681 +0000 UTC m=+613.972934082" Oct 30 01:28:08.439212 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:08.439178 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:08.656835 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:08.656724 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:08.877545 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:08.877513 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk"] Oct 30 01:28:08.881492 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:08.881468 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 01:28:08.900445 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:08.900411 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:08.902861 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:08.902828 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-803472-openshift-default-dockercfg-hqdb6\"" Oct 30 01:28:08.914989 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:08.914955 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk"] Oct 30 01:28:08.956474 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:08.956439 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f3636d-3177-48b9-88a1-14c250c18bba-istio-podinfo\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk\" (UID: \"f4f3636d-3177-48b9-88a1-14c250c18bba\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 01:28:08.956474 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:08.956479 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f3636d-3177-48b9-88a1-14c250c18bba-istiod-ca-cert\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk\" (UID: \"f4f3636d-3177-48b9-88a1-14c250c18bba\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 01:28:08.956735 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:08.956562 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f3636d-3177-48b9-88a1-14c250c18bba-workload-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk\" (UID: \"f4f3636d-3177-48b9-88a1-14c250c18bba\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 01:28:08.956735 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:08.956612 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f3636d-3177-48b9-88a1-14c250c18bba-workload-certs\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk\" (UID: \"f4f3636d-3177-48b9-88a1-14c250c18bba\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 01:28:08.956884 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:08.956847 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f3636d-3177-48b9-88a1-14c250c18bba-istio-data\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk\" (UID: \"f4f3636d-3177-48b9-88a1-14c250c18bba\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 01:28:08.956936 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:08.956902 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f3636d-3177-48b9-88a1-14c250c18bba-istio-envoy\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk\" (UID: \"f4f3636d-3177-48b9-88a1-14c250c18bba\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 01:28:08.956976 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:08.956939 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f3636d-3177-48b9-88a1-14c250c18bba-istio-token\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk\" (UID: \"f4f3636d-3177-48b9-88a1-14c250c18bba\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 01:28:08.957010 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:08.956975 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z4hf\" (UniqueName: \"kubernetes.io/projected/f4f3636d-3177-48b9-88a1-14c250c18bba-kube-api-access-9z4hf\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk\" (UID: \"f4f3636d-3177-48b9-88a1-14c250c18bba\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 01:28:08.957010 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:08.956999 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f3636d-3177-48b9-88a1-14c250c18bba-credential-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk\" (UID: \"f4f3636d-3177-48b9-88a1-14c250c18bba\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 01:28:09.058246 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:09.058208 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f3636d-3177-48b9-88a1-14c250c18bba-istio-token\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk\" (UID: \"f4f3636d-3177-48b9-88a1-14c250c18bba\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 01:28:09.058461 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:09.058268 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z4hf\" (UniqueName: \"kubernetes.io/projected/f4f3636d-3177-48b9-88a1-14c250c18bba-kube-api-access-9z4hf\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk\" (UID: \"f4f3636d-3177-48b9-88a1-14c250c18bba\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 01:28:09.058461 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:09.058298 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f3636d-3177-48b9-88a1-14c250c18bba-credential-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk\" (UID: \"f4f3636d-3177-48b9-88a1-14c250c18bba\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 01:28:09.058461 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:09.058327 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f3636d-3177-48b9-88a1-14c250c18bba-istio-podinfo\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk\" (UID: \"f4f3636d-3177-48b9-88a1-14c250c18bba\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 01:28:09.058461 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:09.058343 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f3636d-3177-48b9-88a1-14c250c18bba-istiod-ca-cert\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk\" (UID: \"f4f3636d-3177-48b9-88a1-14c250c18bba\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 01:28:09.058461 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:09.058376 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f3636d-3177-48b9-88a1-14c250c18bba-workload-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk\" (UID: \"f4f3636d-3177-48b9-88a1-14c250c18bba\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 01:28:09.058461 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:09.058406 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f3636d-3177-48b9-88a1-14c250c18bba-workload-certs\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk\" (UID: \"f4f3636d-3177-48b9-88a1-14c250c18bba\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 01:28:09.058771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:09.058527 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f3636d-3177-48b9-88a1-14c250c18bba-istio-data\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk\" (UID: \"f4f3636d-3177-48b9-88a1-14c250c18bba\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 01:28:09.058771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:09.058564 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f3636d-3177-48b9-88a1-14c250c18bba-istio-envoy\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk\" (UID: \"f4f3636d-3177-48b9-88a1-14c250c18bba\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 01:28:09.058964 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:09.058901 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f3636d-3177-48b9-88a1-14c250c18bba-workload-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk\" (UID: \"f4f3636d-3177-48b9-88a1-14c250c18bba\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 01:28:09.058964 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:09.058948 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f3636d-3177-48b9-88a1-14c250c18bba-credential-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk\" (UID: \"f4f3636d-3177-48b9-88a1-14c250c18bba\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 01:28:09.059146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:09.059091 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f3636d-3177-48b9-88a1-14c250c18bba-workload-certs\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk\" (UID: \"f4f3636d-3177-48b9-88a1-14c250c18bba\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 01:28:09.059209 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:09.059198 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f3636d-3177-48b9-88a1-14c250c18bba-istio-data\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk\" (UID: \"f4f3636d-3177-48b9-88a1-14c250c18bba\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 01:28:09.059341 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:09.059320 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f3636d-3177-48b9-88a1-14c250c18bba-istiod-ca-cert\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk\" (UID: \"f4f3636d-3177-48b9-88a1-14c250c18bba\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 01:28:09.060970 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:09.060945 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f3636d-3177-48b9-88a1-14c250c18bba-istio-envoy\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk\" (UID: \"f4f3636d-3177-48b9-88a1-14c250c18bba\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 01:28:09.061061 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:09.061006 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f3636d-3177-48b9-88a1-14c250c18bba-istio-podinfo\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk\" (UID: \"f4f3636d-3177-48b9-88a1-14c250c18bba\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 01:28:09.083355 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:09.083318 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z4hf\" (UniqueName: \"kubernetes.io/projected/f4f3636d-3177-48b9-88a1-14c250c18bba-kube-api-access-9z4hf\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk\" (UID: \"f4f3636d-3177-48b9-88a1-14c250c18bba\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 01:28:09.083507 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:09.083319 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f3636d-3177-48b9-88a1-14c250c18bba-istio-token\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk\" (UID: \"f4f3636d-3177-48b9-88a1-14c250c18bba\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 01:28:09.128511 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:09.128475 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:09.196353 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:09.196268 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 01:28:09.327361 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:09.327316 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:09.517501 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:09.517407 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:09.527815 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:09.527789 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk"] Oct 30 01:28:09.529587 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:28:09.529563 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4f3636d_3177_48b9_88a1_14c250c18bba.slice/crio-4f8feef36bed139b6dda96638e6a57f41930d774e807cdb3be71d5ac01e5a33f WatchSource:0}: Error finding container 4f8feef36bed139b6dda96638e6a57f41930d774e807cdb3be71d5ac01e5a33f: Status 404 returned error can't find the container with id 4f8feef36bed139b6dda96638e6a57f41930d774e807cdb3be71d5ac01e5a33f Oct 30 01:28:09.532105 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:09.532069 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:28:09.532178 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:09.532143 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:28:09.532178 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:09.532174 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:28:09.923095 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:09.923058 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:10.349302 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:10.348993 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" event={"ID":"f4f3636d-3177-48b9-88a1-14c250c18bba","Type":"ContainerStarted","Data":"a58560fdefc97caffea04f97a78f5e744d0da3554019f5f85c813ab4a27ff796"} Oct 30 01:28:10.349302 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:10.349068 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" event={"ID":"f4f3636d-3177-48b9-88a1-14c250c18bba","Type":"ContainerStarted","Data":"4f8feef36bed139b6dda96638e6a57f41930d774e807cdb3be71d5ac01e5a33f"} Oct 30 01:28:10.412769 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:10.404355 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:10.412769 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:10.412259 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" podStartSLOduration=2.41223564 podStartE2EDuration="2.41223564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:10.399306071 +0000 UTC m=+615.975731477" watchObservedRunningTime="2026-10-30 01:28:10.41223564 +0000 UTC m=+615.988661042" Oct 30 01:28:10.634674 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:10.634586 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:10.879305 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:10.879262 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-18c05df8-dfcbccf8-vvssk" podUID="7e77a029-818a-4930-8ad5-6f341fa0a3b3" containerName="payload-processing" containerID="cri-o://03ffa576a14e105aadda91ae774a38891a40d8785ba7c2e32c21afa38bd080f1" gracePeriod=25 Oct 30 01:28:10.911861 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:10.911815 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-18c05df8-679684cfd7-dvp7m" podUID="85045c0a-97ae-432c-a346-d60115100fad" containerName="payload-pre-processing" containerID="cri-o://9877c2a88a28bd1bc3a6c9c679f05c3a1fd4eedec7b1688fb73e28f6b3799816" gracePeriod=25 Oct 30 01:28:11.183933 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:11.183837 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-18c05df8-dfcbccf8-vvssk" Oct 30 01:28:11.196613 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:11.196583 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 01:28:11.203459 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:11.203428 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 01:28:11.206615 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:11.206592 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-18c05df8-679684cfd7-dvp7m" Oct 30 01:28:11.283260 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:11.283221 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85045c0a-97ae-432c-a346-d60115100fad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85045c0a-97ae-432c-a346-d60115100fad-plugins-config-volume\") pod \"85045c0a-97ae-432c-a346-d60115100fad\" (UID: \"85045c0a-97ae-432c-a346-d60115100fad\") " Oct 30 01:28:11.283260 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:11.283268 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85045c0a-97ae-432c-a346-d60115100fad-kube-api-access-gj9cd\" (UniqueName: \"kubernetes.io/projected/85045c0a-97ae-432c-a346-d60115100fad-kube-api-access-gj9cd\") pod \"85045c0a-97ae-432c-a346-d60115100fad\" (UID: \"85045c0a-97ae-432c-a346-d60115100fad\") " Oct 30 01:28:11.283510 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:11.283321 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e77a029-818a-4930-8ad5-6f341fa0a3b3-kube-api-access-pkf9s\" (UniqueName: \"kubernetes.io/projected/7e77a029-818a-4930-8ad5-6f341fa0a3b3-kube-api-access-pkf9s\") pod \"7e77a029-818a-4930-8ad5-6f341fa0a3b3\" (UID: \"7e77a029-818a-4930-8ad5-6f341fa0a3b3\") " Oct 30 01:28:11.283510 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:11.283372 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e77a029-818a-4930-8ad5-6f341fa0a3b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e77a029-818a-4930-8ad5-6f341fa0a3b3-plugins-config-volume\") pod \"7e77a029-818a-4930-8ad5-6f341fa0a3b3\" (UID: \"7e77a029-818a-4930-8ad5-6f341fa0a3b3\") " Oct 30 01:28:11.283703 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:11.283672 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85045c0a-97ae-432c-a346-d60115100fad-plugins-config-volume" pod "85045c0a-97ae-432c-a346-d60115100fad" (UID: "85045c0a-97ae-432c-a346-d60115100fad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:11.283864 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:11.283821 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e77a029-818a-4930-8ad5-6f341fa0a3b3-plugins-config-volume" pod "7e77a029-818a-4930-8ad5-6f341fa0a3b3" (UID: "7e77a029-818a-4930-8ad5-6f341fa0a3b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:11.283864 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:11.283850 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85045c0a-97ae-432c-a346-d60115100fad-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:28:11.285546 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:11.285524 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e77a029-818a-4930-8ad5-6f341fa0a3b3-kube-api-access-pkf9s" pod "7e77a029-818a-4930-8ad5-6f341fa0a3b3" (UID: "7e77a029-818a-4930-8ad5-6f341fa0a3b3"). InnerVolumeSpecName "kube-api-access-pkf9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:11.285630 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:11.285580 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85045c0a-97ae-432c-a346-d60115100fad-kube-api-access-gj9cd" pod "85045c0a-97ae-432c-a346-d60115100fad" (UID: "85045c0a-97ae-432c-a346-d60115100fad"). InnerVolumeSpecName "kube-api-access-gj9cd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:11.368376 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:11.368342 6572 generic.go:356] "Generic (PLEG): container finished" podID="7e77a029-818a-4930-8ad5-6f341fa0a3b3" containerID="03ffa576a14e105aadda91ae774a38891a40d8785ba7c2e32c21afa38bd080f1" exitCode=0 Oct 30 01:28:11.368577 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:11.368399 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-18c05df8-dfcbccf8-vvssk" event={"ID":"7e77a029-818a-4930-8ad5-6f341fa0a3b3","Type":"ContainerDied","Data":"03ffa576a14e105aadda91ae774a38891a40d8785ba7c2e32c21afa38bd080f1"} Oct 30 01:28:11.368577 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:11.368407 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-18c05df8-dfcbccf8-vvssk" Oct 30 01:28:11.368577 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:11.368459 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-18c05df8-dfcbccf8-vvssk" event={"ID":"7e77a029-818a-4930-8ad5-6f341fa0a3b3","Type":"ContainerDied","Data":"9599170eb0656e4b2b8bd6413ec72e68bb73e313e173161937c24481b64809bb"} Oct 30 01:28:11.368577 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:11.368486 6572 scope.go:122] "RemoveContainer" containerID="03ffa576a14e105aadda91ae774a38891a40d8785ba7c2e32c21afa38bd080f1" Oct 30 01:28:11.370267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:11.370241 6572 generic.go:356] "Generic (PLEG): container finished" podID="85045c0a-97ae-432c-a346-d60115100fad" containerID="9877c2a88a28bd1bc3a6c9c679f05c3a1fd4eedec7b1688fb73e28f6b3799816" exitCode=0 Oct 30 01:28:11.370387 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:11.370300 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-18c05df8-679684cfd7-dvp7m" Oct 30 01:28:11.370387 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:11.370323 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-18c05df8-679684cfd7-dvp7m" event={"ID":"85045c0a-97ae-432c-a346-d60115100fad","Type":"ContainerDied","Data":"9877c2a88a28bd1bc3a6c9c679f05c3a1fd4eedec7b1688fb73e28f6b3799816"} Oct 30 01:28:11.370387 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:11.370382 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-18c05df8-679684cfd7-dvp7m" event={"ID":"85045c0a-97ae-432c-a346-d60115100fad","Type":"ContainerDied","Data":"2a2ca160d95b159458a02a7d9b3c47c7dde40193ccb86b3f3159d300bf3eb961"} Oct 30 01:28:11.370915 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:11.370737 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 01:28:11.372213 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:11.372191 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 01:28:11.380541 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:11.380519 6572 scope.go:122] "RemoveContainer" containerID="03ffa576a14e105aadda91ae774a38891a40d8785ba7c2e32c21afa38bd080f1" Oct 30 01:28:11.380972 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:28:11.380944 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03ffa576a14e105aadda91ae774a38891a40d8785ba7c2e32c21afa38bd080f1\": container with ID starting with 03ffa576a14e105aadda91ae774a38891a40d8785ba7c2e32c21afa38bd080f1 not found: ID does not exist" containerID="03ffa576a14e105aadda91ae774a38891a40d8785ba7c2e32c21afa38bd080f1" Oct 30 01:28:11.381083 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:11.380983 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03ffa576a14e105aadda91ae774a38891a40d8785ba7c2e32c21afa38bd080f1"} err="failed to get container status \"03ffa576a14e105aadda91ae774a38891a40d8785ba7c2e32c21afa38bd080f1\": rpc error: code = NotFound desc = could not find container \"03ffa576a14e105aadda91ae774a38891a40d8785ba7c2e32c21afa38bd080f1\": container with ID starting with 03ffa576a14e105aadda91ae774a38891a40d8785ba7c2e32c21afa38bd080f1 not found: ID does not exist" Oct 30 01:28:11.381083 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:11.381002 6572 scope.go:122] "RemoveContainer" containerID="9877c2a88a28bd1bc3a6c9c679f05c3a1fd4eedec7b1688fb73e28f6b3799816" Oct 30 01:28:11.384791 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:11.384733 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e77a029-818a-4930-8ad5-6f341fa0a3b3-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:28:11.384873 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:11.384795 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj9cd\" (UniqueName: \"kubernetes.io/projected/85045c0a-97ae-432c-a346-d60115100fad-kube-api-access-gj9cd\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:28:11.384873 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:11.384808 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkf9s\" (UniqueName: \"kubernetes.io/projected/7e77a029-818a-4930-8ad5-6f341fa0a3b3-kube-api-access-pkf9s\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:28:11.392598 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:11.392571 6572 scope.go:122] "RemoveContainer" containerID="9877c2a88a28bd1bc3a6c9c679f05c3a1fd4eedec7b1688fb73e28f6b3799816" Oct 30 01:28:11.393031 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:28:11.392979 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9877c2a88a28bd1bc3a6c9c679f05c3a1fd4eedec7b1688fb73e28f6b3799816\": container with ID starting with 9877c2a88a28bd1bc3a6c9c679f05c3a1fd4eedec7b1688fb73e28f6b3799816 not found: ID does not exist" containerID="9877c2a88a28bd1bc3a6c9c679f05c3a1fd4eedec7b1688fb73e28f6b3799816" Oct 30 01:28:11.393126 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:11.393029 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9877c2a88a28bd1bc3a6c9c679f05c3a1fd4eedec7b1688fb73e28f6b3799816"} err="failed to get container status \"9877c2a88a28bd1bc3a6c9c679f05c3a1fd4eedec7b1688fb73e28f6b3799816\": rpc error: code = NotFound desc = could not find container \"9877c2a88a28bd1bc3a6c9c679f05c3a1fd4eedec7b1688fb73e28f6b3799816\": container with ID starting with 9877c2a88a28bd1bc3a6c9c679f05c3a1fd4eedec7b1688fb73e28f6b3799816 not found: ID does not exist" Oct 30 01:28:11.465306 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:11.465270 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-18c05df8-dfcbccf8-vvssk"] Oct 30 01:28:11.468944 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:11.468911 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-18c05df8-dfcbccf8-vvssk"] Oct 30 01:28:11.497761 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:11.497710 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-18c05df8-679684cfd7-dvp7m"] Oct 30 01:28:11.502144 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:11.502110 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-18c05df8-679684cfd7-dvp7m"] Oct 30 01:28:12.272770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:12.268052 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:12.455719 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:12.455669 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:12.559863 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:12.553580 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e77a029-818a-4930-8ad5-6f341fa0a3b3" path="/var/lib/kubelet/pods/7e77a029-818a-4930-8ad5-6f341fa0a3b3/volumes" Oct 30 01:28:12.560459 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:12.560414 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85045c0a-97ae-432c-a346-d60115100fad" path="/var/lib/kubelet/pods/85045c0a-97ae-432c-a346-d60115100fad/volumes" Oct 30 01:28:12.994981 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:12.994951 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:13.223490 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:13.223452 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:13.737129 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:13.737084 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:13.946907 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:13.946866 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:14.187973 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:14.187926 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:14.291604 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:14.291574 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-558327-6bc8566f5f-gbqkb" Oct 30 01:28:14.688977 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:28:14.688934 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:28:15.232506 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:15.232471 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:15.246834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:15.246794 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-803472-d7dc9c774-4cr8j"] Oct 30 01:28:15.247412 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:15.247392 6572 cpu_manager.go:414] "containerMap: removing container" podUID="7e77a029-818a-4930-8ad5-6f341fa0a3b3" containerName="payload-processing" Oct 30 01:28:15.247476 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:15.247414 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e77a029-818a-4930-8ad5-6f341fa0a3b3" containerName="payload-processing" Oct 30 01:28:15.247476 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:15.247438 6572 cpu_manager.go:414] "containerMap: removing container" podUID="85045c0a-97ae-432c-a346-d60115100fad" containerName="payload-pre-processing" Oct 30 01:28:15.247476 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:15.247443 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85045c0a-97ae-432c-a346-d60115100fad" containerName="payload-pre-processing" Oct 30 01:28:15.247576 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:15.247526 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e77a029-818a-4930-8ad5-6f341fa0a3b3" containerName="payload-processing" Oct 30 01:28:15.247576 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:15.247536 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="85045c0a-97ae-432c-a346-d60115100fad" containerName="payload-pre-processing" Oct 30 01:28:15.252427 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:15.252402 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-803472-d7dc9c774-4cr8j" Oct 30 01:28:15.270819 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:15.270785 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-803472-d7dc9c774-4cr8j"] Oct 30 01:28:15.276369 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:15.276339 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-803472\"" Oct 30 01:28:15.295122 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:15.295085 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-803472-dockercfg-jkmx8\"" Oct 30 01:28:15.338073 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:15.338038 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-803472-5ffb57c959-qll7m"] Oct 30 01:28:15.341856 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:15.341826 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-803472-5ffb57c959-qll7m" Oct 30 01:28:15.347032 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:15.346993 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-803472-5ffb57c959-qll7m"] Oct 30 01:28:15.359361 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:15.359322 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3884225-2803-4ea6-8f09-96e0696ac6d3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-803472-d7dc9c774-4cr8j\" (UID: \"c3884225-2803-4ea6-8f09-96e0696ac6d3\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-803472-d7dc9c774-4cr8j" Oct 30 01:28:15.359523 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:15.359386 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgd56\" (UniqueName: \"kubernetes.io/projected/c3884225-2803-4ea6-8f09-96e0696ac6d3-kube-api-access-rgd56\") pod \"payload-processing-e2e-worker-w4-803472-d7dc9c774-4cr8j\" (UID: \"c3884225-2803-4ea6-8f09-96e0696ac6d3\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-803472-d7dc9c774-4cr8j" Oct 30 01:28:15.431711 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:15.431658 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:15.462228 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:15.460492 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgd56\" (UniqueName: \"kubernetes.io/projected/c3884225-2803-4ea6-8f09-96e0696ac6d3-kube-api-access-rgd56\") pod \"payload-processing-e2e-worker-w4-803472-d7dc9c774-4cr8j\" (UID: \"c3884225-2803-4ea6-8f09-96e0696ac6d3\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-803472-d7dc9c774-4cr8j" Oct 30 01:28:15.462228 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:15.460556 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfzw7\" (UniqueName: \"kubernetes.io/projected/d888789f-78c5-4606-94e1-7fcc77a6c16e-kube-api-access-tfzw7\") pod \"payload-pre-processing-e2e-worker-w4-803472-5ffb57c959-qll7m\" (UID: \"d888789f-78c5-4606-94e1-7fcc77a6c16e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-803472-5ffb57c959-qll7m" Oct 30 01:28:15.462228 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:15.460623 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d888789f-78c5-4606-94e1-7fcc77a6c16e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-803472-5ffb57c959-qll7m\" (UID: \"d888789f-78c5-4606-94e1-7fcc77a6c16e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-803472-5ffb57c959-qll7m" Oct 30 01:28:15.462228 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:15.460739 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3884225-2803-4ea6-8f09-96e0696ac6d3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-803472-d7dc9c774-4cr8j\" (UID: \"c3884225-2803-4ea6-8f09-96e0696ac6d3\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-803472-d7dc9c774-4cr8j" Oct 30 01:28:15.462228 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:15.461840 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3884225-2803-4ea6-8f09-96e0696ac6d3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-803472-d7dc9c774-4cr8j\" (UID: \"c3884225-2803-4ea6-8f09-96e0696ac6d3\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-803472-d7dc9c774-4cr8j" Oct 30 01:28:15.480806 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:15.480775 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgd56\" (UniqueName: \"kubernetes.io/projected/c3884225-2803-4ea6-8f09-96e0696ac6d3-kube-api-access-rgd56\") pod \"payload-processing-e2e-worker-w4-803472-d7dc9c774-4cr8j\" (UID: \"c3884225-2803-4ea6-8f09-96e0696ac6d3\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-803472-d7dc9c774-4cr8j" Oct 30 01:28:15.561589 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:15.561475 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfzw7\" (UniqueName: \"kubernetes.io/projected/d888789f-78c5-4606-94e1-7fcc77a6c16e-kube-api-access-tfzw7\") pod \"payload-pre-processing-e2e-worker-w4-803472-5ffb57c959-qll7m\" (UID: \"d888789f-78c5-4606-94e1-7fcc77a6c16e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-803472-5ffb57c959-qll7m" Oct 30 01:28:15.561589 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:15.561559 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d888789f-78c5-4606-94e1-7fcc77a6c16e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-803472-5ffb57c959-qll7m\" (UID: \"d888789f-78c5-4606-94e1-7fcc77a6c16e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-803472-5ffb57c959-qll7m" Oct 30 01:28:15.562399 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:15.562372 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d888789f-78c5-4606-94e1-7fcc77a6c16e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-803472-5ffb57c959-qll7m\" (UID: \"d888789f-78c5-4606-94e1-7fcc77a6c16e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-803472-5ffb57c959-qll7m" Oct 30 01:28:15.566333 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:15.566297 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-803472-d7dc9c774-4cr8j" Oct 30 01:28:15.578364 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:15.578330 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfzw7\" (UniqueName: \"kubernetes.io/projected/d888789f-78c5-4606-94e1-7fcc77a6c16e-kube-api-access-tfzw7\") pod \"payload-pre-processing-e2e-worker-w4-803472-5ffb57c959-qll7m\" (UID: \"d888789f-78c5-4606-94e1-7fcc77a6c16e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-803472-5ffb57c959-qll7m" Oct 30 01:28:15.634699 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:15.634652 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:15.659203 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:15.658919 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-803472-5ffb57c959-qll7m" Oct 30 01:28:15.903520 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:15.903487 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-803472-d7dc9c774-4cr8j"] Oct 30 01:28:15.905291 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:28:15.905260 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3884225_2803_4ea6_8f09_96e0696ac6d3.slice/crio-340f4879b9233e22d067db22a77f30c140361c3fc9175140e78ce6adc3a298ad WatchSource:0}: Error finding container 340f4879b9233e22d067db22a77f30c140361c3fc9175140e78ce6adc3a298ad: Status 404 returned error can't find the container with id 340f4879b9233e22d067db22a77f30c140361c3fc9175140e78ce6adc3a298ad Oct 30 01:28:16.034233 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:16.034017 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-803472-5ffb57c959-qll7m"] Oct 30 01:28:16.037482 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:28:16.037453 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd888789f_78c5_4606_94e1_7fcc77a6c16e.slice/crio-3b3235c7d646fca355e22ba8156fc7f8b33224829c739f052a5341c91021fddd WatchSource:0}: Error finding container 3b3235c7d646fca355e22ba8156fc7f8b33224829c739f052a5341c91021fddd: Status 404 returned error can't find the container with id 3b3235c7d646fca355e22ba8156fc7f8b33224829c739f052a5341c91021fddd Oct 30 01:28:16.161384 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:16.161344 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:16.355669 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:16.355632 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:16.433449 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:16.432660 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-803472-5ffb57c959-qll7m" event={"ID":"d888789f-78c5-4606-94e1-7fcc77a6c16e","Type":"ContainerStarted","Data":"0ef5b3485d8e8aadbeec2fb15675471d134d8610db80cf70182b00a3d54e661d"} Oct 30 01:28:16.433449 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:16.432761 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-803472-5ffb57c959-qll7m" event={"ID":"d888789f-78c5-4606-94e1-7fcc77a6c16e","Type":"ContainerStarted","Data":"3b3235c7d646fca355e22ba8156fc7f8b33224829c739f052a5341c91021fddd"} Oct 30 01:28:16.433449 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:16.432843 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-803472-5ffb57c959-qll7m" Oct 30 01:28:16.442844 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:16.442792 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-803472-d7dc9c774-4cr8j" event={"ID":"c3884225-2803-4ea6-8f09-96e0696ac6d3","Type":"ContainerStarted","Data":"163c7455042f57e14b1363a3d980771807048ab9a07ea33452bc91145117d08a"} Oct 30 01:28:16.443012 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:16.442909 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-803472-d7dc9c774-4cr8j" event={"ID":"c3884225-2803-4ea6-8f09-96e0696ac6d3","Type":"ContainerStarted","Data":"340f4879b9233e22d067db22a77f30c140361c3fc9175140e78ce6adc3a298ad"} Oct 30 01:28:16.443105 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:16.443082 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-803472-d7dc9c774-4cr8j" Oct 30 01:28:16.477603 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:16.477536 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-803472-5ffb57c959-qll7m" podStartSLOduration=1.4775206920000001 podStartE2EDuration="1.477520692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:16.475382366 +0000 UTC m=+622.051807768" watchObservedRunningTime="2026-10-30 01:28:16.477520692 +0000 UTC m=+622.053946094" Oct 30 01:28:16.932603 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:16.932554 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:17.035653 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:17.035571 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-803472-d7dc9c774-4cr8j" podStartSLOduration=2.035546071 podStartE2EDuration="2.035546071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:16.532097908 +0000 UTC m=+622.108523313" watchObservedRunningTime="2026-10-30 01:28:17.035546071 +0000 UTC m=+622.611971474" Oct 30 01:28:17.039537 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:17.039496 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-cf82f8-5bcf4d65c9-jv8q2"] Oct 30 01:28:17.044496 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:17.044464 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cf82f8-5bcf4d65c9-jv8q2" Oct 30 01:28:17.062039 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:17.061992 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-cf82f8-dockercfg-xrtnv\"" Oct 30 01:28:17.064796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:17.064762 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-cf82f8-5bcf4d65c9-jv8q2"] Oct 30 01:28:17.067283 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:17.067241 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-cf82f8\"" Oct 30 01:28:17.087808 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:17.087762 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-cf82f8-6fcc5c744b-gv5bq"] Oct 30 01:28:17.090075 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:17.090046 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fcc6142-79ba-4dcc-b037-6b2fa054a9e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-cf82f8-5bcf4d65c9-jv8q2\" (UID: \"4fcc6142-79ba-4dcc-b037-6b2fa054a9e3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cf82f8-5bcf4d65c9-jv8q2" Oct 30 01:28:17.090217 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:17.090114 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx5rp\" (UniqueName: \"kubernetes.io/projected/4fcc6142-79ba-4dcc-b037-6b2fa054a9e3-kube-api-access-qx5rp\") pod \"payload-pre-processing-e2e-worker-w1-cf82f8-5bcf4d65c9-jv8q2\" (UID: \"4fcc6142-79ba-4dcc-b037-6b2fa054a9e3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cf82f8-5bcf4d65c9-jv8q2" Oct 30 01:28:17.092624 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:17.092590 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-cf82f8-6fcc5c744b-gv5bq" Oct 30 01:28:17.138430 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:17.135254 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-cf82f8-6fcc5c744b-gv5bq"] Oct 30 01:28:17.158815 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:17.158733 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:17.197897 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:17.197565 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fcc6142-79ba-4dcc-b037-6b2fa054a9e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-cf82f8-5bcf4d65c9-jv8q2\" (UID: \"4fcc6142-79ba-4dcc-b037-6b2fa054a9e3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cf82f8-5bcf4d65c9-jv8q2" Oct 30 01:28:17.200704 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:17.199941 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx5rp\" (UniqueName: \"kubernetes.io/projected/4fcc6142-79ba-4dcc-b037-6b2fa054a9e3-kube-api-access-qx5rp\") pod \"payload-pre-processing-e2e-worker-w1-cf82f8-5bcf4d65c9-jv8q2\" (UID: \"4fcc6142-79ba-4dcc-b037-6b2fa054a9e3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cf82f8-5bcf4d65c9-jv8q2" Oct 30 01:28:17.200704 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:17.199993 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27t48\" (UniqueName: \"kubernetes.io/projected/b1a65b5e-39ec-48ca-9bfc-34e207413e49-kube-api-access-27t48\") pod \"payload-processing-e2e-worker-w1-cf82f8-6fcc5c744b-gv5bq\" (UID: \"b1a65b5e-39ec-48ca-9bfc-34e207413e49\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-cf82f8-6fcc5c744b-gv5bq" Oct 30 01:28:17.200704 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:17.200087 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1a65b5e-39ec-48ca-9bfc-34e207413e49-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-cf82f8-6fcc5c744b-gv5bq\" (UID: \"b1a65b5e-39ec-48ca-9bfc-34e207413e49\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-cf82f8-6fcc5c744b-gv5bq" Oct 30 01:28:17.200704 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:17.200482 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fcc6142-79ba-4dcc-b037-6b2fa054a9e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-cf82f8-5bcf4d65c9-jv8q2\" (UID: \"4fcc6142-79ba-4dcc-b037-6b2fa054a9e3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cf82f8-5bcf4d65c9-jv8q2" Oct 30 01:28:17.237837 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:17.237803 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx5rp\" (UniqueName: \"kubernetes.io/projected/4fcc6142-79ba-4dcc-b037-6b2fa054a9e3-kube-api-access-qx5rp\") pod \"payload-pre-processing-e2e-worker-w1-cf82f8-5bcf4d65c9-jv8q2\" (UID: \"4fcc6142-79ba-4dcc-b037-6b2fa054a9e3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cf82f8-5bcf4d65c9-jv8q2" Oct 30 01:28:17.300897 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:17.300851 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27t48\" (UniqueName: \"kubernetes.io/projected/b1a65b5e-39ec-48ca-9bfc-34e207413e49-kube-api-access-27t48\") pod \"payload-processing-e2e-worker-w1-cf82f8-6fcc5c744b-gv5bq\" (UID: \"b1a65b5e-39ec-48ca-9bfc-34e207413e49\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-cf82f8-6fcc5c744b-gv5bq" Oct 30 01:28:17.301094 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:17.301016 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1a65b5e-39ec-48ca-9bfc-34e207413e49-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-cf82f8-6fcc5c744b-gv5bq\" (UID: \"b1a65b5e-39ec-48ca-9bfc-34e207413e49\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-cf82f8-6fcc5c744b-gv5bq" Oct 30 01:28:17.301875 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:17.301854 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1a65b5e-39ec-48ca-9bfc-34e207413e49-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-cf82f8-6fcc5c744b-gv5bq\" (UID: \"b1a65b5e-39ec-48ca-9bfc-34e207413e49\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-cf82f8-6fcc5c744b-gv5bq" Oct 30 01:28:17.336422 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:17.335513 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27t48\" (UniqueName: \"kubernetes.io/projected/b1a65b5e-39ec-48ca-9bfc-34e207413e49-kube-api-access-27t48\") pod \"payload-processing-e2e-worker-w1-cf82f8-6fcc5c744b-gv5bq\" (UID: \"b1a65b5e-39ec-48ca-9bfc-34e207413e49\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-cf82f8-6fcc5c744b-gv5bq" Oct 30 01:28:17.360639 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:17.360598 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cf82f8-5bcf4d65c9-jv8q2" Oct 30 01:28:17.465983 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:17.465957 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-cf82f8-6fcc5c744b-gv5bq" Oct 30 01:28:17.582551 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:17.582514 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:17.740718 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:17.740329 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-cf82f8-5bcf4d65c9-jv8q2"] Oct 30 01:28:17.742257 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:28:17.742217 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fcc6142_79ba_4dcc_b037_6b2fa054a9e3.slice/crio-020562d7a532e4a0bb4d049ba4a45795d3047877f6f7c6eea0bb3559100efeeb WatchSource:0}: Error finding container 020562d7a532e4a0bb4d049ba4a45795d3047877f6f7c6eea0bb3559100efeeb: Status 404 returned error can't find the container with id 020562d7a532e4a0bb4d049ba4a45795d3047877f6f7c6eea0bb3559100efeeb Oct 30 01:28:17.813394 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:17.813362 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-cf82f8-6fcc5c744b-gv5bq"] Oct 30 01:28:17.818165 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:28:17.818125 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1a65b5e_39ec_48ca_9bfc_34e207413e49.slice/crio-1dc891eebd4c7305e0208316a9fd5f2de2f05a0e112e1cbfb52c9a9af5194f93 WatchSource:0}: Error finding container 1dc891eebd4c7305e0208316a9fd5f2de2f05a0e112e1cbfb52c9a9af5194f93: Status 404 returned error can't find the container with id 1dc891eebd4c7305e0208316a9fd5f2de2f05a0e112e1cbfb52c9a9af5194f93 Oct 30 01:28:17.993422 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:17.993316 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:18.498893 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:18.498850 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-cf82f8-6fcc5c744b-gv5bq" event={"ID":"b1a65b5e-39ec-48ca-9bfc-34e207413e49","Type":"ContainerStarted","Data":"782dcae789f98a60b662b84c9d6b7c664225e758873ec5c9815f53e11b8c44ea"} Oct 30 01:28:18.499379 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:18.498921 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-cf82f8-6fcc5c744b-gv5bq" event={"ID":"b1a65b5e-39ec-48ca-9bfc-34e207413e49","Type":"ContainerStarted","Data":"1dc891eebd4c7305e0208316a9fd5f2de2f05a0e112e1cbfb52c9a9af5194f93"} Oct 30 01:28:18.499379 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:18.499078 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-cf82f8-6fcc5c744b-gv5bq" Oct 30 01:28:18.500705 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:18.500666 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cf82f8-5bcf4d65c9-jv8q2" event={"ID":"4fcc6142-79ba-4dcc-b037-6b2fa054a9e3","Type":"ContainerStarted","Data":"b9a9d63d4cf89a45aa3e46991910ffb5a7c96d6d6d2a79bf257ec40262c57672"} Oct 30 01:28:18.500896 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:18.500726 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cf82f8-5bcf4d65c9-jv8q2" event={"ID":"4fcc6142-79ba-4dcc-b037-6b2fa054a9e3","Type":"ContainerStarted","Data":"020562d7a532e4a0bb4d049ba4a45795d3047877f6f7c6eea0bb3559100efeeb"} Oct 30 01:28:18.500896 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:18.500794 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cf82f8-5bcf4d65c9-jv8q2" Oct 30 01:28:18.533348 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:18.533289 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-cf82f8-6fcc5c744b-gv5bq" podStartSLOduration=1.533273065 podStartE2EDuration="1.533273065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:18.531701895 +0000 UTC m=+624.108127296" watchObservedRunningTime="2026-10-30 01:28:18.533273065 +0000 UTC m=+624.109698466" Oct 30 01:28:18.567594 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:18.567532 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cf82f8-5bcf4d65c9-jv8q2" podStartSLOduration=1.567513222 podStartE2EDuration="1.567513222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:18.564518791 +0000 UTC m=+624.140944192" watchObservedRunningTime="2026-10-30 01:28:18.567513222 +0000 UTC m=+624.143938623" Oct 30 01:28:19.311991 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:19.311935 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:28:19.312182 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:19.312018 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" podUID="eb45f4d6-fea9-4316-97c7-0fa7911073cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:28:19.341644 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:19.341611 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eda881-67d94b78bc-6n9dq" Oct 30 01:28:19.342060 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:19.342039 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-eda881-5c7cd4d9b-hqnjh" Oct 30 01:28:19.778535 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:19.778501 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:19.967737 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:19.967699 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:20.494094 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:20.494048 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:20.696262 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:20.696225 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:21.481181 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:21.481141 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:21.523978 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:21.523944 6572 generic.go:356] "Generic (PLEG): container finished" podID="eb45f4d6-fea9-4316-97c7-0fa7911073cc" containerID="44f47ba8ff23967e0342eae0a84955d77338ce62231d4e898a68f1dd83c74a9a" exitCode=0 Oct 30 01:28:21.524162 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:21.524036 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" event={"ID":"eb45f4d6-fea9-4316-97c7-0fa7911073cc","Type":"ContainerDied","Data":"44f47ba8ff23967e0342eae0a84955d77338ce62231d4e898a68f1dd83c74a9a"} Oct 30 01:28:21.524649 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:21.524625 6572 scope.go:122] "RemoveContainer" containerID="44f47ba8ff23967e0342eae0a84955d77338ce62231d4e898a68f1dd83c74a9a" Oct 30 01:28:21.526385 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:21.526345 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:28:21.526468 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:21.526452 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:28:21.526525 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:21.526508 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:28:22.541326 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:22.541287 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" event={"ID":"eb45f4d6-fea9-4316-97c7-0fa7911073cc","Type":"ContainerStarted","Data":"26ee68b38be5a726933e3a4b35f1bdfd18674eb69cc9b9f71fb39aa566f38bd5"} Oct 30 01:28:23.149582 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:23.149538 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:23.277919 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:23.277880 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:28:23.278186 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:23.278160 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 01:28:23.278281 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:23.278233 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" podUID="eb45f4d6-fea9-4316-97c7-0fa7911073cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 01:28:23.438478 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:23.438444 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:24.278117 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:24.278074 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 01:28:24.278550 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:24.278135 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" podUID="eb45f4d6-fea9-4316-97c7-0fa7911073cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 01:28:25.082688 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:25.082659 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:25.284112 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:25.284072 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 01:28:25.284725 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:25.284691 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" podUID="eb45f4d6-fea9-4316-97c7-0fa7911073cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 01:28:25.863565 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:25.863511 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:26.277910 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:26.277869 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 01:28:26.278096 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:26.277927 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" podUID="eb45f4d6-fea9-4316-97c7-0fa7911073cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 01:28:26.486661 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:26.486612 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:26.693703 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:26.693675 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:26.906620 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:26.906583 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:27.278212 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:27.278172 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 01:28:27.278416 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:27.278251 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" podUID="eb45f4d6-fea9-4316-97c7-0fa7911073cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 01:28:27.494388 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:27.494354 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-803472-d7dc9c774-4cr8j" Oct 30 01:28:27.494920 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:27.494879 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-803472-5ffb57c959-qll7m" Oct 30 01:28:28.281473 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:28.281434 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:28:28.281773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:28.281732 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:28:28.282450 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:28.282432 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:28:28.432115 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:28.432080 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:29.274467 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:29.274424 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:29.536490 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:29.536390 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cf82f8-5bcf4d65c9-jv8q2" Oct 30 01:28:29.537463 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:29.537437 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-cf82f8-6fcc5c744b-gv5bq" Oct 30 01:28:30.106261 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:30.106228 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:31.551998 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:31.551963 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:31.737015 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:31.736972 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:32.989968 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:32.989927 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:33.475893 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:33.475811 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:33.687072 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:33.687040 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:33.877623 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:33.877532 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:35.308780 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:35.308651 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:35.628140 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:35.628042 6572 generic.go:356] "Generic (PLEG): container finished" podID="f4f3636d-3177-48b9-88a1-14c250c18bba" containerID="a58560fdefc97caffea04f97a78f5e744d0da3554019f5f85c813ab4a27ff796" exitCode=0 Oct 30 01:28:35.628301 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:35.628125 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" event={"ID":"f4f3636d-3177-48b9-88a1-14c250c18bba","Type":"ContainerDied","Data":"a58560fdefc97caffea04f97a78f5e744d0da3554019f5f85c813ab4a27ff796"} Oct 30 01:28:35.628701 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:35.628668 6572 scope.go:122] "RemoveContainer" containerID="a58560fdefc97caffea04f97a78f5e744d0da3554019f5f85c813ab4a27ff796" Oct 30 01:28:35.630353 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:35.630312 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:28:35.630529 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:35.630505 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:28:35.630607 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:35.630571 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:28:36.644313 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:36.644271 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" event={"ID":"f4f3636d-3177-48b9-88a1-14c250c18bba","Type":"ContainerStarted","Data":"dbb780afbb7c78259bc02247888634c5e49991148e4f036b8be5f7af7003edac"} Oct 30 01:28:37.141364 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:37.141327 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:37.196764 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:37.196721 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 01:28:37.197985 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:37.197960 6572 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 01:28:37.198100 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:37.198030 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" podUID="f4f3636d-3177-48b9-88a1-14c250c18bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 01:28:38.196961 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:38.196926 6572 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 01:28:38.197368 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:38.197015 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" podUID="f4f3636d-3177-48b9-88a1-14c250c18bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 01:28:38.730814 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:38.727662 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:38.943863 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:38.943825 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:39.197766 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:39.197725 6572 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 01:28:39.198279 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:39.197816 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" podUID="f4f3636d-3177-48b9-88a1-14c250c18bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 01:28:39.536546 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:39.536459 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:39.743978 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:39.743939 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:40.196906 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:40.196865 6572 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 01:28:40.197089 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:40.196933 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" podUID="f4f3636d-3177-48b9-88a1-14c250c18bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 01:28:40.455719 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:40.455638 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:40.738689 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:40.738602 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn"] Oct 30 01:28:40.743996 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:40.743969 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" Oct 30 01:28:40.762851 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:40.762821 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn"] Oct 30 01:28:40.769566 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:40.769537 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-803472\"/\"openshift-service-ca.crt\"" Oct 30 01:28:40.769708 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:40.769591 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-803472\"/\"kube-root-ca.crt\"" Oct 30 01:28:40.841937 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:40.841905 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-803472\"/\"default-dockercfg-q52mv\"" Oct 30 01:28:40.841937 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:40.841932 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-803472\"/\"simulator-w4-803472-kserve-self-signed-certs\"" Oct 30 01:28:40.848476 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:40.848442 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fac3ca55-64af-42d9-90b1-0e821b893282-home\") pod \"simulator-w4-803472-kserve-69d5d5f45c-n75kn\" (UID: \"fac3ca55-64af-42d9-90b1-0e821b893282\") " pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" Oct 30 01:28:40.848615 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:40.848484 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fac3ca55-64af-42d9-90b1-0e821b893282-model-cache\") pod \"simulator-w4-803472-kserve-69d5d5f45c-n75kn\" (UID: \"fac3ca55-64af-42d9-90b1-0e821b893282\") " pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" Oct 30 01:28:40.848615 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:40.848517 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fac3ca55-64af-42d9-90b1-0e821b893282-tmp-dir\") pod \"simulator-w4-803472-kserve-69d5d5f45c-n75kn\" (UID: \"fac3ca55-64af-42d9-90b1-0e821b893282\") " pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" Oct 30 01:28:40.848699 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:40.848635 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fac3ca55-64af-42d9-90b1-0e821b893282-dshm\") pod \"simulator-w4-803472-kserve-69d5d5f45c-n75kn\" (UID: \"fac3ca55-64af-42d9-90b1-0e821b893282\") " pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" Oct 30 01:28:40.848699 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:40.848680 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rlnn\" (UniqueName: \"kubernetes.io/projected/fac3ca55-64af-42d9-90b1-0e821b893282-kube-api-access-8rlnn\") pod \"simulator-w4-803472-kserve-69d5d5f45c-n75kn\" (UID: \"fac3ca55-64af-42d9-90b1-0e821b893282\") " pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" Oct 30 01:28:40.848843 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:40.848823 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fac3ca55-64af-42d9-90b1-0e821b893282-tls-certs\") pod \"simulator-w4-803472-kserve-69d5d5f45c-n75kn\" (UID: \"fac3ca55-64af-42d9-90b1-0e821b893282\") " pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" Oct 30 01:28:40.889684 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:40.889655 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:40.950499 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:40.950462 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fac3ca55-64af-42d9-90b1-0e821b893282-home\") pod \"simulator-w4-803472-kserve-69d5d5f45c-n75kn\" (UID: \"fac3ca55-64af-42d9-90b1-0e821b893282\") " pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" Oct 30 01:28:40.950499 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:40.950502 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fac3ca55-64af-42d9-90b1-0e821b893282-model-cache\") pod \"simulator-w4-803472-kserve-69d5d5f45c-n75kn\" (UID: \"fac3ca55-64af-42d9-90b1-0e821b893282\") " pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" Oct 30 01:28:40.950736 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:40.950524 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fac3ca55-64af-42d9-90b1-0e821b893282-tmp-dir\") pod \"simulator-w4-803472-kserve-69d5d5f45c-n75kn\" (UID: \"fac3ca55-64af-42d9-90b1-0e821b893282\") " pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" Oct 30 01:28:40.950736 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:40.950559 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fac3ca55-64af-42d9-90b1-0e821b893282-dshm\") pod \"simulator-w4-803472-kserve-69d5d5f45c-n75kn\" (UID: \"fac3ca55-64af-42d9-90b1-0e821b893282\") " pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" Oct 30 01:28:40.950736 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:40.950602 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rlnn\" (UniqueName: \"kubernetes.io/projected/fac3ca55-64af-42d9-90b1-0e821b893282-kube-api-access-8rlnn\") pod \"simulator-w4-803472-kserve-69d5d5f45c-n75kn\" (UID: \"fac3ca55-64af-42d9-90b1-0e821b893282\") " pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" Oct 30 01:28:40.950736 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:40.950650 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fac3ca55-64af-42d9-90b1-0e821b893282-tls-certs\") pod \"simulator-w4-803472-kserve-69d5d5f45c-n75kn\" (UID: \"fac3ca55-64af-42d9-90b1-0e821b893282\") " pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" Oct 30 01:28:40.951006 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:40.950977 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fac3ca55-64af-42d9-90b1-0e821b893282-home\") pod \"simulator-w4-803472-kserve-69d5d5f45c-n75kn\" (UID: \"fac3ca55-64af-42d9-90b1-0e821b893282\") " pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" Oct 30 01:28:40.951057 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:40.951029 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fac3ca55-64af-42d9-90b1-0e821b893282-model-cache\") pod \"simulator-w4-803472-kserve-69d5d5f45c-n75kn\" (UID: \"fac3ca55-64af-42d9-90b1-0e821b893282\") " pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" Oct 30 01:28:40.951057 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:40.951044 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fac3ca55-64af-42d9-90b1-0e821b893282-tmp-dir\") pod \"simulator-w4-803472-kserve-69d5d5f45c-n75kn\" (UID: \"fac3ca55-64af-42d9-90b1-0e821b893282\") " pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" Oct 30 01:28:40.952973 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:40.952951 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fac3ca55-64af-42d9-90b1-0e821b893282-dshm\") pod \"simulator-w4-803472-kserve-69d5d5f45c-n75kn\" (UID: \"fac3ca55-64af-42d9-90b1-0e821b893282\") " pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" Oct 30 01:28:40.953240 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:40.953223 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fac3ca55-64af-42d9-90b1-0e821b893282-tls-certs\") pod \"simulator-w4-803472-kserve-69d5d5f45c-n75kn\" (UID: \"fac3ca55-64af-42d9-90b1-0e821b893282\") " pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" Oct 30 01:28:40.980696 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:40.980650 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rlnn\" (UniqueName: \"kubernetes.io/projected/fac3ca55-64af-42d9-90b1-0e821b893282-kube-api-access-8rlnn\") pod \"simulator-w4-803472-kserve-69d5d5f45c-n75kn\" (UID: \"fac3ca55-64af-42d9-90b1-0e821b893282\") " pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" Oct 30 01:28:41.060714 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:41.060623 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" Oct 30 01:28:41.197593 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:41.197560 6572 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 01:28:41.197774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:41.197617 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" podUID="f4f3636d-3177-48b9-88a1-14c250c18bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 01:28:41.370753 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:41.370662 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 01:28:41.377797 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:41.377253 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn"] Oct 30 01:28:41.388773 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:28:41.386950 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfac3ca55_64af_42d9_90b1_0e821b893282.slice/crio-531ae5a539bc2904251050dbadbb9b0a3a598eafc68163c44b51308c545f1ec0 WatchSource:0}: Error finding container 531ae5a539bc2904251050dbadbb9b0a3a598eafc68163c44b51308c545f1ec0: Status 404 returned error can't find the container with id 531ae5a539bc2904251050dbadbb9b0a3a598eafc68163c44b51308c545f1ec0 Oct 30 01:28:41.682075 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:41.682028 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" event={"ID":"fac3ca55-64af-42d9-90b1-0e821b893282","Type":"ContainerStarted","Data":"a27836cd757f1537db62bc82910b80f5d5cbf7494695ef25a9f49c9f95e2b840"} Oct 30 01:28:41.682592 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:41.682093 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" event={"ID":"fac3ca55-64af-42d9-90b1-0e821b893282","Type":"ContainerStarted","Data":"531ae5a539bc2904251050dbadbb9b0a3a598eafc68163c44b51308c545f1ec0"} Oct 30 01:28:41.715777 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:41.715683 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" podStartSLOduration=1.715661629 podStartE2EDuration="1.715661629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:41.714680169 +0000 UTC m=+647.291105572" watchObservedRunningTime="2026-10-30 01:28:41.715661629 +0000 UTC m=+647.292087032" Oct 30 01:28:41.980244 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:41.980119 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:42.061568 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:42.061527 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" Oct 30 01:28:42.201169 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:42.201136 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 01:28:42.202472 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:42.202451 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 01:28:43.572713 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:43.572666 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:44.688431 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:28:44.688391 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:28:45.082409 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:45.082304 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:46.764526 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:46.764489 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:46.819604 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:46.819553 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6"] Oct 30 01:28:46.825034 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:46.825003 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" Oct 30 01:28:46.839770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:46.839721 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6"] Oct 30 01:28:46.846798 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:46.845859 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-558327\"/\"openshift-service-ca.crt\"" Oct 30 01:28:46.846798 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:46.845871 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-558327\"/\"kube-root-ca.crt\"" Oct 30 01:28:46.847088 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:46.847006 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-558327\"/\"default-dockercfg-bz6bf\"" Oct 30 01:28:46.847088 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:46.847040 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-558327\"/\"premium-w3-558327-kserve-self-signed-certs\"" Oct 30 01:28:46.970372 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:46.970337 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:47.031595 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:47.031494 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/317505ef-238c-458e-a74e-1198806b94d2-tls-certs\") pod \"premium-w3-558327-kserve-5dc6b96f5-z9wx6\" (UID: \"317505ef-238c-458e-a74e-1198806b94d2\") " pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" Oct 30 01:28:47.031951 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:47.031915 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj8b9\" (UniqueName: \"kubernetes.io/projected/317505ef-238c-458e-a74e-1198806b94d2-kube-api-access-kj8b9\") pod \"premium-w3-558327-kserve-5dc6b96f5-z9wx6\" (UID: \"317505ef-238c-458e-a74e-1198806b94d2\") " pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" Oct 30 01:28:47.032127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:47.032044 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/317505ef-238c-458e-a74e-1198806b94d2-dshm\") pod \"premium-w3-558327-kserve-5dc6b96f5-z9wx6\" (UID: \"317505ef-238c-458e-a74e-1198806b94d2\") " pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" Oct 30 01:28:47.032127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:47.032080 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/317505ef-238c-458e-a74e-1198806b94d2-model-cache\") pod \"premium-w3-558327-kserve-5dc6b96f5-z9wx6\" (UID: \"317505ef-238c-458e-a74e-1198806b94d2\") " pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" Oct 30 01:28:47.032250 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:47.032170 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/317505ef-238c-458e-a74e-1198806b94d2-tmp-dir\") pod \"premium-w3-558327-kserve-5dc6b96f5-z9wx6\" (UID: \"317505ef-238c-458e-a74e-1198806b94d2\") " pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" Oct 30 01:28:47.032250 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:47.032209 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/317505ef-238c-458e-a74e-1198806b94d2-home\") pod \"premium-w3-558327-kserve-5dc6b96f5-z9wx6\" (UID: \"317505ef-238c-458e-a74e-1198806b94d2\") " pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" Oct 30 01:28:47.133058 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:47.133018 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/317505ef-238c-458e-a74e-1198806b94d2-tls-certs\") pod \"premium-w3-558327-kserve-5dc6b96f5-z9wx6\" (UID: \"317505ef-238c-458e-a74e-1198806b94d2\") " pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" Oct 30 01:28:47.133058 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:47.133073 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj8b9\" (UniqueName: \"kubernetes.io/projected/317505ef-238c-458e-a74e-1198806b94d2-kube-api-access-kj8b9\") pod \"premium-w3-558327-kserve-5dc6b96f5-z9wx6\" (UID: \"317505ef-238c-458e-a74e-1198806b94d2\") " pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" Oct 30 01:28:47.133341 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:47.133164 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/317505ef-238c-458e-a74e-1198806b94d2-dshm\") pod \"premium-w3-558327-kserve-5dc6b96f5-z9wx6\" (UID: \"317505ef-238c-458e-a74e-1198806b94d2\") " pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" Oct 30 01:28:47.133341 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:47.133193 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/317505ef-238c-458e-a74e-1198806b94d2-model-cache\") pod \"premium-w3-558327-kserve-5dc6b96f5-z9wx6\" (UID: \"317505ef-238c-458e-a74e-1198806b94d2\") " pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" Oct 30 01:28:47.133341 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:47.133275 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/317505ef-238c-458e-a74e-1198806b94d2-tmp-dir\") pod \"premium-w3-558327-kserve-5dc6b96f5-z9wx6\" (UID: \"317505ef-238c-458e-a74e-1198806b94d2\") " pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" Oct 30 01:28:47.133341 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:47.133311 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/317505ef-238c-458e-a74e-1198806b94d2-home\") pod \"premium-w3-558327-kserve-5dc6b96f5-z9wx6\" (UID: \"317505ef-238c-458e-a74e-1198806b94d2\") " pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" Oct 30 01:28:47.133763 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:47.133715 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/317505ef-238c-458e-a74e-1198806b94d2-model-cache\") pod \"premium-w3-558327-kserve-5dc6b96f5-z9wx6\" (UID: \"317505ef-238c-458e-a74e-1198806b94d2\") " pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" Oct 30 01:28:47.133882 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:47.133736 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/317505ef-238c-458e-a74e-1198806b94d2-tmp-dir\") pod \"premium-w3-558327-kserve-5dc6b96f5-z9wx6\" (UID: \"317505ef-238c-458e-a74e-1198806b94d2\") " pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" Oct 30 01:28:47.133882 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:47.133829 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/317505ef-238c-458e-a74e-1198806b94d2-home\") pod \"premium-w3-558327-kserve-5dc6b96f5-z9wx6\" (UID: \"317505ef-238c-458e-a74e-1198806b94d2\") " pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" Oct 30 01:28:47.135452 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:47.135430 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/317505ef-238c-458e-a74e-1198806b94d2-dshm\") pod \"premium-w3-558327-kserve-5dc6b96f5-z9wx6\" (UID: \"317505ef-238c-458e-a74e-1198806b94d2\") " pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" Oct 30 01:28:47.135612 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:47.135594 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/317505ef-238c-458e-a74e-1198806b94d2-tls-certs\") pod \"premium-w3-558327-kserve-5dc6b96f5-z9wx6\" (UID: \"317505ef-238c-458e-a74e-1198806b94d2\") " pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" Oct 30 01:28:47.151121 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:47.151078 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj8b9\" (UniqueName: \"kubernetes.io/projected/317505ef-238c-458e-a74e-1198806b94d2-kube-api-access-kj8b9\") pod \"premium-w3-558327-kserve-5dc6b96f5-z9wx6\" (UID: \"317505ef-238c-458e-a74e-1198806b94d2\") " pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" Oct 30 01:28:47.443653 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:47.443619 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" Oct 30 01:28:47.774898 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:47.774869 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6"] Oct 30 01:28:47.776486 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:28:47.776456 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod317505ef_238c_458e_a74e_1198806b94d2.slice/crio-fe02478f2c89ab73c0d020cd3e732269d00de83c7fc4249920d4a97ae568dd42 WatchSource:0}: Error finding container fe02478f2c89ab73c0d020cd3e732269d00de83c7fc4249920d4a97ae568dd42: Status 404 returned error can't find the container with id fe02478f2c89ab73c0d020cd3e732269d00de83c7fc4249920d4a97ae568dd42 Oct 30 01:28:48.365218 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:48.365179 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:48.728187 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:48.728153 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" event={"ID":"317505ef-238c-458e-a74e-1198806b94d2","Type":"ContainerStarted","Data":"9a962bf0d349c460708f4489ecc07c1a6d5c6365fc749b0b5328ca390ace6af7"} Oct 30 01:28:48.728187 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:48.728194 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" event={"ID":"317505ef-238c-458e-a74e-1198806b94d2","Type":"ContainerStarted","Data":"fe02478f2c89ab73c0d020cd3e732269d00de83c7fc4249920d4a97ae568dd42"} Oct 30 01:28:48.761730 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:48.761662 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" podStartSLOduration=2.761643903 podStartE2EDuration="2.761643903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:48.760644033 +0000 UTC m=+654.337069434" watchObservedRunningTime="2026-10-30 01:28:48.761643903 +0000 UTC m=+654.338069305" Oct 30 01:28:49.444173 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:49.444132 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" Oct 30 01:28:49.995178 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:49.995138 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:50.743313 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:50.743268 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:51.061799 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:51.061688 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" Oct 30 01:28:51.076532 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:51.076500 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" Oct 30 01:28:51.585363 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:51.585316 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:51.759380 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:51.759354 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" Oct 30 01:28:53.162661 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:53.162604 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:54.720738 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:54.720699 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:56.372247 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:56.372192 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:57.443854 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:57.443809 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" Oct 30 01:28:57.456811 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:57.456771 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" Oct 30 01:28:57.801040 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:57.800947 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" Oct 30 01:28:57.979936 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:57.979891 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:58.225655 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:58.225619 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:59.690486 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:28:59.690440 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:01.210721 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:01.210678 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:02.780891 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:02.780853 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:03.048817 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:03.048719 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:04.286069 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:04.286029 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:05.768563 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:05.768528 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:07.281507 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:07.281473 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:07.589155 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:07.589072 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:08.871435 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:08.871400 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:10.431629 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:10.431586 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:11.312647 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:11.312605 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86"] Oct 30 01:29:11.317415 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:11.317394 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" Oct 30 01:29:11.340230 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:11.340199 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-803472\"/\"premium-w4-803472-kserve-self-signed-certs\"" Oct 30 01:29:11.342919 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:11.342886 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86"] Oct 30 01:29:11.406362 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:11.406319 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-home\") pod \"premium-w4-803472-kserve-68694cd7b8-b5j86\" (UID: \"0ee3f8e3-9b55-4ce5-afaf-86780a26391e\") " pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" Oct 30 01:29:11.406558 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:11.406374 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-model-cache\") pod \"premium-w4-803472-kserve-68694cd7b8-b5j86\" (UID: \"0ee3f8e3-9b55-4ce5-afaf-86780a26391e\") " pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" Oct 30 01:29:11.406558 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:11.406500 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-tls-certs\") pod \"premium-w4-803472-kserve-68694cd7b8-b5j86\" (UID: \"0ee3f8e3-9b55-4ce5-afaf-86780a26391e\") " pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" Oct 30 01:29:11.406558 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:11.406537 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-tmp-dir\") pod \"premium-w4-803472-kserve-68694cd7b8-b5j86\" (UID: \"0ee3f8e3-9b55-4ce5-afaf-86780a26391e\") " pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" Oct 30 01:29:11.406740 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:11.406638 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-dshm\") pod \"premium-w4-803472-kserve-68694cd7b8-b5j86\" (UID: \"0ee3f8e3-9b55-4ce5-afaf-86780a26391e\") " pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" Oct 30 01:29:11.406740 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:11.406725 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8rdj\" (UniqueName: \"kubernetes.io/projected/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-kube-api-access-m8rdj\") pod \"premium-w4-803472-kserve-68694cd7b8-b5j86\" (UID: \"0ee3f8e3-9b55-4ce5-afaf-86780a26391e\") " pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" Oct 30 01:29:11.451985 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:11.451944 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:11.509329 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:11.508630 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8rdj\" (UniqueName: \"kubernetes.io/projected/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-kube-api-access-m8rdj\") pod \"premium-w4-803472-kserve-68694cd7b8-b5j86\" (UID: \"0ee3f8e3-9b55-4ce5-afaf-86780a26391e\") " pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" Oct 30 01:29:11.509329 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:11.508738 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-home\") pod \"premium-w4-803472-kserve-68694cd7b8-b5j86\" (UID: \"0ee3f8e3-9b55-4ce5-afaf-86780a26391e\") " pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" Oct 30 01:29:11.509329 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:11.508810 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-model-cache\") pod \"premium-w4-803472-kserve-68694cd7b8-b5j86\" (UID: \"0ee3f8e3-9b55-4ce5-afaf-86780a26391e\") " pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" Oct 30 01:29:11.509329 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:11.508894 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-tls-certs\") pod \"premium-w4-803472-kserve-68694cd7b8-b5j86\" (UID: \"0ee3f8e3-9b55-4ce5-afaf-86780a26391e\") " pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" Oct 30 01:29:11.509329 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:11.508933 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-tmp-dir\") pod \"premium-w4-803472-kserve-68694cd7b8-b5j86\" (UID: \"0ee3f8e3-9b55-4ce5-afaf-86780a26391e\") " pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" Oct 30 01:29:11.509329 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:11.508988 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-dshm\") pod \"premium-w4-803472-kserve-68694cd7b8-b5j86\" (UID: \"0ee3f8e3-9b55-4ce5-afaf-86780a26391e\") " pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" Oct 30 01:29:11.509973 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:11.509938 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-tmp-dir\") pod \"premium-w4-803472-kserve-68694cd7b8-b5j86\" (UID: \"0ee3f8e3-9b55-4ce5-afaf-86780a26391e\") " pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" Oct 30 01:29:11.514734 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:11.514701 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-home\") pod \"premium-w4-803472-kserve-68694cd7b8-b5j86\" (UID: \"0ee3f8e3-9b55-4ce5-afaf-86780a26391e\") " pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" Oct 30 01:29:11.515368 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:11.515341 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-model-cache\") pod \"premium-w4-803472-kserve-68694cd7b8-b5j86\" (UID: \"0ee3f8e3-9b55-4ce5-afaf-86780a26391e\") " pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" Oct 30 01:29:11.519092 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:11.519066 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-tls-certs\") pod \"premium-w4-803472-kserve-68694cd7b8-b5j86\" (UID: \"0ee3f8e3-9b55-4ce5-afaf-86780a26391e\") " pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" Oct 30 01:29:11.519339 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:11.519294 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-dshm\") pod \"premium-w4-803472-kserve-68694cd7b8-b5j86\" (UID: \"0ee3f8e3-9b55-4ce5-afaf-86780a26391e\") " pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" Oct 30 01:29:11.530680 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:11.530649 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8rdj\" (UniqueName: \"kubernetes.io/projected/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-kube-api-access-m8rdj\") pod \"premium-w4-803472-kserve-68694cd7b8-b5j86\" (UID: \"0ee3f8e3-9b55-4ce5-afaf-86780a26391e\") " pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" Oct 30 01:29:11.630560 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:11.630457 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" Oct 30 01:29:11.928897 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:11.928867 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86"] Oct 30 01:29:11.977604 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:11.977569 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:12.886662 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:12.886619 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" event={"ID":"0ee3f8e3-9b55-4ce5-afaf-86780a26391e","Type":"ContainerStarted","Data":"4b95215344c7160a1f57af88d29d3c69196f43edc38c6deb6ff4684dc8f7de69"} Oct 30 01:29:12.887089 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:12.886672 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" event={"ID":"0ee3f8e3-9b55-4ce5-afaf-86780a26391e","Type":"ContainerStarted","Data":"8fd1d60406b0c9a2863c6fb3a360379efefaaedf3e2d0e7f7be65febd3481888"} Oct 30 01:29:12.933729 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:12.933658 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" podStartSLOduration=1.933622146 podStartE2EDuration="1.933622146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:12.930312064 +0000 UTC m=+678.506737464" watchObservedRunningTime="2026-10-30 01:29:12.933622146 +0000 UTC m=+678.510047549" Oct 30 01:29:13.630817 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:13.630777 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" Oct 30 01:29:13.667167 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:13.667135 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:14.687807 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:29:14.687769 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:29:15.281542 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:15.281502 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:16.868396 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:16.868349 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:18.385367 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:18.385330 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:19.964206 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:19.964170 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:20.398070 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:20.397981 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:21.631355 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:21.631314 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" Oct 30 01:29:21.644137 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:21.644110 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" Oct 30 01:29:21.698680 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:21.698642 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:21.997222 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:21.997190 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" Oct 30 01:29:23.276537 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:23.276500 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:24.878875 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:24.878833 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:26.410672 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:26.410636 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:27.896164 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:27.896124 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:29.472770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:29.471895 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:31.245723 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:31.245682 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:32.808807 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:32.808764 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:34.386058 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:34.386023 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:35.884731 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:35.884689 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:37.437369 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:37.437328 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:39.324788 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:39.324736 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:40.889455 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:40.889409 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:40.974304 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:40.974265 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be8d0c4f-l2kt9"] Oct 30 01:29:40.978062 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:40.978042 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8d0c4f-l2kt9" Oct 30 01:29:41.003199 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.001964 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:29:41.003199 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.002059 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:29:41.003199 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.002071 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:29:41.003199 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.002136 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:29:41.005117 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.005090 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be8d0c4f-l2kt9"] Oct 30 01:29:41.039812 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.039783 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82531f0a-89ec-4156-889a-af4b9b949bfd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be8d0c4f-l2kt9\" (UID: \"82531f0a-89ec-4156-889a-af4b9b949bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8d0c4f-l2kt9" Oct 30 01:29:41.039955 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.039860 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82531f0a-89ec-4156-889a-af4b9b949bfd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be8d0c4f-l2kt9\" (UID: \"82531f0a-89ec-4156-889a-af4b9b949bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8d0c4f-l2kt9" Oct 30 01:29:41.040047 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.040025 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj6wq\" (UniqueName: \"kubernetes.io/projected/82531f0a-89ec-4156-889a-af4b9b949bfd-kube-api-access-mj6wq\") pod \"maas-api-revoke-sub-be8d0c4f-l2kt9\" (UID: \"82531f0a-89ec-4156-889a-af4b9b949bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8d0c4f-l2kt9" Oct 30 01:29:41.141443 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.141372 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82531f0a-89ec-4156-889a-af4b9b949bfd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be8d0c4f-l2kt9\" (UID: \"82531f0a-89ec-4156-889a-af4b9b949bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8d0c4f-l2kt9" Oct 30 01:29:41.141570 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.141467 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj6wq\" (UniqueName: \"kubernetes.io/projected/82531f0a-89ec-4156-889a-af4b9b949bfd-kube-api-access-mj6wq\") pod \"maas-api-revoke-sub-be8d0c4f-l2kt9\" (UID: \"82531f0a-89ec-4156-889a-af4b9b949bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8d0c4f-l2kt9" Oct 30 01:29:41.141616 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.141574 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82531f0a-89ec-4156-889a-af4b9b949bfd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be8d0c4f-l2kt9\" (UID: \"82531f0a-89ec-4156-889a-af4b9b949bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8d0c4f-l2kt9" Oct 30 01:29:41.142209 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.142184 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82531f0a-89ec-4156-889a-af4b9b949bfd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be8d0c4f-l2kt9\" (UID: \"82531f0a-89ec-4156-889a-af4b9b949bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8d0c4f-l2kt9" Oct 30 01:29:41.143602 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.143581 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82531f0a-89ec-4156-889a-af4b9b949bfd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be8d0c4f-l2kt9\" (UID: \"82531f0a-89ec-4156-889a-af4b9b949bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8d0c4f-l2kt9" Oct 30 01:29:41.158869 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.158838 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj6wq\" (UniqueName: \"kubernetes.io/projected/82531f0a-89ec-4156-889a-af4b9b949bfd-kube-api-access-mj6wq\") pod \"maas-api-revoke-sub-be8d0c4f-l2kt9\" (UID: \"82531f0a-89ec-4156-889a-af4b9b949bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8d0c4f-l2kt9" Oct 30 01:29:41.292339 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.292300 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8d0c4f-l2kt9" Oct 30 01:29:41.584781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.584739 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be8d0c4f-l2kt9"] Oct 30 01:29:41.586726 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:29:41.586691 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82531f0a_89ec_4156_889a_af4b9b949bfd.slice/crio-998f987dbe0591f83165d8b30d1fad9d228cb01a90e19130f54a1c3dd5cdca5f WatchSource:0}: Error finding container 998f987dbe0591f83165d8b30d1fad9d228cb01a90e19130f54a1c3dd5cdca5f: Status 404 returned error can't find the container with id 998f987dbe0591f83165d8b30d1fad9d228cb01a90e19130f54a1c3dd5cdca5f Oct 30 01:29:41.708931 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.708891 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv"] Oct 30 01:29:41.714071 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.714037 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" Oct 30 01:29:41.739835 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.739807 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-558327-openshift-default-dockercfg-rmc66\"" Oct 30 01:29:41.743870 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.743840 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv"] Oct 30 01:29:41.746346 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.746320 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9462ac3-f977-4dce-8c37-a89b70928484-istio-data\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-f8bsv\" (UID: \"e9462ac3-f977-4dce-8c37-a89b70928484\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" Oct 30 01:29:41.746442 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.746369 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9462ac3-f977-4dce-8c37-a89b70928484-credential-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-f8bsv\" (UID: \"e9462ac3-f977-4dce-8c37-a89b70928484\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" Oct 30 01:29:41.746497 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.746466 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9462ac3-f977-4dce-8c37-a89b70928484-workload-certs\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-f8bsv\" (UID: \"e9462ac3-f977-4dce-8c37-a89b70928484\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" Oct 30 01:29:41.746540 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.746519 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9462ac3-f977-4dce-8c37-a89b70928484-istio-envoy\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-f8bsv\" (UID: \"e9462ac3-f977-4dce-8c37-a89b70928484\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" Oct 30 01:29:41.746606 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.746586 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9462ac3-f977-4dce-8c37-a89b70928484-istiod-ca-cert\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-f8bsv\" (UID: \"e9462ac3-f977-4dce-8c37-a89b70928484\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" Oct 30 01:29:41.746647 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.746618 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdd9k\" (UniqueName: \"kubernetes.io/projected/e9462ac3-f977-4dce-8c37-a89b70928484-kube-api-access-gdd9k\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-f8bsv\" (UID: \"e9462ac3-f977-4dce-8c37-a89b70928484\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" Oct 30 01:29:41.746696 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.746657 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9462ac3-f977-4dce-8c37-a89b70928484-istio-podinfo\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-f8bsv\" (UID: \"e9462ac3-f977-4dce-8c37-a89b70928484\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" Oct 30 01:29:41.746860 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.746838 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9462ac3-f977-4dce-8c37-a89b70928484-istio-token\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-f8bsv\" (UID: \"e9462ac3-f977-4dce-8c37-a89b70928484\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" Oct 30 01:29:41.746913 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.746881 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9462ac3-f977-4dce-8c37-a89b70928484-workload-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-f8bsv\" (UID: \"e9462ac3-f977-4dce-8c37-a89b70928484\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" Oct 30 01:29:41.849163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.848141 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9462ac3-f977-4dce-8c37-a89b70928484-istio-data\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-f8bsv\" (UID: \"e9462ac3-f977-4dce-8c37-a89b70928484\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" Oct 30 01:29:41.849163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.848193 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9462ac3-f977-4dce-8c37-a89b70928484-credential-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-f8bsv\" (UID: \"e9462ac3-f977-4dce-8c37-a89b70928484\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" Oct 30 01:29:41.849163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.848230 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9462ac3-f977-4dce-8c37-a89b70928484-workload-certs\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-f8bsv\" (UID: \"e9462ac3-f977-4dce-8c37-a89b70928484\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" Oct 30 01:29:41.849163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.848264 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9462ac3-f977-4dce-8c37-a89b70928484-istio-envoy\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-f8bsv\" (UID: \"e9462ac3-f977-4dce-8c37-a89b70928484\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" Oct 30 01:29:41.849163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.848300 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9462ac3-f977-4dce-8c37-a89b70928484-istiod-ca-cert\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-f8bsv\" (UID: \"e9462ac3-f977-4dce-8c37-a89b70928484\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" Oct 30 01:29:41.849163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.848338 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdd9k\" (UniqueName: \"kubernetes.io/projected/e9462ac3-f977-4dce-8c37-a89b70928484-kube-api-access-gdd9k\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-f8bsv\" (UID: \"e9462ac3-f977-4dce-8c37-a89b70928484\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" Oct 30 01:29:41.849163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.848392 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9462ac3-f977-4dce-8c37-a89b70928484-istio-podinfo\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-f8bsv\" (UID: \"e9462ac3-f977-4dce-8c37-a89b70928484\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" Oct 30 01:29:41.849163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.848471 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9462ac3-f977-4dce-8c37-a89b70928484-istio-token\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-f8bsv\" (UID: \"e9462ac3-f977-4dce-8c37-a89b70928484\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" Oct 30 01:29:41.849163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.848543 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9462ac3-f977-4dce-8c37-a89b70928484-workload-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-f8bsv\" (UID: \"e9462ac3-f977-4dce-8c37-a89b70928484\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" Oct 30 01:29:41.849163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.848639 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9462ac3-f977-4dce-8c37-a89b70928484-istio-data\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-f8bsv\" (UID: \"e9462ac3-f977-4dce-8c37-a89b70928484\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" Oct 30 01:29:41.849163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.848705 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9462ac3-f977-4dce-8c37-a89b70928484-credential-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-f8bsv\" (UID: \"e9462ac3-f977-4dce-8c37-a89b70928484\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" Oct 30 01:29:41.849163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.848898 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9462ac3-f977-4dce-8c37-a89b70928484-workload-certs\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-f8bsv\" (UID: \"e9462ac3-f977-4dce-8c37-a89b70928484\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" Oct 30 01:29:41.849163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.848919 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9462ac3-f977-4dce-8c37-a89b70928484-workload-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-f8bsv\" (UID: \"e9462ac3-f977-4dce-8c37-a89b70928484\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" Oct 30 01:29:41.849163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.849131 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9462ac3-f977-4dce-8c37-a89b70928484-istiod-ca-cert\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-f8bsv\" (UID: \"e9462ac3-f977-4dce-8c37-a89b70928484\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" Oct 30 01:29:41.851331 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.851292 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9462ac3-f977-4dce-8c37-a89b70928484-istio-envoy\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-f8bsv\" (UID: \"e9462ac3-f977-4dce-8c37-a89b70928484\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" Oct 30 01:29:41.851462 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.851407 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9462ac3-f977-4dce-8c37-a89b70928484-istio-podinfo\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-f8bsv\" (UID: \"e9462ac3-f977-4dce-8c37-a89b70928484\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" Oct 30 01:29:41.865230 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.865202 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdd9k\" (UniqueName: \"kubernetes.io/projected/e9462ac3-f977-4dce-8c37-a89b70928484-kube-api-access-gdd9k\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-f8bsv\" (UID: \"e9462ac3-f977-4dce-8c37-a89b70928484\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" Oct 30 01:29:41.865724 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:41.865703 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9462ac3-f977-4dce-8c37-a89b70928484-istio-token\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-f8bsv\" (UID: \"e9462ac3-f977-4dce-8c37-a89b70928484\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" Oct 30 01:29:42.027712 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:42.027681 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" Oct 30 01:29:42.217060 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:42.217022 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8d0c4f-l2kt9" event={"ID":"82531f0a-89ec-4156-889a-af4b9b949bfd","Type":"ContainerStarted","Data":"e21ae05e3857f20b05057742525deef06bd80555abda7e95b3e99dc094339290"} Oct 30 01:29:42.217237 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:42.217083 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8d0c4f-l2kt9" event={"ID":"82531f0a-89ec-4156-889a-af4b9b949bfd","Type":"ContainerStarted","Data":"998f987dbe0591f83165d8b30d1fad9d228cb01a90e19130f54a1c3dd5cdca5f"} Oct 30 01:29:42.249569 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:42.249263 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8d0c4f-l2kt9" podStartSLOduration=2.249217936 podStartE2EDuration="2.249217936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:42.248905985 +0000 UTC m=+707.825331444" watchObservedRunningTime="2026-10-30 01:29:42.249217936 +0000 UTC m=+707.825643340" Oct 30 01:29:42.337295 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:42.337270 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv"] Oct 30 01:29:42.339425 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:29:42.339399 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9462ac3_f977_4dce_8c37_a89b70928484.slice/crio-0b961230497d339260ad2e300c508d126061f87fb65d625bfe140d2938e30d39 WatchSource:0}: Error finding container 0b961230497d339260ad2e300c508d126061f87fb65d625bfe140d2938e30d39: Status 404 returned error can't find the container with id 0b961230497d339260ad2e300c508d126061f87fb65d625bfe140d2938e30d39 Oct 30 01:29:42.341987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:42.341955 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:29:42.342062 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:42.342021 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:29:42.342062 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:42.342047 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:29:42.491123 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:42.491050 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:43.237256 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:43.237216 6572 generic.go:356] "Generic (PLEG): container finished" podID="82531f0a-89ec-4156-889a-af4b9b949bfd" containerID="e21ae05e3857f20b05057742525deef06bd80555abda7e95b3e99dc094339290" exitCode=0 Oct 30 01:29:43.237682 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:43.237290 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8d0c4f-l2kt9" event={"ID":"82531f0a-89ec-4156-889a-af4b9b949bfd","Type":"ContainerDied","Data":"e21ae05e3857f20b05057742525deef06bd80555abda7e95b3e99dc094339290"} Oct 30 01:29:43.239175 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:43.239145 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" event={"ID":"e9462ac3-f977-4dce-8c37-a89b70928484","Type":"ContainerStarted","Data":"55bf86878ed086665fe9f88163407d0bd96c516f1fefb061687ab54a114d99c3"} Oct 30 01:29:43.239333 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:43.239195 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" event={"ID":"e9462ac3-f977-4dce-8c37-a89b70928484","Type":"ContainerStarted","Data":"0b961230497d339260ad2e300c508d126061f87fb65d625bfe140d2938e30d39"} Oct 30 01:29:44.030362 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:44.030167 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" Oct 30 01:29:44.031610 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:44.031567 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:44.032087 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:44.032065 6572 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 01:29:44.032142 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:44.032113 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" podUID="e9462ac3-f977-4dce-8c37-a89b70928484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 01:29:44.465355 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:44.465319 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8d0c4f-l2kt9" Oct 30 01:29:44.508444 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:44.508373 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" podStartSLOduration=3.508350209 podStartE2EDuration="3.508350209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:43.338066013 +0000 UTC m=+708.914491415" watchObservedRunningTime="2026-10-30 01:29:44.508350209 +0000 UTC m=+710.084775614" Oct 30 01:29:44.580232 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:44.580198 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82531f0a-89ec-4156-889a-af4b9b949bfd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82531f0a-89ec-4156-889a-af4b9b949bfd-maas-api-openshift-ca\") pod \"82531f0a-89ec-4156-889a-af4b9b949bfd\" (UID: \"82531f0a-89ec-4156-889a-af4b9b949bfd\") " Oct 30 01:29:44.580403 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:44.580250 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82531f0a-89ec-4156-889a-af4b9b949bfd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82531f0a-89ec-4156-889a-af4b9b949bfd-maas-api-xks-ca\") pod \"82531f0a-89ec-4156-889a-af4b9b949bfd\" (UID: \"82531f0a-89ec-4156-889a-af4b9b949bfd\") " Oct 30 01:29:44.580403 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:44.580280 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82531f0a-89ec-4156-889a-af4b9b949bfd-kube-api-access-mj6wq\" (UniqueName: \"kubernetes.io/projected/82531f0a-89ec-4156-889a-af4b9b949bfd-kube-api-access-mj6wq\") pod \"82531f0a-89ec-4156-889a-af4b9b949bfd\" (UID: \"82531f0a-89ec-4156-889a-af4b9b949bfd\") " Oct 30 01:29:44.581164 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:44.581135 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82531f0a-89ec-4156-889a-af4b9b949bfd-maas-api-openshift-ca" pod "82531f0a-89ec-4156-889a-af4b9b949bfd" (UID: "82531f0a-89ec-4156-889a-af4b9b949bfd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:44.582928 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:44.582902 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82531f0a-89ec-4156-889a-af4b9b949bfd-maas-api-xks-ca" pod "82531f0a-89ec-4156-889a-af4b9b949bfd" (UID: "82531f0a-89ec-4156-889a-af4b9b949bfd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:44.583018 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:44.582932 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82531f0a-89ec-4156-889a-af4b9b949bfd-kube-api-access-mj6wq" pod "82531f0a-89ec-4156-889a-af4b9b949bfd" (UID: "82531f0a-89ec-4156-889a-af4b9b949bfd"). InnerVolumeSpecName "kube-api-access-mj6wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:44.682213 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:44.682177 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82531f0a-89ec-4156-889a-af4b9b949bfd-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:29:44.682213 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:44.682208 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82531f0a-89ec-4156-889a-af4b9b949bfd-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:29:44.682213 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:44.682218 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj6wq\" (UniqueName: \"kubernetes.io/projected/82531f0a-89ec-4156-889a-af4b9b949bfd-kube-api-access-mj6wq\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:29:44.689853 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:29:44.689815 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:29:45.029278 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:45.029177 6572 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 01:29:45.029548 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:45.029516 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" podUID="e9462ac3-f977-4dce-8c37-a89b70928484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 01:29:45.259602 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:45.259569 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8d0c4f-l2kt9" Oct 30 01:29:45.259840 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:45.259600 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8d0c4f-l2kt9" event={"ID":"82531f0a-89ec-4156-889a-af4b9b949bfd","Type":"ContainerDied","Data":"998f987dbe0591f83165d8b30d1fad9d228cb01a90e19130f54a1c3dd5cdca5f"} Oct 30 01:29:45.259840 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:45.259652 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="998f987dbe0591f83165d8b30d1fad9d228cb01a90e19130f54a1c3dd5cdca5f" Oct 30 01:29:45.732417 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:45.732380 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:46.028294 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:46.028195 6572 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 01:29:46.028580 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:46.028551 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" podUID="e9462ac3-f977-4dce-8c37-a89b70928484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 01:29:47.031214 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:47.031174 6572 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 01:29:47.031812 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:47.031230 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" podUID="e9462ac3-f977-4dce-8c37-a89b70928484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 01:29:47.291933 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:47.291857 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:48.028770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:48.028295 6572 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 01:29:48.028770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:48.028370 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" podUID="e9462ac3-f977-4dce-8c37-a89b70928484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 01:29:48.886383 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:48.886353 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:49.907684 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:49.907646 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" Oct 30 01:29:49.908140 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:49.908037 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" Oct 30 01:29:49.909529 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:49.909502 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" Oct 30 01:29:50.603112 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:50.603074 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:52.172761 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:52.172707 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:53.833595 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:53.833552 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:53.975207 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:53.975175 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77309087-rpq8g"] Oct 30 01:29:53.975662 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:53.975649 6572 cpu_manager.go:414] "containerMap: removing container" podUID="82531f0a-89ec-4156-889a-af4b9b949bfd" containerName="revoke-keys" Oct 30 01:29:53.975719 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:53.975663 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82531f0a-89ec-4156-889a-af4b9b949bfd" containerName="revoke-keys" Oct 30 01:29:53.975799 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:53.975785 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="82531f0a-89ec-4156-889a-af4b9b949bfd" containerName="revoke-keys" Oct 30 01:29:53.979346 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:53.979320 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77309087-rpq8g" Oct 30 01:29:53.997277 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:53.997244 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77309087-rpq8g"] Oct 30 01:29:54.006774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:54.003822 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:29:54.006774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:54.003823 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:29:54.006774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:54.003873 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:29:54.006774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:54.003915 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:29:54.115985 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:54.115888 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hz88\" (UniqueName: \"kubernetes.io/projected/560bdb6b-fe12-4c95-8db1-fdc29792f939-kube-api-access-5hz88\") pod \"maas-api-revoke-sub-77309087-rpq8g\" (UID: \"560bdb6b-fe12-4c95-8db1-fdc29792f939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77309087-rpq8g" Oct 30 01:29:54.116150 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:54.116009 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/560bdb6b-fe12-4c95-8db1-fdc29792f939-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77309087-rpq8g\" (UID: \"560bdb6b-fe12-4c95-8db1-fdc29792f939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77309087-rpq8g" Oct 30 01:29:54.116150 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:54.116059 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/560bdb6b-fe12-4c95-8db1-fdc29792f939-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77309087-rpq8g\" (UID: \"560bdb6b-fe12-4c95-8db1-fdc29792f939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77309087-rpq8g" Oct 30 01:29:54.217117 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:54.217075 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hz88\" (UniqueName: \"kubernetes.io/projected/560bdb6b-fe12-4c95-8db1-fdc29792f939-kube-api-access-5hz88\") pod \"maas-api-revoke-sub-77309087-rpq8g\" (UID: \"560bdb6b-fe12-4c95-8db1-fdc29792f939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77309087-rpq8g" Oct 30 01:29:54.217315 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:54.217185 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/560bdb6b-fe12-4c95-8db1-fdc29792f939-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77309087-rpq8g\" (UID: \"560bdb6b-fe12-4c95-8db1-fdc29792f939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77309087-rpq8g" Oct 30 01:29:54.217315 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:54.217236 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/560bdb6b-fe12-4c95-8db1-fdc29792f939-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77309087-rpq8g\" (UID: \"560bdb6b-fe12-4c95-8db1-fdc29792f939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77309087-rpq8g" Oct 30 01:29:54.218065 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:54.218034 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/560bdb6b-fe12-4c95-8db1-fdc29792f939-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77309087-rpq8g\" (UID: \"560bdb6b-fe12-4c95-8db1-fdc29792f939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77309087-rpq8g" Oct 30 01:29:54.231278 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:54.220181 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/560bdb6b-fe12-4c95-8db1-fdc29792f939-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77309087-rpq8g\" (UID: \"560bdb6b-fe12-4c95-8db1-fdc29792f939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77309087-rpq8g" Oct 30 01:29:54.234374 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:54.234348 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hz88\" (UniqueName: \"kubernetes.io/projected/560bdb6b-fe12-4c95-8db1-fdc29792f939-kube-api-access-5hz88\") pod \"maas-api-revoke-sub-77309087-rpq8g\" (UID: \"560bdb6b-fe12-4c95-8db1-fdc29792f939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77309087-rpq8g" Oct 30 01:29:54.292662 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:54.292630 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77309087-rpq8g" Oct 30 01:29:54.599918 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:54.599878 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77309087-rpq8g"] Oct 30 01:29:54.601614 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:29:54.601590 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod560bdb6b_fe12_4c95_8db1_fdc29792f939.slice/crio-8245177adbc374e8b90dcdeeba504c2b652bdfc44b80d5607bab606efec19625 WatchSource:0}: Error finding container 8245177adbc374e8b90dcdeeba504c2b652bdfc44b80d5607bab606efec19625: Status 404 returned error can't find the container with id 8245177adbc374e8b90dcdeeba504c2b652bdfc44b80d5607bab606efec19625 Oct 30 01:29:55.342842 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:55.342729 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77309087-rpq8g" event={"ID":"560bdb6b-fe12-4c95-8db1-fdc29792f939","Type":"ContainerStarted","Data":"6647e5c506db41b246cde5d447e565d9e42cc82734252386e4e9defc3658bbec"} Oct 30 01:29:55.343376 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:55.343350 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77309087-rpq8g" event={"ID":"560bdb6b-fe12-4c95-8db1-fdc29792f939","Type":"ContainerStarted","Data":"8245177adbc374e8b90dcdeeba504c2b652bdfc44b80d5607bab606efec19625"} Oct 30 01:29:55.381148 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:55.380619 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77309087-rpq8g" podStartSLOduration=2.380598476 podStartE2EDuration="2.380598476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:55.379189209 +0000 UTC m=+720.955614682" watchObservedRunningTime="2026-10-30 01:29:55.380598476 +0000 UTC m=+720.957023879" Oct 30 01:29:55.410277 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:55.410242 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:56.349868 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:56.349831 6572 generic.go:356] "Generic (PLEG): container finished" podID="560bdb6b-fe12-4c95-8db1-fdc29792f939" containerID="6647e5c506db41b246cde5d447e565d9e42cc82734252386e4e9defc3658bbec" exitCode=0 Oct 30 01:29:56.349868 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:56.349871 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77309087-rpq8g" event={"ID":"560bdb6b-fe12-4c95-8db1-fdc29792f939","Type":"ContainerDied","Data":"6647e5c506db41b246cde5d447e565d9e42cc82734252386e4e9defc3658bbec"} Oct 30 01:29:56.378882 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:56.378845 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa3985f0-lbpwz"] Oct 30 01:29:56.382875 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:56.382844 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3985f0-lbpwz" Oct 30 01:29:56.401611 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:56.401582 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa3985f0-lbpwz"] Oct 30 01:29:56.469938 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:56.469902 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa3985f0-lbpwz\" (UID: \"93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3985f0-lbpwz" Oct 30 01:29:56.470142 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:56.469944 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa3985f0-lbpwz\" (UID: \"93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3985f0-lbpwz" Oct 30 01:29:56.470142 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:56.470083 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8tdh\" (UniqueName: \"kubernetes.io/projected/93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117-kube-api-access-n8tdh\") pod \"maas-api-revoke-sub-fa3985f0-lbpwz\" (UID: \"93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3985f0-lbpwz" Oct 30 01:29:56.570979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:56.570941 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa3985f0-lbpwz\" (UID: \"93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3985f0-lbpwz" Oct 30 01:29:56.570979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:56.570979 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa3985f0-lbpwz\" (UID: \"93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3985f0-lbpwz" Oct 30 01:29:56.571236 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:56.571073 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8tdh\" (UniqueName: \"kubernetes.io/projected/93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117-kube-api-access-n8tdh\") pod \"maas-api-revoke-sub-fa3985f0-lbpwz\" (UID: \"93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3985f0-lbpwz" Oct 30 01:29:56.571781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:56.571735 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa3985f0-lbpwz\" (UID: \"93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3985f0-lbpwz" Oct 30 01:29:56.573412 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:56.573375 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa3985f0-lbpwz\" (UID: \"93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3985f0-lbpwz" Oct 30 01:29:56.589980 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:56.589948 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8tdh\" (UniqueName: \"kubernetes.io/projected/93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117-kube-api-access-n8tdh\") pod \"maas-api-revoke-sub-fa3985f0-lbpwz\" (UID: \"93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3985f0-lbpwz" Oct 30 01:29:56.699203 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:56.699161 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3985f0-lbpwz" Oct 30 01:29:57.018413 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:57.018321 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa3985f0-lbpwz"] Oct 30 01:29:57.024209 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:29:57.024167 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93ebd1c7_2b98_49ad_8c4a_f0e3f37d3117.slice/crio-06abbf98343d4795e9dca0e4be0244b37217765a13e5ee7dcf93f118b411c6b0 WatchSource:0}: Error finding container 06abbf98343d4795e9dca0e4be0244b37217765a13e5ee7dcf93f118b411c6b0: Status 404 returned error can't find the container with id 06abbf98343d4795e9dca0e4be0244b37217765a13e5ee7dcf93f118b411c6b0 Oct 30 01:29:57.054511 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:57.054344 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:57.382162 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:57.382078 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3985f0-lbpwz" event={"ID":"93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117","Type":"ContainerStarted","Data":"06abbf98343d4795e9dca0e4be0244b37217765a13e5ee7dcf93f118b411c6b0"} Oct 30 01:29:57.643573 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:57.643543 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77309087-rpq8g" Oct 30 01:29:57.696895 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:57.696863 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/560bdb6b-fe12-4c95-8db1-fdc29792f939-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/560bdb6b-fe12-4c95-8db1-fdc29792f939-maas-api-openshift-ca\") pod \"560bdb6b-fe12-4c95-8db1-fdc29792f939\" (UID: \"560bdb6b-fe12-4c95-8db1-fdc29792f939\") " Oct 30 01:29:57.697060 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:57.697023 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/560bdb6b-fe12-4c95-8db1-fdc29792f939-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/560bdb6b-fe12-4c95-8db1-fdc29792f939-maas-api-xks-ca\") pod \"560bdb6b-fe12-4c95-8db1-fdc29792f939\" (UID: \"560bdb6b-fe12-4c95-8db1-fdc29792f939\") " Oct 30 01:29:57.697126 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:57.697104 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/560bdb6b-fe12-4c95-8db1-fdc29792f939-kube-api-access-5hz88\" (UniqueName: \"kubernetes.io/projected/560bdb6b-fe12-4c95-8db1-fdc29792f939-kube-api-access-5hz88\") pod \"560bdb6b-fe12-4c95-8db1-fdc29792f939\" (UID: \"560bdb6b-fe12-4c95-8db1-fdc29792f939\") " Oct 30 01:29:57.697268 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:57.697249 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/560bdb6b-fe12-4c95-8db1-fdc29792f939-maas-api-openshift-ca" pod "560bdb6b-fe12-4c95-8db1-fdc29792f939" (UID: "560bdb6b-fe12-4c95-8db1-fdc29792f939"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:57.697441 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:57.697423 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/560bdb6b-fe12-4c95-8db1-fdc29792f939-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:29:57.699181 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:57.699149 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/560bdb6b-fe12-4c95-8db1-fdc29792f939-maas-api-xks-ca" pod "560bdb6b-fe12-4c95-8db1-fdc29792f939" (UID: "560bdb6b-fe12-4c95-8db1-fdc29792f939"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:57.699356 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:57.699327 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/560bdb6b-fe12-4c95-8db1-fdc29792f939-kube-api-access-5hz88" pod "560bdb6b-fe12-4c95-8db1-fdc29792f939" (UID: "560bdb6b-fe12-4c95-8db1-fdc29792f939"). InnerVolumeSpecName "kube-api-access-5hz88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:57.798602 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:57.798565 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/560bdb6b-fe12-4c95-8db1-fdc29792f939-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:29:57.798602 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:57.798599 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hz88\" (UniqueName: \"kubernetes.io/projected/560bdb6b-fe12-4c95-8db1-fdc29792f939-kube-api-access-5hz88\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:29:58.390162 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:58.390080 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77309087-rpq8g" Oct 30 01:29:58.390162 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:58.390113 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77309087-rpq8g" event={"ID":"560bdb6b-fe12-4c95-8db1-fdc29792f939","Type":"ContainerDied","Data":"8245177adbc374e8b90dcdeeba504c2b652bdfc44b80d5607bab606efec19625"} Oct 30 01:29:58.390651 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:58.390163 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8245177adbc374e8b90dcdeeba504c2b652bdfc44b80d5607bab606efec19625" Oct 30 01:29:58.392018 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:58.391989 6572 generic.go:356] "Generic (PLEG): container finished" podID="93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117" containerID="812b77051b7d7b7b876fa5c00ae709bf8dece87733feaf4b178c442eb56b480b" exitCode=0 Oct 30 01:29:58.392154 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:58.392039 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3985f0-lbpwz" event={"ID":"93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117","Type":"ContainerDied","Data":"812b77051b7d7b7b876fa5c00ae709bf8dece87733feaf4b178c442eb56b480b"} Oct 30 01:29:58.584036 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:58.584005 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:59.540611 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:59.540581 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3985f0-lbpwz" Oct 30 01:29:59.619283 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:59.619250 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117-maas-api-xks-ca\") pod \"93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117\" (UID: \"93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117\") " Oct 30 01:29:59.619479 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:59.619293 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117-maas-api-openshift-ca\") pod \"93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117\" (UID: \"93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117\") " Oct 30 01:29:59.619479 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:59.619351 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117-kube-api-access-n8tdh\" (UniqueName: \"kubernetes.io/projected/93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117-kube-api-access-n8tdh\") pod \"93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117\" (UID: \"93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117\") " Oct 30 01:29:59.619721 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:59.619685 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117-maas-api-openshift-ca" pod "93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117" (UID: "93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:59.619966 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:59.619942 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:29:59.621479 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:59.621454 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117-maas-api-xks-ca" pod "93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117" (UID: "93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:59.621568 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:59.621538 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117-kube-api-access-n8tdh" pod "93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117" (UID: "93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117"). InnerVolumeSpecName "kube-api-access-n8tdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:59.721266 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:59.721226 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:29:59.721266 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:29:59.721259 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8tdh\" (UniqueName: \"kubernetes.io/projected/93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117-kube-api-access-n8tdh\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:30:00.081623 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.081529 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:00.156675 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.156644 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888730-vkcb8"] Oct 30 01:30:00.157230 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.157209 6572 cpu_manager.go:414] "containerMap: removing container" podUID="93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117" containerName="revoke-keys" Oct 30 01:30:00.157230 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.157227 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117" containerName="revoke-keys" Oct 30 01:30:00.157325 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.157252 6572 cpu_manager.go:414] "containerMap: removing container" podUID="560bdb6b-fe12-4c95-8db1-fdc29792f939" containerName="revoke-keys" Oct 30 01:30:00.157325 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.157257 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="560bdb6b-fe12-4c95-8db1-fdc29792f939" containerName="revoke-keys" Oct 30 01:30:00.157392 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.157330 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117" containerName="revoke-keys" Oct 30 01:30:00.157392 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.157340 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="560bdb6b-fe12-4c95-8db1-fdc29792f939" containerName="revoke-keys" Oct 30 01:30:00.161468 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.161444 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888730-t2vmk"] Oct 30 01:30:00.161611 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.161558 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888730-vkcb8" Oct 30 01:30:00.166104 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.166077 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888730-5f9p2"] Oct 30 01:30:00.166243 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.166190 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-t2vmk" Oct 30 01:30:00.170113 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.170092 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888730-7fprx"] Oct 30 01:30:00.170230 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.170204 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888730-5f9p2" Oct 30 01:30:00.174271 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.174248 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888730-7fprx" Oct 30 01:30:00.192297 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.192268 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888730-vkcb8"] Oct 30 01:30:00.212911 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.210924 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888730-t2vmk"] Oct 30 01:30:00.214307 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.214277 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888730-5f9p2"] Oct 30 01:30:00.227842 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.227807 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b5aa772-726a-466a-ab5e-fd03013c1306-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-558327-29888730-vkcb8\" (UID: \"2b5aa772-726a-466a-ab5e-fd03013c1306\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888730-vkcb8" Oct 30 01:30:00.228006 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.227868 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79840e8d-cb39-4428-9c5a-f37fb037a6e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888730-7fprx\" (UID: \"79840e8d-cb39-4428-9c5a-f37fb037a6e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888730-7fprx" Oct 30 01:30:00.228006 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.227921 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhv4p\" (UniqueName: \"kubernetes.io/projected/c4f21909-1f45-4556-ac3d-3c6e4d9f0770-kube-api-access-mhv4p\") pod \"maas-api-key-cleanup-e2e-worker-w4-803472-29888730-5f9p2\" (UID: \"c4f21909-1f45-4556-ac3d-3c6e4d9f0770\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888730-5f9p2" Oct 30 01:30:00.228006 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.227995 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8272\" (UniqueName: \"kubernetes.io/projected/2b5aa772-726a-466a-ab5e-fd03013c1306-kube-api-access-j8272\") pod \"maas-api-key-cleanup-e2e-worker-w3-558327-29888730-vkcb8\" (UID: \"2b5aa772-726a-466a-ab5e-fd03013c1306\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888730-vkcb8" Oct 30 01:30:00.228143 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.228049 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20b63c16-472b-435d-a1cc-49c0e471b20c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888730-t2vmk\" (UID: \"20b63c16-472b-435d-a1cc-49c0e471b20c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-t2vmk" Oct 30 01:30:00.228143 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.228100 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4526d\" (UniqueName: \"kubernetes.io/projected/79840e8d-cb39-4428-9c5a-f37fb037a6e8-kube-api-access-4526d\") pod \"maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888730-7fprx\" (UID: \"79840e8d-cb39-4428-9c5a-f37fb037a6e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888730-7fprx" Oct 30 01:30:00.228215 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.228173 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxgsv\" (UniqueName: \"kubernetes.io/projected/20b63c16-472b-435d-a1cc-49c0e471b20c-kube-api-access-cxgsv\") pod \"maas-api-key-cleanup-29888730-t2vmk\" (UID: \"20b63c16-472b-435d-a1cc-49c0e471b20c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-t2vmk" Oct 30 01:30:00.228247 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.228212 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f21909-1f45-4556-ac3d-3c6e4d9f0770-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-803472-29888730-5f9p2\" (UID: \"c4f21909-1f45-4556-ac3d-3c6e4d9f0770\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888730-5f9p2" Oct 30 01:30:00.229887 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.229856 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888730-7fprx"] Oct 30 01:30:00.329048 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.329011 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b5aa772-726a-466a-ab5e-fd03013c1306-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-558327-29888730-vkcb8\" (UID: \"2b5aa772-726a-466a-ab5e-fd03013c1306\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888730-vkcb8" Oct 30 01:30:00.329195 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.329052 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79840e8d-cb39-4428-9c5a-f37fb037a6e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888730-7fprx\" (UID: \"79840e8d-cb39-4428-9c5a-f37fb037a6e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888730-7fprx" Oct 30 01:30:00.329195 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.329101 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhv4p\" (UniqueName: \"kubernetes.io/projected/c4f21909-1f45-4556-ac3d-3c6e4d9f0770-kube-api-access-mhv4p\") pod \"maas-api-key-cleanup-e2e-worker-w4-803472-29888730-5f9p2\" (UID: \"c4f21909-1f45-4556-ac3d-3c6e4d9f0770\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888730-5f9p2" Oct 30 01:30:00.329195 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.329162 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8272\" (UniqueName: \"kubernetes.io/projected/2b5aa772-726a-466a-ab5e-fd03013c1306-kube-api-access-j8272\") pod \"maas-api-key-cleanup-e2e-worker-w3-558327-29888730-vkcb8\" (UID: \"2b5aa772-726a-466a-ab5e-fd03013c1306\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888730-vkcb8" Oct 30 01:30:00.329375 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.329208 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20b63c16-472b-435d-a1cc-49c0e471b20c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888730-t2vmk\" (UID: \"20b63c16-472b-435d-a1cc-49c0e471b20c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-t2vmk" Oct 30 01:30:00.329375 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.329255 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4526d\" (UniqueName: \"kubernetes.io/projected/79840e8d-cb39-4428-9c5a-f37fb037a6e8-kube-api-access-4526d\") pod \"maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888730-7fprx\" (UID: \"79840e8d-cb39-4428-9c5a-f37fb037a6e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888730-7fprx" Oct 30 01:30:00.329375 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.329323 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxgsv\" (UniqueName: \"kubernetes.io/projected/20b63c16-472b-435d-a1cc-49c0e471b20c-kube-api-access-cxgsv\") pod \"maas-api-key-cleanup-29888730-t2vmk\" (UID: \"20b63c16-472b-435d-a1cc-49c0e471b20c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-t2vmk" Oct 30 01:30:00.329375 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.329362 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f21909-1f45-4556-ac3d-3c6e4d9f0770-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-803472-29888730-5f9p2\" (UID: \"c4f21909-1f45-4556-ac3d-3c6e4d9f0770\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888730-5f9p2" Oct 30 01:30:00.329945 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.329926 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79840e8d-cb39-4428-9c5a-f37fb037a6e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888730-7fprx\" (UID: \"79840e8d-cb39-4428-9c5a-f37fb037a6e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888730-7fprx" Oct 30 01:30:00.330000 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.329974 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b5aa772-726a-466a-ab5e-fd03013c1306-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-558327-29888730-vkcb8\" (UID: \"2b5aa772-726a-466a-ab5e-fd03013c1306\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888730-vkcb8" Oct 30 01:30:00.330119 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.330097 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20b63c16-472b-435d-a1cc-49c0e471b20c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888730-t2vmk\" (UID: \"20b63c16-472b-435d-a1cc-49c0e471b20c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-t2vmk" Oct 30 01:30:00.330159 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.330121 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f21909-1f45-4556-ac3d-3c6e4d9f0770-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-803472-29888730-5f9p2\" (UID: \"c4f21909-1f45-4556-ac3d-3c6e4d9f0770\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888730-5f9p2" Oct 30 01:30:00.346138 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.346068 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhv4p\" (UniqueName: \"kubernetes.io/projected/c4f21909-1f45-4556-ac3d-3c6e4d9f0770-kube-api-access-mhv4p\") pod \"maas-api-key-cleanup-e2e-worker-w4-803472-29888730-5f9p2\" (UID: \"c4f21909-1f45-4556-ac3d-3c6e4d9f0770\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888730-5f9p2" Oct 30 01:30:00.346138 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.346130 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxgsv\" (UniqueName: \"kubernetes.io/projected/20b63c16-472b-435d-a1cc-49c0e471b20c-kube-api-access-cxgsv\") pod \"maas-api-key-cleanup-29888730-t2vmk\" (UID: \"20b63c16-472b-435d-a1cc-49c0e471b20c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-t2vmk" Oct 30 01:30:00.346314 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.346175 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4526d\" (UniqueName: \"kubernetes.io/projected/79840e8d-cb39-4428-9c5a-f37fb037a6e8-kube-api-access-4526d\") pod \"maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888730-7fprx\" (UID: \"79840e8d-cb39-4428-9c5a-f37fb037a6e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888730-7fprx" Oct 30 01:30:00.346353 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.346321 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8272\" (UniqueName: \"kubernetes.io/projected/2b5aa772-726a-466a-ab5e-fd03013c1306-kube-api-access-j8272\") pod \"maas-api-key-cleanup-e2e-worker-w3-558327-29888730-vkcb8\" (UID: \"2b5aa772-726a-466a-ab5e-fd03013c1306\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888730-vkcb8" Oct 30 01:30:00.406348 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.406308 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3985f0-lbpwz" event={"ID":"93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117","Type":"ContainerDied","Data":"06abbf98343d4795e9dca0e4be0244b37217765a13e5ee7dcf93f118b411c6b0"} Oct 30 01:30:00.406348 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.406337 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3985f0-lbpwz" Oct 30 01:30:00.406588 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.406361 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06abbf98343d4795e9dca0e4be0244b37217765a13e5ee7dcf93f118b411c6b0" Oct 30 01:30:00.477299 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.477269 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888730-vkcb8" Oct 30 01:30:00.487659 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.487623 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-t2vmk" Oct 30 01:30:00.509146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.509117 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888730-5f9p2" Oct 30 01:30:00.516788 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.516741 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888730-7fprx" Oct 30 01:30:00.781465 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.781437 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888730-vkcb8"] Oct 30 01:30:00.782543 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:30:00.782520 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b5aa772_726a_466a_ab5e_fd03013c1306.slice/crio-b6dc3d01a14d0fc0b3007ae788ac74a5fbb7f038088e9e697b5505d1e82620ac WatchSource:0}: Error finding container b6dc3d01a14d0fc0b3007ae788ac74a5fbb7f038088e9e697b5505d1e82620ac: Status 404 returned error can't find the container with id b6dc3d01a14d0fc0b3007ae788ac74a5fbb7f038088e9e697b5505d1e82620ac Oct 30 01:30:00.800831 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.800808 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888730-t2vmk"] Oct 30 01:30:00.803938 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:30:00.803906 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20b63c16_472b_435d_a1cc_49c0e471b20c.slice/crio-0f34a0600639557c121fa4cd5a8da563cda2df38128645c4c61eb76fc227ab1f WatchSource:0}: Error finding container 0f34a0600639557c121fa4cd5a8da563cda2df38128645c4c61eb76fc227ab1f: Status 404 returned error can't find the container with id 0f34a0600639557c121fa4cd5a8da563cda2df38128645c4c61eb76fc227ab1f Oct 30 01:30:00.832470 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.832442 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888730-5f9p2"] Oct 30 01:30:00.834963 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:30:00.834927 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4f21909_1f45_4556_ac3d_3c6e4d9f0770.slice/crio-a92614c3ce91fa67f6c33dcd0d6fece23e64ea6fa03796101bbb23970949999e WatchSource:0}: Error finding container a92614c3ce91fa67f6c33dcd0d6fece23e64ea6fa03796101bbb23970949999e: Status 404 returned error can't find the container with id a92614c3ce91fa67f6c33dcd0d6fece23e64ea6fa03796101bbb23970949999e Oct 30 01:30:00.836651 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:00.836521 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888730-7fprx"] Oct 30 01:30:00.841882 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:30:00.841858 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79840e8d_cb39_4428_9c5a_f37fb037a6e8.slice/crio-506fe1aede96fd8455ef4f7201dfb9e33699c159f884e991ea15de4dd5752e14 WatchSource:0}: Error finding container 506fe1aede96fd8455ef4f7201dfb9e33699c159f884e991ea15de4dd5752e14: Status 404 returned error can't find the container with id 506fe1aede96fd8455ef4f7201dfb9e33699c159f884e991ea15de4dd5752e14 Oct 30 01:30:01.415063 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:01.415028 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888730-5f9p2" event={"ID":"c4f21909-1f45-4556-ac3d-3c6e4d9f0770","Type":"ContainerStarted","Data":"bc2db100820df635d3573ee598e27bdc8ede24e4b3770f92c343367bd6a81e21"} Oct 30 01:30:01.415241 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:01.415081 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888730-5f9p2" event={"ID":"c4f21909-1f45-4556-ac3d-3c6e4d9f0770","Type":"ContainerStarted","Data":"a92614c3ce91fa67f6c33dcd0d6fece23e64ea6fa03796101bbb23970949999e"} Oct 30 01:30:01.416770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:01.416716 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888730-7fprx" event={"ID":"79840e8d-cb39-4428-9c5a-f37fb037a6e8","Type":"ContainerStarted","Data":"6e81fdea28c8dbe1a558a084af17d21461ed179c348171d16591ba83d82b7680"} Oct 30 01:30:01.416901 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:01.416800 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888730-7fprx" event={"ID":"79840e8d-cb39-4428-9c5a-f37fb037a6e8","Type":"ContainerStarted","Data":"506fe1aede96fd8455ef4f7201dfb9e33699c159f884e991ea15de4dd5752e14"} Oct 30 01:30:01.418416 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:01.418387 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888730-vkcb8" event={"ID":"2b5aa772-726a-466a-ab5e-fd03013c1306","Type":"ContainerStarted","Data":"3380df5f97395370fe318d4db4873e944c1f3a22289d5fce1c6a2ebf36b61dc8"} Oct 30 01:30:01.418501 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:01.418430 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888730-vkcb8" event={"ID":"2b5aa772-726a-466a-ab5e-fd03013c1306","Type":"ContainerStarted","Data":"b6dc3d01a14d0fc0b3007ae788ac74a5fbb7f038088e9e697b5505d1e82620ac"} Oct 30 01:30:01.419853 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:01.419833 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-t2vmk" event={"ID":"20b63c16-472b-435d-a1cc-49c0e471b20c","Type":"ContainerStarted","Data":"0bc5f27de33103e85192c401f8675c49f08c5f0ae253f7a6b221d08e07849ad4"} Oct 30 01:30:01.419938 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:01.419870 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-t2vmk" event={"ID":"20b63c16-472b-435d-a1cc-49c0e471b20c","Type":"ContainerStarted","Data":"0f34a0600639557c121fa4cd5a8da563cda2df38128645c4c61eb76fc227ab1f"} Oct 30 01:30:01.449136 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:01.449081 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888730-5f9p2" podStartSLOduration=1.449066213 podStartE2EDuration="1.449066213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:01.447794306 +0000 UTC m=+727.024219704" watchObservedRunningTime="2026-10-30 01:30:01.449066213 +0000 UTC m=+727.025491671" Oct 30 01:30:01.503037 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:01.502975 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888730-7fprx" podStartSLOduration=1.5029553629999999 podStartE2EDuration="1.502955363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:01.498130185 +0000 UTC m=+727.074555597" watchObservedRunningTime="2026-10-30 01:30:01.502955363 +0000 UTC m=+727.079380766" Oct 30 01:30:01.565021 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:01.564966 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888730-vkcb8" podStartSLOduration=1.564951183 podStartE2EDuration="1.564951183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:01.562602752 +0000 UTC m=+727.139028154" watchObservedRunningTime="2026-10-30 01:30:01.564951183 +0000 UTC m=+727.141376585" Oct 30 01:30:01.623854 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:01.623799 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-t2vmk" podStartSLOduration=1.6237843939999999 podStartE2EDuration="1.623784394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:01.621934693 +0000 UTC m=+727.198360109" watchObservedRunningTime="2026-10-30 01:30:01.623784394 +0000 UTC m=+727.200209792" Oct 30 01:30:01.666035 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:01.665935 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:02.427632 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:02.427598 6572 generic.go:356] "Generic (PLEG): container finished" podID="c4f21909-1f45-4556-ac3d-3c6e4d9f0770" containerID="bc2db100820df635d3573ee598e27bdc8ede24e4b3770f92c343367bd6a81e21" exitCode=60 Oct 30 01:30:02.428046 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:02.427686 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888730-5f9p2" event={"ID":"c4f21909-1f45-4556-ac3d-3c6e4d9f0770","Type":"ContainerDied","Data":"bc2db100820df635d3573ee598e27bdc8ede24e4b3770f92c343367bd6a81e21"} Oct 30 01:30:02.428046 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:02.428021 6572 scope.go:122] "RemoveContainer" containerID="bc2db100820df635d3573ee598e27bdc8ede24e4b3770f92c343367bd6a81e21" Oct 30 01:30:02.429401 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:02.429379 6572 generic.go:356] "Generic (PLEG): container finished" podID="79840e8d-cb39-4428-9c5a-f37fb037a6e8" containerID="6e81fdea28c8dbe1a558a084af17d21461ed179c348171d16591ba83d82b7680" exitCode=60 Oct 30 01:30:02.429514 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:02.429403 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888730-7fprx" event={"ID":"79840e8d-cb39-4428-9c5a-f37fb037a6e8","Type":"ContainerDied","Data":"6e81fdea28c8dbe1a558a084af17d21461ed179c348171d16591ba83d82b7680"} Oct 30 01:30:02.429676 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:02.429653 6572 scope.go:122] "RemoveContainer" containerID="6e81fdea28c8dbe1a558a084af17d21461ed179c348171d16591ba83d82b7680" Oct 30 01:30:02.431295 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:02.431228 6572 generic.go:356] "Generic (PLEG): container finished" podID="2b5aa772-726a-466a-ab5e-fd03013c1306" containerID="3380df5f97395370fe318d4db4873e944c1f3a22289d5fce1c6a2ebf36b61dc8" exitCode=60 Oct 30 01:30:02.431490 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:02.431433 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888730-vkcb8" event={"ID":"2b5aa772-726a-466a-ab5e-fd03013c1306","Type":"ContainerDied","Data":"3380df5f97395370fe318d4db4873e944c1f3a22289d5fce1c6a2ebf36b61dc8"} Oct 30 01:30:02.431626 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:02.431606 6572 scope.go:122] "RemoveContainer" containerID="3380df5f97395370fe318d4db4873e944c1f3a22289d5fce1c6a2ebf36b61dc8" Oct 30 01:30:02.439050 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:02.439032 6572 generic.go:356] "Generic (PLEG): container finished" podID="20b63c16-472b-435d-a1cc-49c0e471b20c" containerID="0bc5f27de33103e85192c401f8675c49f08c5f0ae253f7a6b221d08e07849ad4" exitCode=60 Oct 30 01:30:02.439155 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:02.439061 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-t2vmk" event={"ID":"20b63c16-472b-435d-a1cc-49c0e471b20c","Type":"ContainerDied","Data":"0bc5f27de33103e85192c401f8675c49f08c5f0ae253f7a6b221d08e07849ad4"} Oct 30 01:30:02.439350 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:02.439331 6572 scope.go:122] "RemoveContainer" containerID="0bc5f27de33103e85192c401f8675c49f08c5f0ae253f7a6b221d08e07849ad4" Oct 30 01:30:03.177263 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:03.177218 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:03.381202 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:03.381170 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:03.447929 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:03.447792 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888730-5f9p2" event={"ID":"c4f21909-1f45-4556-ac3d-3c6e4d9f0770","Type":"ContainerStarted","Data":"f697e2d1bd6cc18eb55e17e4ed18971672f6acc5227a24621eb113ac3ac0b2a9"} Oct 30 01:30:03.449874 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:03.449848 6572 generic.go:356] "Generic (PLEG): container finished" podID="79840e8d-cb39-4428-9c5a-f37fb037a6e8" containerID="46774cbfbe03e4b3b95bffae8367c4fcfd0411bb1241758861a3aaa135308e65" exitCode=60 Oct 30 01:30:03.450000 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:03.449926 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888730-7fprx" event={"ID":"79840e8d-cb39-4428-9c5a-f37fb037a6e8","Type":"ContainerDied","Data":"46774cbfbe03e4b3b95bffae8367c4fcfd0411bb1241758861a3aaa135308e65"} Oct 30 01:30:03.450000 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:03.449982 6572 scope.go:122] "RemoveContainer" containerID="6e81fdea28c8dbe1a558a084af17d21461ed179c348171d16591ba83d82b7680" Oct 30 01:30:03.450167 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:03.450152 6572 scope.go:122] "RemoveContainer" containerID="46774cbfbe03e4b3b95bffae8367c4fcfd0411bb1241758861a3aaa135308e65" Oct 30 01:30:03.450422 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:30:03.450396 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888730-7fprx_odh-ai-gateway-infra(79840e8d-cb39-4428-9c5a-f37fb037a6e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888730-7fprx" podUID="79840e8d-cb39-4428-9c5a-f37fb037a6e8" Oct 30 01:30:03.452021 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:03.452000 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888730-vkcb8" event={"ID":"2b5aa772-726a-466a-ab5e-fd03013c1306","Type":"ContainerStarted","Data":"d82172d845104fb3486fb0a939a9fb98378d9a65376ce4d6ad1c2c9e75e09718"} Oct 30 01:30:03.453967 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:03.453945 6572 generic.go:356] "Generic (PLEG): container finished" podID="20b63c16-472b-435d-a1cc-49c0e471b20c" containerID="14850bcff893be5378ea90dd7c69b197ceb180537298c20065dbb413df47b30f" exitCode=60 Oct 30 01:30:03.454068 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:03.454010 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-t2vmk" event={"ID":"20b63c16-472b-435d-a1cc-49c0e471b20c","Type":"ContainerDied","Data":"14850bcff893be5378ea90dd7c69b197ceb180537298c20065dbb413df47b30f"} Oct 30 01:30:03.454261 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:03.454249 6572 scope.go:122] "RemoveContainer" containerID="14850bcff893be5378ea90dd7c69b197ceb180537298c20065dbb413df47b30f" Oct 30 01:30:03.454460 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:30:03.454444 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888730-t2vmk_odh-ai-gateway-infra(20b63c16-472b-435d-a1cc-49c0e471b20c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-t2vmk" podUID="20b63c16-472b-435d-a1cc-49c0e471b20c" Oct 30 01:30:03.462468 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:03.462443 6572 scope.go:122] "RemoveContainer" containerID="0bc5f27de33103e85192c401f8675c49f08c5f0ae253f7a6b221d08e07849ad4" Oct 30 01:30:04.462625 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:04.462585 6572 generic.go:356] "Generic (PLEG): container finished" podID="2b5aa772-726a-466a-ab5e-fd03013c1306" containerID="d82172d845104fb3486fb0a939a9fb98378d9a65376ce4d6ad1c2c9e75e09718" exitCode=60 Oct 30 01:30:04.463115 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:04.462634 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888730-vkcb8" event={"ID":"2b5aa772-726a-466a-ab5e-fd03013c1306","Type":"ContainerDied","Data":"d82172d845104fb3486fb0a939a9fb98378d9a65376ce4d6ad1c2c9e75e09718"} Oct 30 01:30:04.463115 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:04.462731 6572 scope.go:122] "RemoveContainer" containerID="3380df5f97395370fe318d4db4873e944c1f3a22289d5fce1c6a2ebf36b61dc8" Oct 30 01:30:04.463115 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:04.463074 6572 scope.go:122] "RemoveContainer" containerID="d82172d845104fb3486fb0a939a9fb98378d9a65376ce4d6ad1c2c9e75e09718" Oct 30 01:30:04.463425 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:30:04.463401 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-558327-29888730-vkcb8_odh-ai-gateway-infra(2b5aa772-726a-466a-ab5e-fd03013c1306)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888730-vkcb8" podUID="2b5aa772-726a-466a-ab5e-fd03013c1306" Oct 30 01:30:04.464897 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:04.464878 6572 scope.go:122] "RemoveContainer" containerID="14850bcff893be5378ea90dd7c69b197ceb180537298c20065dbb413df47b30f" Oct 30 01:30:04.465139 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:30:04.465122 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888730-t2vmk_odh-ai-gateway-infra(20b63c16-472b-435d-a1cc-49c0e471b20c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-t2vmk" podUID="20b63c16-472b-435d-a1cc-49c0e471b20c" Oct 30 01:30:04.466487 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:04.466468 6572 generic.go:356] "Generic (PLEG): container finished" podID="c4f21909-1f45-4556-ac3d-3c6e4d9f0770" containerID="f697e2d1bd6cc18eb55e17e4ed18971672f6acc5227a24621eb113ac3ac0b2a9" exitCode=60 Oct 30 01:30:04.466578 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:04.466496 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888730-5f9p2" event={"ID":"c4f21909-1f45-4556-ac3d-3c6e4d9f0770","Type":"ContainerDied","Data":"f697e2d1bd6cc18eb55e17e4ed18971672f6acc5227a24621eb113ac3ac0b2a9"} Oct 30 01:30:04.466788 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:04.466774 6572 scope.go:122] "RemoveContainer" containerID="f697e2d1bd6cc18eb55e17e4ed18971672f6acc5227a24621eb113ac3ac0b2a9" Oct 30 01:30:04.467030 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:30:04.467009 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-803472-29888730-5f9p2_odh-ai-gateway-infra(c4f21909-1f45-4556-ac3d-3c6e4d9f0770)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888730-5f9p2" podUID="c4f21909-1f45-4556-ac3d-3c6e4d9f0770" Oct 30 01:30:04.468357 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:04.468340 6572 scope.go:122] "RemoveContainer" containerID="46774cbfbe03e4b3b95bffae8367c4fcfd0411bb1241758861a3aaa135308e65" Oct 30 01:30:04.468521 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:30:04.468505 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888730-7fprx_odh-ai-gateway-infra(79840e8d-cb39-4428-9c5a-f37fb037a6e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888730-7fprx" podUID="79840e8d-cb39-4428-9c5a-f37fb037a6e8" Oct 30 01:30:04.475089 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:04.475065 6572 scope.go:122] "RemoveContainer" containerID="bc2db100820df635d3573ee598e27bdc8ede24e4b3770f92c343367bd6a81e21" Oct 30 01:30:04.944964 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:04.944924 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:05.477453 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:05.477423 6572 scope.go:122] "RemoveContainer" containerID="d82172d845104fb3486fb0a939a9fb98378d9a65376ce4d6ad1c2c9e75e09718" Oct 30 01:30:05.477933 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:30:05.477724 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-558327-29888730-vkcb8_odh-ai-gateway-infra(2b5aa772-726a-466a-ab5e-fd03013c1306)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888730-vkcb8" podUID="2b5aa772-726a-466a-ab5e-fd03013c1306" Oct 30 01:30:05.479232 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:05.479214 6572 scope.go:122] "RemoveContainer" containerID="f697e2d1bd6cc18eb55e17e4ed18971672f6acc5227a24621eb113ac3ac0b2a9" Oct 30 01:30:05.479453 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:30:05.479433 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-803472-29888730-5f9p2_odh-ai-gateway-infra(c4f21909-1f45-4556-ac3d-3c6e4d9f0770)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888730-5f9p2" podUID="c4f21909-1f45-4556-ac3d-3c6e4d9f0770" Oct 30 01:30:06.727191 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:06.727158 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:08.475862 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:08.475809 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:10.193450 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:10.193418 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:10.934670 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:10.934635 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-136a9875-pcfzv"] Oct 30 01:30:10.939274 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:10.939257 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-136a9875-pcfzv" Oct 30 01:30:10.952068 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:10.952030 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:30:10.964834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:10.964806 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-136a9875-pcfzv"] Oct 30 01:30:11.053892 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:11.053856 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ed2e616-52c0-4575-8a58-27c122e24b31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-136a9875-pcfzv\" (UID: \"5ed2e616-52c0-4575-8a58-27c122e24b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-136a9875-pcfzv" Oct 30 01:30:11.054034 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:11.053905 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ed2e616-52c0-4575-8a58-27c122e24b31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-136a9875-pcfzv\" (UID: \"5ed2e616-52c0-4575-8a58-27c122e24b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-136a9875-pcfzv" Oct 30 01:30:11.054034 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:11.053928 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jmxb\" (UniqueName: \"kubernetes.io/projected/5ed2e616-52c0-4575-8a58-27c122e24b31-kube-api-access-5jmxb\") pod \"maas-api-revoke-sub-136a9875-pcfzv\" (UID: \"5ed2e616-52c0-4575-8a58-27c122e24b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-136a9875-pcfzv" Oct 30 01:30:11.154583 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:11.154555 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ed2e616-52c0-4575-8a58-27c122e24b31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-136a9875-pcfzv\" (UID: \"5ed2e616-52c0-4575-8a58-27c122e24b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-136a9875-pcfzv" Oct 30 01:30:11.154727 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:11.154604 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ed2e616-52c0-4575-8a58-27c122e24b31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-136a9875-pcfzv\" (UID: \"5ed2e616-52c0-4575-8a58-27c122e24b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-136a9875-pcfzv" Oct 30 01:30:11.154790 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:11.154720 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jmxb\" (UniqueName: \"kubernetes.io/projected/5ed2e616-52c0-4575-8a58-27c122e24b31-kube-api-access-5jmxb\") pod \"maas-api-revoke-sub-136a9875-pcfzv\" (UID: \"5ed2e616-52c0-4575-8a58-27c122e24b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-136a9875-pcfzv" Oct 30 01:30:11.155211 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:11.155191 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ed2e616-52c0-4575-8a58-27c122e24b31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-136a9875-pcfzv\" (UID: \"5ed2e616-52c0-4575-8a58-27c122e24b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-136a9875-pcfzv" Oct 30 01:30:11.156798 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:11.156781 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ed2e616-52c0-4575-8a58-27c122e24b31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-136a9875-pcfzv\" (UID: \"5ed2e616-52c0-4575-8a58-27c122e24b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-136a9875-pcfzv" Oct 30 01:30:11.172118 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:11.172090 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jmxb\" (UniqueName: \"kubernetes.io/projected/5ed2e616-52c0-4575-8a58-27c122e24b31-kube-api-access-5jmxb\") pod \"maas-api-revoke-sub-136a9875-pcfzv\" (UID: \"5ed2e616-52c0-4575-8a58-27c122e24b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-136a9875-pcfzv" Oct 30 01:30:11.255402 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:11.255318 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-136a9875-pcfzv" Oct 30 01:30:11.560080 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:11.560042 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-136a9875-pcfzv"] Oct 30 01:30:11.560974 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:30:11.560948 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ed2e616_52c0_4575_8a58_27c122e24b31.slice/crio-53238e4c909fcf8e413113d8ffcbc8637ad510753697df5c3b1ad939556d1211 WatchSource:0}: Error finding container 53238e4c909fcf8e413113d8ffcbc8637ad510753697df5c3b1ad939556d1211: Status 404 returned error can't find the container with id 53238e4c909fcf8e413113d8ffcbc8637ad510753697df5c3b1ad939556d1211 Oct 30 01:30:11.920409 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:11.920370 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:12.529937 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:12.529902 6572 generic.go:356] "Generic (PLEG): container finished" podID="5ed2e616-52c0-4575-8a58-27c122e24b31" containerID="c8a4817dcbd10d466d116b4d163c58211d0f2dc69266635e88c47dad0183ca31" exitCode=0 Oct 30 01:30:12.530567 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:12.529977 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-136a9875-pcfzv" event={"ID":"5ed2e616-52c0-4575-8a58-27c122e24b31","Type":"ContainerDied","Data":"c8a4817dcbd10d466d116b4d163c58211d0f2dc69266635e88c47dad0183ca31"} Oct 30 01:30:12.530567 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:12.530039 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-136a9875-pcfzv" event={"ID":"5ed2e616-52c0-4575-8a58-27c122e24b31","Type":"ContainerStarted","Data":"53238e4c909fcf8e413113d8ffcbc8637ad510753697df5c3b1ad939556d1211"} Oct 30 01:30:13.509860 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:13.509819 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:13.802880 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:13.802849 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-136a9875-pcfzv" Oct 30 01:30:13.885984 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:13.885944 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ed2e616-52c0-4575-8a58-27c122e24b31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ed2e616-52c0-4575-8a58-27c122e24b31-maas-api-openshift-ca\") pod \"5ed2e616-52c0-4575-8a58-27c122e24b31\" (UID: \"5ed2e616-52c0-4575-8a58-27c122e24b31\") " Oct 30 01:30:13.886195 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:13.886041 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ed2e616-52c0-4575-8a58-27c122e24b31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ed2e616-52c0-4575-8a58-27c122e24b31-maas-api-xks-ca\") pod \"5ed2e616-52c0-4575-8a58-27c122e24b31\" (UID: \"5ed2e616-52c0-4575-8a58-27c122e24b31\") " Oct 30 01:30:13.886261 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:13.886234 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ed2e616-52c0-4575-8a58-27c122e24b31-kube-api-access-5jmxb\" (UniqueName: \"kubernetes.io/projected/5ed2e616-52c0-4575-8a58-27c122e24b31-kube-api-access-5jmxb\") pod \"5ed2e616-52c0-4575-8a58-27c122e24b31\" (UID: \"5ed2e616-52c0-4575-8a58-27c122e24b31\") " Oct 30 01:30:13.886504 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:13.886475 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ed2e616-52c0-4575-8a58-27c122e24b31-maas-api-openshift-ca" pod "5ed2e616-52c0-4575-8a58-27c122e24b31" (UID: "5ed2e616-52c0-4575-8a58-27c122e24b31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:13.886797 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:13.886771 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ed2e616-52c0-4575-8a58-27c122e24b31-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:30:13.888279 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:13.888245 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ed2e616-52c0-4575-8a58-27c122e24b31-maas-api-xks-ca" pod "5ed2e616-52c0-4575-8a58-27c122e24b31" (UID: "5ed2e616-52c0-4575-8a58-27c122e24b31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:13.888361 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:13.888311 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ed2e616-52c0-4575-8a58-27c122e24b31-kube-api-access-5jmxb" pod "5ed2e616-52c0-4575-8a58-27c122e24b31" (UID: "5ed2e616-52c0-4575-8a58-27c122e24b31"). InnerVolumeSpecName "kube-api-access-5jmxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:13.979526 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:13.979489 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:13.987901 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:13.987863 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ed2e616-52c0-4575-8a58-27c122e24b31-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:30:13.987901 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:13.987898 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jmxb\" (UniqueName: \"kubernetes.io/projected/5ed2e616-52c0-4575-8a58-27c122e24b31-kube-api-access-5jmxb\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:30:14.546478 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:14.546448 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-136a9875-pcfzv" Oct 30 01:30:14.546651 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:14.546501 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-136a9875-pcfzv" event={"ID":"5ed2e616-52c0-4575-8a58-27c122e24b31","Type":"ContainerDied","Data":"53238e4c909fcf8e413113d8ffcbc8637ad510753697df5c3b1ad939556d1211"} Oct 30 01:30:14.546651 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:14.546556 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53238e4c909fcf8e413113d8ffcbc8637ad510753697df5c3b1ad939556d1211" Oct 30 01:30:14.687466 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:30:14.687434 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:30:15.318016 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:15.317981 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:15.535721 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:15.535690 6572 scope.go:122] "RemoveContainer" containerID="f697e2d1bd6cc18eb55e17e4ed18971672f6acc5227a24621eb113ac3ac0b2a9" Oct 30 01:30:15.535901 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:15.535843 6572 scope.go:122] "RemoveContainer" containerID="46774cbfbe03e4b3b95bffae8367c4fcfd0411bb1241758861a3aaa135308e65" Oct 30 01:30:15.535984 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:15.535961 6572 scope.go:122] "RemoveContainer" containerID="d82172d845104fb3486fb0a939a9fb98378d9a65376ce4d6ad1c2c9e75e09718" Oct 30 01:30:16.561877 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:16.561795 6572 generic.go:356] "Generic (PLEG): container finished" podID="c4f21909-1f45-4556-ac3d-3c6e4d9f0770" containerID="423d9b86af142e0db1017bf8d7298ffddbb5fa8bbc1b6a11fcd8605bbaf82591" exitCode=60 Oct 30 01:30:16.562308 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:16.561862 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888730-5f9p2" event={"ID":"c4f21909-1f45-4556-ac3d-3c6e4d9f0770","Type":"ContainerDied","Data":"423d9b86af142e0db1017bf8d7298ffddbb5fa8bbc1b6a11fcd8605bbaf82591"} Oct 30 01:30:16.562308 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:16.561933 6572 scope.go:122] "RemoveContainer" containerID="f697e2d1bd6cc18eb55e17e4ed18971672f6acc5227a24621eb113ac3ac0b2a9" Oct 30 01:30:16.562442 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:16.562360 6572 scope.go:122] "RemoveContainer" containerID="423d9b86af142e0db1017bf8d7298ffddbb5fa8bbc1b6a11fcd8605bbaf82591" Oct 30 01:30:16.562683 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:30:16.562656 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-803472-29888730-5f9p2_odh-ai-gateway-infra(c4f21909-1f45-4556-ac3d-3c6e4d9f0770)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888730-5f9p2" podUID="c4f21909-1f45-4556-ac3d-3c6e4d9f0770" Oct 30 01:30:16.563882 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:16.563861 6572 generic.go:356] "Generic (PLEG): container finished" podID="79840e8d-cb39-4428-9c5a-f37fb037a6e8" containerID="3448792d6407b39ccccb5d63328f5d52fe710ea402b412908fb9ee8a882d39c6" exitCode=60 Oct 30 01:30:16.563977 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:16.563918 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888730-7fprx" event={"ID":"79840e8d-cb39-4428-9c5a-f37fb037a6e8","Type":"ContainerDied","Data":"3448792d6407b39ccccb5d63328f5d52fe710ea402b412908fb9ee8a882d39c6"} Oct 30 01:30:16.564276 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:16.564259 6572 scope.go:122] "RemoveContainer" containerID="3448792d6407b39ccccb5d63328f5d52fe710ea402b412908fb9ee8a882d39c6" Oct 30 01:30:16.564556 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:30:16.564534 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888730-7fprx_odh-ai-gateway-infra(79840e8d-cb39-4428-9c5a-f37fb037a6e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888730-7fprx" podUID="79840e8d-cb39-4428-9c5a-f37fb037a6e8" Oct 30 01:30:16.565889 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:16.565870 6572 generic.go:356] "Generic (PLEG): container finished" podID="2b5aa772-726a-466a-ab5e-fd03013c1306" containerID="8fd1645d323275c3d27be148030b6d74ffffe027f2270cf69b397bbb90cfd73d" exitCode=60 Oct 30 01:30:16.565986 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:16.565945 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888730-vkcb8" event={"ID":"2b5aa772-726a-466a-ab5e-fd03013c1306","Type":"ContainerDied","Data":"8fd1645d323275c3d27be148030b6d74ffffe027f2270cf69b397bbb90cfd73d"} Oct 30 01:30:16.566282 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:16.566265 6572 scope.go:122] "RemoveContainer" containerID="8fd1645d323275c3d27be148030b6d74ffffe027f2270cf69b397bbb90cfd73d" Oct 30 01:30:16.566480 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:30:16.566461 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-558327-29888730-vkcb8_odh-ai-gateway-infra(2b5aa772-726a-466a-ab5e-fd03013c1306)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888730-vkcb8" podUID="2b5aa772-726a-466a-ab5e-fd03013c1306" Oct 30 01:30:16.574740 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:16.574720 6572 scope.go:122] "RemoveContainer" containerID="46774cbfbe03e4b3b95bffae8367c4fcfd0411bb1241758861a3aaa135308e65" Oct 30 01:30:16.588581 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:16.588557 6572 scope.go:122] "RemoveContainer" containerID="d82172d845104fb3486fb0a939a9fb98378d9a65376ce4d6ad1c2c9e75e09718" Oct 30 01:30:17.126616 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:17.126582 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:17.535407 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:17.535365 6572 scope.go:122] "RemoveContainer" containerID="14850bcff893be5378ea90dd7c69b197ceb180537298c20065dbb413df47b30f" Oct 30 01:30:17.618358 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:17.618323 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888730-5f9p2"] Oct 30 01:30:17.669530 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:17.669499 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888730-vkcb8"] Oct 30 01:30:17.694041 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:17.693993 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888730-7fprx"] Oct 30 01:30:17.785282 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:17.785109 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888730-5f9p2" Oct 30 01:30:17.873919 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:17.873892 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888730-vkcb8" Oct 30 01:30:17.887190 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:17.887167 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888730-7fprx" Oct 30 01:30:17.932916 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:17.932888 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4f21909-1f45-4556-ac3d-3c6e4d9f0770-kube-api-access-mhv4p\" (UniqueName: \"kubernetes.io/projected/c4f21909-1f45-4556-ac3d-3c6e4d9f0770-kube-api-access-mhv4p\") pod \"c4f21909-1f45-4556-ac3d-3c6e4d9f0770\" (UID: \"c4f21909-1f45-4556-ac3d-3c6e4d9f0770\") " Oct 30 01:30:17.933081 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:17.932943 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4f21909-1f45-4556-ac3d-3c6e4d9f0770-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f21909-1f45-4556-ac3d-3c6e4d9f0770-maas-api-service-ca\") pod \"c4f21909-1f45-4556-ac3d-3c6e4d9f0770\" (UID: \"c4f21909-1f45-4556-ac3d-3c6e4d9f0770\") " Oct 30 01:30:17.933373 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:17.933347 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4f21909-1f45-4556-ac3d-3c6e4d9f0770-maas-api-service-ca" pod "c4f21909-1f45-4556-ac3d-3c6e4d9f0770" (UID: "c4f21909-1f45-4556-ac3d-3c6e4d9f0770"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:17.935166 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:17.935142 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f21909-1f45-4556-ac3d-3c6e4d9f0770-kube-api-access-mhv4p" pod "c4f21909-1f45-4556-ac3d-3c6e4d9f0770" (UID: "c4f21909-1f45-4556-ac3d-3c6e4d9f0770"). InnerVolumeSpecName "kube-api-access-mhv4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:18.034335 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:18.034302 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b5aa772-726a-466a-ab5e-fd03013c1306-kube-api-access-j8272\" (UniqueName: \"kubernetes.io/projected/2b5aa772-726a-466a-ab5e-fd03013c1306-kube-api-access-j8272\") pod \"2b5aa772-726a-466a-ab5e-fd03013c1306\" (UID: \"2b5aa772-726a-466a-ab5e-fd03013c1306\") " Oct 30 01:30:18.034536 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:18.034403 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b5aa772-726a-466a-ab5e-fd03013c1306-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b5aa772-726a-466a-ab5e-fd03013c1306-maas-api-service-ca\") pod \"2b5aa772-726a-466a-ab5e-fd03013c1306\" (UID: \"2b5aa772-726a-466a-ab5e-fd03013c1306\") " Oct 30 01:30:18.034536 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:18.034445 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79840e8d-cb39-4428-9c5a-f37fb037a6e8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79840e8d-cb39-4428-9c5a-f37fb037a6e8-maas-api-service-ca\") pod \"79840e8d-cb39-4428-9c5a-f37fb037a6e8\" (UID: \"79840e8d-cb39-4428-9c5a-f37fb037a6e8\") " Oct 30 01:30:18.034536 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:18.034531 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79840e8d-cb39-4428-9c5a-f37fb037a6e8-kube-api-access-4526d\" (UniqueName: \"kubernetes.io/projected/79840e8d-cb39-4428-9c5a-f37fb037a6e8-kube-api-access-4526d\") pod \"79840e8d-cb39-4428-9c5a-f37fb037a6e8\" (UID: \"79840e8d-cb39-4428-9c5a-f37fb037a6e8\") " Oct 30 01:30:18.034797 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:18.034775 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b5aa772-726a-466a-ab5e-fd03013c1306-maas-api-service-ca" pod "2b5aa772-726a-466a-ab5e-fd03013c1306" (UID: "2b5aa772-726a-466a-ab5e-fd03013c1306"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:18.034898 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:18.034868 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79840e8d-cb39-4428-9c5a-f37fb037a6e8-maas-api-service-ca" pod "79840e8d-cb39-4428-9c5a-f37fb037a6e8" (UID: "79840e8d-cb39-4428-9c5a-f37fb037a6e8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:18.035017 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:18.035003 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhv4p\" (UniqueName: \"kubernetes.io/projected/c4f21909-1f45-4556-ac3d-3c6e4d9f0770-kube-api-access-mhv4p\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:30:18.035073 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:18.035019 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b5aa772-726a-466a-ab5e-fd03013c1306-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:30:18.035073 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:18.035028 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79840e8d-cb39-4428-9c5a-f37fb037a6e8-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:30:18.035073 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:18.035042 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f21909-1f45-4556-ac3d-3c6e4d9f0770-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:30:18.036604 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:18.036586 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b5aa772-726a-466a-ab5e-fd03013c1306-kube-api-access-j8272" pod "2b5aa772-726a-466a-ab5e-fd03013c1306" (UID: "2b5aa772-726a-466a-ab5e-fd03013c1306"). InnerVolumeSpecName "kube-api-access-j8272". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:18.036671 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:18.036617 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79840e8d-cb39-4428-9c5a-f37fb037a6e8-kube-api-access-4526d" pod "79840e8d-cb39-4428-9c5a-f37fb037a6e8" (UID: "79840e8d-cb39-4428-9c5a-f37fb037a6e8"). InnerVolumeSpecName "kube-api-access-4526d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:18.136595 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:18.136514 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8272\" (UniqueName: \"kubernetes.io/projected/2b5aa772-726a-466a-ab5e-fd03013c1306-kube-api-access-j8272\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:30:18.136595 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:18.136544 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4526d\" (UniqueName: \"kubernetes.io/projected/79840e8d-cb39-4428-9c5a-f37fb037a6e8-kube-api-access-4526d\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:30:18.593283 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:18.593249 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888730-5f9p2" Oct 30 01:30:18.593479 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:18.593255 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888730-5f9p2" event={"ID":"c4f21909-1f45-4556-ac3d-3c6e4d9f0770","Type":"ContainerDied","Data":"a92614c3ce91fa67f6c33dcd0d6fece23e64ea6fa03796101bbb23970949999e"} Oct 30 01:30:18.593479 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:18.593413 6572 scope.go:122] "RemoveContainer" containerID="423d9b86af142e0db1017bf8d7298ffddbb5fa8bbc1b6a11fcd8605bbaf82591" Oct 30 01:30:18.594656 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:18.594635 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888730-7fprx" Oct 30 01:30:18.594763 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:18.594704 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888730-7fprx" event={"ID":"79840e8d-cb39-4428-9c5a-f37fb037a6e8","Type":"ContainerDied","Data":"506fe1aede96fd8455ef4f7201dfb9e33699c159f884e991ea15de4dd5752e14"} Oct 30 01:30:18.596341 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:18.596317 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888730-vkcb8" event={"ID":"2b5aa772-726a-466a-ab5e-fd03013c1306","Type":"ContainerDied","Data":"b6dc3d01a14d0fc0b3007ae788ac74a5fbb7f038088e9e697b5505d1e82620ac"} Oct 30 01:30:18.596341 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:18.596329 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888730-vkcb8" Oct 30 01:30:18.598390 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:18.598370 6572 generic.go:356] "Generic (PLEG): container finished" podID="20b63c16-472b-435d-a1cc-49c0e471b20c" containerID="8bc5005d13284b64dbfbe6d7fe55dfbeec7d3389c1d13216fa725263f7b81f75" exitCode=60 Oct 30 01:30:18.598494 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:18.598442 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-t2vmk" event={"ID":"20b63c16-472b-435d-a1cc-49c0e471b20c","Type":"ContainerDied","Data":"8bc5005d13284b64dbfbe6d7fe55dfbeec7d3389c1d13216fa725263f7b81f75"} Oct 30 01:30:18.598815 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:18.598798 6572 scope.go:122] "RemoveContainer" containerID="8bc5005d13284b64dbfbe6d7fe55dfbeec7d3389c1d13216fa725263f7b81f75" Oct 30 01:30:18.599031 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:30:18.599014 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888730-t2vmk_odh-ai-gateway-infra(20b63c16-472b-435d-a1cc-49c0e471b20c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-t2vmk" podUID="20b63c16-472b-435d-a1cc-49c0e471b20c" Oct 30 01:30:18.604499 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:18.604474 6572 scope.go:122] "RemoveContainer" containerID="3448792d6407b39ccccb5d63328f5d52fe710ea402b412908fb9ee8a882d39c6" Oct 30 01:30:18.613330 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:18.613313 6572 scope.go:122] "RemoveContainer" containerID="8fd1645d323275c3d27be148030b6d74ffffe027f2270cf69b397bbb90cfd73d" Oct 30 01:30:18.622038 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:18.621839 6572 scope.go:122] "RemoveContainer" containerID="14850bcff893be5378ea90dd7c69b197ceb180537298c20065dbb413df47b30f" Oct 30 01:30:18.649069 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:18.649041 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888730-5f9p2"] Oct 30 01:30:18.665208 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:18.665183 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888730-5f9p2"] Oct 30 01:30:18.700638 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:18.700564 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888730-vkcb8"] Oct 30 01:30:18.716030 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:18.716003 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888730-vkcb8"] Oct 30 01:30:18.802966 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:18.802935 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888730-7fprx"] Oct 30 01:30:18.807235 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:18.807213 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888730-7fprx"] Oct 30 01:30:19.062549 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:19.062461 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:19.780378 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:19.780343 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888730-t2vmk"] Oct 30 01:30:19.925836 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:19.925812 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-t2vmk" Oct 30 01:30:20.059341 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:20.059250 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20b63c16-472b-435d-a1cc-49c0e471b20c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20b63c16-472b-435d-a1cc-49c0e471b20c-maas-api-service-ca\") pod \"20b63c16-472b-435d-a1cc-49c0e471b20c\" (UID: \"20b63c16-472b-435d-a1cc-49c0e471b20c\") " Oct 30 01:30:20.059504 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:20.059345 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20b63c16-472b-435d-a1cc-49c0e471b20c-kube-api-access-cxgsv\" (UniqueName: \"kubernetes.io/projected/20b63c16-472b-435d-a1cc-49c0e471b20c-kube-api-access-cxgsv\") pod \"20b63c16-472b-435d-a1cc-49c0e471b20c\" (UID: \"20b63c16-472b-435d-a1cc-49c0e471b20c\") " Oct 30 01:30:20.059658 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:20.059635 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20b63c16-472b-435d-a1cc-49c0e471b20c-maas-api-service-ca" pod "20b63c16-472b-435d-a1cc-49c0e471b20c" (UID: "20b63c16-472b-435d-a1cc-49c0e471b20c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:20.061488 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:20.061458 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20b63c16-472b-435d-a1cc-49c0e471b20c-kube-api-access-cxgsv" pod "20b63c16-472b-435d-a1cc-49c0e471b20c" (UID: "20b63c16-472b-435d-a1cc-49c0e471b20c"). InnerVolumeSpecName "kube-api-access-cxgsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:20.160814 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:20.160774 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20b63c16-472b-435d-a1cc-49c0e471b20c-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:30:20.160814 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:20.160806 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxgsv\" (UniqueName: \"kubernetes.io/projected/20b63c16-472b-435d-a1cc-49c0e471b20c-kube-api-access-cxgsv\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:30:20.541609 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:20.541567 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b5aa772-726a-466a-ab5e-fd03013c1306" path="/var/lib/kubelet/pods/2b5aa772-726a-466a-ab5e-fd03013c1306/volumes" Oct 30 01:30:20.542086 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:20.542064 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79840e8d-cb39-4428-9c5a-f37fb037a6e8" path="/var/lib/kubelet/pods/79840e8d-cb39-4428-9c5a-f37fb037a6e8/volumes" Oct 30 01:30:20.542493 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:20.542475 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4f21909-1f45-4556-ac3d-3c6e4d9f0770" path="/var/lib/kubelet/pods/c4f21909-1f45-4556-ac3d-3c6e4d9f0770/volumes" Oct 30 01:30:20.617352 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:20.617324 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-t2vmk" Oct 30 01:30:20.617532 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:20.617359 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-t2vmk" event={"ID":"20b63c16-472b-435d-a1cc-49c0e471b20c","Type":"ContainerDied","Data":"0f34a0600639557c121fa4cd5a8da563cda2df38128645c4c61eb76fc227ab1f"} Oct 30 01:30:20.617532 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:20.617425 6572 scope.go:122] "RemoveContainer" containerID="8bc5005d13284b64dbfbe6d7fe55dfbeec7d3389c1d13216fa725263f7b81f75" Oct 30 01:30:20.665760 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:20.665717 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888730-t2vmk"] Oct 30 01:30:20.672398 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:20.672368 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888730-t2vmk"] Oct 30 01:30:20.868319 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:20.868224 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:22.542620 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:22.542574 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20b63c16-472b-435d-a1cc-49c0e471b20c" path="/var/lib/kubelet/pods/20b63c16-472b-435d-a1cc-49c0e471b20c/volumes" Oct 30 01:30:22.767356 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:22.767318 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:24.665150 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:24.665116 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:25.149607 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.149512 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c63a0579-j2qk9"] Oct 30 01:30:25.150144 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.150123 6572 cpu_manager.go:414] "containerMap: removing container" podUID="79840e8d-cb39-4428-9c5a-f37fb037a6e8" containerName="cleanup" Oct 30 01:30:25.150243 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.150145 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79840e8d-cb39-4428-9c5a-f37fb037a6e8" containerName="cleanup" Oct 30 01:30:25.150243 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.150162 6572 cpu_manager.go:414] "containerMap: removing container" podUID="2b5aa772-726a-466a-ab5e-fd03013c1306" containerName="cleanup" Oct 30 01:30:25.150243 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.150171 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b5aa772-726a-466a-ab5e-fd03013c1306" containerName="cleanup" Oct 30 01:30:25.150243 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.150186 6572 cpu_manager.go:414] "containerMap: removing container" podUID="c4f21909-1f45-4556-ac3d-3c6e4d9f0770" containerName="cleanup" Oct 30 01:30:25.150243 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.150194 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4f21909-1f45-4556-ac3d-3c6e4d9f0770" containerName="cleanup" Oct 30 01:30:25.150243 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.150204 6572 cpu_manager.go:414] "containerMap: removing container" podUID="5ed2e616-52c0-4575-8a58-27c122e24b31" containerName="revoke-keys" Oct 30 01:30:25.150243 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.150211 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ed2e616-52c0-4575-8a58-27c122e24b31" containerName="revoke-keys" Oct 30 01:30:25.150243 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.150228 6572 cpu_manager.go:414] "containerMap: removing container" podUID="c4f21909-1f45-4556-ac3d-3c6e4d9f0770" containerName="cleanup" Oct 30 01:30:25.150243 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.150236 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4f21909-1f45-4556-ac3d-3c6e4d9f0770" containerName="cleanup" Oct 30 01:30:25.150243 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.150245 6572 cpu_manager.go:414] "containerMap: removing container" podUID="2b5aa772-726a-466a-ab5e-fd03013c1306" containerName="cleanup" Oct 30 01:30:25.150723 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.150252 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b5aa772-726a-466a-ab5e-fd03013c1306" containerName="cleanup" Oct 30 01:30:25.150723 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.150264 6572 cpu_manager.go:414] "containerMap: removing container" podUID="79840e8d-cb39-4428-9c5a-f37fb037a6e8" containerName="cleanup" Oct 30 01:30:25.150723 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.150270 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79840e8d-cb39-4428-9c5a-f37fb037a6e8" containerName="cleanup" Oct 30 01:30:25.150723 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.150282 6572 cpu_manager.go:414] "containerMap: removing container" podUID="c4f21909-1f45-4556-ac3d-3c6e4d9f0770" containerName="cleanup" Oct 30 01:30:25.150723 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.150289 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4f21909-1f45-4556-ac3d-3c6e4d9f0770" containerName="cleanup" Oct 30 01:30:25.150723 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.150309 6572 cpu_manager.go:414] "containerMap: removing container" podUID="20b63c16-472b-435d-a1cc-49c0e471b20c" containerName="cleanup" Oct 30 01:30:25.150723 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.150316 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20b63c16-472b-435d-a1cc-49c0e471b20c" containerName="cleanup" Oct 30 01:30:25.150723 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.150354 6572 cpu_manager.go:414] "containerMap: removing container" podUID="20b63c16-472b-435d-a1cc-49c0e471b20c" containerName="cleanup" Oct 30 01:30:25.150723 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.150362 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20b63c16-472b-435d-a1cc-49c0e471b20c" containerName="cleanup" Oct 30 01:30:25.150723 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.150492 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="20b63c16-472b-435d-a1cc-49c0e471b20c" containerName="cleanup" Oct 30 01:30:25.150723 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.150504 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b5aa772-726a-466a-ab5e-fd03013c1306" containerName="cleanup" Oct 30 01:30:25.150723 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.150513 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4f21909-1f45-4556-ac3d-3c6e4d9f0770" containerName="cleanup" Oct 30 01:30:25.150723 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.150522 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b5aa772-726a-466a-ab5e-fd03013c1306" containerName="cleanup" Oct 30 01:30:25.150723 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.150532 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4f21909-1f45-4556-ac3d-3c6e4d9f0770" containerName="cleanup" Oct 30 01:30:25.150723 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.150541 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="79840e8d-cb39-4428-9c5a-f37fb037a6e8" containerName="cleanup" Oct 30 01:30:25.150723 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.150549 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="20b63c16-472b-435d-a1cc-49c0e471b20c" containerName="cleanup" Oct 30 01:30:25.150723 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.150563 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="79840e8d-cb39-4428-9c5a-f37fb037a6e8" containerName="cleanup" Oct 30 01:30:25.150723 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.150571 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ed2e616-52c0-4575-8a58-27c122e24b31" containerName="revoke-keys" Oct 30 01:30:25.155894 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.155846 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63a0579-j2qk9" Oct 30 01:30:25.170962 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.170942 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:30:25.174876 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.174854 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:30:25.184471 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.184440 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c63a0579-j2qk9"] Oct 30 01:30:25.202774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.202720 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:30:25.278051 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.278027 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:30:25.319017 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.318989 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac7d7752-da8b-4385-9f5b-48f401708d87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c63a0579-j2qk9\" (UID: \"ac7d7752-da8b-4385-9f5b-48f401708d87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63a0579-j2qk9" Oct 30 01:30:25.319142 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.319058 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac7d7752-da8b-4385-9f5b-48f401708d87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c63a0579-j2qk9\" (UID: \"ac7d7752-da8b-4385-9f5b-48f401708d87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63a0579-j2qk9" Oct 30 01:30:25.319142 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.319091 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8cld\" (UniqueName: \"kubernetes.io/projected/ac7d7752-da8b-4385-9f5b-48f401708d87-kube-api-access-l8cld\") pod \"maas-api-revoke-sub-c63a0579-j2qk9\" (UID: \"ac7d7752-da8b-4385-9f5b-48f401708d87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63a0579-j2qk9" Oct 30 01:30:25.420872 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.420839 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac7d7752-da8b-4385-9f5b-48f401708d87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c63a0579-j2qk9\" (UID: \"ac7d7752-da8b-4385-9f5b-48f401708d87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63a0579-j2qk9" Oct 30 01:30:25.421016 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.420888 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac7d7752-da8b-4385-9f5b-48f401708d87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c63a0579-j2qk9\" (UID: \"ac7d7752-da8b-4385-9f5b-48f401708d87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63a0579-j2qk9" Oct 30 01:30:25.421016 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.420912 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8cld\" (UniqueName: \"kubernetes.io/projected/ac7d7752-da8b-4385-9f5b-48f401708d87-kube-api-access-l8cld\") pod \"maas-api-revoke-sub-c63a0579-j2qk9\" (UID: \"ac7d7752-da8b-4385-9f5b-48f401708d87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63a0579-j2qk9" Oct 30 01:30:25.421546 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.421526 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac7d7752-da8b-4385-9f5b-48f401708d87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c63a0579-j2qk9\" (UID: \"ac7d7752-da8b-4385-9f5b-48f401708d87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63a0579-j2qk9" Oct 30 01:30:25.423223 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.423204 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac7d7752-da8b-4385-9f5b-48f401708d87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c63a0579-j2qk9\" (UID: \"ac7d7752-da8b-4385-9f5b-48f401708d87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63a0579-j2qk9" Oct 30 01:30:25.441835 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.441811 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8cld\" (UniqueName: \"kubernetes.io/projected/ac7d7752-da8b-4385-9f5b-48f401708d87-kube-api-access-l8cld\") pod \"maas-api-revoke-sub-c63a0579-j2qk9\" (UID: \"ac7d7752-da8b-4385-9f5b-48f401708d87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63a0579-j2qk9" Oct 30 01:30:25.470989 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.470962 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63a0579-j2qk9" Oct 30 01:30:25.767041 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:25.767006 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c63a0579-j2qk9"] Oct 30 01:30:25.768407 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:30:25.768377 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac7d7752_da8b_4385_9f5b_48f401708d87.slice/crio-1a66069f15653995fa5bdc24d8c0320eafea68339cb0ecf29d1191f9fd27a5b6 WatchSource:0}: Error finding container 1a66069f15653995fa5bdc24d8c0320eafea68339cb0ecf29d1191f9fd27a5b6: Status 404 returned error can't find the container with id 1a66069f15653995fa5bdc24d8c0320eafea68339cb0ecf29d1191f9fd27a5b6 Oct 30 01:30:26.559156 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:26.559111 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:26.664419 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:26.664384 6572 generic.go:356] "Generic (PLEG): container finished" podID="ac7d7752-da8b-4385-9f5b-48f401708d87" containerID="5ca99ecaf118e30faea18a8dd53d6b7afdb935ef1651a6156d679d0ca990e5ab" exitCode=0 Oct 30 01:30:26.664617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:26.664464 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63a0579-j2qk9" event={"ID":"ac7d7752-da8b-4385-9f5b-48f401708d87","Type":"ContainerDied","Data":"5ca99ecaf118e30faea18a8dd53d6b7afdb935ef1651a6156d679d0ca990e5ab"} Oct 30 01:30:26.664617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:26.664514 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63a0579-j2qk9" event={"ID":"ac7d7752-da8b-4385-9f5b-48f401708d87","Type":"ContainerStarted","Data":"1a66069f15653995fa5bdc24d8c0320eafea68339cb0ecf29d1191f9fd27a5b6"} Oct 30 01:30:27.813618 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:27.813594 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63a0579-j2qk9" Oct 30 01:30:27.948290 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:27.948211 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac7d7752-da8b-4385-9f5b-48f401708d87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac7d7752-da8b-4385-9f5b-48f401708d87-maas-api-xks-ca\") pod \"ac7d7752-da8b-4385-9f5b-48f401708d87\" (UID: \"ac7d7752-da8b-4385-9f5b-48f401708d87\") " Oct 30 01:30:27.948290 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:27.948282 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac7d7752-da8b-4385-9f5b-48f401708d87-kube-api-access-l8cld\" (UniqueName: \"kubernetes.io/projected/ac7d7752-da8b-4385-9f5b-48f401708d87-kube-api-access-l8cld\") pod \"ac7d7752-da8b-4385-9f5b-48f401708d87\" (UID: \"ac7d7752-da8b-4385-9f5b-48f401708d87\") " Oct 30 01:30:27.948511 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:27.948329 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac7d7752-da8b-4385-9f5b-48f401708d87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac7d7752-da8b-4385-9f5b-48f401708d87-maas-api-openshift-ca\") pod \"ac7d7752-da8b-4385-9f5b-48f401708d87\" (UID: \"ac7d7752-da8b-4385-9f5b-48f401708d87\") " Oct 30 01:30:27.948730 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:27.948696 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac7d7752-da8b-4385-9f5b-48f401708d87-maas-api-openshift-ca" pod "ac7d7752-da8b-4385-9f5b-48f401708d87" (UID: "ac7d7752-da8b-4385-9f5b-48f401708d87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:27.950381 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:27.950360 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac7d7752-da8b-4385-9f5b-48f401708d87-maas-api-xks-ca" pod "ac7d7752-da8b-4385-9f5b-48f401708d87" (UID: "ac7d7752-da8b-4385-9f5b-48f401708d87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:27.950439 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:27.950401 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac7d7752-da8b-4385-9f5b-48f401708d87-kube-api-access-l8cld" pod "ac7d7752-da8b-4385-9f5b-48f401708d87" (UID: "ac7d7752-da8b-4385-9f5b-48f401708d87"). InnerVolumeSpecName "kube-api-access-l8cld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:28.049726 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:28.049693 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac7d7752-da8b-4385-9f5b-48f401708d87-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:30:28.049726 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:28.049719 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8cld\" (UniqueName: \"kubernetes.io/projected/ac7d7752-da8b-4385-9f5b-48f401708d87-kube-api-access-l8cld\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:30:28.049726 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:28.049729 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac7d7752-da8b-4385-9f5b-48f401708d87-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:30:28.382730 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:28.382652 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:28.677908 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:28.677879 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63a0579-j2qk9" Oct 30 01:30:28.678043 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:28.677911 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63a0579-j2qk9" event={"ID":"ac7d7752-da8b-4385-9f5b-48f401708d87","Type":"ContainerDied","Data":"1a66069f15653995fa5bdc24d8c0320eafea68339cb0ecf29d1191f9fd27a5b6"} Oct 30 01:30:28.678043 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:28.677960 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a66069f15653995fa5bdc24d8c0320eafea68339cb0ecf29d1191f9fd27a5b6" Oct 30 01:30:30.340001 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:30.339968 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:32.240481 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:32.240444 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:33.358564 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.358530 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd"] Oct 30 01:30:33.359188 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.359170 6572 cpu_manager.go:414] "containerMap: removing container" podUID="ac7d7752-da8b-4385-9f5b-48f401708d87" containerName="revoke-keys" Oct 30 01:30:33.359242 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.359189 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac7d7752-da8b-4385-9f5b-48f401708d87" containerName="revoke-keys" Oct 30 01:30:33.359242 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.359209 6572 cpu_manager.go:414] "containerMap: removing container" podUID="20b63c16-472b-435d-a1cc-49c0e471b20c" containerName="cleanup" Oct 30 01:30:33.359242 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.359215 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20b63c16-472b-435d-a1cc-49c0e471b20c" containerName="cleanup" Oct 30 01:30:33.359242 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.359238 6572 cpu_manager.go:414] "containerMap: removing container" podUID="2b5aa772-726a-466a-ab5e-fd03013c1306" containerName="cleanup" Oct 30 01:30:33.359242 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.359243 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b5aa772-726a-466a-ab5e-fd03013c1306" containerName="cleanup" Oct 30 01:30:33.359385 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.359250 6572 cpu_manager.go:414] "containerMap: removing container" podUID="79840e8d-cb39-4428-9c5a-f37fb037a6e8" containerName="cleanup" Oct 30 01:30:33.359385 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.359256 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79840e8d-cb39-4428-9c5a-f37fb037a6e8" containerName="cleanup" Oct 30 01:30:33.359385 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.359333 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b5aa772-726a-466a-ab5e-fd03013c1306" containerName="cleanup" Oct 30 01:30:33.359385 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.359341 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4f21909-1f45-4556-ac3d-3c6e4d9f0770" containerName="cleanup" Oct 30 01:30:33.359385 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.359346 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="79840e8d-cb39-4428-9c5a-f37fb037a6e8" containerName="cleanup" Oct 30 01:30:33.359385 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.359352 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="20b63c16-472b-435d-a1cc-49c0e471b20c" containerName="cleanup" Oct 30 01:30:33.359385 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.359362 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac7d7752-da8b-4385-9f5b-48f401708d87" containerName="revoke-keys" Oct 30 01:30:33.362928 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.362911 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" Oct 30 01:30:33.381556 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.381532 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-eda881\"/\"kube-root-ca.crt\"" Oct 30 01:30:33.391430 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.391406 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-eda881\"/\"openshift-service-ca.crt\"" Oct 30 01:30:33.391824 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.391805 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-eda881\"/\"default-dockercfg-fg5tb\"" Oct 30 01:30:33.391930 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.391863 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-eda881\"/\"distinct-w2-eda881-kserve-self-signed-certs\"" Oct 30 01:30:33.396188 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.396166 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd"] Oct 30 01:30:33.404555 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.404527 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a56da7a-9b92-40e2-b08b-43cf591b5221-tls-certs\") pod \"distinct-w2-eda881-kserve-7d9d4c5588-dqwqd\" (UID: \"9a56da7a-9b92-40e2-b08b-43cf591b5221\") " pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" Oct 30 01:30:33.404659 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.404611 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfmhf\" (UniqueName: \"kubernetes.io/projected/9a56da7a-9b92-40e2-b08b-43cf591b5221-kube-api-access-zfmhf\") pod \"distinct-w2-eda881-kserve-7d9d4c5588-dqwqd\" (UID: \"9a56da7a-9b92-40e2-b08b-43cf591b5221\") " pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" Oct 30 01:30:33.404696 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.404674 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a56da7a-9b92-40e2-b08b-43cf591b5221-tmp-dir\") pod \"distinct-w2-eda881-kserve-7d9d4c5588-dqwqd\" (UID: \"9a56da7a-9b92-40e2-b08b-43cf591b5221\") " pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" Oct 30 01:30:33.404806 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.404788 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a56da7a-9b92-40e2-b08b-43cf591b5221-dshm\") pod \"distinct-w2-eda881-kserve-7d9d4c5588-dqwqd\" (UID: \"9a56da7a-9b92-40e2-b08b-43cf591b5221\") " pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" Oct 30 01:30:33.404867 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.404851 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a56da7a-9b92-40e2-b08b-43cf591b5221-home\") pod \"distinct-w2-eda881-kserve-7d9d4c5588-dqwqd\" (UID: \"9a56da7a-9b92-40e2-b08b-43cf591b5221\") " pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" Oct 30 01:30:33.404921 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.404907 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a56da7a-9b92-40e2-b08b-43cf591b5221-model-cache\") pod \"distinct-w2-eda881-kserve-7d9d4c5588-dqwqd\" (UID: \"9a56da7a-9b92-40e2-b08b-43cf591b5221\") " pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" Oct 30 01:30:33.506444 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.506405 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a56da7a-9b92-40e2-b08b-43cf591b5221-tmp-dir\") pod \"distinct-w2-eda881-kserve-7d9d4c5588-dqwqd\" (UID: \"9a56da7a-9b92-40e2-b08b-43cf591b5221\") " pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" Oct 30 01:30:33.506612 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.506498 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a56da7a-9b92-40e2-b08b-43cf591b5221-dshm\") pod \"distinct-w2-eda881-kserve-7d9d4c5588-dqwqd\" (UID: \"9a56da7a-9b92-40e2-b08b-43cf591b5221\") " pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" Oct 30 01:30:33.506612 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.506562 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a56da7a-9b92-40e2-b08b-43cf591b5221-home\") pod \"distinct-w2-eda881-kserve-7d9d4c5588-dqwqd\" (UID: \"9a56da7a-9b92-40e2-b08b-43cf591b5221\") " pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" Oct 30 01:30:33.506794 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.506612 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a56da7a-9b92-40e2-b08b-43cf591b5221-model-cache\") pod \"distinct-w2-eda881-kserve-7d9d4c5588-dqwqd\" (UID: \"9a56da7a-9b92-40e2-b08b-43cf591b5221\") " pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" Oct 30 01:30:33.506794 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.506665 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a56da7a-9b92-40e2-b08b-43cf591b5221-tls-certs\") pod \"distinct-w2-eda881-kserve-7d9d4c5588-dqwqd\" (UID: \"9a56da7a-9b92-40e2-b08b-43cf591b5221\") " pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" Oct 30 01:30:33.506794 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.506719 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfmhf\" (UniqueName: \"kubernetes.io/projected/9a56da7a-9b92-40e2-b08b-43cf591b5221-kube-api-access-zfmhf\") pod \"distinct-w2-eda881-kserve-7d9d4c5588-dqwqd\" (UID: \"9a56da7a-9b92-40e2-b08b-43cf591b5221\") " pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" Oct 30 01:30:33.507005 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.506881 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a56da7a-9b92-40e2-b08b-43cf591b5221-tmp-dir\") pod \"distinct-w2-eda881-kserve-7d9d4c5588-dqwqd\" (UID: \"9a56da7a-9b92-40e2-b08b-43cf591b5221\") " pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" Oct 30 01:30:33.507186 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.507162 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a56da7a-9b92-40e2-b08b-43cf591b5221-model-cache\") pod \"distinct-w2-eda881-kserve-7d9d4c5588-dqwqd\" (UID: \"9a56da7a-9b92-40e2-b08b-43cf591b5221\") " pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" Oct 30 01:30:33.507305 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.507287 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a56da7a-9b92-40e2-b08b-43cf591b5221-home\") pod \"distinct-w2-eda881-kserve-7d9d4c5588-dqwqd\" (UID: \"9a56da7a-9b92-40e2-b08b-43cf591b5221\") " pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" Oct 30 01:30:33.509047 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.509021 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a56da7a-9b92-40e2-b08b-43cf591b5221-dshm\") pod \"distinct-w2-eda881-kserve-7d9d4c5588-dqwqd\" (UID: \"9a56da7a-9b92-40e2-b08b-43cf591b5221\") " pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" Oct 30 01:30:33.509415 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.509398 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a56da7a-9b92-40e2-b08b-43cf591b5221-tls-certs\") pod \"distinct-w2-eda881-kserve-7d9d4c5588-dqwqd\" (UID: \"9a56da7a-9b92-40e2-b08b-43cf591b5221\") " pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" Oct 30 01:30:33.526629 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.526608 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfmhf\" (UniqueName: \"kubernetes.io/projected/9a56da7a-9b92-40e2-b08b-43cf591b5221-kube-api-access-zfmhf\") pod \"distinct-w2-eda881-kserve-7d9d4c5588-dqwqd\" (UID: \"9a56da7a-9b92-40e2-b08b-43cf591b5221\") " pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" Oct 30 01:30:33.576677 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.576639 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:33.675954 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.675925 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" Oct 30 01:30:34.000767 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:33.999912 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd"] Oct 30 01:30:34.207082 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:34.207040 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:34.717655 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:34.717621 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" event={"ID":"9a56da7a-9b92-40e2-b08b-43cf591b5221","Type":"ContainerStarted","Data":"a183fcef174a0070df08e7ee8b84a662ee426e569957c4b47e109619135c666e"} Oct 30 01:30:34.718073 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:34.717668 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" event={"ID":"9a56da7a-9b92-40e2-b08b-43cf591b5221","Type":"ContainerStarted","Data":"d1f7ff677eb0162b34704c6b45b68d57cd5cd8bbb9fb2152cc7b9cd5c2f64824"} Oct 30 01:30:34.803690 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:34.803634 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" podStartSLOduration=1.8036209699999999 podStartE2EDuration="1.80362097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:34.800141567 +0000 UTC m=+760.376566969" watchObservedRunningTime="2026-10-30 01:30:34.80362097 +0000 UTC m=+760.380046371" Oct 30 01:30:35.676708 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:35.676674 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" Oct 30 01:30:36.105360 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:36.105283 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:37.888984 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:37.888941 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:39.787627 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:39.787575 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:41.885774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:41.885069 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:42.112663 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:42.112628 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:43.676462 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:43.676430 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" Oct 30 01:30:43.688880 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:43.688857 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" Oct 30 01:30:43.839480 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:43.839450 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" Oct 30 01:30:43.950254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:43.950161 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:44.686426 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:30:44.686392 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:30:45.818369 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:45.818331 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:47.713098 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:47.713056 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:49.832864 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:49.832828 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:51.487369 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:51.487328 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:52.521346 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:52.521316 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl"] Oct 30 01:30:52.525446 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:52.525425 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl" Oct 30 01:30:52.551465 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:52.551427 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl"] Oct 30 01:30:52.556783 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:52.556724 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-cf82f8\"/\"openshift-service-ca.crt\"" Oct 30 01:30:52.556783 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:52.556758 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-cf82f8\"/\"kube-root-ca.crt\"" Oct 30 01:30:52.556783 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:52.556758 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-cf82f8\"/\"default-dockercfg-d6pv6\"" Oct 30 01:30:52.557049 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:52.556827 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-cf82f8\"/\"e2e-conflict-b-d3477dc4-kserve-self-signed-certs\"" Oct 30 01:30:52.650625 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:52.650579 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8t6t\" (UniqueName: \"kubernetes.io/projected/be800e84-dc15-4618-8d51-95404c9b6c5d-kube-api-access-q8t6t\") pod \"e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl\" (UID: \"be800e84-dc15-4618-8d51-95404c9b6c5d\") " pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl" Oct 30 01:30:52.650859 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:52.650631 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be800e84-dc15-4618-8d51-95404c9b6c5d-model-cache\") pod \"e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl\" (UID: \"be800e84-dc15-4618-8d51-95404c9b6c5d\") " pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl" Oct 30 01:30:52.650859 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:52.650740 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be800e84-dc15-4618-8d51-95404c9b6c5d-dshm\") pod \"e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl\" (UID: \"be800e84-dc15-4618-8d51-95404c9b6c5d\") " pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl" Oct 30 01:30:52.650963 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:52.650887 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be800e84-dc15-4618-8d51-95404c9b6c5d-home\") pod \"e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl\" (UID: \"be800e84-dc15-4618-8d51-95404c9b6c5d\") " pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl" Oct 30 01:30:52.650963 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:52.650958 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be800e84-dc15-4618-8d51-95404c9b6c5d-tls-certs\") pod \"e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl\" (UID: \"be800e84-dc15-4618-8d51-95404c9b6c5d\") " pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl" Oct 30 01:30:52.651051 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:52.650996 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be800e84-dc15-4618-8d51-95404c9b6c5d-tmp-dir\") pod \"e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl\" (UID: \"be800e84-dc15-4618-8d51-95404c9b6c5d\") " pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl" Oct 30 01:30:52.729654 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:52.729556 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:52.752358 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:52.752323 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8t6t\" (UniqueName: \"kubernetes.io/projected/be800e84-dc15-4618-8d51-95404c9b6c5d-kube-api-access-q8t6t\") pod \"e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl\" (UID: \"be800e84-dc15-4618-8d51-95404c9b6c5d\") " pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl" Oct 30 01:30:52.752574 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:52.752374 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be800e84-dc15-4618-8d51-95404c9b6c5d-model-cache\") pod \"e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl\" (UID: \"be800e84-dc15-4618-8d51-95404c9b6c5d\") " pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl" Oct 30 01:30:52.752574 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:52.752419 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be800e84-dc15-4618-8d51-95404c9b6c5d-dshm\") pod \"e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl\" (UID: \"be800e84-dc15-4618-8d51-95404c9b6c5d\") " pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl" Oct 30 01:30:52.752574 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:52.752482 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be800e84-dc15-4618-8d51-95404c9b6c5d-home\") pod \"e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl\" (UID: \"be800e84-dc15-4618-8d51-95404c9b6c5d\") " pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl" Oct 30 01:30:52.752574 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:52.752525 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be800e84-dc15-4618-8d51-95404c9b6c5d-tls-certs\") pod \"e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl\" (UID: \"be800e84-dc15-4618-8d51-95404c9b6c5d\") " pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl" Oct 30 01:30:52.752574 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:52.752569 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be800e84-dc15-4618-8d51-95404c9b6c5d-tmp-dir\") pod \"e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl\" (UID: \"be800e84-dc15-4618-8d51-95404c9b6c5d\") " pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl" Oct 30 01:30:52.752987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:52.752963 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be800e84-dc15-4618-8d51-95404c9b6c5d-model-cache\") pod \"e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl\" (UID: \"be800e84-dc15-4618-8d51-95404c9b6c5d\") " pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl" Oct 30 01:30:52.753058 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:52.753039 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be800e84-dc15-4618-8d51-95404c9b6c5d-home\") pod \"e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl\" (UID: \"be800e84-dc15-4618-8d51-95404c9b6c5d\") " pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl" Oct 30 01:30:52.753118 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:52.753083 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be800e84-dc15-4618-8d51-95404c9b6c5d-tmp-dir\") pod \"e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl\" (UID: \"be800e84-dc15-4618-8d51-95404c9b6c5d\") " pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl" Oct 30 01:30:52.755379 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:52.755353 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be800e84-dc15-4618-8d51-95404c9b6c5d-tls-certs\") pod \"e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl\" (UID: \"be800e84-dc15-4618-8d51-95404c9b6c5d\") " pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl" Oct 30 01:30:52.755509 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:52.755419 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be800e84-dc15-4618-8d51-95404c9b6c5d-dshm\") pod \"e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl\" (UID: \"be800e84-dc15-4618-8d51-95404c9b6c5d\") " pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl" Oct 30 01:30:52.779408 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:52.779328 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8t6t\" (UniqueName: \"kubernetes.io/projected/be800e84-dc15-4618-8d51-95404c9b6c5d-kube-api-access-q8t6t\") pod \"e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl\" (UID: \"be800e84-dc15-4618-8d51-95404c9b6c5d\") " pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl" Oct 30 01:30:52.838712 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:52.838667 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl" Oct 30 01:30:53.145959 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:53.145872 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl"] Oct 30 01:30:53.148466 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:30:53.148436 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe800e84_dc15_4618_8d51_95404c9b6c5d.slice/crio-fe731cb5f8b23e0c930de6448fc0e44eaa17db4b47582d18f6d8bf3393f7282d WatchSource:0}: Error finding container fe731cb5f8b23e0c930de6448fc0e44eaa17db4b47582d18f6d8bf3393f7282d: Status 404 returned error can't find the container with id fe731cb5f8b23e0c930de6448fc0e44eaa17db4b47582d18f6d8bf3393f7282d Oct 30 01:30:53.412444 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:53.412407 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:53.889866 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:53.889739 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl" event={"ID":"be800e84-dc15-4618-8d51-95404c9b6c5d","Type":"ContainerStarted","Data":"e40a518b40dace7e5c2fb1da16b67bad7d97c220c18fb7ac488987ae20b97ef5"} Oct 30 01:30:53.889866 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:53.889823 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl" event={"ID":"be800e84-dc15-4618-8d51-95404c9b6c5d","Type":"ContainerStarted","Data":"fe731cb5f8b23e0c930de6448fc0e44eaa17db4b47582d18f6d8bf3393f7282d"} Oct 30 01:30:53.935368 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:53.935310 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl" podStartSLOduration=1.935295913 podStartE2EDuration="1.935295913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:53.932941214 +0000 UTC m=+779.509366626" watchObservedRunningTime="2026-10-30 01:30:53.935295913 +0000 UTC m=+779.511721315" Oct 30 01:30:54.839146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:54.839110 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl" Oct 30 01:30:55.146916 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:55.146828 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:55.370174 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:55.370133 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:55.607829 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:55.607788 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:56.871332 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:56.871292 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:58.588699 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:58.588659 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:58.860213 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:30:58.860126 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:00.512537 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:00.512503 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:02.457082 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:02.457046 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:02.838911 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:02.838806 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl" Oct 30 01:31:02.851808 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:02.851760 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl" Oct 30 01:31:02.958585 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:02.958545 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl" Oct 30 01:31:03.375630 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:03.375579 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:04.600769 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:04.600363 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:06.509214 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:06.509176 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:08.570483 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:08.570446 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:08.870924 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:08.870848 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n"] Oct 30 01:31:08.874552 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:08.874530 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" Oct 30 01:31:08.892855 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:08.892826 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-eda881\"/\"distinct-2-w2-eda881-kserve-self-signed-certs\"" Oct 30 01:31:08.910418 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:08.910381 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n"] Oct 30 01:31:08.951934 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:08.951894 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v529k\" (UniqueName: \"kubernetes.io/projected/e76339df-9659-4bea-80a4-5f18c4a73fd6-kube-api-access-v529k\") pod \"distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n\" (UID: \"e76339df-9659-4bea-80a4-5f18c4a73fd6\") " pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" Oct 30 01:31:08.952107 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:08.951944 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e76339df-9659-4bea-80a4-5f18c4a73fd6-dshm\") pod \"distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n\" (UID: \"e76339df-9659-4bea-80a4-5f18c4a73fd6\") " pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" Oct 30 01:31:08.952107 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:08.952076 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e76339df-9659-4bea-80a4-5f18c4a73fd6-tmp-dir\") pod \"distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n\" (UID: \"e76339df-9659-4bea-80a4-5f18c4a73fd6\") " pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" Oct 30 01:31:08.952204 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:08.952187 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e76339df-9659-4bea-80a4-5f18c4a73fd6-home\") pod \"distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n\" (UID: \"e76339df-9659-4bea-80a4-5f18c4a73fd6\") " pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" Oct 30 01:31:08.952241 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:08.952213 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e76339df-9659-4bea-80a4-5f18c4a73fd6-tls-certs\") pod \"distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n\" (UID: \"e76339df-9659-4bea-80a4-5f18c4a73fd6\") " pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" Oct 30 01:31:08.952318 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:08.952304 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e76339df-9659-4bea-80a4-5f18c4a73fd6-model-cache\") pod \"distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n\" (UID: \"e76339df-9659-4bea-80a4-5f18c4a73fd6\") " pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" Oct 30 01:31:09.053086 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:09.053046 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e76339df-9659-4bea-80a4-5f18c4a73fd6-home\") pod \"distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n\" (UID: \"e76339df-9659-4bea-80a4-5f18c4a73fd6\") " pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" Oct 30 01:31:09.053086 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:09.053091 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e76339df-9659-4bea-80a4-5f18c4a73fd6-tls-certs\") pod \"distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n\" (UID: \"e76339df-9659-4bea-80a4-5f18c4a73fd6\") " pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" Oct 30 01:31:09.053326 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:09.053130 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e76339df-9659-4bea-80a4-5f18c4a73fd6-model-cache\") pod \"distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n\" (UID: \"e76339df-9659-4bea-80a4-5f18c4a73fd6\") " pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" Oct 30 01:31:09.053326 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:09.053180 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v529k\" (UniqueName: \"kubernetes.io/projected/e76339df-9659-4bea-80a4-5f18c4a73fd6-kube-api-access-v529k\") pod \"distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n\" (UID: \"e76339df-9659-4bea-80a4-5f18c4a73fd6\") " pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" Oct 30 01:31:09.053326 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:09.053204 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e76339df-9659-4bea-80a4-5f18c4a73fd6-dshm\") pod \"distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n\" (UID: \"e76339df-9659-4bea-80a4-5f18c4a73fd6\") " pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" Oct 30 01:31:09.053326 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:09.053264 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e76339df-9659-4bea-80a4-5f18c4a73fd6-tmp-dir\") pod \"distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n\" (UID: \"e76339df-9659-4bea-80a4-5f18c4a73fd6\") " pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" Oct 30 01:31:09.053540 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:09.053469 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e76339df-9659-4bea-80a4-5f18c4a73fd6-home\") pod \"distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n\" (UID: \"e76339df-9659-4bea-80a4-5f18c4a73fd6\") " pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" Oct 30 01:31:09.053540 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:09.053509 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e76339df-9659-4bea-80a4-5f18c4a73fd6-model-cache\") pod \"distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n\" (UID: \"e76339df-9659-4bea-80a4-5f18c4a73fd6\") " pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" Oct 30 01:31:09.053671 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:09.053643 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e76339df-9659-4bea-80a4-5f18c4a73fd6-tmp-dir\") pod \"distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n\" (UID: \"e76339df-9659-4bea-80a4-5f18c4a73fd6\") " pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" Oct 30 01:31:09.055493 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:09.055471 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e76339df-9659-4bea-80a4-5f18c4a73fd6-dshm\") pod \"distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n\" (UID: \"e76339df-9659-4bea-80a4-5f18c4a73fd6\") " pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" Oct 30 01:31:09.055638 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:09.055620 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e76339df-9659-4bea-80a4-5f18c4a73fd6-tls-certs\") pod \"distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n\" (UID: \"e76339df-9659-4bea-80a4-5f18c4a73fd6\") " pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" Oct 30 01:31:09.088170 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:09.088137 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v529k\" (UniqueName: \"kubernetes.io/projected/e76339df-9659-4bea-80a4-5f18c4a73fd6-kube-api-access-v529k\") pod \"distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n\" (UID: \"e76339df-9659-4bea-80a4-5f18c4a73fd6\") " pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" Oct 30 01:31:09.119091 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:09.119047 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:09.187059 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:09.187023 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" Oct 30 01:31:09.499600 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:09.499565 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n"] Oct 30 01:31:09.994811 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:09.994773 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" event={"ID":"e76339df-9659-4bea-80a4-5f18c4a73fd6","Type":"ContainerStarted","Data":"b6aff951c15f7b34689a6c804f655051fb9bd1254c1a35c8bba7cfd5b379701a"} Oct 30 01:31:09.995183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:09.994826 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" event={"ID":"e76339df-9659-4bea-80a4-5f18c4a73fd6","Type":"ContainerStarted","Data":"8f70f23b95f9231eeea315d81b9147d3ec3fb39683c87ce0b013502b6cd9ecc4"} Oct 30 01:31:10.039586 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:10.039528 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" podStartSLOduration=2.039514878 podStartE2EDuration="2.039514878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:31:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:31:10.038773263 +0000 UTC m=+795.615198664" watchObservedRunningTime="2026-10-30 01:31:10.039514878 +0000 UTC m=+795.615940279" Oct 30 01:31:10.187374 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:10.187339 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" Oct 30 01:31:10.744311 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:10.744263 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:12.792393 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:12.792340 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:14.688307 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:31:14.688267 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:31:15.047703 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:15.047669 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:17.203603 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:17.203561 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:17.516970 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:17.516861 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:19.187588 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:19.187545 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" Oct 30 01:31:19.200547 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:19.200517 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" Oct 30 01:31:19.484207 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:19.484123 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:20.077918 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:20.077884 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" Oct 30 01:31:21.315314 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:21.315275 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:23.153243 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:23.153207 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:23.376025 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:23.375988 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:25.120528 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.120490 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:25.466428 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.466388 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg"] Oct 30 01:31:25.470547 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.470520 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" Oct 30 01:31:25.489806 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.489774 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v"] Oct 30 01:31:25.493611 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.493590 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" Oct 30 01:31:25.498778 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.498731 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-cf82f8-openshift-default-dockercfg-xt5zt\"" Oct 30 01:31:25.504551 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.504524 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg"] Oct 30 01:31:25.521896 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.521864 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v"] Oct 30 01:31:25.643575 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.643529 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29807745-a828-46e6-ab0e-994057d74363-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg\" (UID: \"29807745-a828-46e6-ab0e-994057d74363\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" Oct 30 01:31:25.643809 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.643596 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b445121f-4bf6-4f3a-886c-a02f45574a57-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v\" (UID: \"b445121f-4bf6-4f3a-886c-a02f45574a57\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" Oct 30 01:31:25.643809 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.643650 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29807745-a828-46e6-ab0e-994057d74363-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg\" (UID: \"29807745-a828-46e6-ab0e-994057d74363\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" Oct 30 01:31:25.643809 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.643676 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b445121f-4bf6-4f3a-886c-a02f45574a57-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v\" (UID: \"b445121f-4bf6-4f3a-886c-a02f45574a57\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" Oct 30 01:31:25.643972 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.643827 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b445121f-4bf6-4f3a-886c-a02f45574a57-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v\" (UID: \"b445121f-4bf6-4f3a-886c-a02f45574a57\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" Oct 30 01:31:25.643972 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.643859 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29807745-a828-46e6-ab0e-994057d74363-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg\" (UID: \"29807745-a828-46e6-ab0e-994057d74363\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" Oct 30 01:31:25.643972 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.643897 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b445121f-4bf6-4f3a-886c-a02f45574a57-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v\" (UID: \"b445121f-4bf6-4f3a-886c-a02f45574a57\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" Oct 30 01:31:25.643972 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.643916 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29807745-a828-46e6-ab0e-994057d74363-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg\" (UID: \"29807745-a828-46e6-ab0e-994057d74363\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" Oct 30 01:31:25.644101 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.644007 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b445121f-4bf6-4f3a-886c-a02f45574a57-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v\" (UID: \"b445121f-4bf6-4f3a-886c-a02f45574a57\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" Oct 30 01:31:25.644101 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.644037 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgn94\" (UniqueName: \"kubernetes.io/projected/29807745-a828-46e6-ab0e-994057d74363-kube-api-access-lgn94\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg\" (UID: \"29807745-a828-46e6-ab0e-994057d74363\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" Oct 30 01:31:25.644163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.644107 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29807745-a828-46e6-ab0e-994057d74363-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg\" (UID: \"29807745-a828-46e6-ab0e-994057d74363\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" Oct 30 01:31:25.644196 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.644167 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b445121f-4bf6-4f3a-886c-a02f45574a57-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v\" (UID: \"b445121f-4bf6-4f3a-886c-a02f45574a57\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" Oct 30 01:31:25.644231 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.644200 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29807745-a828-46e6-ab0e-994057d74363-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg\" (UID: \"29807745-a828-46e6-ab0e-994057d74363\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" Oct 30 01:31:25.644231 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.644225 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cb5s\" (UniqueName: \"kubernetes.io/projected/b445121f-4bf6-4f3a-886c-a02f45574a57-kube-api-access-8cb5s\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v\" (UID: \"b445121f-4bf6-4f3a-886c-a02f45574a57\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" Oct 30 01:31:25.644295 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.644279 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29807745-a828-46e6-ab0e-994057d74363-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg\" (UID: \"29807745-a828-46e6-ab0e-994057d74363\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" Oct 30 01:31:25.644336 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.644307 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b445121f-4bf6-4f3a-886c-a02f45574a57-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v\" (UID: \"b445121f-4bf6-4f3a-886c-a02f45574a57\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" Oct 30 01:31:25.644369 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.644353 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29807745-a828-46e6-ab0e-994057d74363-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg\" (UID: \"29807745-a828-46e6-ab0e-994057d74363\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" Oct 30 01:31:25.644411 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.644375 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b445121f-4bf6-4f3a-886c-a02f45574a57-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v\" (UID: \"b445121f-4bf6-4f3a-886c-a02f45574a57\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" Oct 30 01:31:25.746060 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.745966 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29807745-a828-46e6-ab0e-994057d74363-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg\" (UID: \"29807745-a828-46e6-ab0e-994057d74363\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" Oct 30 01:31:25.746060 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.746036 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b445121f-4bf6-4f3a-886c-a02f45574a57-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v\" (UID: \"b445121f-4bf6-4f3a-886c-a02f45574a57\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" Oct 30 01:31:25.746283 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.746074 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29807745-a828-46e6-ab0e-994057d74363-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg\" (UID: \"29807745-a828-46e6-ab0e-994057d74363\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" Oct 30 01:31:25.746283 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.746102 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cb5s\" (UniqueName: \"kubernetes.io/projected/b445121f-4bf6-4f3a-886c-a02f45574a57-kube-api-access-8cb5s\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v\" (UID: \"b445121f-4bf6-4f3a-886c-a02f45574a57\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" Oct 30 01:31:25.746283 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.746122 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29807745-a828-46e6-ab0e-994057d74363-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg\" (UID: \"29807745-a828-46e6-ab0e-994057d74363\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" Oct 30 01:31:25.746283 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.746146 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b445121f-4bf6-4f3a-886c-a02f45574a57-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v\" (UID: \"b445121f-4bf6-4f3a-886c-a02f45574a57\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" Oct 30 01:31:25.746283 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.746173 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29807745-a828-46e6-ab0e-994057d74363-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg\" (UID: \"29807745-a828-46e6-ab0e-994057d74363\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" Oct 30 01:31:25.746283 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.746217 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b445121f-4bf6-4f3a-886c-a02f45574a57-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v\" (UID: \"b445121f-4bf6-4f3a-886c-a02f45574a57\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" Oct 30 01:31:25.746662 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.746632 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29807745-a828-46e6-ab0e-994057d74363-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg\" (UID: \"29807745-a828-46e6-ab0e-994057d74363\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" Oct 30 01:31:25.746886 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.746662 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29807745-a828-46e6-ab0e-994057d74363-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg\" (UID: \"29807745-a828-46e6-ab0e-994057d74363\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" Oct 30 01:31:25.746886 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.746550 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29807745-a828-46e6-ab0e-994057d74363-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg\" (UID: \"29807745-a828-46e6-ab0e-994057d74363\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" Oct 30 01:31:25.746886 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.746719 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b445121f-4bf6-4f3a-886c-a02f45574a57-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v\" (UID: \"b445121f-4bf6-4f3a-886c-a02f45574a57\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" Oct 30 01:31:25.746886 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.746823 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29807745-a828-46e6-ab0e-994057d74363-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg\" (UID: \"29807745-a828-46e6-ab0e-994057d74363\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" Oct 30 01:31:25.746886 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.746850 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b445121f-4bf6-4f3a-886c-a02f45574a57-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v\" (UID: \"b445121f-4bf6-4f3a-886c-a02f45574a57\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" Oct 30 01:31:25.747184 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.746906 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b445121f-4bf6-4f3a-886c-a02f45574a57-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v\" (UID: \"b445121f-4bf6-4f3a-886c-a02f45574a57\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" Oct 30 01:31:25.747184 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.746938 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29807745-a828-46e6-ab0e-994057d74363-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg\" (UID: \"29807745-a828-46e6-ab0e-994057d74363\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" Oct 30 01:31:25.747184 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.746997 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b445121f-4bf6-4f3a-886c-a02f45574a57-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v\" (UID: \"b445121f-4bf6-4f3a-886c-a02f45574a57\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" Oct 30 01:31:25.747184 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.747028 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29807745-a828-46e6-ab0e-994057d74363-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg\" (UID: \"29807745-a828-46e6-ab0e-994057d74363\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" Oct 30 01:31:25.747184 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.747034 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b445121f-4bf6-4f3a-886c-a02f45574a57-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v\" (UID: \"b445121f-4bf6-4f3a-886c-a02f45574a57\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" Oct 30 01:31:25.747184 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.747089 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b445121f-4bf6-4f3a-886c-a02f45574a57-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v\" (UID: \"b445121f-4bf6-4f3a-886c-a02f45574a57\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" Oct 30 01:31:25.747184 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.747089 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b445121f-4bf6-4f3a-886c-a02f45574a57-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v\" (UID: \"b445121f-4bf6-4f3a-886c-a02f45574a57\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" Oct 30 01:31:25.747184 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.747118 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgn94\" (UniqueName: \"kubernetes.io/projected/29807745-a828-46e6-ab0e-994057d74363-kube-api-access-lgn94\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg\" (UID: \"29807745-a828-46e6-ab0e-994057d74363\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" Oct 30 01:31:25.747600 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.747323 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29807745-a828-46e6-ab0e-994057d74363-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg\" (UID: \"29807745-a828-46e6-ab0e-994057d74363\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" Oct 30 01:31:25.747600 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.747323 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29807745-a828-46e6-ab0e-994057d74363-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg\" (UID: \"29807745-a828-46e6-ab0e-994057d74363\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" Oct 30 01:31:25.747600 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.747372 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b445121f-4bf6-4f3a-886c-a02f45574a57-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v\" (UID: \"b445121f-4bf6-4f3a-886c-a02f45574a57\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" Oct 30 01:31:25.747600 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.747580 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b445121f-4bf6-4f3a-886c-a02f45574a57-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v\" (UID: \"b445121f-4bf6-4f3a-886c-a02f45574a57\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" Oct 30 01:31:25.747837 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.747657 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b445121f-4bf6-4f3a-886c-a02f45574a57-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v\" (UID: \"b445121f-4bf6-4f3a-886c-a02f45574a57\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" Oct 30 01:31:25.748194 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.748168 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29807745-a828-46e6-ab0e-994057d74363-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg\" (UID: \"29807745-a828-46e6-ab0e-994057d74363\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" Oct 30 01:31:25.749213 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.749186 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29807745-a828-46e6-ab0e-994057d74363-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg\" (UID: \"29807745-a828-46e6-ab0e-994057d74363\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" Oct 30 01:31:25.749213 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.749205 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29807745-a828-46e6-ab0e-994057d74363-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg\" (UID: \"29807745-a828-46e6-ab0e-994057d74363\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" Oct 30 01:31:25.750231 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.750212 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b445121f-4bf6-4f3a-886c-a02f45574a57-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v\" (UID: \"b445121f-4bf6-4f3a-886c-a02f45574a57\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" Oct 30 01:31:25.750579 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.750561 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b445121f-4bf6-4f3a-886c-a02f45574a57-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v\" (UID: \"b445121f-4bf6-4f3a-886c-a02f45574a57\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" Oct 30 01:31:25.767062 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.767031 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29807745-a828-46e6-ab0e-994057d74363-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg\" (UID: \"29807745-a828-46e6-ab0e-994057d74363\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" Oct 30 01:31:25.767236 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.767179 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cb5s\" (UniqueName: \"kubernetes.io/projected/b445121f-4bf6-4f3a-886c-a02f45574a57-kube-api-access-8cb5s\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v\" (UID: \"b445121f-4bf6-4f3a-886c-a02f45574a57\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" Oct 30 01:31:25.767236 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.767189 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgn94\" (UniqueName: \"kubernetes.io/projected/29807745-a828-46e6-ab0e-994057d74363-kube-api-access-lgn94\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg\" (UID: \"29807745-a828-46e6-ab0e-994057d74363\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" Oct 30 01:31:25.767370 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.767353 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b445121f-4bf6-4f3a-886c-a02f45574a57-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v\" (UID: \"b445121f-4bf6-4f3a-886c-a02f45574a57\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" Oct 30 01:31:25.784982 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.784942 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" Oct 30 01:31:25.810314 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:25.810280 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" Oct 30 01:31:26.109158 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:26.109063 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg"] Oct 30 01:31:26.115239 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:31:26.115205 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29807745_a828_46e6_ab0e_994057d74363.slice/crio-4a3d97f3bed1d8b173040a8268525a61cd41be971a174d7e06f2155d4a17e589 WatchSource:0}: Error finding container 4a3d97f3bed1d8b173040a8268525a61cd41be971a174d7e06f2155d4a17e589: Status 404 returned error can't find the container with id 4a3d97f3bed1d8b173040a8268525a61cd41be971a174d7e06f2155d4a17e589 Oct 30 01:31:26.117779 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:26.117725 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:31:26.117916 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:26.117828 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:31:26.117916 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:26.117874 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:31:26.132959 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:26.132925 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v"] Oct 30 01:31:26.137940 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:31:26.137785 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb445121f_4bf6_4f3a_886c_a02f45574a57.slice/crio-dc31336e7c0d77a2f9cbfd754f358bd0a97fee4a02b6240e47f707d8d7c3f501 WatchSource:0}: Error finding container dc31336e7c0d77a2f9cbfd754f358bd0a97fee4a02b6240e47f707d8d7c3f501: Status 404 returned error can't find the container with id dc31336e7c0d77a2f9cbfd754f358bd0a97fee4a02b6240e47f707d8d7c3f501 Oct 30 01:31:26.140716 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:26.140665 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:31:26.140850 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:26.140824 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:31:26.140935 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:26.140917 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:31:26.944831 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:26.938048 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:27.159159 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:27.159117 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" event={"ID":"29807745-a828-46e6-ab0e-994057d74363","Type":"ContainerStarted","Data":"50f5f648e8fcbb3767a2ff2447a53fce4196ed835371402653a7bfff9939e1b2"} Oct 30 01:31:27.159654 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:27.159210 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" event={"ID":"29807745-a828-46e6-ab0e-994057d74363","Type":"ContainerStarted","Data":"4a3d97f3bed1d8b173040a8268525a61cd41be971a174d7e06f2155d4a17e589"} Oct 30 01:31:27.161623 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:27.161590 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" event={"ID":"b445121f-4bf6-4f3a-886c-a02f45574a57","Type":"ContainerStarted","Data":"e120bd8bc486c96df059b013155362b62a313e6abbdec528201637dd3a01de92"} Oct 30 01:31:27.161766 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:27.161665 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" event={"ID":"b445121f-4bf6-4f3a-886c-a02f45574a57","Type":"ContainerStarted","Data":"dc31336e7c0d77a2f9cbfd754f358bd0a97fee4a02b6240e47f707d8d7c3f501"} Oct 30 01:31:27.244843 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:27.244653 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" podStartSLOduration=2.244629632 podStartE2EDuration="2.244629632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:31:27.228118057 +0000 UTC m=+812.804543462" watchObservedRunningTime="2026-10-30 01:31:27.244629632 +0000 UTC m=+812.821055036" Oct 30 01:31:27.278320 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:27.278272 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" podStartSLOduration=2.278259571 podStartE2EDuration="2.278259571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:31:27.274195034 +0000 UTC m=+812.850620428" watchObservedRunningTime="2026-10-30 01:31:27.278259571 +0000 UTC m=+812.854684972" Oct 30 01:31:27.786015 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:27.785976 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" Oct 30 01:31:27.787502 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:27.787474 6572 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 01:31:27.787640 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:27.787522 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" podUID="29807745-a828-46e6-ab0e-994057d74363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 01:31:27.815774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:27.815161 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" Oct 30 01:31:27.840812 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:27.840777 6572 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 01:31:27.841095 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:27.841063 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" podUID="b445121f-4bf6-4f3a-886c-a02f45574a57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 01:31:28.786177 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:28.786145 6572 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 01:31:28.787242 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:28.787211 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" podUID="29807745-a828-46e6-ab0e-994057d74363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 01:31:28.811432 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:28.811384 6572 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 01:31:28.811685 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:28.811660 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" podUID="b445121f-4bf6-4f3a-886c-a02f45574a57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 01:31:28.901452 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:28.901420 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:29.785835 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:29.785803 6572 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 01:31:29.786014 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:29.785872 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" podUID="29807745-a828-46e6-ab0e-994057d74363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 01:31:29.840229 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:29.840186 6572 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 01:31:29.840842 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:29.840808 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" podUID="b445121f-4bf6-4f3a-886c-a02f45574a57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 01:31:30.057139 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:30.057044 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl"] Oct 30 01:31:30.174644 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:30.174574 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:30.733773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:30.733455 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:30.786000 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:30.785967 6572 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 01:31:30.786260 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:30.786236 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" podUID="29807745-a828-46e6-ab0e-994057d74363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 01:31:30.813246 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:30.813199 6572 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 01:31:30.814777 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:30.814717 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" podUID="b445121f-4bf6-4f3a-886c-a02f45574a57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 01:31:31.786061 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:31.786022 6572 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 01:31:31.786485 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:31.786076 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" podUID="29807745-a828-46e6-ab0e-994057d74363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 01:31:31.814122 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:31.814083 6572 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 01:31:31.814411 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:31.814376 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" podUID="b445121f-4bf6-4f3a-886c-a02f45574a57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 01:31:32.563913 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:32.563870 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:32.785591 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:32.785552 6572 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 01:31:32.785790 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:32.785608 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" podUID="29807745-a828-46e6-ab0e-994057d74363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 01:31:32.811415 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:32.811372 6572 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 01:31:32.811839 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:32.811444 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" podUID="b445121f-4bf6-4f3a-886c-a02f45574a57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 01:31:33.790869 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:33.790782 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" Oct 30 01:31:33.791183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:33.791163 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" Oct 30 01:31:33.791942 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:33.791920 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" Oct 30 01:31:33.815090 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:33.815062 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" Oct 30 01:31:33.815446 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:33.815326 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" Oct 30 01:31:33.816115 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:33.816098 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" Oct 30 01:31:34.464608 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:34.464568 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:36.407909 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:36.407870 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:36.664013 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:36.663975 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:38.622470 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:38.622436 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:40.608903 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:40.608865 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:40.667705 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:40.667669 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16725ed7-t895t"] Oct 30 01:31:40.671706 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:40.671688 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16725ed7-t895t" Oct 30 01:31:40.691241 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:40.691219 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:31:40.697394 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:40.697338 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:31:40.697394 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:40.697343 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:31:40.697585 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:40.697414 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:31:40.701086 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:40.701061 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16725ed7-t895t"] Oct 30 01:31:40.777709 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:40.777674 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7d28ee6-0e8a-45dc-91ca-73c70f96576c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16725ed7-t895t\" (UID: \"f7d28ee6-0e8a-45dc-91ca-73c70f96576c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16725ed7-t895t" Oct 30 01:31:40.777930 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:40.777887 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srjnn\" (UniqueName: \"kubernetes.io/projected/f7d28ee6-0e8a-45dc-91ca-73c70f96576c-kube-api-access-srjnn\") pod \"maas-api-revoke-sub-16725ed7-t895t\" (UID: \"f7d28ee6-0e8a-45dc-91ca-73c70f96576c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16725ed7-t895t" Oct 30 01:31:40.777930 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:40.777922 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7d28ee6-0e8a-45dc-91ca-73c70f96576c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16725ed7-t895t\" (UID: \"f7d28ee6-0e8a-45dc-91ca-73c70f96576c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16725ed7-t895t" Oct 30 01:31:40.879686 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:40.879608 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7d28ee6-0e8a-45dc-91ca-73c70f96576c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16725ed7-t895t\" (UID: \"f7d28ee6-0e8a-45dc-91ca-73c70f96576c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16725ed7-t895t" Oct 30 01:31:40.879862 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:40.879732 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srjnn\" (UniqueName: \"kubernetes.io/projected/f7d28ee6-0e8a-45dc-91ca-73c70f96576c-kube-api-access-srjnn\") pod \"maas-api-revoke-sub-16725ed7-t895t\" (UID: \"f7d28ee6-0e8a-45dc-91ca-73c70f96576c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16725ed7-t895t" Oct 30 01:31:40.879862 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:40.879774 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7d28ee6-0e8a-45dc-91ca-73c70f96576c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16725ed7-t895t\" (UID: \"f7d28ee6-0e8a-45dc-91ca-73c70f96576c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16725ed7-t895t" Oct 30 01:31:40.880465 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:40.880439 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7d28ee6-0e8a-45dc-91ca-73c70f96576c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16725ed7-t895t\" (UID: \"f7d28ee6-0e8a-45dc-91ca-73c70f96576c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16725ed7-t895t" Oct 30 01:31:40.882086 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:40.882063 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7d28ee6-0e8a-45dc-91ca-73c70f96576c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16725ed7-t895t\" (UID: \"f7d28ee6-0e8a-45dc-91ca-73c70f96576c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16725ed7-t895t" Oct 30 01:31:40.900638 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:40.900615 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srjnn\" (UniqueName: \"kubernetes.io/projected/f7d28ee6-0e8a-45dc-91ca-73c70f96576c-kube-api-access-srjnn\") pod \"maas-api-revoke-sub-16725ed7-t895t\" (UID: \"f7d28ee6-0e8a-45dc-91ca-73c70f96576c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16725ed7-t895t" Oct 30 01:31:40.985608 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:40.985570 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16725ed7-t895t" Oct 30 01:31:41.296897 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:41.296863 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16725ed7-t895t"] Oct 30 01:31:41.298234 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:31:41.298196 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d28ee6_0e8a_45dc_91ca_73c70f96576c.slice/crio-f8a8a97ecbbc4fc1b586b29188c7b445617fa9841d82c19655c19ec5d8f68f9a WatchSource:0}: Error finding container f8a8a97ecbbc4fc1b586b29188c7b445617fa9841d82c19655c19ec5d8f68f9a: Status 404 returned error can't find the container with id f8a8a97ecbbc4fc1b586b29188c7b445617fa9841d82c19655c19ec5d8f68f9a Oct 30 01:31:41.354894 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:41.354865 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:42.291552 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:42.291519 6572 generic.go:356] "Generic (PLEG): container finished" podID="f7d28ee6-0e8a-45dc-91ca-73c70f96576c" containerID="384025cc5788363ecf1be475709e853e7837e8321a76608bad2e2ab344c55a58" exitCode=0 Oct 30 01:31:42.291961 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:42.291608 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16725ed7-t895t" event={"ID":"f7d28ee6-0e8a-45dc-91ca-73c70f96576c","Type":"ContainerDied","Data":"384025cc5788363ecf1be475709e853e7837e8321a76608bad2e2ab344c55a58"} Oct 30 01:31:42.291961 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:42.291662 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16725ed7-t895t" event={"ID":"f7d28ee6-0e8a-45dc-91ca-73c70f96576c","Type":"ContainerStarted","Data":"f8a8a97ecbbc4fc1b586b29188c7b445617fa9841d82c19655c19ec5d8f68f9a"} Oct 30 01:31:42.649696 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:42.649661 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:43.455164 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:43.455131 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16725ed7-t895t" Oct 30 01:31:43.611146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:43.611048 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7d28ee6-0e8a-45dc-91ca-73c70f96576c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7d28ee6-0e8a-45dc-91ca-73c70f96576c-maas-api-xks-ca\") pod \"f7d28ee6-0e8a-45dc-91ca-73c70f96576c\" (UID: \"f7d28ee6-0e8a-45dc-91ca-73c70f96576c\") " Oct 30 01:31:43.611146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:43.611103 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7d28ee6-0e8a-45dc-91ca-73c70f96576c-kube-api-access-srjnn\" (UniqueName: \"kubernetes.io/projected/f7d28ee6-0e8a-45dc-91ca-73c70f96576c-kube-api-access-srjnn\") pod \"f7d28ee6-0e8a-45dc-91ca-73c70f96576c\" (UID: \"f7d28ee6-0e8a-45dc-91ca-73c70f96576c\") " Oct 30 01:31:43.611405 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:43.611244 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7d28ee6-0e8a-45dc-91ca-73c70f96576c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7d28ee6-0e8a-45dc-91ca-73c70f96576c-maas-api-openshift-ca\") pod \"f7d28ee6-0e8a-45dc-91ca-73c70f96576c\" (UID: \"f7d28ee6-0e8a-45dc-91ca-73c70f96576c\") " Oct 30 01:31:43.611637 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:43.611614 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7d28ee6-0e8a-45dc-91ca-73c70f96576c-maas-api-openshift-ca" pod "f7d28ee6-0e8a-45dc-91ca-73c70f96576c" (UID: "f7d28ee6-0e8a-45dc-91ca-73c70f96576c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:31:43.611760 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:43.611726 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7d28ee6-0e8a-45dc-91ca-73c70f96576c-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:31:43.613262 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:43.613237 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7d28ee6-0e8a-45dc-91ca-73c70f96576c-maas-api-xks-ca" pod "f7d28ee6-0e8a-45dc-91ca-73c70f96576c" (UID: "f7d28ee6-0e8a-45dc-91ca-73c70f96576c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:31:43.613346 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:43.613284 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7d28ee6-0e8a-45dc-91ca-73c70f96576c-kube-api-access-srjnn" pod "f7d28ee6-0e8a-45dc-91ca-73c70f96576c" (UID: "f7d28ee6-0e8a-45dc-91ca-73c70f96576c"). InnerVolumeSpecName "kube-api-access-srjnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:31:43.713380 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:43.713340 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7d28ee6-0e8a-45dc-91ca-73c70f96576c-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:31:43.713380 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:43.713382 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srjnn\" (UniqueName: \"kubernetes.io/projected/f7d28ee6-0e8a-45dc-91ca-73c70f96576c-kube-api-access-srjnn\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:31:44.306791 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:44.306759 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16725ed7-t895t" Oct 30 01:31:44.306791 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:44.306777 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16725ed7-t895t" event={"ID":"f7d28ee6-0e8a-45dc-91ca-73c70f96576c","Type":"ContainerDied","Data":"f8a8a97ecbbc4fc1b586b29188c7b445617fa9841d82c19655c19ec5d8f68f9a"} Oct 30 01:31:44.307015 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:44.306856 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8a8a97ecbbc4fc1b586b29188c7b445617fa9841d82c19655c19ec5d8f68f9a" Oct 30 01:31:44.585350 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:44.585252 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:44.689543 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:31:44.689506 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:31:45.079065 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:45.079021 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl" podUID="be800e84-dc15-4618-8d51-95404c9b6c5d" containerName="main" containerID="cri-o://e40a518b40dace7e5c2fb1da16b67bad7d97c220c18fb7ac488987ae20b97ef5" gracePeriod=45 Oct 30 01:31:45.318310 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:45.318280 6572 generic.go:356] "Generic (PLEG): container finished" podID="be800e84-dc15-4618-8d51-95404c9b6c5d" containerID="e40a518b40dace7e5c2fb1da16b67bad7d97c220c18fb7ac488987ae20b97ef5" exitCode=0 Oct 30 01:31:45.318465 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:45.318354 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl" event={"ID":"be800e84-dc15-4618-8d51-95404c9b6c5d","Type":"ContainerDied","Data":"e40a518b40dace7e5c2fb1da16b67bad7d97c220c18fb7ac488987ae20b97ef5"} Oct 30 01:31:45.348297 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:45.348222 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl" Oct 30 01:31:45.433068 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:45.433034 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be800e84-dc15-4618-8d51-95404c9b6c5d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/be800e84-dc15-4618-8d51-95404c9b6c5d-dshm\") pod \"be800e84-dc15-4618-8d51-95404c9b6c5d\" (UID: \"be800e84-dc15-4618-8d51-95404c9b6c5d\") " Oct 30 01:31:45.433253 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:45.433078 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be800e84-dc15-4618-8d51-95404c9b6c5d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be800e84-dc15-4618-8d51-95404c9b6c5d-tmp-dir\") pod \"be800e84-dc15-4618-8d51-95404c9b6c5d\" (UID: \"be800e84-dc15-4618-8d51-95404c9b6c5d\") " Oct 30 01:31:45.433253 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:45.433116 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be800e84-dc15-4618-8d51-95404c9b6c5d-kube-api-access-q8t6t\" (UniqueName: \"kubernetes.io/projected/be800e84-dc15-4618-8d51-95404c9b6c5d-kube-api-access-q8t6t\") pod \"be800e84-dc15-4618-8d51-95404c9b6c5d\" (UID: \"be800e84-dc15-4618-8d51-95404c9b6c5d\") " Oct 30 01:31:45.433253 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:45.433224 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be800e84-dc15-4618-8d51-95404c9b6c5d-home\" (UniqueName: \"kubernetes.io/empty-dir/be800e84-dc15-4618-8d51-95404c9b6c5d-home\") pod \"be800e84-dc15-4618-8d51-95404c9b6c5d\" (UID: \"be800e84-dc15-4618-8d51-95404c9b6c5d\") " Oct 30 01:31:45.433442 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:45.433333 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be800e84-dc15-4618-8d51-95404c9b6c5d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be800e84-dc15-4618-8d51-95404c9b6c5d-model-cache\") pod \"be800e84-dc15-4618-8d51-95404c9b6c5d\" (UID: \"be800e84-dc15-4618-8d51-95404c9b6c5d\") " Oct 30 01:31:45.433442 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:45.433338 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be800e84-dc15-4618-8d51-95404c9b6c5d-tmp-dir" pod "be800e84-dc15-4618-8d51-95404c9b6c5d" (UID: "be800e84-dc15-4618-8d51-95404c9b6c5d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:45.433546 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:45.433467 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be800e84-dc15-4618-8d51-95404c9b6c5d-tls-certs\" (UniqueName: \"kubernetes.io/secret/be800e84-dc15-4618-8d51-95404c9b6c5d-tls-certs\") pod \"be800e84-dc15-4618-8d51-95404c9b6c5d\" (UID: \"be800e84-dc15-4618-8d51-95404c9b6c5d\") " Oct 30 01:31:45.433546 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:45.433501 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be800e84-dc15-4618-8d51-95404c9b6c5d-home" pod "be800e84-dc15-4618-8d51-95404c9b6c5d" (UID: "be800e84-dc15-4618-8d51-95404c9b6c5d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:45.433647 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:45.433570 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be800e84-dc15-4618-8d51-95404c9b6c5d-model-cache" pod "be800e84-dc15-4618-8d51-95404c9b6c5d" (UID: "be800e84-dc15-4618-8d51-95404c9b6c5d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:45.433900 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:45.433881 6572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be800e84-dc15-4618-8d51-95404c9b6c5d-home\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:31:45.434004 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:45.433903 6572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be800e84-dc15-4618-8d51-95404c9b6c5d-model-cache\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:31:45.434004 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:45.433916 6572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be800e84-dc15-4618-8d51-95404c9b6c5d-tmp-dir\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:31:45.435480 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:45.435446 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be800e84-dc15-4618-8d51-95404c9b6c5d-dshm" pod "be800e84-dc15-4618-8d51-95404c9b6c5d" (UID: "be800e84-dc15-4618-8d51-95404c9b6c5d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:45.435480 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:45.435453 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be800e84-dc15-4618-8d51-95404c9b6c5d-kube-api-access-q8t6t" pod "be800e84-dc15-4618-8d51-95404c9b6c5d" (UID: "be800e84-dc15-4618-8d51-95404c9b6c5d"). InnerVolumeSpecName "kube-api-access-q8t6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:31:45.435721 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:45.435704 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be800e84-dc15-4618-8d51-95404c9b6c5d-tls-certs" pod "be800e84-dc15-4618-8d51-95404c9b6c5d" (UID: "be800e84-dc15-4618-8d51-95404c9b6c5d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:31:45.535293 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:45.535254 6572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be800e84-dc15-4618-8d51-95404c9b6c5d-tls-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:31:45.535293 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:45.535287 6572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be800e84-dc15-4618-8d51-95404c9b6c5d-dshm\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:31:45.535293 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:45.535297 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8t6t\" (UniqueName: \"kubernetes.io/projected/be800e84-dc15-4618-8d51-95404c9b6c5d-kube-api-access-q8t6t\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:31:46.326427 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:46.326394 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl" event={"ID":"be800e84-dc15-4618-8d51-95404c9b6c5d","Type":"ContainerDied","Data":"fe731cb5f8b23e0c930de6448fc0e44eaa17db4b47582d18f6d8bf3393f7282d"} Oct 30 01:31:46.327108 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:46.326448 6572 scope.go:122] "RemoveContainer" containerID="e40a518b40dace7e5c2fb1da16b67bad7d97c220c18fb7ac488987ae20b97ef5" Oct 30 01:31:46.327108 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:46.326614 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl" Oct 30 01:31:46.379987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:46.379950 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl"] Oct 30 01:31:46.389429 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:46.389388 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-b-d3477dc4-kserve-5fb9c9bbd5-pcbpl"] Oct 30 01:31:46.419038 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:46.418995 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:46.542537 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:46.542504 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be800e84-dc15-4618-8d51-95404c9b6c5d" path="/var/lib/kubelet/pods/be800e84-dc15-4618-8d51-95404c9b6c5d/volumes" Oct 30 01:31:48.109289 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:48.109248 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:49.927416 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:49.927378 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:51.937368 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:51.937330 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:53.727221 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:53.727165 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:55.401200 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:55.401166 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:55.668041 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:55.668005 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3eb7deca-jlslv"] Oct 30 01:31:55.668527 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:55.668514 6572 cpu_manager.go:414] "containerMap: removing container" podUID="f7d28ee6-0e8a-45dc-91ca-73c70f96576c" containerName="revoke-keys" Oct 30 01:31:55.668574 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:55.668528 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7d28ee6-0e8a-45dc-91ca-73c70f96576c" containerName="revoke-keys" Oct 30 01:31:55.668574 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:55.668538 6572 cpu_manager.go:414] "containerMap: removing container" podUID="be800e84-dc15-4618-8d51-95404c9b6c5d" containerName="main" Oct 30 01:31:55.668574 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:55.668542 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be800e84-dc15-4618-8d51-95404c9b6c5d" containerName="main" Oct 30 01:31:55.668665 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:55.668619 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7d28ee6-0e8a-45dc-91ca-73c70f96576c" containerName="revoke-keys" Oct 30 01:31:55.668665 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:55.668629 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="be800e84-dc15-4618-8d51-95404c9b6c5d" containerName="main" Oct 30 01:31:55.673662 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:55.673641 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eb7deca-jlslv" Oct 30 01:31:55.689571 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:55.689537 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3eb7deca-jlslv"] Oct 30 01:31:55.695512 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:55.695479 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:31:55.695512 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:55.695481 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:31:55.695727 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:55.695510 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:31:55.695727 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:55.695554 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:31:55.765194 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:55.765161 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e4f9c14-e607-49c0-88ae-804ab307b074-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3eb7deca-jlslv\" (UID: \"5e4f9c14-e607-49c0-88ae-804ab307b074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eb7deca-jlslv" Oct 30 01:31:55.765363 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:55.765233 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e4f9c14-e607-49c0-88ae-804ab307b074-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3eb7deca-jlslv\" (UID: \"5e4f9c14-e607-49c0-88ae-804ab307b074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eb7deca-jlslv" Oct 30 01:31:55.765410 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:55.765358 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrf7g\" (UniqueName: \"kubernetes.io/projected/5e4f9c14-e607-49c0-88ae-804ab307b074-kube-api-access-lrf7g\") pod \"maas-api-revoke-sub-3eb7deca-jlslv\" (UID: \"5e4f9c14-e607-49c0-88ae-804ab307b074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eb7deca-jlslv" Oct 30 01:31:55.867235 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:55.867195 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e4f9c14-e607-49c0-88ae-804ab307b074-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3eb7deca-jlslv\" (UID: \"5e4f9c14-e607-49c0-88ae-804ab307b074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eb7deca-jlslv" Oct 30 01:31:55.867418 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:55.867288 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e4f9c14-e607-49c0-88ae-804ab307b074-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3eb7deca-jlslv\" (UID: \"5e4f9c14-e607-49c0-88ae-804ab307b074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eb7deca-jlslv" Oct 30 01:31:55.867418 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:55.867323 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrf7g\" (UniqueName: \"kubernetes.io/projected/5e4f9c14-e607-49c0-88ae-804ab307b074-kube-api-access-lrf7g\") pod \"maas-api-revoke-sub-3eb7deca-jlslv\" (UID: \"5e4f9c14-e607-49c0-88ae-804ab307b074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eb7deca-jlslv" Oct 30 01:31:55.868088 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:55.868069 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e4f9c14-e607-49c0-88ae-804ab307b074-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3eb7deca-jlslv\" (UID: \"5e4f9c14-e607-49c0-88ae-804ab307b074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eb7deca-jlslv" Oct 30 01:31:55.869759 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:55.869726 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e4f9c14-e607-49c0-88ae-804ab307b074-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3eb7deca-jlslv\" (UID: \"5e4f9c14-e607-49c0-88ae-804ab307b074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eb7deca-jlslv" Oct 30 01:31:55.888561 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:55.888526 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrf7g\" (UniqueName: \"kubernetes.io/projected/5e4f9c14-e607-49c0-88ae-804ab307b074-kube-api-access-lrf7g\") pod \"maas-api-revoke-sub-3eb7deca-jlslv\" (UID: \"5e4f9c14-e607-49c0-88ae-804ab307b074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eb7deca-jlslv" Oct 30 01:31:55.991559 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:55.991452 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eb7deca-jlslv" Oct 30 01:31:56.299968 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:56.299930 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3eb7deca-jlslv"] Oct 30 01:31:56.307474 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:31:56.307430 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e4f9c14_e607_49c0_88ae_804ab307b074.slice/crio-4aa519bb44b18ddc0864304aa86375ef6f8f80ba3799817b795541744ddf97b2 WatchSource:0}: Error finding container 4aa519bb44b18ddc0864304aa86375ef6f8f80ba3799817b795541744ddf97b2: Status 404 returned error can't find the container with id 4aa519bb44b18ddc0864304aa86375ef6f8f80ba3799817b795541744ddf97b2 Oct 30 01:31:56.435396 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:56.435359 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eb7deca-jlslv" event={"ID":"5e4f9c14-e607-49c0-88ae-804ab307b074","Type":"ContainerStarted","Data":"4aa519bb44b18ddc0864304aa86375ef6f8f80ba3799817b795541744ddf97b2"} Oct 30 01:31:57.179278 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.179241 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w"] Oct 30 01:31:57.183452 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.183424 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" Oct 30 01:31:57.216018 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.215986 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w"] Oct 30 01:31:57.225251 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.225218 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:57.291999 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.291956 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-istio-token\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-fdz5w\" (UID: \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" Oct 30 01:31:57.292187 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.292030 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-workload-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-fdz5w\" (UID: \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" Oct 30 01:31:57.292187 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.292087 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-credential-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-fdz5w\" (UID: \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" Oct 30 01:31:57.292187 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.292157 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-istiod-ca-cert\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-fdz5w\" (UID: \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" Oct 30 01:31:57.292298 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.292271 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-istio-podinfo\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-fdz5w\" (UID: \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" Oct 30 01:31:57.292343 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.292306 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-istio-data\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-fdz5w\" (UID: \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" Oct 30 01:31:57.292343 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.292328 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn5n8\" (UniqueName: \"kubernetes.io/projected/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-kube-api-access-tn5n8\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-fdz5w\" (UID: \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" Oct 30 01:31:57.292423 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.292403 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-istio-envoy\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-fdz5w\" (UID: \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" Oct 30 01:31:57.292461 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.292444 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-workload-certs\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-fdz5w\" (UID: \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" Oct 30 01:31:57.394161 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.394123 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-workload-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-fdz5w\" (UID: \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" Oct 30 01:31:57.394161 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.394162 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-credential-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-fdz5w\" (UID: \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" Oct 30 01:31:57.394428 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.394180 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-istiod-ca-cert\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-fdz5w\" (UID: \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" Oct 30 01:31:57.394428 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.394279 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-istio-podinfo\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-fdz5w\" (UID: \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" Oct 30 01:31:57.394428 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.394326 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-istio-data\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-fdz5w\" (UID: \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" Oct 30 01:31:57.394428 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.394355 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn5n8\" (UniqueName: \"kubernetes.io/projected/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-kube-api-access-tn5n8\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-fdz5w\" (UID: \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" Oct 30 01:31:57.394637 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.394488 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-istio-envoy\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-fdz5w\" (UID: \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" Oct 30 01:31:57.394637 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.394558 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-workload-certs\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-fdz5w\" (UID: \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" Oct 30 01:31:57.394781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.394662 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-istio-token\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-fdz5w\" (UID: \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" Oct 30 01:31:57.394781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.394675 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-credential-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-fdz5w\" (UID: \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" Oct 30 01:31:57.394898 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.394775 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-istio-data\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-fdz5w\" (UID: \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" Oct 30 01:31:57.395037 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.395007 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-workload-certs\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-fdz5w\" (UID: \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" Oct 30 01:31:57.395129 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.395090 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-workload-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-fdz5w\" (UID: \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" Oct 30 01:31:57.395183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.395122 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-istiod-ca-cert\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-fdz5w\" (UID: \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" Oct 30 01:31:57.396970 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.396947 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-istio-envoy\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-fdz5w\" (UID: \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" Oct 30 01:31:57.397238 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.397219 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-istio-podinfo\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-fdz5w\" (UID: \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" Oct 30 01:31:57.412704 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.412665 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-istio-token\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-fdz5w\" (UID: \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" Oct 30 01:31:57.412885 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.412864 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn5n8\" (UniqueName: \"kubernetes.io/projected/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-kube-api-access-tn5n8\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-fdz5w\" (UID: \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" Oct 30 01:31:57.446761 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.446647 6572 generic.go:356] "Generic (PLEG): container finished" podID="5e4f9c14-e607-49c0-88ae-804ab307b074" containerID="c6eed83069a67c1a85a6d56020eed776746799faa575b9e60521d9daef96b907" exitCode=0 Oct 30 01:31:57.446761 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.446717 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eb7deca-jlslv" event={"ID":"5e4f9c14-e607-49c0-88ae-804ab307b074","Type":"ContainerDied","Data":"c6eed83069a67c1a85a6d56020eed776746799faa575b9e60521d9daef96b907"} Oct 30 01:31:57.502521 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.502482 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" Oct 30 01:31:57.815555 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.815510 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w"] Oct 30 01:31:57.816701 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:31:57.816669 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ba2dcf2_ec73_413b_8190_5e2b31cd692c.slice/crio-d9b30d1ab93cdb9085f76f065445ae5dd1c1733f5902050f755863d1781e5981 WatchSource:0}: Error finding container d9b30d1ab93cdb9085f76f065445ae5dd1c1733f5902050f755863d1781e5981: Status 404 returned error can't find the container with id d9b30d1ab93cdb9085f76f065445ae5dd1c1733f5902050f755863d1781e5981 Oct 30 01:31:57.819812 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.819714 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:31:57.819922 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.819858 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:31:57.819922 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:57.819909 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:31:58.456500 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:58.456462 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" event={"ID":"6ba2dcf2-ec73-413b-8190-5e2b31cd692c","Type":"ContainerStarted","Data":"38e8aa2bb06b9712c749767dca3d4fe5b92e43db83d6cd6c636d3050a28718a0"} Oct 30 01:31:58.456954 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:58.456529 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" event={"ID":"6ba2dcf2-ec73-413b-8190-5e2b31cd692c","Type":"ContainerStarted","Data":"d9b30d1ab93cdb9085f76f065445ae5dd1c1733f5902050f755863d1781e5981"} Oct 30 01:31:58.502653 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:58.502614 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" Oct 30 01:31:58.504428 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:58.504379 6572 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 01:31:58.504590 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:58.504451 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" podUID="6ba2dcf2-ec73-413b-8190-5e2b31cd692c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 01:31:58.526168 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:58.526096 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" podStartSLOduration=1.526072706 podStartE2EDuration="1.526072706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:31:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:31:58.502737214 +0000 UTC m=+844.079162615" watchObservedRunningTime="2026-10-30 01:31:58.526072706 +0000 UTC m=+844.102498110" Oct 30 01:31:58.646818 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:58.646785 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eb7deca-jlslv" Oct 30 01:31:58.740275 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:58.740177 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e4f9c14-e607-49c0-88ae-804ab307b074-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e4f9c14-e607-49c0-88ae-804ab307b074-maas-api-openshift-ca\") pod \"5e4f9c14-e607-49c0-88ae-804ab307b074\" (UID: \"5e4f9c14-e607-49c0-88ae-804ab307b074\") " Oct 30 01:31:58.740275 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:58.740246 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e4f9c14-e607-49c0-88ae-804ab307b074-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e4f9c14-e607-49c0-88ae-804ab307b074-maas-api-xks-ca\") pod \"5e4f9c14-e607-49c0-88ae-804ab307b074\" (UID: \"5e4f9c14-e607-49c0-88ae-804ab307b074\") " Oct 30 01:31:58.740531 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:58.740291 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e4f9c14-e607-49c0-88ae-804ab307b074-kube-api-access-lrf7g\" (UniqueName: \"kubernetes.io/projected/5e4f9c14-e607-49c0-88ae-804ab307b074-kube-api-access-lrf7g\") pod \"5e4f9c14-e607-49c0-88ae-804ab307b074\" (UID: \"5e4f9c14-e607-49c0-88ae-804ab307b074\") " Oct 30 01:31:58.741599 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:58.741571 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e4f9c14-e607-49c0-88ae-804ab307b074-maas-api-openshift-ca" pod "5e4f9c14-e607-49c0-88ae-804ab307b074" (UID: "5e4f9c14-e607-49c0-88ae-804ab307b074"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:31:58.751705 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:58.751646 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e4f9c14-e607-49c0-88ae-804ab307b074-kube-api-access-lrf7g" pod "5e4f9c14-e607-49c0-88ae-804ab307b074" (UID: "5e4f9c14-e607-49c0-88ae-804ab307b074"). InnerVolumeSpecName "kube-api-access-lrf7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:31:58.753253 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:58.753186 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e4f9c14-e607-49c0-88ae-804ab307b074-maas-api-xks-ca" pod "5e4f9c14-e607-49c0-88ae-804ab307b074" (UID: "5e4f9c14-e607-49c0-88ae-804ab307b074"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:31:58.842521 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:58.842482 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e4f9c14-e607-49c0-88ae-804ab307b074-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:31:58.842521 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:58.842514 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrf7g\" (UniqueName: \"kubernetes.io/projected/5e4f9c14-e607-49c0-88ae-804ab307b074-kube-api-access-lrf7g\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:31:58.842521 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:58.842526 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e4f9c14-e607-49c0-88ae-804ab307b074-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:31:58.932628 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:58.932577 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:58.970325 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:58.970289 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f69974da-gpnw2"] Oct 30 01:31:58.970911 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:58.970893 6572 cpu_manager.go:414] "containerMap: removing container" podUID="5e4f9c14-e607-49c0-88ae-804ab307b074" containerName="revoke-keys" Oct 30 01:31:58.970974 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:58.970914 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e4f9c14-e607-49c0-88ae-804ab307b074" containerName="revoke-keys" Oct 30 01:31:58.971053 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:58.971042 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e4f9c14-e607-49c0-88ae-804ab307b074" containerName="revoke-keys" Oct 30 01:31:58.975022 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:58.974995 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f69974da-gpnw2" Oct 30 01:31:59.040827 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:59.033820 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f69974da-gpnw2"] Oct 30 01:31:59.149781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:59.149699 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70404cb5-1c78-4fdb-9dcb-c06667b44754-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f69974da-gpnw2\" (UID: \"70404cb5-1c78-4fdb-9dcb-c06667b44754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f69974da-gpnw2" Oct 30 01:31:59.149971 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:59.149932 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70404cb5-1c78-4fdb-9dcb-c06667b44754-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f69974da-gpnw2\" (UID: \"70404cb5-1c78-4fdb-9dcb-c06667b44754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f69974da-gpnw2" Oct 30 01:31:59.149971 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:59.149965 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qgcc\" (UniqueName: \"kubernetes.io/projected/70404cb5-1c78-4fdb-9dcb-c06667b44754-kube-api-access-4qgcc\") pod \"maas-api-revoke-sub-f69974da-gpnw2\" (UID: \"70404cb5-1c78-4fdb-9dcb-c06667b44754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f69974da-gpnw2" Oct 30 01:31:59.251536 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:59.251491 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70404cb5-1c78-4fdb-9dcb-c06667b44754-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f69974da-gpnw2\" (UID: \"70404cb5-1c78-4fdb-9dcb-c06667b44754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f69974da-gpnw2" Oct 30 01:31:59.251738 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:59.251635 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70404cb5-1c78-4fdb-9dcb-c06667b44754-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f69974da-gpnw2\" (UID: \"70404cb5-1c78-4fdb-9dcb-c06667b44754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f69974da-gpnw2" Oct 30 01:31:59.251738 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:59.251680 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qgcc\" (UniqueName: \"kubernetes.io/projected/70404cb5-1c78-4fdb-9dcb-c06667b44754-kube-api-access-4qgcc\") pod \"maas-api-revoke-sub-f69974da-gpnw2\" (UID: \"70404cb5-1c78-4fdb-9dcb-c06667b44754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f69974da-gpnw2" Oct 30 01:31:59.252452 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:59.252425 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70404cb5-1c78-4fdb-9dcb-c06667b44754-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f69974da-gpnw2\" (UID: \"70404cb5-1c78-4fdb-9dcb-c06667b44754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f69974da-gpnw2" Oct 30 01:31:59.254081 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:59.254056 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70404cb5-1c78-4fdb-9dcb-c06667b44754-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f69974da-gpnw2\" (UID: \"70404cb5-1c78-4fdb-9dcb-c06667b44754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f69974da-gpnw2" Oct 30 01:31:59.269200 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:59.269165 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qgcc\" (UniqueName: \"kubernetes.io/projected/70404cb5-1c78-4fdb-9dcb-c06667b44754-kube-api-access-4qgcc\") pod \"maas-api-revoke-sub-f69974da-gpnw2\" (UID: \"70404cb5-1c78-4fdb-9dcb-c06667b44754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f69974da-gpnw2" Oct 30 01:31:59.336077 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:59.335965 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f69974da-gpnw2" Oct 30 01:31:59.471725 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:59.471691 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eb7deca-jlslv" Oct 30 01:31:59.471725 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:59.471718 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eb7deca-jlslv" event={"ID":"5e4f9c14-e607-49c0-88ae-804ab307b074","Type":"ContainerDied","Data":"4aa519bb44b18ddc0864304aa86375ef6f8f80ba3799817b795541744ddf97b2"} Oct 30 01:31:59.472226 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:59.471789 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4aa519bb44b18ddc0864304aa86375ef6f8f80ba3799817b795541744ddf97b2" Oct 30 01:31:59.503810 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:59.503765 6572 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 01:31:59.503991 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:59.503835 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" podUID="6ba2dcf2-ec73-413b-8190-5e2b31cd692c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 01:31:59.654217 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:31:59.654184 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f69974da-gpnw2"] Oct 30 01:31:59.656940 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:31:59.656896 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70404cb5_1c78_4fdb_9dcb_c06667b44754.slice/crio-40545415340da988e33da6e6055f712b542cb65a614db6f8fc5c1e0907ad3dcf WatchSource:0}: Error finding container 40545415340da988e33da6e6055f712b542cb65a614db6f8fc5c1e0907ad3dcf: Status 404 returned error can't find the container with id 40545415340da988e33da6e6055f712b542cb65a614db6f8fc5c1e0907ad3dcf Oct 30 01:32:00.480683 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:00.480641 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f69974da-gpnw2" event={"ID":"70404cb5-1c78-4fdb-9dcb-c06667b44754","Type":"ContainerStarted","Data":"d3cfd15de9ecf912b6d1940236b94e2bbd5ca5a94db178e9d2a004a13a11334b"} Oct 30 01:32:00.481150 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:00.480783 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f69974da-gpnw2" event={"ID":"70404cb5-1c78-4fdb-9dcb-c06667b44754","Type":"ContainerStarted","Data":"40545415340da988e33da6e6055f712b542cb65a614db6f8fc5c1e0907ad3dcf"} Oct 30 01:32:00.503840 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:00.503785 6572 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 01:32:00.504027 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:00.503857 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" podUID="6ba2dcf2-ec73-413b-8190-5e2b31cd692c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 01:32:00.532003 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:00.531934 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f69974da-gpnw2" podStartSLOduration=2.5319092899999998 podStartE2EDuration="2.53190929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:00.523140826 +0000 UTC m=+846.099566207" watchObservedRunningTime="2026-10-30 01:32:00.53190929 +0000 UTC m=+846.108334696" Oct 30 01:32:00.832019 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:00.831975 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:01.489687 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:01.489650 6572 generic.go:356] "Generic (PLEG): container finished" podID="70404cb5-1c78-4fdb-9dcb-c06667b44754" containerID="d3cfd15de9ecf912b6d1940236b94e2bbd5ca5a94db178e9d2a004a13a11334b" exitCode=0 Oct 30 01:32:01.490453 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:01.489782 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f69974da-gpnw2" event={"ID":"70404cb5-1c78-4fdb-9dcb-c06667b44754","Type":"ContainerDied","Data":"d3cfd15de9ecf912b6d1940236b94e2bbd5ca5a94db178e9d2a004a13a11334b"} Oct 30 01:32:01.503303 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:01.503260 6572 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 01:32:01.503399 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:01.503336 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" podUID="6ba2dcf2-ec73-413b-8190-5e2b31cd692c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 01:32:01.634773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:01.621090 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:02.503665 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:02.503634 6572 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 01:32:02.504063 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:02.503688 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" podUID="6ba2dcf2-ec73-413b-8190-5e2b31cd692c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 01:32:02.738768 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:02.738282 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f69974da-gpnw2" Oct 30 01:32:02.751319 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:02.751213 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:02.804614 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:02.804577 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70404cb5-1c78-4fdb-9dcb-c06667b44754-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70404cb5-1c78-4fdb-9dcb-c06667b44754-maas-api-openshift-ca\") pod \"70404cb5-1c78-4fdb-9dcb-c06667b44754\" (UID: \"70404cb5-1c78-4fdb-9dcb-c06667b44754\") " Oct 30 01:32:02.804843 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:02.804634 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70404cb5-1c78-4fdb-9dcb-c06667b44754-kube-api-access-4qgcc\" (UniqueName: \"kubernetes.io/projected/70404cb5-1c78-4fdb-9dcb-c06667b44754-kube-api-access-4qgcc\") pod \"70404cb5-1c78-4fdb-9dcb-c06667b44754\" (UID: \"70404cb5-1c78-4fdb-9dcb-c06667b44754\") " Oct 30 01:32:02.804843 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:02.804694 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70404cb5-1c78-4fdb-9dcb-c06667b44754-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70404cb5-1c78-4fdb-9dcb-c06667b44754-maas-api-xks-ca\") pod \"70404cb5-1c78-4fdb-9dcb-c06667b44754\" (UID: \"70404cb5-1c78-4fdb-9dcb-c06667b44754\") " Oct 30 01:32:02.805415 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:02.805386 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70404cb5-1c78-4fdb-9dcb-c06667b44754-maas-api-openshift-ca" pod "70404cb5-1c78-4fdb-9dcb-c06667b44754" (UID: "70404cb5-1c78-4fdb-9dcb-c06667b44754"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:32:02.830979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:02.829802 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70404cb5-1c78-4fdb-9dcb-c06667b44754-maas-api-xks-ca" pod "70404cb5-1c78-4fdb-9dcb-c06667b44754" (UID: "70404cb5-1c78-4fdb-9dcb-c06667b44754"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:32:02.844844 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:02.844695 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70404cb5-1c78-4fdb-9dcb-c06667b44754-kube-api-access-4qgcc" pod "70404cb5-1c78-4fdb-9dcb-c06667b44754" (UID: "70404cb5-1c78-4fdb-9dcb-c06667b44754"). InnerVolumeSpecName "kube-api-access-4qgcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:02.929775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:02.906606 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70404cb5-1c78-4fdb-9dcb-c06667b44754-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:02.929775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:02.906651 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qgcc\" (UniqueName: \"kubernetes.io/projected/70404cb5-1c78-4fdb-9dcb-c06667b44754-kube-api-access-4qgcc\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:02.929775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:02.906667 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70404cb5-1c78-4fdb-9dcb-c06667b44754-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:02.981293 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:02.981257 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7b538df-bq9bk"] Oct 30 01:32:02.981861 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:02.981846 6572 cpu_manager.go:414] "containerMap: removing container" podUID="70404cb5-1c78-4fdb-9dcb-c06667b44754" containerName="revoke-keys" Oct 30 01:32:02.981917 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:02.981862 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70404cb5-1c78-4fdb-9dcb-c06667b44754" containerName="revoke-keys" Oct 30 01:32:02.981982 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:02.981973 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="70404cb5-1c78-4fdb-9dcb-c06667b44754" containerName="revoke-keys" Oct 30 01:32:02.985616 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:02.985590 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b538df-bq9bk" Oct 30 01:32:03.026830 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:03.024460 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3598ded5-aba2-40f9-a3e5-36ad98429c05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7b538df-bq9bk\" (UID: \"3598ded5-aba2-40f9-a3e5-36ad98429c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b538df-bq9bk" Oct 30 01:32:03.026830 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:03.024527 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3598ded5-aba2-40f9-a3e5-36ad98429c05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7b538df-bq9bk\" (UID: \"3598ded5-aba2-40f9-a3e5-36ad98429c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b538df-bq9bk" Oct 30 01:32:03.026830 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:03.024590 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5nn9\" (UniqueName: \"kubernetes.io/projected/3598ded5-aba2-40f9-a3e5-36ad98429c05-kube-api-access-g5nn9\") pod \"maas-api-revoke-sub-b7b538df-bq9bk\" (UID: \"3598ded5-aba2-40f9-a3e5-36ad98429c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b538df-bq9bk" Oct 30 01:32:03.039844 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:03.029777 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7b538df-bq9bk"] Oct 30 01:32:03.129770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:03.129480 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3598ded5-aba2-40f9-a3e5-36ad98429c05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7b538df-bq9bk\" (UID: \"3598ded5-aba2-40f9-a3e5-36ad98429c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b538df-bq9bk" Oct 30 01:32:03.129770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:03.129562 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3598ded5-aba2-40f9-a3e5-36ad98429c05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7b538df-bq9bk\" (UID: \"3598ded5-aba2-40f9-a3e5-36ad98429c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b538df-bq9bk" Oct 30 01:32:03.129770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:03.129626 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5nn9\" (UniqueName: \"kubernetes.io/projected/3598ded5-aba2-40f9-a3e5-36ad98429c05-kube-api-access-g5nn9\") pod \"maas-api-revoke-sub-b7b538df-bq9bk\" (UID: \"3598ded5-aba2-40f9-a3e5-36ad98429c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b538df-bq9bk" Oct 30 01:32:03.131436 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:03.131412 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3598ded5-aba2-40f9-a3e5-36ad98429c05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7b538df-bq9bk\" (UID: \"3598ded5-aba2-40f9-a3e5-36ad98429c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b538df-bq9bk" Oct 30 01:32:03.133358 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:03.133335 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3598ded5-aba2-40f9-a3e5-36ad98429c05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7b538df-bq9bk\" (UID: \"3598ded5-aba2-40f9-a3e5-36ad98429c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b538df-bq9bk" Oct 30 01:32:03.154857 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:03.154824 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5nn9\" (UniqueName: \"kubernetes.io/projected/3598ded5-aba2-40f9-a3e5-36ad98429c05-kube-api-access-g5nn9\") pod \"maas-api-revoke-sub-b7b538df-bq9bk\" (UID: \"3598ded5-aba2-40f9-a3e5-36ad98429c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b538df-bq9bk" Oct 30 01:32:03.300048 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:03.299950 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b538df-bq9bk" Oct 30 01:32:03.503432 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:03.503400 6572 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 01:32:03.503619 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:03.503452 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" podUID="6ba2dcf2-ec73-413b-8190-5e2b31cd692c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 01:32:03.532719 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:03.532670 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f69974da-gpnw2" Oct 30 01:32:03.532719 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:03.532684 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f69974da-gpnw2" event={"ID":"70404cb5-1c78-4fdb-9dcb-c06667b44754","Type":"ContainerDied","Data":"40545415340da988e33da6e6055f712b542cb65a614db6f8fc5c1e0907ad3dcf"} Oct 30 01:32:03.533270 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:03.532780 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40545415340da988e33da6e6055f712b542cb65a614db6f8fc5c1e0907ad3dcf" Oct 30 01:32:03.533270 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:03.533230 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:03.649363 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:03.649332 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7b538df-bq9bk"] Oct 30 01:32:03.650614 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:32:03.650579 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3598ded5_aba2_40f9_a3e5_36ad98429c05.slice/crio-a4cd6a098a57a36a3b2552f56363ea71e1655e0858734671e8116a945fbd2e89 WatchSource:0}: Error finding container a4cd6a098a57a36a3b2552f56363ea71e1655e0858734671e8116a945fbd2e89: Status 404 returned error can't find the container with id a4cd6a098a57a36a3b2552f56363ea71e1655e0858734671e8116a945fbd2e89 Oct 30 01:32:04.200939 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:04.200899 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp"] Oct 30 01:32:04.205161 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:04.205131 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" Oct 30 01:32:04.219556 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:04.219528 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a3a2ac\"/\"default-dockercfg-zpcmr\"" Oct 30 01:32:04.229584 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:04.229548 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a3a2ac\"/\"distinct-w0-a3a2ac-kserve-self-signed-certs\"" Oct 30 01:32:04.229584 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:04.229548 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a3a2ac\"/\"kube-root-ca.crt\"" Oct 30 01:32:04.235028 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:04.234991 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp"] Oct 30 01:32:04.245822 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:04.245741 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsdxs\" (UniqueName: \"kubernetes.io/projected/cb20484a-51e2-4581-a629-e92d4234784e-kube-api-access-qsdxs\") pod \"distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp\" (UID: \"cb20484a-51e2-4581-a629-e92d4234784e\") " pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" Oct 30 01:32:04.246033 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:04.245845 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb20484a-51e2-4581-a629-e92d4234784e-dshm\") pod \"distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp\" (UID: \"cb20484a-51e2-4581-a629-e92d4234784e\") " pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" Oct 30 01:32:04.246118 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:04.246029 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb20484a-51e2-4581-a629-e92d4234784e-model-cache\") pod \"distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp\" (UID: \"cb20484a-51e2-4581-a629-e92d4234784e\") " pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" Oct 30 01:32:04.246118 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:04.246063 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb20484a-51e2-4581-a629-e92d4234784e-tls-certs\") pod \"distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp\" (UID: \"cb20484a-51e2-4581-a629-e92d4234784e\") " pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" Oct 30 01:32:04.246235 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:04.246132 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb20484a-51e2-4581-a629-e92d4234784e-home\") pod \"distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp\" (UID: \"cb20484a-51e2-4581-a629-e92d4234784e\") " pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" Oct 30 01:32:04.246235 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:04.246166 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb20484a-51e2-4581-a629-e92d4234784e-tmp-dir\") pod \"distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp\" (UID: \"cb20484a-51e2-4581-a629-e92d4234784e\") " pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" Oct 30 01:32:04.322283 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:04.322205 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a3a2ac\"/\"openshift-service-ca.crt\"" Oct 30 01:32:04.346920 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:04.346873 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb20484a-51e2-4581-a629-e92d4234784e-model-cache\") pod \"distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp\" (UID: \"cb20484a-51e2-4581-a629-e92d4234784e\") " pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" Oct 30 01:32:04.347089 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:04.346931 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb20484a-51e2-4581-a629-e92d4234784e-tls-certs\") pod \"distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp\" (UID: \"cb20484a-51e2-4581-a629-e92d4234784e\") " pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" Oct 30 01:32:04.347089 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:04.346953 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb20484a-51e2-4581-a629-e92d4234784e-home\") pod \"distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp\" (UID: \"cb20484a-51e2-4581-a629-e92d4234784e\") " pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" Oct 30 01:32:04.347089 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:04.346978 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb20484a-51e2-4581-a629-e92d4234784e-tmp-dir\") pod \"distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp\" (UID: \"cb20484a-51e2-4581-a629-e92d4234784e\") " pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" Oct 30 01:32:04.347089 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:04.347068 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsdxs\" (UniqueName: \"kubernetes.io/projected/cb20484a-51e2-4581-a629-e92d4234784e-kube-api-access-qsdxs\") pod \"distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp\" (UID: \"cb20484a-51e2-4581-a629-e92d4234784e\") " pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" Oct 30 01:32:04.347320 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:04.347094 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb20484a-51e2-4581-a629-e92d4234784e-dshm\") pod \"distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp\" (UID: \"cb20484a-51e2-4581-a629-e92d4234784e\") " pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" Oct 30 01:32:04.347413 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:04.347388 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb20484a-51e2-4581-a629-e92d4234784e-model-cache\") pod \"distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp\" (UID: \"cb20484a-51e2-4581-a629-e92d4234784e\") " pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" Oct 30 01:32:04.347489 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:04.347419 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb20484a-51e2-4581-a629-e92d4234784e-home\") pod \"distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp\" (UID: \"cb20484a-51e2-4581-a629-e92d4234784e\") " pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" Oct 30 01:32:04.347552 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:04.347478 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb20484a-51e2-4581-a629-e92d4234784e-tmp-dir\") pod \"distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp\" (UID: \"cb20484a-51e2-4581-a629-e92d4234784e\") " pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" Oct 30 01:32:04.349531 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:04.349496 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb20484a-51e2-4581-a629-e92d4234784e-dshm\") pod \"distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp\" (UID: \"cb20484a-51e2-4581-a629-e92d4234784e\") " pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" Oct 30 01:32:04.349832 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:04.349810 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb20484a-51e2-4581-a629-e92d4234784e-tls-certs\") pod \"distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp\" (UID: \"cb20484a-51e2-4581-a629-e92d4234784e\") " pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" Oct 30 01:32:04.356239 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:04.356195 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:04.366731 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:04.366692 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsdxs\" (UniqueName: \"kubernetes.io/projected/cb20484a-51e2-4581-a629-e92d4234784e-kube-api-access-qsdxs\") pod \"distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp\" (UID: \"cb20484a-51e2-4581-a629-e92d4234784e\") " pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" Oct 30 01:32:04.503577 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:04.503544 6572 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 01:32:04.503777 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:04.503596 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" podUID="6ba2dcf2-ec73-413b-8190-5e2b31cd692c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 01:32:04.518691 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:04.518651 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" Oct 30 01:32:04.545297 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:04.545257 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b538df-bq9bk" event={"ID":"3598ded5-aba2-40f9-a3e5-36ad98429c05","Type":"ContainerStarted","Data":"92e064dc3aedb1117e3d505de23d71ffe4c5c802af0d57b2e883831dddcdf68c"} Oct 30 01:32:04.545711 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:04.545325 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b538df-bq9bk" event={"ID":"3598ded5-aba2-40f9-a3e5-36ad98429c05","Type":"ContainerStarted","Data":"a4cd6a098a57a36a3b2552f56363ea71e1655e0858734671e8116a945fbd2e89"} Oct 30 01:32:04.582075 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:04.581957 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b538df-bq9bk" podStartSLOduration=2.581942442 podStartE2EDuration="2.581942442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:04.579064616 +0000 UTC m=+850.155490014" watchObservedRunningTime="2026-10-30 01:32:04.581942442 +0000 UTC m=+850.158367874" Oct 30 01:32:04.860675 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:04.860570 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp"] Oct 30 01:32:04.879400 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:04.879369 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:05.507955 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:05.507920 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" Oct 30 01:32:05.508218 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:05.508200 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" Oct 30 01:32:05.509122 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:05.509103 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" Oct 30 01:32:05.554209 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:05.554164 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" event={"ID":"cb20484a-51e2-4581-a629-e92d4234784e","Type":"ContainerStarted","Data":"5d440c306c4be1f8e0b17057f73633f6d604c476fff8d7ce9ece155aee9c0002"} Oct 30 01:32:05.554209 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:05.554208 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" event={"ID":"cb20484a-51e2-4581-a629-e92d4234784e","Type":"ContainerStarted","Data":"0c61535a350120f9e199f9754a36e49b045e6fddcf6500711e473d15582fc01b"} Oct 30 01:32:05.555849 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:05.555828 6572 generic.go:356] "Generic (PLEG): container finished" podID="3598ded5-aba2-40f9-a3e5-36ad98429c05" containerID="92e064dc3aedb1117e3d505de23d71ffe4c5c802af0d57b2e883831dddcdf68c" exitCode=0 Oct 30 01:32:05.555946 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:05.555870 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b538df-bq9bk" event={"ID":"3598ded5-aba2-40f9-a3e5-36ad98429c05","Type":"ContainerDied","Data":"92e064dc3aedb1117e3d505de23d71ffe4c5c802af0d57b2e883831dddcdf68c"} Oct 30 01:32:05.631974 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:05.631916 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" podStartSLOduration=1.631895864 podStartE2EDuration="1.631895864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:05.630057712 +0000 UTC m=+851.206483121" watchObservedRunningTime="2026-10-30 01:32:05.631895864 +0000 UTC m=+851.208321267" Oct 30 01:32:06.519028 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:06.518989 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" Oct 30 01:32:06.522542 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:06.522506 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:06.718469 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:06.718442 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b538df-bq9bk" Oct 30 01:32:06.776908 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:06.776812 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3598ded5-aba2-40f9-a3e5-36ad98429c05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3598ded5-aba2-40f9-a3e5-36ad98429c05-maas-api-openshift-ca\") pod \"3598ded5-aba2-40f9-a3e5-36ad98429c05\" (UID: \"3598ded5-aba2-40f9-a3e5-36ad98429c05\") " Oct 30 01:32:06.776908 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:06.776863 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3598ded5-aba2-40f9-a3e5-36ad98429c05-kube-api-access-g5nn9\" (UniqueName: \"kubernetes.io/projected/3598ded5-aba2-40f9-a3e5-36ad98429c05-kube-api-access-g5nn9\") pod \"3598ded5-aba2-40f9-a3e5-36ad98429c05\" (UID: \"3598ded5-aba2-40f9-a3e5-36ad98429c05\") " Oct 30 01:32:06.777171 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:06.777012 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3598ded5-aba2-40f9-a3e5-36ad98429c05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3598ded5-aba2-40f9-a3e5-36ad98429c05-maas-api-xks-ca\") pod \"3598ded5-aba2-40f9-a3e5-36ad98429c05\" (UID: \"3598ded5-aba2-40f9-a3e5-36ad98429c05\") " Oct 30 01:32:06.777319 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:06.777291 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3598ded5-aba2-40f9-a3e5-36ad98429c05-maas-api-openshift-ca" pod "3598ded5-aba2-40f9-a3e5-36ad98429c05" (UID: "3598ded5-aba2-40f9-a3e5-36ad98429c05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:32:06.777487 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:06.777468 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3598ded5-aba2-40f9-a3e5-36ad98429c05-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:06.779036 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:06.779009 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3598ded5-aba2-40f9-a3e5-36ad98429c05-maas-api-xks-ca" pod "3598ded5-aba2-40f9-a3e5-36ad98429c05" (UID: "3598ded5-aba2-40f9-a3e5-36ad98429c05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:32:06.779204 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:06.779180 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3598ded5-aba2-40f9-a3e5-36ad98429c05-kube-api-access-g5nn9" pod "3598ded5-aba2-40f9-a3e5-36ad98429c05" (UID: "3598ded5-aba2-40f9-a3e5-36ad98429c05"). InnerVolumeSpecName "kube-api-access-g5nn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:06.879162 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:06.879125 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3598ded5-aba2-40f9-a3e5-36ad98429c05-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:06.879162 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:06.879158 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5nn9\" (UniqueName: \"kubernetes.io/projected/3598ded5-aba2-40f9-a3e5-36ad98429c05-kube-api-access-g5nn9\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:07.573465 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:07.573426 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b538df-bq9bk" event={"ID":"3598ded5-aba2-40f9-a3e5-36ad98429c05","Type":"ContainerDied","Data":"a4cd6a098a57a36a3b2552f56363ea71e1655e0858734671e8116a945fbd2e89"} Oct 30 01:32:07.573465 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:07.573455 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b538df-bq9bk" Oct 30 01:32:07.573726 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:07.573490 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4cd6a098a57a36a3b2552f56363ea71e1655e0858734671e8116a945fbd2e89" Oct 30 01:32:08.041736 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:08.041694 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-833c3c85-9stwn"] Oct 30 01:32:08.042340 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:08.042324 6572 cpu_manager.go:414] "containerMap: removing container" podUID="3598ded5-aba2-40f9-a3e5-36ad98429c05" containerName="revoke-keys" Oct 30 01:32:08.042340 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:08.042341 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3598ded5-aba2-40f9-a3e5-36ad98429c05" containerName="revoke-keys" Oct 30 01:32:08.042461 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:08.042451 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="3598ded5-aba2-40f9-a3e5-36ad98429c05" containerName="revoke-keys" Oct 30 01:32:08.045983 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:08.045959 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-833c3c85-9stwn" Oct 30 01:32:08.063508 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:08.063479 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:32:08.063685 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:08.063524 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:32:08.063685 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:08.063579 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:32:08.073793 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:08.073743 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:32:08.078851 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:08.077875 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-833c3c85-9stwn"] Oct 30 01:32:08.093846 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:08.093804 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f869d5c-1aca-49c0-ae2e-bdb13e847d5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-833c3c85-9stwn\" (UID: \"4f869d5c-1aca-49c0-ae2e-bdb13e847d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-833c3c85-9stwn" Oct 30 01:32:08.094022 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:08.093857 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f869d5c-1aca-49c0-ae2e-bdb13e847d5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-833c3c85-9stwn\" (UID: \"4f869d5c-1aca-49c0-ae2e-bdb13e847d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-833c3c85-9stwn" Oct 30 01:32:08.094022 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:08.093960 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95rj2\" (UniqueName: \"kubernetes.io/projected/4f869d5c-1aca-49c0-ae2e-bdb13e847d5f-kube-api-access-95rj2\") pod \"maas-api-revoke-sub-833c3c85-9stwn\" (UID: \"4f869d5c-1aca-49c0-ae2e-bdb13e847d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-833c3c85-9stwn" Oct 30 01:32:08.195509 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:08.195464 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f869d5c-1aca-49c0-ae2e-bdb13e847d5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-833c3c85-9stwn\" (UID: \"4f869d5c-1aca-49c0-ae2e-bdb13e847d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-833c3c85-9stwn" Oct 30 01:32:08.195509 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:08.195514 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f869d5c-1aca-49c0-ae2e-bdb13e847d5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-833c3c85-9stwn\" (UID: \"4f869d5c-1aca-49c0-ae2e-bdb13e847d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-833c3c85-9stwn" Oct 30 01:32:08.195695 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:08.195539 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95rj2\" (UniqueName: \"kubernetes.io/projected/4f869d5c-1aca-49c0-ae2e-bdb13e847d5f-kube-api-access-95rj2\") pod \"maas-api-revoke-sub-833c3c85-9stwn\" (UID: \"4f869d5c-1aca-49c0-ae2e-bdb13e847d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-833c3c85-9stwn" Oct 30 01:32:08.196220 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:08.196189 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f869d5c-1aca-49c0-ae2e-bdb13e847d5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-833c3c85-9stwn\" (UID: \"4f869d5c-1aca-49c0-ae2e-bdb13e847d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-833c3c85-9stwn" Oct 30 01:32:08.198106 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:08.198078 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f869d5c-1aca-49c0-ae2e-bdb13e847d5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-833c3c85-9stwn\" (UID: \"4f869d5c-1aca-49c0-ae2e-bdb13e847d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-833c3c85-9stwn" Oct 30 01:32:08.214317 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:08.214283 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95rj2\" (UniqueName: \"kubernetes.io/projected/4f869d5c-1aca-49c0-ae2e-bdb13e847d5f-kube-api-access-95rj2\") pod \"maas-api-revoke-sub-833c3c85-9stwn\" (UID: \"4f869d5c-1aca-49c0-ae2e-bdb13e847d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-833c3c85-9stwn" Oct 30 01:32:08.319810 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:08.319708 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:08.362840 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:08.362804 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-833c3c85-9stwn" Oct 30 01:32:08.661174 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:08.661142 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-833c3c85-9stwn"] Oct 30 01:32:08.662077 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:32:08.662053 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f869d5c_1aca_49c0_ae2e_bdb13e847d5f.slice/crio-ed89df056f31783e0896ed52ceb13667543912657d37bb2e261e0e09b4c9c727 WatchSource:0}: Error finding container ed89df056f31783e0896ed52ceb13667543912657d37bb2e261e0e09b4c9c727: Status 404 returned error can't find the container with id ed89df056f31783e0896ed52ceb13667543912657d37bb2e261e0e09b4c9c727 Oct 30 01:32:09.310982 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:09.310879 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61b3fc92-2wcqv"] Oct 30 01:32:09.315150 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:09.315119 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b3fc92-2wcqv" Oct 30 01:32:09.333460 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:09.333417 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61b3fc92-2wcqv"] Oct 30 01:32:09.410113 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:09.410071 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87f831d2-1c73-4a89-a171-b5059272e385-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61b3fc92-2wcqv\" (UID: \"87f831d2-1c73-4a89-a171-b5059272e385\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b3fc92-2wcqv" Oct 30 01:32:09.410296 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:09.410281 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whzlf\" (UniqueName: \"kubernetes.io/projected/87f831d2-1c73-4a89-a171-b5059272e385-kube-api-access-whzlf\") pod \"maas-api-revoke-sub-61b3fc92-2wcqv\" (UID: \"87f831d2-1c73-4a89-a171-b5059272e385\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b3fc92-2wcqv" Oct 30 01:32:09.410352 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:09.410334 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87f831d2-1c73-4a89-a171-b5059272e385-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61b3fc92-2wcqv\" (UID: \"87f831d2-1c73-4a89-a171-b5059272e385\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b3fc92-2wcqv" Oct 30 01:32:09.511343 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:09.511309 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whzlf\" (UniqueName: \"kubernetes.io/projected/87f831d2-1c73-4a89-a171-b5059272e385-kube-api-access-whzlf\") pod \"maas-api-revoke-sub-61b3fc92-2wcqv\" (UID: \"87f831d2-1c73-4a89-a171-b5059272e385\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b3fc92-2wcqv" Oct 30 01:32:09.511517 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:09.511369 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87f831d2-1c73-4a89-a171-b5059272e385-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61b3fc92-2wcqv\" (UID: \"87f831d2-1c73-4a89-a171-b5059272e385\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b3fc92-2wcqv" Oct 30 01:32:09.511517 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:09.511473 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87f831d2-1c73-4a89-a171-b5059272e385-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61b3fc92-2wcqv\" (UID: \"87f831d2-1c73-4a89-a171-b5059272e385\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b3fc92-2wcqv" Oct 30 01:32:09.512252 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:09.512225 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87f831d2-1c73-4a89-a171-b5059272e385-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61b3fc92-2wcqv\" (UID: \"87f831d2-1c73-4a89-a171-b5059272e385\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b3fc92-2wcqv" Oct 30 01:32:09.514166 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:09.514145 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87f831d2-1c73-4a89-a171-b5059272e385-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61b3fc92-2wcqv\" (UID: \"87f831d2-1c73-4a89-a171-b5059272e385\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b3fc92-2wcqv" Oct 30 01:32:09.528705 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:09.528675 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whzlf\" (UniqueName: \"kubernetes.io/projected/87f831d2-1c73-4a89-a171-b5059272e385-kube-api-access-whzlf\") pod \"maas-api-revoke-sub-61b3fc92-2wcqv\" (UID: \"87f831d2-1c73-4a89-a171-b5059272e385\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b3fc92-2wcqv" Oct 30 01:32:09.633544 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:09.633435 6572 generic.go:356] "Generic (PLEG): container finished" podID="4f869d5c-1aca-49c0-ae2e-bdb13e847d5f" containerID="44dedf7efa8d2e3fbeba9f5d1da22e7b919192af8691317efdad783a8e4ac85f" exitCode=0 Oct 30 01:32:09.633727 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:09.633677 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-833c3c85-9stwn" event={"ID":"4f869d5c-1aca-49c0-ae2e-bdb13e847d5f","Type":"ContainerDied","Data":"44dedf7efa8d2e3fbeba9f5d1da22e7b919192af8691317efdad783a8e4ac85f"} Oct 30 01:32:09.633838 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:09.633797 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-833c3c85-9stwn" event={"ID":"4f869d5c-1aca-49c0-ae2e-bdb13e847d5f","Type":"ContainerStarted","Data":"ed89df056f31783e0896ed52ceb13667543912657d37bb2e261e0e09b4c9c727"} Oct 30 01:32:09.634199 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:09.634177 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b3fc92-2wcqv" Oct 30 01:32:09.988222 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:09.988158 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61b3fc92-2wcqv"] Oct 30 01:32:10.231251 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:10.231216 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:10.664183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:10.664140 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b3fc92-2wcqv" event={"ID":"87f831d2-1c73-4a89-a171-b5059272e385","Type":"ContainerStarted","Data":"43bb85bee88ebe879c9dd4c59f19f3bb95c59589b883f2dca98659b8d755e043"} Oct 30 01:32:10.676705 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:10.676673 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b3fc92-2wcqv" event={"ID":"87f831d2-1c73-4a89-a171-b5059272e385","Type":"ContainerStarted","Data":"8e9a4a5b1be58a51ef77e05411d8e75378e765c7fb6a858e6da99632437fc1f7"} Oct 30 01:32:11.505013 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:11.504677 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-833c3c85-9stwn" Oct 30 01:32:11.548641 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:11.548579 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b3fc92-2wcqv" podStartSLOduration=2.548558921 podStartE2EDuration="2.548558921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:10.709606028 +0000 UTC m=+856.286031432" watchObservedRunningTime="2026-10-30 01:32:11.548558921 +0000 UTC m=+857.124984324" Oct 30 01:32:11.562040 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:11.561134 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f869d5c-1aca-49c0-ae2e-bdb13e847d5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f869d5c-1aca-49c0-ae2e-bdb13e847d5f-maas-api-xks-ca\") pod \"4f869d5c-1aca-49c0-ae2e-bdb13e847d5f\" (UID: \"4f869d5c-1aca-49c0-ae2e-bdb13e847d5f\") " Oct 30 01:32:11.562040 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:11.561198 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f869d5c-1aca-49c0-ae2e-bdb13e847d5f-kube-api-access-95rj2\" (UniqueName: \"kubernetes.io/projected/4f869d5c-1aca-49c0-ae2e-bdb13e847d5f-kube-api-access-95rj2\") pod \"4f869d5c-1aca-49c0-ae2e-bdb13e847d5f\" (UID: \"4f869d5c-1aca-49c0-ae2e-bdb13e847d5f\") " Oct 30 01:32:11.603274 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:11.598406 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f869d5c-1aca-49c0-ae2e-bdb13e847d5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f869d5c-1aca-49c0-ae2e-bdb13e847d5f-maas-api-openshift-ca\") pod \"4f869d5c-1aca-49c0-ae2e-bdb13e847d5f\" (UID: \"4f869d5c-1aca-49c0-ae2e-bdb13e847d5f\") " Oct 30 01:32:11.603274 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:11.599505 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f869d5c-1aca-49c0-ae2e-bdb13e847d5f-maas-api-openshift-ca" pod "4f869d5c-1aca-49c0-ae2e-bdb13e847d5f" (UID: "4f869d5c-1aca-49c0-ae2e-bdb13e847d5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:32:11.629038 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:11.628964 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f869d5c-1aca-49c0-ae2e-bdb13e847d5f-kube-api-access-95rj2" pod "4f869d5c-1aca-49c0-ae2e-bdb13e847d5f" (UID: "4f869d5c-1aca-49c0-ae2e-bdb13e847d5f"). InnerVolumeSpecName "kube-api-access-95rj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:11.647981 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:11.647570 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f869d5c-1aca-49c0-ae2e-bdb13e847d5f-maas-api-xks-ca" pod "4f869d5c-1aca-49c0-ae2e-bdb13e847d5f" (UID: "4f869d5c-1aca-49c0-ae2e-bdb13e847d5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:32:11.686994 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:11.686899 6572 generic.go:356] "Generic (PLEG): container finished" podID="87f831d2-1c73-4a89-a171-b5059272e385" containerID="43bb85bee88ebe879c9dd4c59f19f3bb95c59589b883f2dca98659b8d755e043" exitCode=0 Oct 30 01:32:11.687773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:11.687727 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b3fc92-2wcqv" event={"ID":"87f831d2-1c73-4a89-a171-b5059272e385","Type":"ContainerDied","Data":"43bb85bee88ebe879c9dd4c59f19f3bb95c59589b883f2dca98659b8d755e043"} Oct 30 01:32:11.690438 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:11.690411 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-833c3c85-9stwn" event={"ID":"4f869d5c-1aca-49c0-ae2e-bdb13e847d5f","Type":"ContainerDied","Data":"ed89df056f31783e0896ed52ceb13667543912657d37bb2e261e0e09b4c9c727"} Oct 30 01:32:11.690595 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:11.690579 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed89df056f31783e0896ed52ceb13667543912657d37bb2e261e0e09b4c9c727" Oct 30 01:32:11.690796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:11.690777 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-833c3c85-9stwn" Oct 30 01:32:11.700078 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:11.700048 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f869d5c-1aca-49c0-ae2e-bdb13e847d5f-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:11.700290 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:11.700274 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f869d5c-1aca-49c0-ae2e-bdb13e847d5f-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:11.700393 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:11.700379 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95rj2\" (UniqueName: \"kubernetes.io/projected/4f869d5c-1aca-49c0-ae2e-bdb13e847d5f-kube-api-access-95rj2\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.207228 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:12.207191 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:12.848139 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:12.848112 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b3fc92-2wcqv" Oct 30 01:32:13.035183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:13.035101 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87f831d2-1c73-4a89-a171-b5059272e385-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87f831d2-1c73-4a89-a171-b5059272e385-maas-api-openshift-ca\") pod \"87f831d2-1c73-4a89-a171-b5059272e385\" (UID: \"87f831d2-1c73-4a89-a171-b5059272e385\") " Oct 30 01:32:13.035183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:13.035138 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87f831d2-1c73-4a89-a171-b5059272e385-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87f831d2-1c73-4a89-a171-b5059272e385-maas-api-xks-ca\") pod \"87f831d2-1c73-4a89-a171-b5059272e385\" (UID: \"87f831d2-1c73-4a89-a171-b5059272e385\") " Oct 30 01:32:13.035421 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:13.035198 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87f831d2-1c73-4a89-a171-b5059272e385-kube-api-access-whzlf\" (UniqueName: \"kubernetes.io/projected/87f831d2-1c73-4a89-a171-b5059272e385-kube-api-access-whzlf\") pod \"87f831d2-1c73-4a89-a171-b5059272e385\" (UID: \"87f831d2-1c73-4a89-a171-b5059272e385\") " Oct 30 01:32:13.035525 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:13.035476 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87f831d2-1c73-4a89-a171-b5059272e385-maas-api-openshift-ca" pod "87f831d2-1c73-4a89-a171-b5059272e385" (UID: "87f831d2-1c73-4a89-a171-b5059272e385"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:32:13.035736 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:13.035715 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87f831d2-1c73-4a89-a171-b5059272e385-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:13.037514 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:13.037483 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87f831d2-1c73-4a89-a171-b5059272e385-kube-api-access-whzlf" pod "87f831d2-1c73-4a89-a171-b5059272e385" (UID: "87f831d2-1c73-4a89-a171-b5059272e385"). InnerVolumeSpecName "kube-api-access-whzlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:13.037514 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:13.037490 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87f831d2-1c73-4a89-a171-b5059272e385-maas-api-xks-ca" pod "87f831d2-1c73-4a89-a171-b5059272e385" (UID: "87f831d2-1c73-4a89-a171-b5059272e385"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:32:13.137003 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:13.136968 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87f831d2-1c73-4a89-a171-b5059272e385-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:13.137003 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:13.136998 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whzlf\" (UniqueName: \"kubernetes.io/projected/87f831d2-1c73-4a89-a171-b5059272e385-kube-api-access-whzlf\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:13.706547 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:13.706519 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b3fc92-2wcqv" Oct 30 01:32:13.706709 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:13.706550 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b3fc92-2wcqv" event={"ID":"87f831d2-1c73-4a89-a171-b5059272e385","Type":"ContainerDied","Data":"8e9a4a5b1be58a51ef77e05411d8e75378e765c7fb6a858e6da99632437fc1f7"} Oct 30 01:32:13.706709 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:13.706600 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e9a4a5b1be58a51ef77e05411d8e75378e765c7fb6a858e6da99632437fc1f7" Oct 30 01:32:13.916726 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:13.916684 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:14.519447 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:14.519409 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" Oct 30 01:32:14.532893 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:14.532861 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" Oct 30 01:32:14.689131 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:32:14.689089 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:32:14.725130 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:14.725038 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" Oct 30 01:32:15.700216 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:15.700174 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:15.894506 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:15.894470 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0388b014-p6zjm"] Oct 30 01:32:15.897077 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:15.894997 6572 cpu_manager.go:414] "containerMap: removing container" podUID="4f869d5c-1aca-49c0-ae2e-bdb13e847d5f" containerName="revoke-keys" Oct 30 01:32:15.897077 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:15.895009 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f869d5c-1aca-49c0-ae2e-bdb13e847d5f" containerName="revoke-keys" Oct 30 01:32:15.897077 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:15.895037 6572 cpu_manager.go:414] "containerMap: removing container" podUID="87f831d2-1c73-4a89-a171-b5059272e385" containerName="revoke-keys" Oct 30 01:32:15.897077 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:15.895042 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87f831d2-1c73-4a89-a171-b5059272e385" containerName="revoke-keys" Oct 30 01:32:15.897077 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:15.895147 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="87f831d2-1c73-4a89-a171-b5059272e385" containerName="revoke-keys" Oct 30 01:32:15.897077 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:15.895156 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f869d5c-1aca-49c0-ae2e-bdb13e847d5f" containerName="revoke-keys" Oct 30 01:32:15.898813 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:15.898796 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0388b014-p6zjm" Oct 30 01:32:15.918411 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:15.918379 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0388b014-p6zjm"] Oct 30 01:32:15.924038 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:15.924002 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:32:15.924313 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:15.924279 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:32:15.924446 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:15.924345 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:32:15.924510 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:15.924449 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:32:15.968878 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:15.968730 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af93de6e-815c-4103-9437-3a5a45afecaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0388b014-p6zjm\" (UID: \"af93de6e-815c-4103-9437-3a5a45afecaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0388b014-p6zjm" Oct 30 01:32:15.969023 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:15.968890 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kw5n\" (UniqueName: \"kubernetes.io/projected/af93de6e-815c-4103-9437-3a5a45afecaf-kube-api-access-7kw5n\") pod \"maas-api-revoke-sub-0388b014-p6zjm\" (UID: \"af93de6e-815c-4103-9437-3a5a45afecaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0388b014-p6zjm" Oct 30 01:32:15.969023 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:15.968913 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af93de6e-815c-4103-9437-3a5a45afecaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0388b014-p6zjm\" (UID: \"af93de6e-815c-4103-9437-3a5a45afecaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0388b014-p6zjm" Oct 30 01:32:16.070441 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.070399 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af93de6e-815c-4103-9437-3a5a45afecaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0388b014-p6zjm\" (UID: \"af93de6e-815c-4103-9437-3a5a45afecaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0388b014-p6zjm" Oct 30 01:32:16.070621 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.070501 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kw5n\" (UniqueName: \"kubernetes.io/projected/af93de6e-815c-4103-9437-3a5a45afecaf-kube-api-access-7kw5n\") pod \"maas-api-revoke-sub-0388b014-p6zjm\" (UID: \"af93de6e-815c-4103-9437-3a5a45afecaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0388b014-p6zjm" Oct 30 01:32:16.070621 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.070530 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af93de6e-815c-4103-9437-3a5a45afecaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0388b014-p6zjm\" (UID: \"af93de6e-815c-4103-9437-3a5a45afecaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0388b014-p6zjm" Oct 30 01:32:16.071124 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.071104 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af93de6e-815c-4103-9437-3a5a45afecaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0388b014-p6zjm\" (UID: \"af93de6e-815c-4103-9437-3a5a45afecaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0388b014-p6zjm" Oct 30 01:32:16.072999 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.072980 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af93de6e-815c-4103-9437-3a5a45afecaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0388b014-p6zjm\" (UID: \"af93de6e-815c-4103-9437-3a5a45afecaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0388b014-p6zjm" Oct 30 01:32:16.087375 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.087347 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kw5n\" (UniqueName: \"kubernetes.io/projected/af93de6e-815c-4103-9437-3a5a45afecaf-kube-api-access-7kw5n\") pod \"maas-api-revoke-sub-0388b014-p6zjm\" (UID: \"af93de6e-815c-4103-9437-3a5a45afecaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0388b014-p6zjm" Oct 30 01:32:16.212326 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.212292 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0388b014-p6zjm" Oct 30 01:32:16.512065 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.512035 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0388b014-p6zjm"] Oct 30 01:32:16.513484 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:32:16.513454 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf93de6e_815c_4103_9437_3a5a45afecaf.slice/crio-2540677d98c99249b0cc4aa7d1094bec984bcc82f77a68f22b8d5976f90a7467 WatchSource:0}: Error finding container 2540677d98c99249b0cc4aa7d1094bec984bcc82f77a68f22b8d5976f90a7467: Status 404 returned error can't find the container with id 2540677d98c99249b0cc4aa7d1094bec984bcc82f77a68f22b8d5976f90a7467 Oct 30 01:32:16.729897 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.729863 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0388b014-p6zjm" event={"ID":"af93de6e-815c-4103-9437-3a5a45afecaf","Type":"ContainerStarted","Data":"2540677d98c99249b0cc4aa7d1094bec984bcc82f77a68f22b8d5976f90a7467"} Oct 30 01:32:16.853798 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.853760 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx"] Oct 30 01:32:16.857971 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.857945 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" Oct 30 01:32:16.879847 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.879789 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s"] Oct 30 01:32:16.888381 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.888349 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11caba41-1413-482d-8c53-03da9c396b26-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx\" (UID: \"11caba41-1413-482d-8c53-03da9c396b26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" Oct 30 01:32:16.888547 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.888421 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11caba41-1413-482d-8c53-03da9c396b26-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx\" (UID: \"11caba41-1413-482d-8c53-03da9c396b26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" Oct 30 01:32:16.888547 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.888471 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11caba41-1413-482d-8c53-03da9c396b26-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx\" (UID: \"11caba41-1413-482d-8c53-03da9c396b26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" Oct 30 01:32:16.888676 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.888566 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m56nm\" (UniqueName: \"kubernetes.io/projected/11caba41-1413-482d-8c53-03da9c396b26-kube-api-access-m56nm\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx\" (UID: \"11caba41-1413-482d-8c53-03da9c396b26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" Oct 30 01:32:16.888676 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.888608 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11caba41-1413-482d-8c53-03da9c396b26-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx\" (UID: \"11caba41-1413-482d-8c53-03da9c396b26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" Oct 30 01:32:16.888676 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.888637 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11caba41-1413-482d-8c53-03da9c396b26-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx\" (UID: \"11caba41-1413-482d-8c53-03da9c396b26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" Oct 30 01:32:16.888676 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.888670 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11caba41-1413-482d-8c53-03da9c396b26-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx\" (UID: \"11caba41-1413-482d-8c53-03da9c396b26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" Oct 30 01:32:16.888931 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.888761 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11caba41-1413-482d-8c53-03da9c396b26-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx\" (UID: \"11caba41-1413-482d-8c53-03da9c396b26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" Oct 30 01:32:16.888931 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.888894 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11caba41-1413-482d-8c53-03da9c396b26-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx\" (UID: \"11caba41-1413-482d-8c53-03da9c396b26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" Oct 30 01:32:16.889404 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.889384 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" Oct 30 01:32:16.892495 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.892467 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx"] Oct 30 01:32:16.918926 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.918462 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s"] Oct 30 01:32:16.990840 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.990802 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fa0269-eb44-4cf3-91df-50cf13230d1e-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s\" (UID: \"35fa0269-eb44-4cf3-91df-50cf13230d1e\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" Oct 30 01:32:16.991073 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.990864 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11caba41-1413-482d-8c53-03da9c396b26-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx\" (UID: \"11caba41-1413-482d-8c53-03da9c396b26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" Oct 30 01:32:16.991073 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.990954 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35fa0269-eb44-4cf3-91df-50cf13230d1e-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s\" (UID: \"35fa0269-eb44-4cf3-91df-50cf13230d1e\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" Oct 30 01:32:16.991073 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.990998 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35fa0269-eb44-4cf3-91df-50cf13230d1e-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s\" (UID: \"35fa0269-eb44-4cf3-91df-50cf13230d1e\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" Oct 30 01:32:16.991073 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.991053 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35fa0269-eb44-4cf3-91df-50cf13230d1e-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s\" (UID: \"35fa0269-eb44-4cf3-91df-50cf13230d1e\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" Oct 30 01:32:16.991267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.991114 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11caba41-1413-482d-8c53-03da9c396b26-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx\" (UID: \"11caba41-1413-482d-8c53-03da9c396b26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" Oct 30 01:32:16.991267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.991187 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35fa0269-eb44-4cf3-91df-50cf13230d1e-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s\" (UID: \"35fa0269-eb44-4cf3-91df-50cf13230d1e\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" Oct 30 01:32:16.991267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.991232 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11caba41-1413-482d-8c53-03da9c396b26-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx\" (UID: \"11caba41-1413-482d-8c53-03da9c396b26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" Oct 30 01:32:16.991424 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.991298 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fa0269-eb44-4cf3-91df-50cf13230d1e-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s\" (UID: \"35fa0269-eb44-4cf3-91df-50cf13230d1e\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" Oct 30 01:32:16.991424 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.991389 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m56nm\" (UniqueName: \"kubernetes.io/projected/11caba41-1413-482d-8c53-03da9c396b26-kube-api-access-m56nm\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx\" (UID: \"11caba41-1413-482d-8c53-03da9c396b26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" Oct 30 01:32:16.991526 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.991426 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11caba41-1413-482d-8c53-03da9c396b26-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx\" (UID: \"11caba41-1413-482d-8c53-03da9c396b26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" Oct 30 01:32:16.991526 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.991455 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35fa0269-eb44-4cf3-91df-50cf13230d1e-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s\" (UID: \"35fa0269-eb44-4cf3-91df-50cf13230d1e\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" Oct 30 01:32:16.991526 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.991498 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11caba41-1413-482d-8c53-03da9c396b26-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx\" (UID: \"11caba41-1413-482d-8c53-03da9c396b26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" Oct 30 01:32:16.991668 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.991585 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11caba41-1413-482d-8c53-03da9c396b26-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx\" (UID: \"11caba41-1413-482d-8c53-03da9c396b26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" Oct 30 01:32:16.991727 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.991681 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35fa0269-eb44-4cf3-91df-50cf13230d1e-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s\" (UID: \"35fa0269-eb44-4cf3-91df-50cf13230d1e\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" Oct 30 01:32:16.991727 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.991714 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11caba41-1413-482d-8c53-03da9c396b26-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx\" (UID: \"11caba41-1413-482d-8c53-03da9c396b26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" Oct 30 01:32:16.991881 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.991821 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11caba41-1413-482d-8c53-03da9c396b26-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx\" (UID: \"11caba41-1413-482d-8c53-03da9c396b26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" Oct 30 01:32:16.991881 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.991838 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11caba41-1413-482d-8c53-03da9c396b26-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx\" (UID: \"11caba41-1413-482d-8c53-03da9c396b26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" Oct 30 01:32:16.991881 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.991856 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85f65\" (UniqueName: \"kubernetes.io/projected/35fa0269-eb44-4cf3-91df-50cf13230d1e-kube-api-access-85f65\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s\" (UID: \"35fa0269-eb44-4cf3-91df-50cf13230d1e\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" Oct 30 01:32:16.992042 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.991980 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11caba41-1413-482d-8c53-03da9c396b26-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx\" (UID: \"11caba41-1413-482d-8c53-03da9c396b26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" Oct 30 01:32:16.992128 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.992108 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11caba41-1413-482d-8c53-03da9c396b26-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx\" (UID: \"11caba41-1413-482d-8c53-03da9c396b26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" Oct 30 01:32:16.992298 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.992274 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11caba41-1413-482d-8c53-03da9c396b26-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx\" (UID: \"11caba41-1413-482d-8c53-03da9c396b26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" Oct 30 01:32:16.992362 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.992279 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11caba41-1413-482d-8c53-03da9c396b26-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx\" (UID: \"11caba41-1413-482d-8c53-03da9c396b26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" Oct 30 01:32:16.993589 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.993565 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11caba41-1413-482d-8c53-03da9c396b26-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx\" (UID: \"11caba41-1413-482d-8c53-03da9c396b26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" Oct 30 01:32:16.993963 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:16.993946 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11caba41-1413-482d-8c53-03da9c396b26-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx\" (UID: \"11caba41-1413-482d-8c53-03da9c396b26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" Oct 30 01:32:17.011578 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.011545 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11caba41-1413-482d-8c53-03da9c396b26-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx\" (UID: \"11caba41-1413-482d-8c53-03da9c396b26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" Oct 30 01:32:17.011729 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.011710 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m56nm\" (UniqueName: \"kubernetes.io/projected/11caba41-1413-482d-8c53-03da9c396b26-kube-api-access-m56nm\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx\" (UID: \"11caba41-1413-482d-8c53-03da9c396b26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" Oct 30 01:32:17.092734 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.092638 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fa0269-eb44-4cf3-91df-50cf13230d1e-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s\" (UID: \"35fa0269-eb44-4cf3-91df-50cf13230d1e\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" Oct 30 01:32:17.092734 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.092690 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35fa0269-eb44-4cf3-91df-50cf13230d1e-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s\" (UID: \"35fa0269-eb44-4cf3-91df-50cf13230d1e\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" Oct 30 01:32:17.092734 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.092729 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35fa0269-eb44-4cf3-91df-50cf13230d1e-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s\" (UID: \"35fa0269-eb44-4cf3-91df-50cf13230d1e\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" Oct 30 01:32:17.093024 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.092814 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35fa0269-eb44-4cf3-91df-50cf13230d1e-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s\" (UID: \"35fa0269-eb44-4cf3-91df-50cf13230d1e\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" Oct 30 01:32:17.093024 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.092840 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35fa0269-eb44-4cf3-91df-50cf13230d1e-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s\" (UID: \"35fa0269-eb44-4cf3-91df-50cf13230d1e\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" Oct 30 01:32:17.093024 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.092910 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fa0269-eb44-4cf3-91df-50cf13230d1e-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s\" (UID: \"35fa0269-eb44-4cf3-91df-50cf13230d1e\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" Oct 30 01:32:17.093024 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.092981 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35fa0269-eb44-4cf3-91df-50cf13230d1e-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s\" (UID: \"35fa0269-eb44-4cf3-91df-50cf13230d1e\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" Oct 30 01:32:17.093224 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.093054 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35fa0269-eb44-4cf3-91df-50cf13230d1e-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s\" (UID: \"35fa0269-eb44-4cf3-91df-50cf13230d1e\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" Oct 30 01:32:17.093224 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.093127 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85f65\" (UniqueName: \"kubernetes.io/projected/35fa0269-eb44-4cf3-91df-50cf13230d1e-kube-api-access-85f65\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s\" (UID: \"35fa0269-eb44-4cf3-91df-50cf13230d1e\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" Oct 30 01:32:17.093224 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.093155 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fa0269-eb44-4cf3-91df-50cf13230d1e-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s\" (UID: \"35fa0269-eb44-4cf3-91df-50cf13230d1e\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" Oct 30 01:32:17.093365 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.093249 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35fa0269-eb44-4cf3-91df-50cf13230d1e-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s\" (UID: \"35fa0269-eb44-4cf3-91df-50cf13230d1e\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" Oct 30 01:32:17.093404 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.093389 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fa0269-eb44-4cf3-91df-50cf13230d1e-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s\" (UID: \"35fa0269-eb44-4cf3-91df-50cf13230d1e\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" Oct 30 01:32:17.093443 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.093408 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35fa0269-eb44-4cf3-91df-50cf13230d1e-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s\" (UID: \"35fa0269-eb44-4cf3-91df-50cf13230d1e\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" Oct 30 01:32:17.093815 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.093793 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35fa0269-eb44-4cf3-91df-50cf13230d1e-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s\" (UID: \"35fa0269-eb44-4cf3-91df-50cf13230d1e\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" Oct 30 01:32:17.095354 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.095331 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35fa0269-eb44-4cf3-91df-50cf13230d1e-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s\" (UID: \"35fa0269-eb44-4cf3-91df-50cf13230d1e\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" Oct 30 01:32:17.095642 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.095623 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35fa0269-eb44-4cf3-91df-50cf13230d1e-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s\" (UID: \"35fa0269-eb44-4cf3-91df-50cf13230d1e\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" Oct 30 01:32:17.110411 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.110384 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35fa0269-eb44-4cf3-91df-50cf13230d1e-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s\" (UID: \"35fa0269-eb44-4cf3-91df-50cf13230d1e\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" Oct 30 01:32:17.110532 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.110478 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85f65\" (UniqueName: \"kubernetes.io/projected/35fa0269-eb44-4cf3-91df-50cf13230d1e-kube-api-access-85f65\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s\" (UID: \"35fa0269-eb44-4cf3-91df-50cf13230d1e\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" Oct 30 01:32:17.174841 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.174803 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" Oct 30 01:32:17.220600 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.220568 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" Oct 30 01:32:17.502435 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.502394 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:32:17.502602 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.502488 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:32:17.502602 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.502531 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:32:17.505045 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.505017 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx"] Oct 30 01:32:17.521808 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.521777 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:17.549559 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.549534 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s"] Oct 30 01:32:17.557994 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:32:17.557970 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35fa0269_eb44_4cf3_91df_50cf13230d1e.slice/crio-2e24cf56fc89b78b626c95845be9966098f80d289f035262f717caff0910fac3 WatchSource:0}: Error finding container 2e24cf56fc89b78b626c95845be9966098f80d289f035262f717caff0910fac3: Status 404 returned error can't find the container with id 2e24cf56fc89b78b626c95845be9966098f80d289f035262f717caff0910fac3 Oct 30 01:32:17.560840 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.560807 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:32:17.560949 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.560887 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:32:17.560949 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.560940 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:32:17.739865 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.739828 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" event={"ID":"35fa0269-eb44-4cf3-91df-50cf13230d1e","Type":"ContainerStarted","Data":"03137dcd2352e94cc2a793cada615a905f3d7e1096fc10fe889c04e670280f5e"} Oct 30 01:32:17.740322 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.739882 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" event={"ID":"35fa0269-eb44-4cf3-91df-50cf13230d1e","Type":"ContainerStarted","Data":"2e24cf56fc89b78b626c95845be9966098f80d289f035262f717caff0910fac3"} Oct 30 01:32:17.742041 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.742010 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" event={"ID":"11caba41-1413-482d-8c53-03da9c396b26","Type":"ContainerStarted","Data":"aa426fc35ad5dda05e6838068e05d634ca4a6e7f4169f24f322e17e46a31581c"} Oct 30 01:32:17.742193 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.742062 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" event={"ID":"11caba41-1413-482d-8c53-03da9c396b26","Type":"ContainerStarted","Data":"8167e3193f2f17090e559215a91ac4b3b9d90f2933af92945d2ec5af47cf30e2"} Oct 30 01:32:17.744116 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.744088 6572 generic.go:356] "Generic (PLEG): container finished" podID="af93de6e-815c-4103-9437-3a5a45afecaf" containerID="7aac4fc1580136722c073a0d4c60f2735c33cc18b4807bd5ce7ae0126af15da4" exitCode=0 Oct 30 01:32:17.744231 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.744123 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0388b014-p6zjm" event={"ID":"af93de6e-815c-4103-9437-3a5a45afecaf","Type":"ContainerDied","Data":"7aac4fc1580136722c073a0d4c60f2735c33cc18b4807bd5ce7ae0126af15da4"} Oct 30 01:32:17.782132 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.782072 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" podStartSLOduration=1.782055074 podStartE2EDuration="1.782055074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:17.77761632 +0000 UTC m=+863.354041721" watchObservedRunningTime="2026-10-30 01:32:17.782055074 +0000 UTC m=+863.358480477" Oct 30 01:32:17.818268 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.818206 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" podStartSLOduration=1.818189314 podStartE2EDuration="1.818189314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:17.814847207 +0000 UTC m=+863.391272634" watchObservedRunningTime="2026-10-30 01:32:17.818189314 +0000 UTC m=+863.394614717" Oct 30 01:32:17.977496 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.977461 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld"] Oct 30 01:32:17.982181 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:17.982147 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" Oct 30 01:32:18.008334 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.007790 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld"] Oct 30 01:32:18.008593 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.008562 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37079845-e0dc-4eef-844e-eb7f15ae8cb3-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld\" (UID: \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" Oct 30 01:32:18.008862 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.008827 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37079845-e0dc-4eef-844e-eb7f15ae8cb3-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld\" (UID: \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" Oct 30 01:32:18.009016 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.008999 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37079845-e0dc-4eef-844e-eb7f15ae8cb3-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld\" (UID: \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" Oct 30 01:32:18.009170 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.009134 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmdsm\" (UniqueName: \"kubernetes.io/projected/37079845-e0dc-4eef-844e-eb7f15ae8cb3-kube-api-access-dmdsm\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld\" (UID: \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" Oct 30 01:32:18.009283 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.009262 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37079845-e0dc-4eef-844e-eb7f15ae8cb3-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld\" (UID: \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" Oct 30 01:32:18.009408 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.009393 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37079845-e0dc-4eef-844e-eb7f15ae8cb3-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld\" (UID: \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" Oct 30 01:32:18.009561 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.009537 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37079845-e0dc-4eef-844e-eb7f15ae8cb3-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld\" (UID: \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" Oct 30 01:32:18.009690 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.009670 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37079845-e0dc-4eef-844e-eb7f15ae8cb3-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld\" (UID: \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" Oct 30 01:32:18.009815 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.009801 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37079845-e0dc-4eef-844e-eb7f15ae8cb3-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld\" (UID: \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" Oct 30 01:32:18.079059 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.079030 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a3a2ac-openshift-default-dockercfg-mwwtj\"" Oct 30 01:32:18.119771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.113802 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37079845-e0dc-4eef-844e-eb7f15ae8cb3-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld\" (UID: \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" Oct 30 01:32:18.119771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.114003 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37079845-e0dc-4eef-844e-eb7f15ae8cb3-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld\" (UID: \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" Oct 30 01:32:18.119771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.114039 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37079845-e0dc-4eef-844e-eb7f15ae8cb3-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld\" (UID: \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" Oct 30 01:32:18.119771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.114106 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmdsm\" (UniqueName: \"kubernetes.io/projected/37079845-e0dc-4eef-844e-eb7f15ae8cb3-kube-api-access-dmdsm\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld\" (UID: \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" Oct 30 01:32:18.119771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.114153 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37079845-e0dc-4eef-844e-eb7f15ae8cb3-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld\" (UID: \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" Oct 30 01:32:18.119771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.114181 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37079845-e0dc-4eef-844e-eb7f15ae8cb3-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld\" (UID: \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" Oct 30 01:32:18.119771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.115276 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37079845-e0dc-4eef-844e-eb7f15ae8cb3-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld\" (UID: \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" Oct 30 01:32:18.119771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.115556 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37079845-e0dc-4eef-844e-eb7f15ae8cb3-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld\" (UID: \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" Oct 30 01:32:18.119771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.117179 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37079845-e0dc-4eef-844e-eb7f15ae8cb3-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld\" (UID: \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" Oct 30 01:32:18.132848 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.128915 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37079845-e0dc-4eef-844e-eb7f15ae8cb3-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld\" (UID: \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" Oct 30 01:32:18.132848 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.129006 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37079845-e0dc-4eef-844e-eb7f15ae8cb3-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld\" (UID: \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" Oct 30 01:32:18.132848 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.129044 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37079845-e0dc-4eef-844e-eb7f15ae8cb3-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld\" (UID: \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" Oct 30 01:32:18.132848 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.129598 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37079845-e0dc-4eef-844e-eb7f15ae8cb3-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld\" (UID: \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" Oct 30 01:32:18.132848 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.130222 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37079845-e0dc-4eef-844e-eb7f15ae8cb3-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld\" (UID: \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" Oct 30 01:32:18.132848 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.132506 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37079845-e0dc-4eef-844e-eb7f15ae8cb3-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld\" (UID: \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" Oct 30 01:32:18.136224 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.136187 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37079845-e0dc-4eef-844e-eb7f15ae8cb3-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld\" (UID: \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" Oct 30 01:32:18.137380 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.137358 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37079845-e0dc-4eef-844e-eb7f15ae8cb3-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld\" (UID: \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" Oct 30 01:32:18.138716 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.138694 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmdsm\" (UniqueName: \"kubernetes.io/projected/37079845-e0dc-4eef-844e-eb7f15ae8cb3-kube-api-access-dmdsm\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld\" (UID: \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" Oct 30 01:32:18.175339 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.175300 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" Oct 30 01:32:18.180769 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.177609 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 01:32:18.180769 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.177666 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" podUID="11caba41-1413-482d-8c53-03da9c396b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 01:32:18.225770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.224784 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" Oct 30 01:32:18.225770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.224956 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 01:32:18.225770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.225011 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" podUID="35fa0269-eb44-4cf3-91df-50cf13230d1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 01:32:18.310075 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.309978 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" Oct 30 01:32:18.681198 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.681168 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld"] Oct 30 01:32:18.683511 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:32:18.683487 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37079845_e0dc_4eef_844e_eb7f15ae8cb3.slice/crio-8b77b2c939a0be216d074754e52da2903c97ee4c3145d45895914b361671dd51 WatchSource:0}: Error finding container 8b77b2c939a0be216d074754e52da2903c97ee4c3145d45895914b361671dd51: Status 404 returned error can't find the container with id 8b77b2c939a0be216d074754e52da2903c97ee4c3145d45895914b361671dd51 Oct 30 01:32:18.685377 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.685344 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:32:18.685468 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.685403 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:32:18.685468 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.685431 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:32:18.767800 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:18.767721 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" event={"ID":"37079845-e0dc-4eef-844e-eb7f15ae8cb3","Type":"ContainerStarted","Data":"8b77b2c939a0be216d074754e52da2903c97ee4c3145d45895914b361671dd51"} Oct 30 01:32:19.072248 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:19.071844 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0388b014-p6zjm" Oct 30 01:32:19.171322 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:19.171282 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af93de6e-815c-4103-9437-3a5a45afecaf-kube-api-access-7kw5n\" (UniqueName: \"kubernetes.io/projected/af93de6e-815c-4103-9437-3a5a45afecaf-kube-api-access-7kw5n\") pod \"af93de6e-815c-4103-9437-3a5a45afecaf\" (UID: \"af93de6e-815c-4103-9437-3a5a45afecaf\") " Oct 30 01:32:19.171679 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:19.171652 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af93de6e-815c-4103-9437-3a5a45afecaf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af93de6e-815c-4103-9437-3a5a45afecaf-maas-api-openshift-ca\") pod \"af93de6e-815c-4103-9437-3a5a45afecaf\" (UID: \"af93de6e-815c-4103-9437-3a5a45afecaf\") " Oct 30 01:32:19.171854 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:19.171836 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af93de6e-815c-4103-9437-3a5a45afecaf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af93de6e-815c-4103-9437-3a5a45afecaf-maas-api-xks-ca\") pod \"af93de6e-815c-4103-9437-3a5a45afecaf\" (UID: \"af93de6e-815c-4103-9437-3a5a45afecaf\") " Oct 30 01:32:19.172644 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:19.172557 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af93de6e-815c-4103-9437-3a5a45afecaf-maas-api-openshift-ca" pod "af93de6e-815c-4103-9437-3a5a45afecaf" (UID: "af93de6e-815c-4103-9437-3a5a45afecaf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:32:19.175447 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:19.175422 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 01:32:19.175558 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:19.175472 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" podUID="11caba41-1413-482d-8c53-03da9c396b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 01:32:19.175619 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:19.175553 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af93de6e-815c-4103-9437-3a5a45afecaf-kube-api-access-7kw5n" pod "af93de6e-815c-4103-9437-3a5a45afecaf" (UID: "af93de6e-815c-4103-9437-3a5a45afecaf"). InnerVolumeSpecName "kube-api-access-7kw5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:19.177797 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:19.176741 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af93de6e-815c-4103-9437-3a5a45afecaf-maas-api-xks-ca" pod "af93de6e-815c-4103-9437-3a5a45afecaf" (UID: "af93de6e-815c-4103-9437-3a5a45afecaf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:32:19.256515 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:19.256479 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 01:32:19.256843 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:19.256794 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" podUID="35fa0269-eb44-4cf3-91df-50cf13230d1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 01:32:19.275476 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:19.275447 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kw5n\" (UniqueName: \"kubernetes.io/projected/af93de6e-815c-4103-9437-3a5a45afecaf-kube-api-access-7kw5n\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:19.275476 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:19.275474 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af93de6e-815c-4103-9437-3a5a45afecaf-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:19.275635 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:19.275484 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af93de6e-815c-4103-9437-3a5a45afecaf-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:19.799070 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:19.799035 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:19.802400 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:19.802366 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" event={"ID":"37079845-e0dc-4eef-844e-eb7f15ae8cb3","Type":"ContainerStarted","Data":"48c63fa656a13f2e80a255e1f267a12b3f31e982b429a760bb7d79b776894c8d"} Oct 30 01:32:19.804806 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:19.804789 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0388b014-p6zjm" Oct 30 01:32:19.805167 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:19.805054 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0388b014-p6zjm" event={"ID":"af93de6e-815c-4103-9437-3a5a45afecaf","Type":"ContainerDied","Data":"2540677d98c99249b0cc4aa7d1094bec984bcc82f77a68f22b8d5976f90a7467"} Oct 30 01:32:19.805247 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:19.805199 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2540677d98c99249b0cc4aa7d1094bec984bcc82f77a68f22b8d5976f90a7467" Oct 30 01:32:19.887276 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:19.887203 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" podStartSLOduration=2.887185159 podStartE2EDuration="2.887185159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:19.870226245 +0000 UTC m=+865.446651660" watchObservedRunningTime="2026-10-30 01:32:19.887185159 +0000 UTC m=+865.463610562" Oct 30 01:32:20.177788 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:20.176190 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 01:32:20.177788 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:20.176254 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" podUID="11caba41-1413-482d-8c53-03da9c396b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 01:32:20.222774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:20.222307 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 01:32:20.222774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:20.222366 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" podUID="35fa0269-eb44-4cf3-91df-50cf13230d1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 01:32:20.327780 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:20.326819 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" Oct 30 01:32:20.347165 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:20.332863 6572 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 01:32:20.347165 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:20.332922 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" podUID="37079845-e0dc-4eef-844e-eb7f15ae8cb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 01:32:21.175774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:21.175157 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 01:32:21.175774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:21.175217 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" podUID="11caba41-1413-482d-8c53-03da9c396b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 01:32:21.221770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:21.221016 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 01:32:21.221770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:21.221076 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" podUID="35fa0269-eb44-4cf3-91df-50cf13230d1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 01:32:21.331771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:21.331120 6572 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 01:32:21.331771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:21.331177 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" podUID="37079845-e0dc-4eef-844e-eb7f15ae8cb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 01:32:21.884773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:21.881189 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:22.182773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:22.176236 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 01:32:22.182773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:22.176297 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" podUID="11caba41-1413-482d-8c53-03da9c396b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 01:32:22.261870 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:22.229135 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 01:32:22.261870 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:22.229196 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" podUID="35fa0269-eb44-4cf3-91df-50cf13230d1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 01:32:22.328772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:22.323714 6572 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 01:32:22.351773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:22.335819 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" podUID="37079845-e0dc-4eef-844e-eb7f15ae8cb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 01:32:23.177936 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:23.177899 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 01:32:23.178226 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:23.178192 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" podUID="11caba41-1413-482d-8c53-03da9c396b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 01:32:23.224164 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:23.224126 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 01:32:23.224728 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:23.224681 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" podUID="35fa0269-eb44-4cf3-91df-50cf13230d1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 01:32:23.330773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:23.325241 6572 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 01:32:23.330773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:23.325350 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" podUID="37079845-e0dc-4eef-844e-eb7f15ae8cb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 01:32:23.462831 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:23.461969 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fea92145-wmbl9"] Oct 30 01:32:23.463007 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:23.462938 6572 cpu_manager.go:414] "containerMap: removing container" podUID="af93de6e-815c-4103-9437-3a5a45afecaf" containerName="revoke-keys" Oct 30 01:32:23.463007 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:23.462962 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af93de6e-815c-4103-9437-3a5a45afecaf" containerName="revoke-keys" Oct 30 01:32:23.463137 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:23.463116 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="af93de6e-815c-4103-9437-3a5a45afecaf" containerName="revoke-keys" Oct 30 01:32:23.488773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:23.488275 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fea92145-wmbl9"] Oct 30 01:32:23.488773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:23.488409 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea92145-wmbl9" Oct 30 01:32:23.517166 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:23.504030 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:32:23.517166 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:23.504121 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:32:23.517166 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:23.504242 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:32:23.541792 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:23.532071 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:32:23.605094 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:23.605036 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fddjs\" (UniqueName: \"kubernetes.io/projected/bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4-kube-api-access-fddjs\") pod \"maas-api-revoke-sub-fea92145-wmbl9\" (UID: \"bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea92145-wmbl9" Oct 30 01:32:23.605289 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:23.605198 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fea92145-wmbl9\" (UID: \"bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea92145-wmbl9" Oct 30 01:32:23.605366 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:23.605322 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fea92145-wmbl9\" (UID: \"bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea92145-wmbl9" Oct 30 01:32:23.718830 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:23.707516 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fea92145-wmbl9\" (UID: \"bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea92145-wmbl9" Oct 30 01:32:23.743005 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:23.742969 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fddjs\" (UniqueName: \"kubernetes.io/projected/bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4-kube-api-access-fddjs\") pod \"maas-api-revoke-sub-fea92145-wmbl9\" (UID: \"bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea92145-wmbl9" Oct 30 01:32:23.756951 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:23.756917 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fea92145-wmbl9\" (UID: \"bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea92145-wmbl9" Oct 30 01:32:23.767931 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:23.767890 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fea92145-wmbl9\" (UID: \"bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea92145-wmbl9" Oct 30 01:32:23.781266 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:23.779476 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fddjs\" (UniqueName: \"kubernetes.io/projected/bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4-kube-api-access-fddjs\") pod \"maas-api-revoke-sub-fea92145-wmbl9\" (UID: \"bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea92145-wmbl9" Oct 30 01:32:23.781266 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:23.779843 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fea92145-wmbl9\" (UID: \"bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea92145-wmbl9" Oct 30 01:32:23.828004 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:23.827968 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea92145-wmbl9" Oct 30 01:32:23.878674 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:23.878602 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:24.179804 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:24.178108 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 01:32:24.179804 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:24.178164 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" podUID="11caba41-1413-482d-8c53-03da9c396b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 01:32:24.225769 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:24.225066 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 01:32:24.225769 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:24.225127 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" podUID="35fa0269-eb44-4cf3-91df-50cf13230d1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 01:32:24.252664 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:24.252637 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fea92145-wmbl9"] Oct 30 01:32:24.345774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:24.342167 6572 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 01:32:24.345774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:24.342242 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" podUID="37079845-e0dc-4eef-844e-eb7f15ae8cb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 01:32:24.882966 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:24.882869 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0324c2b-6gpz8"] Oct 30 01:32:24.888127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:24.888102 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0324c2b-6gpz8" Oct 30 01:32:24.953918 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:24.953800 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0324c2b-6gpz8"] Oct 30 01:32:25.004654 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:25.004619 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cnns\" (UniqueName: \"kubernetes.io/projected/d0c5e0cf-8408-4a9f-b63f-521614b65232-kube-api-access-8cnns\") pod \"maas-api-revoke-sub-e0324c2b-6gpz8\" (UID: \"d0c5e0cf-8408-4a9f-b63f-521614b65232\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0324c2b-6gpz8" Oct 30 01:32:25.004862 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:25.004779 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c5e0cf-8408-4a9f-b63f-521614b65232-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0324c2b-6gpz8\" (UID: \"d0c5e0cf-8408-4a9f-b63f-521614b65232\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0324c2b-6gpz8" Oct 30 01:32:25.005087 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:25.005058 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c5e0cf-8408-4a9f-b63f-521614b65232-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0324c2b-6gpz8\" (UID: \"d0c5e0cf-8408-4a9f-b63f-521614b65232\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0324c2b-6gpz8" Oct 30 01:32:25.017775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:25.013057 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea92145-wmbl9" event={"ID":"bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4","Type":"ContainerStarted","Data":"ba961df5c71ab81c34f613c3a5acbc8795582ae6810febbf1f041c7d116a7a41"} Oct 30 01:32:25.017775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:25.013134 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea92145-wmbl9" event={"ID":"bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4","Type":"ContainerStarted","Data":"63a05a571426cff6a8a1e8c378fe9b0f86681321b43a56d0fe0fd423c443efe2"} Oct 30 01:32:25.078090 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:25.078021 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea92145-wmbl9" podStartSLOduration=2.07799998 podStartE2EDuration="2.07799998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:25.075640704 +0000 UTC m=+870.652066107" watchObservedRunningTime="2026-10-30 01:32:25.07799998 +0000 UTC m=+870.654425386" Oct 30 01:32:25.106812 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:25.106769 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c5e0cf-8408-4a9f-b63f-521614b65232-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0324c2b-6gpz8\" (UID: \"d0c5e0cf-8408-4a9f-b63f-521614b65232\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0324c2b-6gpz8" Oct 30 01:32:25.106994 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:25.106835 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cnns\" (UniqueName: \"kubernetes.io/projected/d0c5e0cf-8408-4a9f-b63f-521614b65232-kube-api-access-8cnns\") pod \"maas-api-revoke-sub-e0324c2b-6gpz8\" (UID: \"d0c5e0cf-8408-4a9f-b63f-521614b65232\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0324c2b-6gpz8" Oct 30 01:32:25.106994 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:25.106903 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c5e0cf-8408-4a9f-b63f-521614b65232-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0324c2b-6gpz8\" (UID: \"d0c5e0cf-8408-4a9f-b63f-521614b65232\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0324c2b-6gpz8" Oct 30 01:32:25.109206 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:25.109176 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c5e0cf-8408-4a9f-b63f-521614b65232-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0324c2b-6gpz8\" (UID: \"d0c5e0cf-8408-4a9f-b63f-521614b65232\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0324c2b-6gpz8" Oct 30 01:32:25.119196 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:25.117899 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c5e0cf-8408-4a9f-b63f-521614b65232-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0324c2b-6gpz8\" (UID: \"d0c5e0cf-8408-4a9f-b63f-521614b65232\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0324c2b-6gpz8" Oct 30 01:32:25.128614 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:25.128573 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cnns\" (UniqueName: \"kubernetes.io/projected/d0c5e0cf-8408-4a9f-b63f-521614b65232-kube-api-access-8cnns\") pod \"maas-api-revoke-sub-e0324c2b-6gpz8\" (UID: \"d0c5e0cf-8408-4a9f-b63f-521614b65232\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0324c2b-6gpz8" Oct 30 01:32:25.210527 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:25.210482 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0324c2b-6gpz8" Oct 30 01:32:25.346031 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:25.331182 6572 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 01:32:25.346031 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:25.331245 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" podUID="37079845-e0dc-4eef-844e-eb7f15ae8cb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 01:32:25.461729 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:25.460266 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" Oct 30 01:32:25.462443 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:25.462120 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" Oct 30 01:32:25.464792 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:25.463390 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" Oct 30 01:32:25.581717 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:25.581546 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0324c2b-6gpz8"] Oct 30 01:32:25.670183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:25.670145 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" Oct 30 01:32:25.671593 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:25.671568 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" Oct 30 01:32:25.672515 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:25.672485 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" Oct 30 01:32:25.803818 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:25.803712 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:26.009628 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:26.002589 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:26.031621 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:26.031587 6572 generic.go:356] "Generic (PLEG): container finished" podID="bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4" containerID="ba961df5c71ab81c34f613c3a5acbc8795582ae6810febbf1f041c7d116a7a41" exitCode=0 Oct 30 01:32:26.031890 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:26.031701 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea92145-wmbl9" event={"ID":"bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4","Type":"ContainerDied","Data":"ba961df5c71ab81c34f613c3a5acbc8795582ae6810febbf1f041c7d116a7a41"} Oct 30 01:32:26.033521 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:26.033476 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0324c2b-6gpz8" event={"ID":"d0c5e0cf-8408-4a9f-b63f-521614b65232","Type":"ContainerStarted","Data":"fc2de41f50a4a92b7fb5700aa81ae4e50e1ba1d217935c38d68ca52be10551f0"} Oct 30 01:32:26.327250 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:26.327218 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" Oct 30 01:32:26.327719 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:26.327695 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" Oct 30 01:32:26.328498 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:26.328481 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" Oct 30 01:32:27.042801 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:27.042742 6572 generic.go:356] "Generic (PLEG): container finished" podID="d0c5e0cf-8408-4a9f-b63f-521614b65232" containerID="b04f8ecdadc5fa7b8a73f3e5f9844dd43ef45d64be8b0aa859857c81c412b07c" exitCode=0 Oct 30 01:32:27.043263 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:27.042841 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0324c2b-6gpz8" event={"ID":"d0c5e0cf-8408-4a9f-b63f-521614b65232","Type":"ContainerDied","Data":"b04f8ecdadc5fa7b8a73f3e5f9844dd43ef45d64be8b0aa859857c81c412b07c"} Oct 30 01:32:27.185112 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:27.185089 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea92145-wmbl9" Oct 30 01:32:27.241387 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:27.241346 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4-kube-api-access-fddjs\" (UniqueName: \"kubernetes.io/projected/bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4-kube-api-access-fddjs\") pod \"bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4\" (UID: \"bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4\") " Oct 30 01:32:27.241548 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:27.241409 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4-maas-api-xks-ca\") pod \"bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4\" (UID: \"bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4\") " Oct 30 01:32:27.241548 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:27.241468 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4-maas-api-openshift-ca\") pod \"bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4\" (UID: \"bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4\") " Oct 30 01:32:27.242027 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:27.241996 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4-maas-api-openshift-ca" pod "bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4" (UID: "bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:32:27.242155 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:27.242110 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:27.243935 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:27.243906 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4-kube-api-access-fddjs" pod "bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4" (UID: "bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4"). InnerVolumeSpecName "kube-api-access-fddjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:27.244170 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:27.244136 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4-maas-api-xks-ca" pod "bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4" (UID: "bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:32:27.343428 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:27.343349 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fddjs\" (UniqueName: \"kubernetes.io/projected/bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4-kube-api-access-fddjs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:27.343428 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:27.343376 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:27.749425 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:27.749391 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:28.054098 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:28.054011 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea92145-wmbl9" Oct 30 01:32:28.054098 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:28.054039 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea92145-wmbl9" event={"ID":"bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4","Type":"ContainerDied","Data":"63a05a571426cff6a8a1e8c378fe9b0f86681321b43a56d0fe0fd423c443efe2"} Oct 30 01:32:28.054098 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:28.054089 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63a05a571426cff6a8a1e8c378fe9b0f86681321b43a56d0fe0fd423c443efe2" Oct 30 01:32:28.080490 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:32:28.080461 6572 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf8dc66_e1f8_41d4_9d9f_a113beaf2bb4.slice\": RecentStats: unable to find data in memory cache]" Oct 30 01:32:28.211719 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:28.211697 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0324c2b-6gpz8" Oct 30 01:32:28.254503 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:28.254471 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0c5e0cf-8408-4a9f-b63f-521614b65232-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c5e0cf-8408-4a9f-b63f-521614b65232-maas-api-xks-ca\") pod \"d0c5e0cf-8408-4a9f-b63f-521614b65232\" (UID: \"d0c5e0cf-8408-4a9f-b63f-521614b65232\") " Oct 30 01:32:28.254668 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:28.254526 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0c5e0cf-8408-4a9f-b63f-521614b65232-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c5e0cf-8408-4a9f-b63f-521614b65232-maas-api-openshift-ca\") pod \"d0c5e0cf-8408-4a9f-b63f-521614b65232\" (UID: \"d0c5e0cf-8408-4a9f-b63f-521614b65232\") " Oct 30 01:32:28.254740 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:28.254683 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c5e0cf-8408-4a9f-b63f-521614b65232-kube-api-access-8cnns\" (UniqueName: \"kubernetes.io/projected/d0c5e0cf-8408-4a9f-b63f-521614b65232-kube-api-access-8cnns\") pod \"d0c5e0cf-8408-4a9f-b63f-521614b65232\" (UID: \"d0c5e0cf-8408-4a9f-b63f-521614b65232\") " Oct 30 01:32:28.254926 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:28.254893 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c5e0cf-8408-4a9f-b63f-521614b65232-maas-api-openshift-ca" pod "d0c5e0cf-8408-4a9f-b63f-521614b65232" (UID: "d0c5e0cf-8408-4a9f-b63f-521614b65232"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:32:28.255179 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:28.255151 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c5e0cf-8408-4a9f-b63f-521614b65232-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:28.256731 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:28.256693 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0c5e0cf-8408-4a9f-b63f-521614b65232-maas-api-xks-ca" pod "d0c5e0cf-8408-4a9f-b63f-521614b65232" (UID: "d0c5e0cf-8408-4a9f-b63f-521614b65232"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:32:28.256971 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:28.256950 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c5e0cf-8408-4a9f-b63f-521614b65232-kube-api-access-8cnns" pod "d0c5e0cf-8408-4a9f-b63f-521614b65232" (UID: "d0c5e0cf-8408-4a9f-b63f-521614b65232"). InnerVolumeSpecName "kube-api-access-8cnns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:28.356161 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:28.356086 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cnns\" (UniqueName: \"kubernetes.io/projected/d0c5e0cf-8408-4a9f-b63f-521614b65232-kube-api-access-8cnns\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:28.356161 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:28.356114 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c5e0cf-8408-4a9f-b63f-521614b65232-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:29.063732 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:29.063705 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0324c2b-6gpz8" Oct 30 01:32:29.064122 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:29.063704 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0324c2b-6gpz8" event={"ID":"d0c5e0cf-8408-4a9f-b63f-521614b65232","Type":"ContainerDied","Data":"fc2de41f50a4a92b7fb5700aa81ae4e50e1ba1d217935c38d68ca52be10551f0"} Oct 30 01:32:29.064122 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:29.063826 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc2de41f50a4a92b7fb5700aa81ae4e50e1ba1d217935c38d68ca52be10551f0" Oct 30 01:32:29.543544 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:29.543508 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:29.577794 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:29.577734 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr"] Oct 30 01:32:29.578375 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:29.578329 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" podUID="51505b47-e483-40b3-bbac-a8aedafc4037" containerName="istio-proxy" containerID="cri-o://57103f7121857b7de41efb8727b9b01bad751fd002eb4fc8f4e3af12efaf7849" gracePeriod=30 Oct 30 01:32:29.581697 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:29.580628 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db"] Oct 30 01:32:29.581697 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:29.580861 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" podUID="432d7644-c50a-49bb-96d5-5f6f14aa1c6e" containerName="istio-proxy" containerID="cri-o://8dd1174b950c9b3bdd11c3d61eb81d0b38513b2be0dc776a8ef5c52dbb25ed66" gracePeriod=30 Oct 30 01:32:31.908686 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:31.906451 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg"] Oct 30 01:32:31.908686 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:31.907076 6572 cpu_manager.go:414] "containerMap: removing container" podUID="bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4" containerName="revoke-keys" Oct 30 01:32:31.908686 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:31.907094 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4" containerName="revoke-keys" Oct 30 01:32:31.908686 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:31.907112 6572 cpu_manager.go:414] "containerMap: removing container" podUID="d0c5e0cf-8408-4a9f-b63f-521614b65232" containerName="revoke-keys" Oct 30 01:32:31.908686 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:31.907117 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0c5e0cf-8408-4a9f-b63f-521614b65232" containerName="revoke-keys" Oct 30 01:32:31.908686 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:31.907227 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0c5e0cf-8408-4a9f-b63f-521614b65232" containerName="revoke-keys" Oct 30 01:32:31.908686 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:31.907244 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4" containerName="revoke-keys" Oct 30 01:32:31.912124 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:31.912095 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" Oct 30 01:32:31.948859 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:31.948780 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg"] Oct 30 01:32:31.949810 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:31.949739 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca3df60-4274-4031-8eef-04b1416d22b8-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg\" (UID: \"7ca3df60-4274-4031-8eef-04b1416d22b8\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" Oct 30 01:32:31.950128 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:31.950106 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca3df60-4274-4031-8eef-04b1416d22b8-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg\" (UID: \"7ca3df60-4274-4031-8eef-04b1416d22b8\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" Oct 30 01:32:31.950342 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:31.950321 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca3df60-4274-4031-8eef-04b1416d22b8-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg\" (UID: \"7ca3df60-4274-4031-8eef-04b1416d22b8\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" Oct 30 01:32:31.950533 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:31.950514 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca3df60-4274-4031-8eef-04b1416d22b8-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg\" (UID: \"7ca3df60-4274-4031-8eef-04b1416d22b8\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" Oct 30 01:32:31.950733 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:31.950714 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca3df60-4274-4031-8eef-04b1416d22b8-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg\" (UID: \"7ca3df60-4274-4031-8eef-04b1416d22b8\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" Oct 30 01:32:31.950943 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:31.950924 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca3df60-4274-4031-8eef-04b1416d22b8-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg\" (UID: \"7ca3df60-4274-4031-8eef-04b1416d22b8\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" Oct 30 01:32:31.951166 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:31.951146 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ca3df60-4274-4031-8eef-04b1416d22b8-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg\" (UID: \"7ca3df60-4274-4031-8eef-04b1416d22b8\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" Oct 30 01:32:31.951355 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:31.951336 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swv8h\" (UniqueName: \"kubernetes.io/projected/7ca3df60-4274-4031-8eef-04b1416d22b8-kube-api-access-swv8h\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg\" (UID: \"7ca3df60-4274-4031-8eef-04b1416d22b8\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" Oct 30 01:32:31.951592 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:31.951571 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca3df60-4274-4031-8eef-04b1416d22b8-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg\" (UID: \"7ca3df60-4274-4031-8eef-04b1416d22b8\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" Oct 30 01:32:32.053116 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:32.053081 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ca3df60-4274-4031-8eef-04b1416d22b8-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg\" (UID: \"7ca3df60-4274-4031-8eef-04b1416d22b8\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" Oct 30 01:32:32.053286 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:32.053124 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swv8h\" (UniqueName: \"kubernetes.io/projected/7ca3df60-4274-4031-8eef-04b1416d22b8-kube-api-access-swv8h\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg\" (UID: \"7ca3df60-4274-4031-8eef-04b1416d22b8\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" Oct 30 01:32:32.053286 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:32.053187 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca3df60-4274-4031-8eef-04b1416d22b8-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg\" (UID: \"7ca3df60-4274-4031-8eef-04b1416d22b8\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" Oct 30 01:32:32.053286 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:32.053256 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca3df60-4274-4031-8eef-04b1416d22b8-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg\" (UID: \"7ca3df60-4274-4031-8eef-04b1416d22b8\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" Oct 30 01:32:32.053430 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:32.053311 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca3df60-4274-4031-8eef-04b1416d22b8-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg\" (UID: \"7ca3df60-4274-4031-8eef-04b1416d22b8\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" Oct 30 01:32:32.053430 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:32.053336 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca3df60-4274-4031-8eef-04b1416d22b8-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg\" (UID: \"7ca3df60-4274-4031-8eef-04b1416d22b8\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" Oct 30 01:32:32.053430 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:32.053369 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca3df60-4274-4031-8eef-04b1416d22b8-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg\" (UID: \"7ca3df60-4274-4031-8eef-04b1416d22b8\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" Oct 30 01:32:32.053430 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:32.053392 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca3df60-4274-4031-8eef-04b1416d22b8-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg\" (UID: \"7ca3df60-4274-4031-8eef-04b1416d22b8\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" Oct 30 01:32:32.053623 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:32.053429 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca3df60-4274-4031-8eef-04b1416d22b8-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg\" (UID: \"7ca3df60-4274-4031-8eef-04b1416d22b8\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" Oct 30 01:32:32.053833 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:32.053807 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca3df60-4274-4031-8eef-04b1416d22b8-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg\" (UID: \"7ca3df60-4274-4031-8eef-04b1416d22b8\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" Oct 30 01:32:32.053946 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:32.053833 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca3df60-4274-4031-8eef-04b1416d22b8-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg\" (UID: \"7ca3df60-4274-4031-8eef-04b1416d22b8\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" Oct 30 01:32:32.053946 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:32.053921 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca3df60-4274-4031-8eef-04b1416d22b8-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg\" (UID: \"7ca3df60-4274-4031-8eef-04b1416d22b8\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" Oct 30 01:32:32.054045 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:32.053971 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca3df60-4274-4031-8eef-04b1416d22b8-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg\" (UID: \"7ca3df60-4274-4031-8eef-04b1416d22b8\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" Oct 30 01:32:32.054080 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:32.054056 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca3df60-4274-4031-8eef-04b1416d22b8-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg\" (UID: \"7ca3df60-4274-4031-8eef-04b1416d22b8\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" Oct 30 01:32:32.055509 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:32.055490 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca3df60-4274-4031-8eef-04b1416d22b8-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg\" (UID: \"7ca3df60-4274-4031-8eef-04b1416d22b8\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" Oct 30 01:32:32.055871 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:32.055853 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca3df60-4274-4031-8eef-04b1416d22b8-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg\" (UID: \"7ca3df60-4274-4031-8eef-04b1416d22b8\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" Oct 30 01:32:32.069406 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:32.069382 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ca3df60-4274-4031-8eef-04b1416d22b8-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg\" (UID: \"7ca3df60-4274-4031-8eef-04b1416d22b8\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" Oct 30 01:32:32.069576 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:32.069558 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swv8h\" (UniqueName: \"kubernetes.io/projected/7ca3df60-4274-4031-8eef-04b1416d22b8-kube-api-access-swv8h\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg\" (UID: \"7ca3df60-4274-4031-8eef-04b1416d22b8\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" Oct 30 01:32:32.272954 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:32.272917 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" Oct 30 01:32:32.585809 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:32.585710 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg"] Oct 30 01:32:32.586493 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:32.586462 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:32.589964 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:32:32.589936 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca3df60_4274_4031_8eef_04b1416d22b8.slice/crio-139f2dc64dd3756744dcc4c9e94d2dd6737481db6f7073d2c3e2be6dc1e23344 WatchSource:0}: Error finding container 139f2dc64dd3756744dcc4c9e94d2dd6737481db6f7073d2c3e2be6dc1e23344: Status 404 returned error can't find the container with id 139f2dc64dd3756744dcc4c9e94d2dd6737481db6f7073d2c3e2be6dc1e23344 Oct 30 01:32:32.592338 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:32.592307 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:32:32.592411 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:32.592374 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:32:32.592449 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:32.592423 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:32:33.126015 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:33.125979 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" event={"ID":"7ca3df60-4274-4031-8eef-04b1416d22b8","Type":"ContainerStarted","Data":"2ef6caa5f52cf812ae4ae9c15500c040ad3d27bb51f303739dbb9470555f3d46"} Oct 30 01:32:33.126371 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:33.126036 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" event={"ID":"7ca3df60-4274-4031-8eef-04b1416d22b8","Type":"ContainerStarted","Data":"139f2dc64dd3756744dcc4c9e94d2dd6737481db6f7073d2c3e2be6dc1e23344"} Oct 30 01:32:33.273151 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:33.273111 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" Oct 30 01:32:33.274920 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:33.274889 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 01:32:33.275038 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:33.274938 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" podUID="7ca3df60-4274-4031-8eef-04b1416d22b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 01:32:34.274178 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.274145 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 01:32:34.274615 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.274200 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" podUID="7ca3df60-4274-4031-8eef-04b1416d22b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 01:32:34.418612 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.418577 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:34.942801 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.942770 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" Oct 30 01:32:34.956534 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.956513 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" Oct 30 01:32:34.996781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.992840 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-istio-token\" (UniqueName: \"kubernetes.io/projected/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-istio-token\") pod \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\" (UID: \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\") " Oct 30 01:32:34.996781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.992889 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-istiod-ca-cert\") pod \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\" (UID: \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\") " Oct 30 01:32:34.996781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.992954 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51505b47-e483-40b3-bbac-a8aedafc4037-istio-token\" (UniqueName: \"kubernetes.io/projected/51505b47-e483-40b3-bbac-a8aedafc4037-istio-token\") pod \"51505b47-e483-40b3-bbac-a8aedafc4037\" (UID: \"51505b47-e483-40b3-bbac-a8aedafc4037\") " Oct 30 01:32:34.996781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.993007 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51505b47-e483-40b3-bbac-a8aedafc4037-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51505b47-e483-40b3-bbac-a8aedafc4037-istio-envoy\") pod \"51505b47-e483-40b3-bbac-a8aedafc4037\" (UID: \"51505b47-e483-40b3-bbac-a8aedafc4037\") " Oct 30 01:32:34.996781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.993044 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51505b47-e483-40b3-bbac-a8aedafc4037-kube-api-access-hwtxt\" (UniqueName: \"kubernetes.io/projected/51505b47-e483-40b3-bbac-a8aedafc4037-kube-api-access-hwtxt\") pod \"51505b47-e483-40b3-bbac-a8aedafc4037\" (UID: \"51505b47-e483-40b3-bbac-a8aedafc4037\") " Oct 30 01:32:34.996781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.993104 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51505b47-e483-40b3-bbac-a8aedafc4037-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51505b47-e483-40b3-bbac-a8aedafc4037-istiod-ca-cert\") pod \"51505b47-e483-40b3-bbac-a8aedafc4037\" (UID: \"51505b47-e483-40b3-bbac-a8aedafc4037\") " Oct 30 01:32:34.996781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.993134 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-istio-envoy\") pod \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\" (UID: \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\") " Oct 30 01:32:34.996781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.993187 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51505b47-e483-40b3-bbac-a8aedafc4037-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51505b47-e483-40b3-bbac-a8aedafc4037-istio-podinfo\") pod \"51505b47-e483-40b3-bbac-a8aedafc4037\" (UID: \"51505b47-e483-40b3-bbac-a8aedafc4037\") " Oct 30 01:32:34.996781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.993229 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-istio-data\") pod \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\" (UID: \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\") " Oct 30 01:32:34.996781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.993273 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-credential-socket\") pod \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\" (UID: \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\") " Oct 30 01:32:34.996781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.993310 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-workload-socket\") pod \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\" (UID: \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\") " Oct 30 01:32:34.996781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.993340 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-kube-api-access-xbdr4\" (UniqueName: \"kubernetes.io/projected/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-kube-api-access-xbdr4\") pod \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\" (UID: \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\") " Oct 30 01:32:34.996781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.993378 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51505b47-e483-40b3-bbac-a8aedafc4037-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51505b47-e483-40b3-bbac-a8aedafc4037-workload-socket\") pod \"51505b47-e483-40b3-bbac-a8aedafc4037\" (UID: \"51505b47-e483-40b3-bbac-a8aedafc4037\") " Oct 30 01:32:34.996781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.993437 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51505b47-e483-40b3-bbac-a8aedafc4037-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51505b47-e483-40b3-bbac-a8aedafc4037-workload-certs\") pod \"51505b47-e483-40b3-bbac-a8aedafc4037\" (UID: \"51505b47-e483-40b3-bbac-a8aedafc4037\") " Oct 30 01:32:34.996781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.993524 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51505b47-e483-40b3-bbac-a8aedafc4037-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51505b47-e483-40b3-bbac-a8aedafc4037-istio-data\") pod \"51505b47-e483-40b3-bbac-a8aedafc4037\" (UID: \"51505b47-e483-40b3-bbac-a8aedafc4037\") " Oct 30 01:32:34.996781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.993566 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-istio-podinfo\") pod \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\" (UID: \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\") " Oct 30 01:32:34.996781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.993595 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51505b47-e483-40b3-bbac-a8aedafc4037-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51505b47-e483-40b3-bbac-a8aedafc4037-credential-socket\") pod \"51505b47-e483-40b3-bbac-a8aedafc4037\" (UID: \"51505b47-e483-40b3-bbac-a8aedafc4037\") " Oct 30 01:32:34.996781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.993627 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-workload-certs\") pod \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\" (UID: \"432d7644-c50a-49bb-96d5-5f6f14aa1c6e\") " Oct 30 01:32:34.996781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.994536 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-istio-data" pod "432d7644-c50a-49bb-96d5-5f6f14aa1c6e" (UID: "432d7644-c50a-49bb-96d5-5f6f14aa1c6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:34.996781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.994711 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51505b47-e483-40b3-bbac-a8aedafc4037-workload-socket" pod "51505b47-e483-40b3-bbac-a8aedafc4037" (UID: "51505b47-e483-40b3-bbac-a8aedafc4037"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:34.996781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.994943 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-credential-socket" pod "432d7644-c50a-49bb-96d5-5f6f14aa1c6e" (UID: "432d7644-c50a-49bb-96d5-5f6f14aa1c6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:34.996781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.995041 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51505b47-e483-40b3-bbac-a8aedafc4037-workload-certs" pod "51505b47-e483-40b3-bbac-a8aedafc4037" (UID: "51505b47-e483-40b3-bbac-a8aedafc4037"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:34.996781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.995054 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-workload-socket" pod "432d7644-c50a-49bb-96d5-5f6f14aa1c6e" (UID: "432d7644-c50a-49bb-96d5-5f6f14aa1c6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:34.996781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.995071 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51505b47-e483-40b3-bbac-a8aedafc4037-istiod-ca-cert" pod "51505b47-e483-40b3-bbac-a8aedafc4037" (UID: "51505b47-e483-40b3-bbac-a8aedafc4037"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:32:34.996781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.995087 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51505b47-e483-40b3-bbac-a8aedafc4037-istio-data" pod "51505b47-e483-40b3-bbac-a8aedafc4037" (UID: "51505b47-e483-40b3-bbac-a8aedafc4037"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:34.996781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.995151 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51505b47-e483-40b3-bbac-a8aedafc4037-credential-socket" pod "51505b47-e483-40b3-bbac-a8aedafc4037" (UID: "51505b47-e483-40b3-bbac-a8aedafc4037"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:34.996781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.995342 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-istiod-ca-cert" pod "432d7644-c50a-49bb-96d5-5f6f14aa1c6e" (UID: "432d7644-c50a-49bb-96d5-5f6f14aa1c6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:32:34.996781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.996225 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-workload-certs" pod "432d7644-c50a-49bb-96d5-5f6f14aa1c6e" (UID: "432d7644-c50a-49bb-96d5-5f6f14aa1c6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:34.998072 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.998012 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51505b47-e483-40b3-bbac-a8aedafc4037-istio-envoy" pod "51505b47-e483-40b3-bbac-a8aedafc4037" (UID: "51505b47-e483-40b3-bbac-a8aedafc4037"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:34.999158 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.999130 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-kube-api-access-xbdr4" pod "432d7644-c50a-49bb-96d5-5f6f14aa1c6e" (UID: "432d7644-c50a-49bb-96d5-5f6f14aa1c6e"). InnerVolumeSpecName "kube-api-access-xbdr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:34.999601 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.999578 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-istio-token" pod "432d7644-c50a-49bb-96d5-5f6f14aa1c6e" (UID: "432d7644-c50a-49bb-96d5-5f6f14aa1c6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:34.999763 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.999671 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51505b47-e483-40b3-bbac-a8aedafc4037-kube-api-access-hwtxt" pod "51505b47-e483-40b3-bbac-a8aedafc4037" (UID: "51505b47-e483-40b3-bbac-a8aedafc4037"). InnerVolumeSpecName "kube-api-access-hwtxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:34.999763 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.999702 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51505b47-e483-40b3-bbac-a8aedafc4037-istio-token" pod "51505b47-e483-40b3-bbac-a8aedafc4037" (UID: "51505b47-e483-40b3-bbac-a8aedafc4037"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:34.999872 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.999741 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51505b47-e483-40b3-bbac-a8aedafc4037-istio-podinfo" pod "51505b47-e483-40b3-bbac-a8aedafc4037" (UID: "51505b47-e483-40b3-bbac-a8aedafc4037"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:32:35.000013 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:34.999979 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-istio-podinfo" pod "432d7644-c50a-49bb-96d5-5f6f14aa1c6e" (UID: "432d7644-c50a-49bb-96d5-5f6f14aa1c6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:32:35.001035 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.000990 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" podStartSLOduration=4.000976234 podStartE2EDuration="4.000976234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:33.183620639 +0000 UTC m=+878.760046039" watchObservedRunningTime="2026-10-30 01:32:35.000976234 +0000 UTC m=+880.577401636" Oct 30 01:32:35.001676 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.001646 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-istio-envoy" pod "432d7644-c50a-49bb-96d5-5f6f14aa1c6e" (UID: "432d7644-c50a-49bb-96d5-5f6f14aa1c6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:35.095093 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.095060 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:35.095093 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.095090 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:35.095093 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.095101 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbdr4\" (UniqueName: \"kubernetes.io/projected/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-kube-api-access-xbdr4\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:35.095309 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.095112 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51505b47-e483-40b3-bbac-a8aedafc4037-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:35.095309 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.095121 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51505b47-e483-40b3-bbac-a8aedafc4037-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:35.095309 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.095130 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51505b47-e483-40b3-bbac-a8aedafc4037-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:35.095309 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.095137 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:35.095309 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.095145 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51505b47-e483-40b3-bbac-a8aedafc4037-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:35.095309 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.095158 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:35.095309 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.095170 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:35.095309 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.095178 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:35.095309 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.095185 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51505b47-e483-40b3-bbac-a8aedafc4037-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:35.095309 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.095192 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51505b47-e483-40b3-bbac-a8aedafc4037-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:35.095309 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.095199 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwtxt\" (UniqueName: \"kubernetes.io/projected/51505b47-e483-40b3-bbac-a8aedafc4037-kube-api-access-hwtxt\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:35.095309 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.095207 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51505b47-e483-40b3-bbac-a8aedafc4037-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:35.095309 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.095215 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:35.095309 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.095222 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51505b47-e483-40b3-bbac-a8aedafc4037-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:35.095309 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.095232 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432d7644-c50a-49bb-96d5-5f6f14aa1c6e-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:35.148272 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.148185 6572 generic.go:356] "Generic (PLEG): container finished" podID="432d7644-c50a-49bb-96d5-5f6f14aa1c6e" containerID="8dd1174b950c9b3bdd11c3d61eb81d0b38513b2be0dc776a8ef5c52dbb25ed66" exitCode=0 Oct 30 01:32:35.148272 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.148248 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" Oct 30 01:32:35.148508 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.148266 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" event={"ID":"432d7644-c50a-49bb-96d5-5f6f14aa1c6e","Type":"ContainerDied","Data":"8dd1174b950c9b3bdd11c3d61eb81d0b38513b2be0dc776a8ef5c52dbb25ed66"} Oct 30 01:32:35.148508 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.148323 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db" event={"ID":"432d7644-c50a-49bb-96d5-5f6f14aa1c6e","Type":"ContainerDied","Data":"bfe612738d604fe453162e9aab01ae1f41430359b9d8936dbce9a6e5a2c9aa38"} Oct 30 01:32:35.148508 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.148348 6572 scope.go:122] "RemoveContainer" containerID="8dd1174b950c9b3bdd11c3d61eb81d0b38513b2be0dc776a8ef5c52dbb25ed66" Oct 30 01:32:35.150147 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.150117 6572 generic.go:356] "Generic (PLEG): container finished" podID="51505b47-e483-40b3-bbac-a8aedafc4037" containerID="57103f7121857b7de41efb8727b9b01bad751fd002eb4fc8f4e3af12efaf7849" exitCode=0 Oct 30 01:32:35.150264 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.150158 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" Oct 30 01:32:35.150264 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.150212 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" event={"ID":"51505b47-e483-40b3-bbac-a8aedafc4037","Type":"ContainerDied","Data":"57103f7121857b7de41efb8727b9b01bad751fd002eb4fc8f4e3af12efaf7849"} Oct 30 01:32:35.150375 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.150282 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr" event={"ID":"51505b47-e483-40b3-bbac-a8aedafc4037","Type":"ContainerDied","Data":"88a4e01ae82a3232ce40384da8366d070f84d705b7b560ab6b79071e0f673d42"} Oct 30 01:32:35.160991 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.160973 6572 scope.go:122] "RemoveContainer" containerID="8dd1174b950c9b3bdd11c3d61eb81d0b38513b2be0dc776a8ef5c52dbb25ed66" Oct 30 01:32:35.161790 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:32:35.161767 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dd1174b950c9b3bdd11c3d61eb81d0b38513b2be0dc776a8ef5c52dbb25ed66\": container with ID starting with 8dd1174b950c9b3bdd11c3d61eb81d0b38513b2be0dc776a8ef5c52dbb25ed66 not found: ID does not exist" containerID="8dd1174b950c9b3bdd11c3d61eb81d0b38513b2be0dc776a8ef5c52dbb25ed66" Oct 30 01:32:35.161915 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.161897 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dd1174b950c9b3bdd11c3d61eb81d0b38513b2be0dc776a8ef5c52dbb25ed66"} err="failed to get container status \"8dd1174b950c9b3bdd11c3d61eb81d0b38513b2be0dc776a8ef5c52dbb25ed66\": rpc error: code = NotFound desc = could not find container \"8dd1174b950c9b3bdd11c3d61eb81d0b38513b2be0dc776a8ef5c52dbb25ed66\": container with ID starting with 8dd1174b950c9b3bdd11c3d61eb81d0b38513b2be0dc776a8ef5c52dbb25ed66 not found: ID does not exist" Oct 30 01:32:35.161986 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.161978 6572 scope.go:122] "RemoveContainer" containerID="57103f7121857b7de41efb8727b9b01bad751fd002eb4fc8f4e3af12efaf7849" Oct 30 01:32:35.170734 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.170719 6572 scope.go:122] "RemoveContainer" containerID="57103f7121857b7de41efb8727b9b01bad751fd002eb4fc8f4e3af12efaf7849" Oct 30 01:32:35.170989 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:32:35.170972 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57103f7121857b7de41efb8727b9b01bad751fd002eb4fc8f4e3af12efaf7849\": container with ID starting with 57103f7121857b7de41efb8727b9b01bad751fd002eb4fc8f4e3af12efaf7849 not found: ID does not exist" containerID="57103f7121857b7de41efb8727b9b01bad751fd002eb4fc8f4e3af12efaf7849" Oct 30 01:32:35.171043 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.170996 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57103f7121857b7de41efb8727b9b01bad751fd002eb4fc8f4e3af12efaf7849"} err="failed to get container status \"57103f7121857b7de41efb8727b9b01bad751fd002eb4fc8f4e3af12efaf7849\": rpc error: code = NotFound desc = could not find container \"57103f7121857b7de41efb8727b9b01bad751fd002eb4fc8f4e3af12efaf7849\": container with ID starting with 57103f7121857b7de41efb8727b9b01bad751fd002eb4fc8f4e3af12efaf7849 not found: ID does not exist" Oct 30 01:32:35.232420 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.232386 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db"] Oct 30 01:32:35.236474 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.236452 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cf2db"] Oct 30 01:32:35.247686 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.247659 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr"] Oct 30 01:32:35.251607 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.251588 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-57zjr"] Oct 30 01:32:35.274032 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.274002 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 01:32:35.274171 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.274055 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" podUID="7ca3df60-4274-4031-8eef-04b1416d22b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 01:32:35.299033 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.298972 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c764d992-xhd7f"] Oct 30 01:32:35.299607 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.299588 6572 cpu_manager.go:414] "containerMap: removing container" podUID="432d7644-c50a-49bb-96d5-5f6f14aa1c6e" containerName="istio-proxy" Oct 30 01:32:35.299607 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.299608 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="432d7644-c50a-49bb-96d5-5f6f14aa1c6e" containerName="istio-proxy" Oct 30 01:32:35.299700 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.299630 6572 cpu_manager.go:414] "containerMap: removing container" podUID="51505b47-e483-40b3-bbac-a8aedafc4037" containerName="istio-proxy" Oct 30 01:32:35.299700 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.299635 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51505b47-e483-40b3-bbac-a8aedafc4037" containerName="istio-proxy" Oct 30 01:32:35.299798 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.299739 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="432d7644-c50a-49bb-96d5-5f6f14aa1c6e" containerName="istio-proxy" Oct 30 01:32:35.299798 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.299770 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="51505b47-e483-40b3-bbac-a8aedafc4037" containerName="istio-proxy" Oct 30 01:32:35.304582 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.304563 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c764d992-xhd7f" Oct 30 01:32:35.331334 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.331293 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c764d992-xhd7f"] Oct 30 01:32:35.331979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.331954 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:32:35.332827 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.332807 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:32:35.333055 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.333039 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:32:35.333275 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.333258 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:32:35.399199 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.399105 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c764d992-xhd7f\" (UID: \"82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c764d992-xhd7f" Oct 30 01:32:35.399351 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.399250 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c764d992-xhd7f\" (UID: \"82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c764d992-xhd7f" Oct 30 01:32:35.399351 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.399315 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dk2c\" (UniqueName: \"kubernetes.io/projected/82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7-kube-api-access-8dk2c\") pod \"maas-api-revoke-sub-c764d992-xhd7f\" (UID: \"82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c764d992-xhd7f" Oct 30 01:32:35.458434 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.458404 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:35.500821 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.500793 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c764d992-xhd7f\" (UID: \"82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c764d992-xhd7f" Oct 30 01:32:35.500981 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.500858 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dk2c\" (UniqueName: \"kubernetes.io/projected/82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7-kube-api-access-8dk2c\") pod \"maas-api-revoke-sub-c764d992-xhd7f\" (UID: \"82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c764d992-xhd7f" Oct 30 01:32:35.501039 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.501012 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c764d992-xhd7f\" (UID: \"82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c764d992-xhd7f" Oct 30 01:32:35.501604 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.501581 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c764d992-xhd7f\" (UID: \"82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c764d992-xhd7f" Oct 30 01:32:35.503162 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.503144 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c764d992-xhd7f\" (UID: \"82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c764d992-xhd7f" Oct 30 01:32:35.532427 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.532399 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dk2c\" (UniqueName: \"kubernetes.io/projected/82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7-kube-api-access-8dk2c\") pod \"maas-api-revoke-sub-c764d992-xhd7f\" (UID: \"82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c764d992-xhd7f" Oct 30 01:32:35.650230 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.650140 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c764d992-xhd7f" Oct 30 01:32:35.795355 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.795330 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc"] Oct 30 01:32:35.801563 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.801533 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:32:35.828190 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.827429 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-278fe7d5-openshift-default-dockercfg-gstz5\"" Oct 30 01:32:35.843161 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.843116 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc"] Oct 30 01:32:35.930836 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.905869 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7578c814-103e-433a-b58e-a173bc4d3a97-istio-token\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc\" (UID: \"7578c814-103e-433a-b58e-a173bc4d3a97\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:32:35.930979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.930886 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7578c814-103e-433a-b58e-a173bc4d3a97-istio-data\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc\" (UID: \"7578c814-103e-433a-b58e-a173bc4d3a97\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:32:35.931036 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.931015 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tbnc\" (UniqueName: \"kubernetes.io/projected/7578c814-103e-433a-b58e-a173bc4d3a97-kube-api-access-9tbnc\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc\" (UID: \"7578c814-103e-433a-b58e-a173bc4d3a97\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:32:35.931113 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.931099 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7578c814-103e-433a-b58e-a173bc4d3a97-workload-socket\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc\" (UID: \"7578c814-103e-433a-b58e-a173bc4d3a97\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:32:35.931208 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.931197 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7578c814-103e-433a-b58e-a173bc4d3a97-workload-certs\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc\" (UID: \"7578c814-103e-433a-b58e-a173bc4d3a97\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:32:35.931246 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.931231 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7578c814-103e-433a-b58e-a173bc4d3a97-istio-podinfo\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc\" (UID: \"7578c814-103e-433a-b58e-a173bc4d3a97\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:32:35.931292 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.931249 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7578c814-103e-433a-b58e-a173bc4d3a97-istiod-ca-cert\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc\" (UID: \"7578c814-103e-433a-b58e-a173bc4d3a97\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:32:35.931359 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.931341 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7578c814-103e-433a-b58e-a173bc4d3a97-istio-envoy\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc\" (UID: \"7578c814-103e-433a-b58e-a173bc4d3a97\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:32:35.931393 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.931372 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7578c814-103e-433a-b58e-a173bc4d3a97-credential-socket\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc\" (UID: \"7578c814-103e-433a-b58e-a173bc4d3a97\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:32:35.960092 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.960049 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:35.976916 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:35.976847 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c764d992-xhd7f"] Oct 30 01:32:35.978163 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:32:35.978138 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82e4a162_d1e0_4d0a_a2ef_b34d22f52ca7.slice/crio-772789ec00d0f85086c2c529c49e1f167b075b7246da18eda517ea82341b6f99 WatchSource:0}: Error finding container 772789ec00d0f85086c2c529c49e1f167b075b7246da18eda517ea82341b6f99: Status 404 returned error can't find the container with id 772789ec00d0f85086c2c529c49e1f167b075b7246da18eda517ea82341b6f99 Oct 30 01:32:36.046473 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:36.043144 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7578c814-103e-433a-b58e-a173bc4d3a97-istio-data\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc\" (UID: \"7578c814-103e-433a-b58e-a173bc4d3a97\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:32:36.046473 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:36.043260 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tbnc\" (UniqueName: \"kubernetes.io/projected/7578c814-103e-433a-b58e-a173bc4d3a97-kube-api-access-9tbnc\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc\" (UID: \"7578c814-103e-433a-b58e-a173bc4d3a97\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:32:36.046473 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:36.043337 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7578c814-103e-433a-b58e-a173bc4d3a97-workload-socket\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc\" (UID: \"7578c814-103e-433a-b58e-a173bc4d3a97\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:32:36.046473 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:36.043416 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7578c814-103e-433a-b58e-a173bc4d3a97-workload-certs\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc\" (UID: \"7578c814-103e-433a-b58e-a173bc4d3a97\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:32:36.046473 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:36.043452 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7578c814-103e-433a-b58e-a173bc4d3a97-istio-podinfo\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc\" (UID: \"7578c814-103e-433a-b58e-a173bc4d3a97\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:32:36.046473 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:36.043480 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7578c814-103e-433a-b58e-a173bc4d3a97-istiod-ca-cert\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc\" (UID: \"7578c814-103e-433a-b58e-a173bc4d3a97\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:32:36.046473 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:36.043529 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7578c814-103e-433a-b58e-a173bc4d3a97-istio-envoy\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc\" (UID: \"7578c814-103e-433a-b58e-a173bc4d3a97\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:32:36.046473 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:36.043562 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7578c814-103e-433a-b58e-a173bc4d3a97-credential-socket\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc\" (UID: \"7578c814-103e-433a-b58e-a173bc4d3a97\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:32:36.046473 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:36.043639 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7578c814-103e-433a-b58e-a173bc4d3a97-istio-token\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc\" (UID: \"7578c814-103e-433a-b58e-a173bc4d3a97\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:32:36.046473 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:36.045210 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7578c814-103e-433a-b58e-a173bc4d3a97-istio-data\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc\" (UID: \"7578c814-103e-433a-b58e-a173bc4d3a97\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:32:36.046473 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:36.045786 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7578c814-103e-433a-b58e-a173bc4d3a97-workload-socket\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc\" (UID: \"7578c814-103e-433a-b58e-a173bc4d3a97\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:32:36.046473 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:36.046007 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7578c814-103e-433a-b58e-a173bc4d3a97-workload-certs\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc\" (UID: \"7578c814-103e-433a-b58e-a173bc4d3a97\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:32:36.053959 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:36.051875 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7578c814-103e-433a-b58e-a173bc4d3a97-istiod-ca-cert\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc\" (UID: \"7578c814-103e-433a-b58e-a173bc4d3a97\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:32:36.053959 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:36.052428 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7578c814-103e-433a-b58e-a173bc4d3a97-credential-socket\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc\" (UID: \"7578c814-103e-433a-b58e-a173bc4d3a97\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:32:36.053959 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:36.053615 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7578c814-103e-433a-b58e-a173bc4d3a97-istio-podinfo\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc\" (UID: \"7578c814-103e-433a-b58e-a173bc4d3a97\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:32:36.053959 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:36.053683 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7578c814-103e-433a-b58e-a173bc4d3a97-istio-envoy\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc\" (UID: \"7578c814-103e-433a-b58e-a173bc4d3a97\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:32:36.070428 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:36.065033 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tbnc\" (UniqueName: \"kubernetes.io/projected/7578c814-103e-433a-b58e-a173bc4d3a97-kube-api-access-9tbnc\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc\" (UID: \"7578c814-103e-433a-b58e-a173bc4d3a97\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:32:36.070428 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:36.069371 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7578c814-103e-433a-b58e-a173bc4d3a97-istio-token\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc\" (UID: \"7578c814-103e-433a-b58e-a173bc4d3a97\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:32:36.158355 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:36.158323 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:32:36.160699 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:36.160670 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:36.167914 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:36.167885 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c764d992-xhd7f" event={"ID":"82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7","Type":"ContainerStarted","Data":"772789ec00d0f85086c2c529c49e1f167b075b7246da18eda517ea82341b6f99"} Oct 30 01:32:36.277607 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:36.277572 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 01:32:36.277880 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:36.277855 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" podUID="7ca3df60-4274-4031-8eef-04b1416d22b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 01:32:36.532173 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:32:36.532123 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7578c814_103e_433a_b58e_a173bc4d3a97.slice/crio-c1cbc306eacd8ccd9ca8fac254b2099e37685eae9d6a312ab52afd58921f294c WatchSource:0}: Error finding container c1cbc306eacd8ccd9ca8fac254b2099e37685eae9d6a312ab52afd58921f294c: Status 404 returned error can't find the container with id c1cbc306eacd8ccd9ca8fac254b2099e37685eae9d6a312ab52afd58921f294c Oct 30 01:32:36.533811 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:36.533783 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc"] Oct 30 01:32:36.536360 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:36.536326 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:32:36.536450 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:36.536397 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:32:36.536450 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:36.536443 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:32:36.542288 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:36.542263 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="432d7644-c50a-49bb-96d5-5f6f14aa1c6e" path="/var/lib/kubelet/pods/432d7644-c50a-49bb-96d5-5f6f14aa1c6e/volumes" Oct 30 01:32:36.542713 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:36.542698 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51505b47-e483-40b3-bbac-a8aedafc4037" path="/var/lib/kubelet/pods/51505b47-e483-40b3-bbac-a8aedafc4037/volumes" Oct 30 01:32:37.182816 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:37.182779 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c764d992-xhd7f" event={"ID":"82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7","Type":"ContainerDied","Data":"d9d67de54334c98147ea6074fe07340d2ce5cfea6df7c26a6fdc9932de4e6c3c"} Oct 30 01:32:37.182952 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:37.182779 6572 generic.go:356] "Generic (PLEG): container finished" podID="82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7" containerID="d9d67de54334c98147ea6074fe07340d2ce5cfea6df7c26a6fdc9932de4e6c3c" exitCode=0 Oct 30 01:32:37.184757 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:37.184719 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" event={"ID":"7578c814-103e-433a-b58e-a173bc4d3a97","Type":"ContainerStarted","Data":"e5327fe09cd3167ce229da0ac44ae5f511e694c63e0318f3f7439835e5508e60"} Oct 30 01:32:37.184842 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:37.184797 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" event={"ID":"7578c814-103e-433a-b58e-a173bc4d3a97","Type":"ContainerStarted","Data":"c1cbc306eacd8ccd9ca8fac254b2099e37685eae9d6a312ab52afd58921f294c"} Oct 30 01:32:37.270424 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:37.270380 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" podStartSLOduration=2.270365178 podStartE2EDuration="2.270365178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:37.268177708 +0000 UTC m=+882.844603110" watchObservedRunningTime="2026-10-30 01:32:37.270365178 +0000 UTC m=+882.846790579" Oct 30 01:32:37.273958 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:37.273935 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 01:32:37.274050 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:37.273993 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" podUID="7ca3df60-4274-4031-8eef-04b1416d22b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 01:32:37.584756 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:37.584700 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:37.931768 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:37.931443 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:38.158488 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:38.158452 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:32:38.165568 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:38.165542 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:32:38.191640 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:38.191574 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:32:38.192476 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:38.192452 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:32:38.278963 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:38.273291 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 01:32:38.278963 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:38.273343 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" podUID="7ca3df60-4274-4031-8eef-04b1416d22b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 01:32:38.386671 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:38.386646 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c764d992-xhd7f" Oct 30 01:32:38.478289 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:38.478205 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7-maas-api-xks-ca\") pod \"82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7\" (UID: \"82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7\") " Oct 30 01:32:38.478443 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:38.478425 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7-maas-api-openshift-ca\") pod \"82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7\" (UID: \"82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7\") " Oct 30 01:32:38.478502 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:38.478461 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7-kube-api-access-8dk2c\" (UniqueName: \"kubernetes.io/projected/82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7-kube-api-access-8dk2c\") pod \"82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7\" (UID: \"82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7\") " Oct 30 01:32:38.478876 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:38.478840 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7-maas-api-openshift-ca" pod "82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7" (UID: "82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:32:38.480523 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:38.480497 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7-maas-api-xks-ca" pod "82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7" (UID: "82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:32:38.480654 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:38.480638 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7-kube-api-access-8dk2c" pod "82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7" (UID: "82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7"). InnerVolumeSpecName "kube-api-access-8dk2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:38.580306 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:38.580274 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:38.580306 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:38.580302 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:38.580490 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:38.580314 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dk2c\" (UniqueName: \"kubernetes.io/projected/82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7-kube-api-access-8dk2c\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:39.201477 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:39.201447 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c764d992-xhd7f" Oct 30 01:32:39.201980 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:39.201495 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c764d992-xhd7f" event={"ID":"82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7","Type":"ContainerDied","Data":"772789ec00d0f85086c2c529c49e1f167b075b7246da18eda517ea82341b6f99"} Oct 30 01:32:39.201980 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:39.201541 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="772789ec00d0f85086c2c529c49e1f167b075b7246da18eda517ea82341b6f99" Oct 30 01:32:39.274011 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:39.273983 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 01:32:39.274157 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:39.274037 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" podUID="7ca3df60-4274-4031-8eef-04b1416d22b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 01:32:39.814604 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:39.814571 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n"] Oct 30 01:32:39.815485 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:39.815462 6572 cpu_manager.go:414] "containerMap: removing container" podUID="82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7" containerName="revoke-keys" Oct 30 01:32:39.815566 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:39.815489 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7" containerName="revoke-keys" Oct 30 01:32:39.815670 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:39.815655 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7" containerName="revoke-keys" Oct 30 01:32:39.819964 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:39.819937 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" Oct 30 01:32:39.847823 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:39.847797 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n"] Oct 30 01:32:39.901571 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:39.901530 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:40.005788 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.005727 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7243362-b004-4393-8a15-8ea62c257b6d-istiod-ca-cert\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n\" (UID: \"d7243362-b004-4393-8a15-8ea62c257b6d\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" Oct 30 01:32:40.006050 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.006012 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7243362-b004-4393-8a15-8ea62c257b6d-istio-data\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n\" (UID: \"d7243362-b004-4393-8a15-8ea62c257b6d\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" Oct 30 01:32:40.006202 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.006054 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7243362-b004-4393-8a15-8ea62c257b6d-istio-token\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n\" (UID: \"d7243362-b004-4393-8a15-8ea62c257b6d\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" Oct 30 01:32:40.006202 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.006136 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7243362-b004-4393-8a15-8ea62c257b6d-workload-certs\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n\" (UID: \"d7243362-b004-4393-8a15-8ea62c257b6d\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" Oct 30 01:32:40.006202 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.006182 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z46nz\" (UniqueName: \"kubernetes.io/projected/d7243362-b004-4393-8a15-8ea62c257b6d-kube-api-access-z46nz\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n\" (UID: \"d7243362-b004-4393-8a15-8ea62c257b6d\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" Oct 30 01:32:40.006405 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.006357 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7243362-b004-4393-8a15-8ea62c257b6d-istio-envoy\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n\" (UID: \"d7243362-b004-4393-8a15-8ea62c257b6d\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" Oct 30 01:32:40.006463 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.006436 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7243362-b004-4393-8a15-8ea62c257b6d-istio-podinfo\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n\" (UID: \"d7243362-b004-4393-8a15-8ea62c257b6d\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" Oct 30 01:32:40.006559 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.006541 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7243362-b004-4393-8a15-8ea62c257b6d-workload-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n\" (UID: \"d7243362-b004-4393-8a15-8ea62c257b6d\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" Oct 30 01:32:40.006619 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.006604 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7243362-b004-4393-8a15-8ea62c257b6d-credential-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n\" (UID: \"d7243362-b004-4393-8a15-8ea62c257b6d\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" Oct 30 01:32:40.101528 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.101433 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:40.107874 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.107843 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7243362-b004-4393-8a15-8ea62c257b6d-istio-envoy\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n\" (UID: \"d7243362-b004-4393-8a15-8ea62c257b6d\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" Oct 30 01:32:40.108052 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.107882 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7243362-b004-4393-8a15-8ea62c257b6d-istio-podinfo\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n\" (UID: \"d7243362-b004-4393-8a15-8ea62c257b6d\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" Oct 30 01:32:40.108052 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.107944 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7243362-b004-4393-8a15-8ea62c257b6d-workload-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n\" (UID: \"d7243362-b004-4393-8a15-8ea62c257b6d\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" Oct 30 01:32:40.108052 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.107977 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7243362-b004-4393-8a15-8ea62c257b6d-credential-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n\" (UID: \"d7243362-b004-4393-8a15-8ea62c257b6d\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" Oct 30 01:32:40.108244 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.108055 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7243362-b004-4393-8a15-8ea62c257b6d-istiod-ca-cert\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n\" (UID: \"d7243362-b004-4393-8a15-8ea62c257b6d\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" Oct 30 01:32:40.108244 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.108080 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7243362-b004-4393-8a15-8ea62c257b6d-istio-data\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n\" (UID: \"d7243362-b004-4393-8a15-8ea62c257b6d\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" Oct 30 01:32:40.108244 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.108104 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7243362-b004-4393-8a15-8ea62c257b6d-istio-token\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n\" (UID: \"d7243362-b004-4393-8a15-8ea62c257b6d\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" Oct 30 01:32:40.108244 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.108174 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7243362-b004-4393-8a15-8ea62c257b6d-workload-certs\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n\" (UID: \"d7243362-b004-4393-8a15-8ea62c257b6d\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" Oct 30 01:32:40.108244 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.108217 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z46nz\" (UniqueName: \"kubernetes.io/projected/d7243362-b004-4393-8a15-8ea62c257b6d-kube-api-access-z46nz\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n\" (UID: \"d7243362-b004-4393-8a15-8ea62c257b6d\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" Oct 30 01:32:40.108453 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.108324 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7243362-b004-4393-8a15-8ea62c257b6d-workload-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n\" (UID: \"d7243362-b004-4393-8a15-8ea62c257b6d\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" Oct 30 01:32:40.108797 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.108739 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7243362-b004-4393-8a15-8ea62c257b6d-istio-data\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n\" (UID: \"d7243362-b004-4393-8a15-8ea62c257b6d\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" Oct 30 01:32:40.108900 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.108797 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7243362-b004-4393-8a15-8ea62c257b6d-credential-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n\" (UID: \"d7243362-b004-4393-8a15-8ea62c257b6d\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" Oct 30 01:32:40.109053 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.109030 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7243362-b004-4393-8a15-8ea62c257b6d-workload-certs\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n\" (UID: \"d7243362-b004-4393-8a15-8ea62c257b6d\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" Oct 30 01:32:40.109270 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.109239 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7243362-b004-4393-8a15-8ea62c257b6d-istiod-ca-cert\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n\" (UID: \"d7243362-b004-4393-8a15-8ea62c257b6d\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" Oct 30 01:32:40.110247 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.110227 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7243362-b004-4393-8a15-8ea62c257b6d-istio-envoy\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n\" (UID: \"d7243362-b004-4393-8a15-8ea62c257b6d\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" Oct 30 01:32:40.110912 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.110894 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7243362-b004-4393-8a15-8ea62c257b6d-istio-podinfo\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n\" (UID: \"d7243362-b004-4393-8a15-8ea62c257b6d\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" Oct 30 01:32:40.128681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.128655 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7243362-b004-4393-8a15-8ea62c257b6d-istio-token\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n\" (UID: \"d7243362-b004-4393-8a15-8ea62c257b6d\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" Oct 30 01:32:40.129229 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.129208 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z46nz\" (UniqueName: \"kubernetes.io/projected/d7243362-b004-4393-8a15-8ea62c257b6d-kube-api-access-z46nz\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n\" (UID: \"d7243362-b004-4393-8a15-8ea62c257b6d\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" Oct 30 01:32:40.136954 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.136932 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" Oct 30 01:32:40.278359 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.278327 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" Oct 30 01:32:40.278727 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.278706 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" Oct 30 01:32:40.279398 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.279377 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" Oct 30 01:32:40.459237 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.459205 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n"] Oct 30 01:32:40.459489 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:32:40.459468 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7243362_b004_4393_8a15_8ea62c257b6d.slice/crio-4568434182b4f0f0b184def829c99824f102ca54acf30a3b2861897404e00234 WatchSource:0}: Error finding container 4568434182b4f0f0b184def829c99824f102ca54acf30a3b2861897404e00234: Status 404 returned error can't find the container with id 4568434182b4f0f0b184def829c99824f102ca54acf30a3b2861897404e00234 Oct 30 01:32:40.462104 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.462068 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:32:40.462211 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.462136 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:32:40.462211 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.462167 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:32:40.565489 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.565461 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fb82088-mn4fn"] Oct 30 01:32:40.569439 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.569412 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fb82088-mn4fn" Oct 30 01:32:40.582475 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.582451 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:32:40.582587 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.582530 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:32:40.589055 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.589030 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fb82088-mn4fn"] Oct 30 01:32:40.638456 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.638428 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:32:40.638605 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.638551 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:32:40.716723 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.716635 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fb82088-mn4fn\" (UID: \"4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fb82088-mn4fn" Oct 30 01:32:40.716723 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.716672 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fb82088-mn4fn\" (UID: \"4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fb82088-mn4fn" Oct 30 01:32:40.716939 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.716848 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgdd7\" (UniqueName: \"kubernetes.io/projected/4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5-kube-api-access-wgdd7\") pod \"maas-api-revoke-sub-6fb82088-mn4fn\" (UID: \"4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fb82088-mn4fn" Oct 30 01:32:40.729316 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.729290 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf"] Oct 30 01:32:40.733469 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.733449 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" Oct 30 01:32:40.761007 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.760977 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf"] Oct 30 01:32:40.818458 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.818425 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgdd7\" (UniqueName: \"kubernetes.io/projected/4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5-kube-api-access-wgdd7\") pod \"maas-api-revoke-sub-6fb82088-mn4fn\" (UID: \"4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fb82088-mn4fn" Oct 30 01:32:40.818613 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.818601 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fb82088-mn4fn\" (UID: \"4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fb82088-mn4fn" Oct 30 01:32:40.818690 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.818635 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fb82088-mn4fn\" (UID: \"4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fb82088-mn4fn" Oct 30 01:32:40.819253 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.819232 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fb82088-mn4fn\" (UID: \"4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fb82088-mn4fn" Oct 30 01:32:40.821508 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.821486 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fb82088-mn4fn\" (UID: \"4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fb82088-mn4fn" Oct 30 01:32:40.835170 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.835147 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgdd7\" (UniqueName: \"kubernetes.io/projected/4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5-kube-api-access-wgdd7\") pod \"maas-api-revoke-sub-6fb82088-mn4fn\" (UID: \"4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fb82088-mn4fn" Oct 30 01:32:40.884977 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.884948 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fb82088-mn4fn" Oct 30 01:32:40.919662 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.919630 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b50de5b-2782-45a9-8b90-44e3508013f8-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf\" (UID: \"6b50de5b-2782-45a9-8b90-44e3508013f8\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" Oct 30 01:32:40.919822 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.919673 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b50de5b-2782-45a9-8b90-44e3508013f8-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf\" (UID: \"6b50de5b-2782-45a9-8b90-44e3508013f8\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" Oct 30 01:32:40.919822 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.919718 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b50de5b-2782-45a9-8b90-44e3508013f8-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf\" (UID: \"6b50de5b-2782-45a9-8b90-44e3508013f8\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" Oct 30 01:32:40.919933 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.919902 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b50de5b-2782-45a9-8b90-44e3508013f8-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf\" (UID: \"6b50de5b-2782-45a9-8b90-44e3508013f8\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" Oct 30 01:32:40.920009 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.919994 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b50de5b-2782-45a9-8b90-44e3508013f8-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf\" (UID: \"6b50de5b-2782-45a9-8b90-44e3508013f8\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" Oct 30 01:32:40.920050 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.920036 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b50de5b-2782-45a9-8b90-44e3508013f8-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf\" (UID: \"6b50de5b-2782-45a9-8b90-44e3508013f8\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" Oct 30 01:32:40.920113 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.920098 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qss2\" (UniqueName: \"kubernetes.io/projected/6b50de5b-2782-45a9-8b90-44e3508013f8-kube-api-access-2qss2\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf\" (UID: \"6b50de5b-2782-45a9-8b90-44e3508013f8\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" Oct 30 01:32:40.920168 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.920153 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b50de5b-2782-45a9-8b90-44e3508013f8-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf\" (UID: \"6b50de5b-2782-45a9-8b90-44e3508013f8\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" Oct 30 01:32:40.920249 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:40.920234 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b50de5b-2782-45a9-8b90-44e3508013f8-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf\" (UID: \"6b50de5b-2782-45a9-8b90-44e3508013f8\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" Oct 30 01:32:41.038738 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.038638 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b50de5b-2782-45a9-8b90-44e3508013f8-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf\" (UID: \"6b50de5b-2782-45a9-8b90-44e3508013f8\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" Oct 30 01:32:41.038939 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.038784 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b50de5b-2782-45a9-8b90-44e3508013f8-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf\" (UID: \"6b50de5b-2782-45a9-8b90-44e3508013f8\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" Oct 30 01:32:41.038939 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.038838 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b50de5b-2782-45a9-8b90-44e3508013f8-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf\" (UID: \"6b50de5b-2782-45a9-8b90-44e3508013f8\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" Oct 30 01:32:41.038939 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.038878 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b50de5b-2782-45a9-8b90-44e3508013f8-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf\" (UID: \"6b50de5b-2782-45a9-8b90-44e3508013f8\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" Oct 30 01:32:41.038939 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.038932 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b50de5b-2782-45a9-8b90-44e3508013f8-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf\" (UID: \"6b50de5b-2782-45a9-8b90-44e3508013f8\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" Oct 30 01:32:41.039071 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.039004 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b50de5b-2782-45a9-8b90-44e3508013f8-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf\" (UID: \"6b50de5b-2782-45a9-8b90-44e3508013f8\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" Oct 30 01:32:41.039106 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.039074 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b50de5b-2782-45a9-8b90-44e3508013f8-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf\" (UID: \"6b50de5b-2782-45a9-8b90-44e3508013f8\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" Oct 30 01:32:41.039138 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.039112 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b50de5b-2782-45a9-8b90-44e3508013f8-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf\" (UID: \"6b50de5b-2782-45a9-8b90-44e3508013f8\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" Oct 30 01:32:41.039200 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.039178 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qss2\" (UniqueName: \"kubernetes.io/projected/6b50de5b-2782-45a9-8b90-44e3508013f8-kube-api-access-2qss2\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf\" (UID: \"6b50de5b-2782-45a9-8b90-44e3508013f8\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" Oct 30 01:32:41.039629 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.039599 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b50de5b-2782-45a9-8b90-44e3508013f8-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf\" (UID: \"6b50de5b-2782-45a9-8b90-44e3508013f8\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" Oct 30 01:32:41.039738 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.039720 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b50de5b-2782-45a9-8b90-44e3508013f8-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf\" (UID: \"6b50de5b-2782-45a9-8b90-44e3508013f8\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" Oct 30 01:32:41.039854 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.039614 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b50de5b-2782-45a9-8b90-44e3508013f8-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf\" (UID: \"6b50de5b-2782-45a9-8b90-44e3508013f8\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" Oct 30 01:32:41.040215 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.040171 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b50de5b-2782-45a9-8b90-44e3508013f8-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf\" (UID: \"6b50de5b-2782-45a9-8b90-44e3508013f8\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" Oct 30 01:32:41.040876 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.040821 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b50de5b-2782-45a9-8b90-44e3508013f8-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf\" (UID: \"6b50de5b-2782-45a9-8b90-44e3508013f8\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" Oct 30 01:32:41.047770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.043084 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b50de5b-2782-45a9-8b90-44e3508013f8-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf\" (UID: \"6b50de5b-2782-45a9-8b90-44e3508013f8\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" Oct 30 01:32:41.047770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.043546 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b50de5b-2782-45a9-8b90-44e3508013f8-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf\" (UID: \"6b50de5b-2782-45a9-8b90-44e3508013f8\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" Oct 30 01:32:41.057239 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.056861 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b50de5b-2782-45a9-8b90-44e3508013f8-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf\" (UID: \"6b50de5b-2782-45a9-8b90-44e3508013f8\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" Oct 30 01:32:41.057542 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.057520 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qss2\" (UniqueName: \"kubernetes.io/projected/6b50de5b-2782-45a9-8b90-44e3508013f8-kube-api-access-2qss2\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf\" (UID: \"6b50de5b-2782-45a9-8b90-44e3508013f8\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" Oct 30 01:32:41.183885 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.183854 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fb82088-mn4fn"] Oct 30 01:32:41.184420 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:32:41.184392 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4107c0e6_8225_4b5d_9b47_5bcef0b5e7f5.slice/crio-c10519f08fe7d9de04d62263dc83227bda4520d0045c605af2d8f3dc0a676c76 WatchSource:0}: Error finding container c10519f08fe7d9de04d62263dc83227bda4520d0045c605af2d8f3dc0a676c76: Status 404 returned error can't find the container with id c10519f08fe7d9de04d62263dc83227bda4520d0045c605af2d8f3dc0a676c76 Oct 30 01:32:41.236813 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.236778 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" event={"ID":"d7243362-b004-4393-8a15-8ea62c257b6d","Type":"ContainerStarted","Data":"9f775cffb5e8cb2436d7c7f18b3586d3a5d4ce1fc4e66ebed075b12a97a6b65c"} Oct 30 01:32:41.236977 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.236844 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" event={"ID":"d7243362-b004-4393-8a15-8ea62c257b6d","Type":"ContainerStarted","Data":"4568434182b4f0f0b184def829c99824f102ca54acf30a3b2861897404e00234"} Oct 30 01:32:41.238368 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.238330 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fb82088-mn4fn" event={"ID":"4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5","Type":"ContainerStarted","Data":"c10519f08fe7d9de04d62263dc83227bda4520d0045c605af2d8f3dc0a676c76"} Oct 30 01:32:41.276605 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.276561 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" podStartSLOduration=2.276548853 podStartE2EDuration="2.276548853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:41.271912395 +0000 UTC m=+886.848337813" watchObservedRunningTime="2026-10-30 01:32:41.276548853 +0000 UTC m=+886.852974255" Oct 30 01:32:41.349035 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.348949 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" Oct 30 01:32:41.582222 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.582184 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:41.642549 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.642516 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-9t2d5"] Oct 30 01:32:41.648929 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.648902 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-9t2d5"] Oct 30 01:32:41.649072 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.649061 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-9t2d5" Oct 30 01:32:41.666830 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.666761 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-278fe7d5\"" Oct 30 01:32:41.667433 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.667394 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-278fe7d5-dockercfg-jlrw2\"" Oct 30 01:32:41.671771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.671713 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf"] Oct 30 01:32:41.672780 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:32:41.672728 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b50de5b_2782_45a9_8b90_44e3508013f8.slice/crio-edffcd27ccb93b205779b1698ca8dcfa7dedfe14e68e2f5f4ef8bdf94da0b4ec WatchSource:0}: Error finding container edffcd27ccb93b205779b1698ca8dcfa7dedfe14e68e2f5f4ef8bdf94da0b4ec: Status 404 returned error can't find the container with id edffcd27ccb93b205779b1698ca8dcfa7dedfe14e68e2f5f4ef8bdf94da0b4ec Oct 30 01:32:41.676458 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.676431 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:32:41.676544 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.676488 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:32:41.676544 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.676517 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:32:41.758183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.748982 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g9nb\" (UniqueName: \"kubernetes.io/projected/485b6169-3cf8-4111-be91-8050f32439d7-kube-api-access-7g9nb\") pod \"payload-processing-e2e-mt-278fe7d5-7b956c9596-9t2d5\" (UID: \"485b6169-3cf8-4111-be91-8050f32439d7\") " pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-9t2d5" Oct 30 01:32:41.758183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.749123 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/485b6169-3cf8-4111-be91-8050f32439d7-plugins-config-volume\") pod \"payload-processing-e2e-mt-278fe7d5-7b956c9596-9t2d5\" (UID: \"485b6169-3cf8-4111-be91-8050f32439d7\") " pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-9t2d5" Oct 30 01:32:41.775081 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.774884 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:41.850978 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.850925 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g9nb\" (UniqueName: \"kubernetes.io/projected/485b6169-3cf8-4111-be91-8050f32439d7-kube-api-access-7g9nb\") pod \"payload-processing-e2e-mt-278fe7d5-7b956c9596-9t2d5\" (UID: \"485b6169-3cf8-4111-be91-8050f32439d7\") " pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-9t2d5" Oct 30 01:32:41.851150 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.851083 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/485b6169-3cf8-4111-be91-8050f32439d7-plugins-config-volume\") pod \"payload-processing-e2e-mt-278fe7d5-7b956c9596-9t2d5\" (UID: \"485b6169-3cf8-4111-be91-8050f32439d7\") " pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-9t2d5" Oct 30 01:32:41.851857 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.851830 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/485b6169-3cf8-4111-be91-8050f32439d7-plugins-config-volume\") pod \"payload-processing-e2e-mt-278fe7d5-7b956c9596-9t2d5\" (UID: \"485b6169-3cf8-4111-be91-8050f32439d7\") " pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-9t2d5" Oct 30 01:32:41.868268 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.868239 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g9nb\" (UniqueName: \"kubernetes.io/projected/485b6169-3cf8-4111-be91-8050f32439d7-kube-api-access-7g9nb\") pod \"payload-processing-e2e-mt-278fe7d5-7b956c9596-9t2d5\" (UID: \"485b6169-3cf8-4111-be91-8050f32439d7\") " pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-9t2d5" Oct 30 01:32:41.992771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.970192 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-9t2d5" Oct 30 01:32:41.992771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:41.981971 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:42.145678 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:42.145446 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" Oct 30 01:32:42.147708 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:42.147686 6572 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 01:32:42.147814 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:42.147737 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" podUID="d7243362-b004-4393-8a15-8ea62c257b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 01:32:42.248540 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:42.248447 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fb82088-mn4fn" event={"ID":"4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5","Type":"ContainerStarted","Data":"6ef1a1ed0f8fdd112a16251180621502c0898fbe6128b2e8a293c28980412a3e"} Oct 30 01:32:42.250136 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:42.250038 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" event={"ID":"6b50de5b-2782-45a9-8b90-44e3508013f8","Type":"ContainerStarted","Data":"15b6f1a4c55c7c82910ce7775a4ab791b96e3f519793e8c5b6fb46c53c74ba9f"} Oct 30 01:32:42.250136 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:42.250096 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" event={"ID":"6b50de5b-2782-45a9-8b90-44e3508013f8","Type":"ContainerStarted","Data":"edffcd27ccb93b205779b1698ca8dcfa7dedfe14e68e2f5f4ef8bdf94da0b4ec"} Oct 30 01:32:42.288666 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:42.288606 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fb82088-mn4fn" podStartSLOduration=2.288587626 podStartE2EDuration="2.288587626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:42.282993823 +0000 UTC m=+887.859419225" watchObservedRunningTime="2026-10-30 01:32:42.288587626 +0000 UTC m=+887.865013029" Oct 30 01:32:42.333772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:42.328352 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" podStartSLOduration=2.328332585 podStartE2EDuration="2.328332585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:42.32431464 +0000 UTC m=+887.900740043" watchObservedRunningTime="2026-10-30 01:32:42.328332585 +0000 UTC m=+887.904757985" Oct 30 01:32:42.350111 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:42.349813 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" Oct 30 01:32:42.351659 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:42.351125 6572 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 01:32:42.351659 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:42.351324 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" podUID="6b50de5b-2782-45a9-8b90-44e3508013f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 01:32:42.397882 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:42.397854 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-9t2d5"] Oct 30 01:32:43.044769 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:43.037735 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5af953b7-ctpc5"] Oct 30 01:32:43.063713 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:43.063683 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af953b7-ctpc5" Oct 30 01:32:43.064566 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:43.063580 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5af953b7-ctpc5"] Oct 30 01:32:43.071967 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:43.071933 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46b4ef76-9c38-436f-a193-dc11782fcfaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5af953b7-ctpc5\" (UID: \"46b4ef76-9c38-436f-a193-dc11782fcfaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af953b7-ctpc5" Oct 30 01:32:43.072139 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:43.072113 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46b4ef76-9c38-436f-a193-dc11782fcfaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5af953b7-ctpc5\" (UID: \"46b4ef76-9c38-436f-a193-dc11782fcfaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af953b7-ctpc5" Oct 30 01:32:43.072424 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:43.072401 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g76fl\" (UniqueName: \"kubernetes.io/projected/46b4ef76-9c38-436f-a193-dc11782fcfaf-kube-api-access-g76fl\") pod \"maas-api-revoke-sub-5af953b7-ctpc5\" (UID: \"46b4ef76-9c38-436f-a193-dc11782fcfaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af953b7-ctpc5" Oct 30 01:32:43.148771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:43.147384 6572 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 01:32:43.148771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:43.147444 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" podUID="d7243362-b004-4393-8a15-8ea62c257b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 01:32:43.174359 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:43.174324 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46b4ef76-9c38-436f-a193-dc11782fcfaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5af953b7-ctpc5\" (UID: \"46b4ef76-9c38-436f-a193-dc11782fcfaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af953b7-ctpc5" Oct 30 01:32:43.174543 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:43.174434 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46b4ef76-9c38-436f-a193-dc11782fcfaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5af953b7-ctpc5\" (UID: \"46b4ef76-9c38-436f-a193-dc11782fcfaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af953b7-ctpc5" Oct 30 01:32:43.174543 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:43.174530 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g76fl\" (UniqueName: \"kubernetes.io/projected/46b4ef76-9c38-436f-a193-dc11782fcfaf-kube-api-access-g76fl\") pod \"maas-api-revoke-sub-5af953b7-ctpc5\" (UID: \"46b4ef76-9c38-436f-a193-dc11782fcfaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af953b7-ctpc5" Oct 30 01:32:43.176802 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:43.176134 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46b4ef76-9c38-436f-a193-dc11782fcfaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5af953b7-ctpc5\" (UID: \"46b4ef76-9c38-436f-a193-dc11782fcfaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af953b7-ctpc5" Oct 30 01:32:43.178709 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:43.178678 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46b4ef76-9c38-436f-a193-dc11782fcfaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5af953b7-ctpc5\" (UID: \"46b4ef76-9c38-436f-a193-dc11782fcfaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af953b7-ctpc5" Oct 30 01:32:43.193200 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:43.193172 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g76fl\" (UniqueName: \"kubernetes.io/projected/46b4ef76-9c38-436f-a193-dc11782fcfaf-kube-api-access-g76fl\") pod \"maas-api-revoke-sub-5af953b7-ctpc5\" (UID: \"46b4ef76-9c38-436f-a193-dc11782fcfaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af953b7-ctpc5" Oct 30 01:32:43.269880 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:43.269837 6572 generic.go:356] "Generic (PLEG): container finished" podID="4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5" containerID="6ef1a1ed0f8fdd112a16251180621502c0898fbe6128b2e8a293c28980412a3e" exitCode=0 Oct 30 01:32:43.270061 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:43.269974 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fb82088-mn4fn" event={"ID":"4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5","Type":"ContainerDied","Data":"6ef1a1ed0f8fdd112a16251180621502c0898fbe6128b2e8a293c28980412a3e"} Oct 30 01:32:43.272844 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:43.272816 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-9t2d5" event={"ID":"485b6169-3cf8-4111-be91-8050f32439d7","Type":"ContainerStarted","Data":"04eb97acfe1994c72454bfaf1078733f44e22f8338a8c0786ca58417d21fce3b"} Oct 30 01:32:43.272973 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:43.272866 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-9t2d5" event={"ID":"485b6169-3cf8-4111-be91-8050f32439d7","Type":"ContainerStarted","Data":"88e5fc10db768257ec6d64b468eb13182e9d15eebc735308cb33c10bd193cc59"} Oct 30 01:32:43.272973 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:43.272962 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-9t2d5" Oct 30 01:32:43.349832 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:43.349337 6572 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 01:32:43.349832 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:43.349403 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" podUID="6b50de5b-2782-45a9-8b90-44e3508013f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 01:32:43.372011 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:43.371961 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-9t2d5" podStartSLOduration=2.371945929 podStartE2EDuration="2.371945929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:43.369564879 +0000 UTC m=+888.945990280" watchObservedRunningTime="2026-10-30 01:32:43.371945929 +0000 UTC m=+888.948371347" Oct 30 01:32:43.382856 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:43.382828 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af953b7-ctpc5" Oct 30 01:32:43.795099 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:43.795069 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5af953b7-ctpc5"] Oct 30 01:32:43.795597 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:32:43.795571 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b4ef76_9c38_436f_a193_dc11782fcfaf.slice/crio-e618e0703bd2cbce1fdb130ac55854f7d232773e609bb69607f5f8fa943a67f8 WatchSource:0}: Error finding container e618e0703bd2cbce1fdb130ac55854f7d232773e609bb69607f5f8fa943a67f8: Status 404 returned error can't find the container with id e618e0703bd2cbce1fdb130ac55854f7d232773e609bb69607f5f8fa943a67f8 Oct 30 01:32:44.156796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:44.145987 6572 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 01:32:44.156796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:44.146066 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" podUID="d7243362-b004-4393-8a15-8ea62c257b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 01:32:44.282119 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:44.282082 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af953b7-ctpc5" event={"ID":"46b4ef76-9c38-436f-a193-dc11782fcfaf","Type":"ContainerStarted","Data":"8870fc64531782ca4670c69cf431145c70a36f90291adcc76f83e58f96ebc845"} Oct 30 01:32:44.282267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:44.282139 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af953b7-ctpc5" event={"ID":"46b4ef76-9c38-436f-a193-dc11782fcfaf","Type":"ContainerStarted","Data":"e618e0703bd2cbce1fdb130ac55854f7d232773e609bb69607f5f8fa943a67f8"} Oct 30 01:32:44.358830 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:44.350830 6572 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 01:32:44.358830 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:44.350886 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" podUID="6b50de5b-2782-45a9-8b90-44e3508013f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 01:32:44.359057 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:44.358969 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af953b7-ctpc5" podStartSLOduration=1.358950074 podStartE2EDuration="1.358950074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:44.343039525 +0000 UTC m=+889.919464929" watchObservedRunningTime="2026-10-30 01:32:44.358950074 +0000 UTC m=+889.935375476" Oct 30 01:32:44.489029 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:44.485024 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:44.591854 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:44.591829 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fb82088-mn4fn" Oct 30 01:32:44.702968 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:32:44.702930 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:32:44.703170 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:44.703144 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5-maas-api-xks-ca\") pod \"4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5\" (UID: \"4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5\") " Oct 30 01:32:44.703313 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:44.703298 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5-maas-api-openshift-ca\") pod \"4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5\" (UID: \"4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5\") " Oct 30 01:32:44.703372 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:44.703331 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5-kube-api-access-wgdd7\" (UniqueName: \"kubernetes.io/projected/4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5-kube-api-access-wgdd7\") pod \"4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5\" (UID: \"4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5\") " Oct 30 01:32:44.704679 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:44.704642 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5-maas-api-openshift-ca" pod "4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5" (UID: "4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:32:44.725980 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:44.719871 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5-maas-api-xks-ca" pod "4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5" (UID: "4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:32:44.750775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:44.746489 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5-kube-api-access-wgdd7" pod "4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5" (UID: "4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5"). InnerVolumeSpecName "kube-api-access-wgdd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:44.804991 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:44.804960 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:44.804991 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:44.804989 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:44.804991 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:44.805001 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgdd7\" (UniqueName: \"kubernetes.io/projected/4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5-kube-api-access-wgdd7\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:45.141784 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:45.141284 6572 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 01:32:45.141784 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:45.141352 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" podUID="d7243362-b004-4393-8a15-8ea62c257b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 01:32:45.292093 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:45.292003 6572 generic.go:356] "Generic (PLEG): container finished" podID="46b4ef76-9c38-436f-a193-dc11782fcfaf" containerID="8870fc64531782ca4670c69cf431145c70a36f90291adcc76f83e58f96ebc845" exitCode=0 Oct 30 01:32:45.292259 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:45.292129 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af953b7-ctpc5" event={"ID":"46b4ef76-9c38-436f-a193-dc11782fcfaf","Type":"ContainerDied","Data":"8870fc64531782ca4670c69cf431145c70a36f90291adcc76f83e58f96ebc845"} Oct 30 01:32:45.293738 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:45.293715 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fb82088-mn4fn" Oct 30 01:32:45.293738 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:45.293723 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fb82088-mn4fn" event={"ID":"4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5","Type":"ContainerDied","Data":"c10519f08fe7d9de04d62263dc83227bda4520d0045c605af2d8f3dc0a676c76"} Oct 30 01:32:45.293917 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:45.293777 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c10519f08fe7d9de04d62263dc83227bda4520d0045c605af2d8f3dc0a676c76" Oct 30 01:32:45.358504 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:45.357916 6572 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 01:32:45.358504 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:45.357971 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" podUID="6b50de5b-2782-45a9-8b90-44e3508013f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 01:32:46.151773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:46.144182 6572 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 01:32:46.151773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:46.144245 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" podUID="d7243362-b004-4393-8a15-8ea62c257b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 01:32:46.361237 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:46.361168 6572 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 01:32:46.361415 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:46.361238 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" podUID="6b50de5b-2782-45a9-8b90-44e3508013f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 01:32:46.583645 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:46.583608 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af953b7-ctpc5" Oct 30 01:32:46.673501 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:46.673460 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46b4ef76-9c38-436f-a193-dc11782fcfaf-kube-api-access-g76fl\" (UniqueName: \"kubernetes.io/projected/46b4ef76-9c38-436f-a193-dc11782fcfaf-kube-api-access-g76fl\") pod \"46b4ef76-9c38-436f-a193-dc11782fcfaf\" (UID: \"46b4ef76-9c38-436f-a193-dc11782fcfaf\") " Oct 30 01:32:46.673811 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:46.673789 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46b4ef76-9c38-436f-a193-dc11782fcfaf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46b4ef76-9c38-436f-a193-dc11782fcfaf-maas-api-openshift-ca\") pod \"46b4ef76-9c38-436f-a193-dc11782fcfaf\" (UID: \"46b4ef76-9c38-436f-a193-dc11782fcfaf\") " Oct 30 01:32:46.674000 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:46.673980 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46b4ef76-9c38-436f-a193-dc11782fcfaf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46b4ef76-9c38-436f-a193-dc11782fcfaf-maas-api-xks-ca\") pod \"46b4ef76-9c38-436f-a193-dc11782fcfaf\" (UID: \"46b4ef76-9c38-436f-a193-dc11782fcfaf\") " Oct 30 01:32:46.674669 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:46.674638 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46b4ef76-9c38-436f-a193-dc11782fcfaf-maas-api-openshift-ca" pod "46b4ef76-9c38-436f-a193-dc11782fcfaf" (UID: "46b4ef76-9c38-436f-a193-dc11782fcfaf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:32:46.674929 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:46.674906 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46b4ef76-9c38-436f-a193-dc11782fcfaf-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:46.678288 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:46.678250 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b4ef76-9c38-436f-a193-dc11782fcfaf-kube-api-access-g76fl" pod "46b4ef76-9c38-436f-a193-dc11782fcfaf" (UID: "46b4ef76-9c38-436f-a193-dc11782fcfaf"). InnerVolumeSpecName "kube-api-access-g76fl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:46.678597 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:46.678580 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46b4ef76-9c38-436f-a193-dc11782fcfaf-maas-api-xks-ca" pod "46b4ef76-9c38-436f-a193-dc11782fcfaf" (UID: "46b4ef76-9c38-436f-a193-dc11782fcfaf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:32:46.776209 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:46.776170 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g76fl\" (UniqueName: \"kubernetes.io/projected/46b4ef76-9c38-436f-a193-dc11782fcfaf-kube-api-access-g76fl\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:46.776209 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:46.776224 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46b4ef76-9c38-436f-a193-dc11782fcfaf-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:46.846778 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:46.828162 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:47.140242 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:47.140146 6572 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 01:32:47.140242 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:47.140206 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" podUID="d7243362-b004-4393-8a15-8ea62c257b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 01:32:47.335774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:47.320472 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af953b7-ctpc5" event={"ID":"46b4ef76-9c38-436f-a193-dc11782fcfaf","Type":"ContainerDied","Data":"e618e0703bd2cbce1fdb130ac55854f7d232773e609bb69607f5f8fa943a67f8"} Oct 30 01:32:47.335774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:47.320539 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e618e0703bd2cbce1fdb130ac55854f7d232773e609bb69607f5f8fa943a67f8" Oct 30 01:32:47.335774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:47.320613 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af953b7-ctpc5" Oct 30 01:32:47.349808 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:47.349726 6572 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 01:32:47.350016 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:47.349838 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" podUID="6b50de5b-2782-45a9-8b90-44e3508013f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 01:32:47.952793 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:47.952725 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:48.176041 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:48.176010 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" Oct 30 01:32:48.176981 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:48.176953 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" Oct 30 01:32:48.178086 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:48.178064 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" Oct 30 01:32:48.349671 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:48.349580 6572 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 01:32:48.349671 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:48.349649 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" podUID="6b50de5b-2782-45a9-8b90-44e3508013f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 01:32:49.355011 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:49.354977 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" Oct 30 01:32:49.355563 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:49.355536 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" Oct 30 01:32:49.356612 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:49.356594 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" Oct 30 01:32:49.518126 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:49.518085 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:51.052502 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:51.051872 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" podUID="0ee3f8e3-9b55-4ce5-afaf-86780a26391e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:51.757800 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:51.735818 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" podUID="cb20484a-51e2-4581-a629-e92d4234784e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:51.780771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:51.758113 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" podUID="fac3ca55-64af-42d9-90b1-0e821b893282" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:51.780771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:51.759989 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff85dbf6-mfl4s"] Oct 30 01:32:51.780771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:51.760849 6572 cpu_manager.go:414] "containerMap: removing container" podUID="4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5" containerName="revoke-keys" Oct 30 01:32:51.780771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:51.760869 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5" containerName="revoke-keys" Oct 30 01:32:51.780771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:51.760891 6572 cpu_manager.go:414] "containerMap: removing container" podUID="46b4ef76-9c38-436f-a193-dc11782fcfaf" containerName="revoke-keys" Oct 30 01:32:51.780771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:51.760899 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46b4ef76-9c38-436f-a193-dc11782fcfaf" containerName="revoke-keys" Oct 30 01:32:51.780771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:51.761116 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5" containerName="revoke-keys" Oct 30 01:32:51.780771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:51.761132 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="46b4ef76-9c38-436f-a193-dc11782fcfaf" containerName="revoke-keys" Oct 30 01:32:51.810774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:51.801827 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" podUID="317505ef-238c-458e-a74e-1198806b94d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:51.852774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:51.841062 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff85dbf6-mfl4s" Oct 30 01:32:51.852774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:51.842504 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" podUID="9a56da7a-9b92-40e2-b08b-43cf591b5221" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:51.860785 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:51.857134 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:32:51.888278 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:51.873998 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff85dbf6-mfl4s"] Oct 30 01:32:51.917774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:51.914240 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:32:51.917774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:51.914386 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:32:51.917774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:51.914441 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:32:52.017855 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:52.013816 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f713498-db56-4752-b3f6-d98cb67f3dd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff85dbf6-mfl4s\" (UID: \"1f713498-db56-4752-b3f6-d98cb67f3dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff85dbf6-mfl4s" Oct 30 01:32:52.031168 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:52.031130 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgff9\" (UniqueName: \"kubernetes.io/projected/1f713498-db56-4752-b3f6-d98cb67f3dd2-kube-api-access-bgff9\") pod \"maas-api-revoke-sub-ff85dbf6-mfl4s\" (UID: \"1f713498-db56-4752-b3f6-d98cb67f3dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff85dbf6-mfl4s" Oct 30 01:32:52.031510 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:52.031495 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f713498-db56-4752-b3f6-d98cb67f3dd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff85dbf6-mfl4s\" (UID: \"1f713498-db56-4752-b3f6-d98cb67f3dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff85dbf6-mfl4s" Oct 30 01:32:52.059945 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:52.059892 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" podUID="0ee3f8e3-9b55-4ce5-afaf-86780a26391e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:52.061068 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:52.061039 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" Oct 30 01:32:52.088774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:52.070850 6572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" podUID="fac3ca55-64af-42d9-90b1-0e821b893282" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:52.088774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:52.070961 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" podUID="e76339df-9659-4bea-80a4-5f18c4a73fd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 30 01:32:52.165775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:52.154616 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgff9\" (UniqueName: \"kubernetes.io/projected/1f713498-db56-4752-b3f6-d98cb67f3dd2-kube-api-access-bgff9\") pod \"maas-api-revoke-sub-ff85dbf6-mfl4s\" (UID: \"1f713498-db56-4752-b3f6-d98cb67f3dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff85dbf6-mfl4s" Oct 30 01:32:52.172123 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:52.172075 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f713498-db56-4752-b3f6-d98cb67f3dd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff85dbf6-mfl4s\" (UID: \"1f713498-db56-4752-b3f6-d98cb67f3dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff85dbf6-mfl4s" Oct 30 01:32:52.180934 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:52.180891 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f713498-db56-4752-b3f6-d98cb67f3dd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff85dbf6-mfl4s\" (UID: \"1f713498-db56-4752-b3f6-d98cb67f3dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff85dbf6-mfl4s" Oct 30 01:32:52.255775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:52.223603 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f713498-db56-4752-b3f6-d98cb67f3dd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff85dbf6-mfl4s\" (UID: \"1f713498-db56-4752-b3f6-d98cb67f3dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff85dbf6-mfl4s" Oct 30 01:32:52.265855 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:52.232271 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f713498-db56-4752-b3f6-d98cb67f3dd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff85dbf6-mfl4s\" (UID: \"1f713498-db56-4752-b3f6-d98cb67f3dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff85dbf6-mfl4s" Oct 30 01:32:52.296693 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:52.296611 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgff9\" (UniqueName: \"kubernetes.io/projected/1f713498-db56-4752-b3f6-d98cb67f3dd2-kube-api-access-bgff9\") pod \"maas-api-revoke-sub-ff85dbf6-mfl4s\" (UID: \"1f713498-db56-4752-b3f6-d98cb67f3dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff85dbf6-mfl4s" Oct 30 01:32:52.609334 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:52.608994 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff85dbf6-mfl4s" Oct 30 01:32:52.658408 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:52.658357 6572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" podUID="0ee3f8e3-9b55-4ce5-afaf-86780a26391e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:52.792366 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:52.792306 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" podUID="fac3ca55-64af-42d9-90b1-0e821b893282" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 01:32:52.821328 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:52.821271 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" podUID="cb20484a-51e2-4581-a629-e92d4234784e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 01:32:52.926798 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:52.909871 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" podUID="9a56da7a-9b92-40e2-b08b-43cf591b5221" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 01:32:53.016787 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:53.015352 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:53.466977 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:53.162275 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" Oct 30 01:32:53.466977 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:53.162559 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" Oct 30 01:32:53.466977 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:53.162857 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" Oct 30 01:32:53.509803 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:53.507487 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" podUID="0ee3f8e3-9b55-4ce5-afaf-86780a26391e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 30 01:32:53.509803 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:53.508365 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" podUID="e76339df-9659-4bea-80a4-5f18c4a73fd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 30 01:32:53.510568 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:53.510212 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" Oct 30 01:32:53.798320 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:53.798241 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" Oct 30 01:32:53.810891 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:53.800135 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" podUID="317505ef-238c-458e-a74e-1198806b94d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 01:32:53.811658 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:53.810895 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" Oct 30 01:32:53.812051 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:53.812034 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" Oct 30 01:32:53.825254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:53.825136 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" podUID="cb20484a-51e2-4581-a629-e92d4234784e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:53.859361 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:53.847155 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" Oct 30 01:32:53.898025 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:53.894837 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" Oct 30 01:32:53.923213 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:53.923186 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" Oct 30 01:32:53.960412 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:53.960264 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" Oct 30 01:32:53.975016 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:53.969109 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff85dbf6-mfl4s"] Oct 30 01:32:54.713207 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:54.713160 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff85dbf6-mfl4s" event={"ID":"1f713498-db56-4752-b3f6-d98cb67f3dd2","Type":"ContainerStarted","Data":"64392540aaa61c122acfc1af0353c2b6543ebca152a4c2c415cb362f17196755"} Oct 30 01:32:54.713807 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:54.713222 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff85dbf6-mfl4s" event={"ID":"1f713498-db56-4752-b3f6-d98cb67f3dd2","Type":"ContainerStarted","Data":"3708d280e42298b78ece4521db95553be7b67737846666af5a23597a53210c52"} Oct 30 01:32:54.753616 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:54.753544 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff85dbf6-mfl4s" podStartSLOduration=3.753521204 podStartE2EDuration="3.753521204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:54.749121986 +0000 UTC m=+900.325547380" watchObservedRunningTime="2026-10-30 01:32:54.753521204 +0000 UTC m=+900.329946607" Oct 30 01:32:55.262657 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:55.262616 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:55.297593 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:55.297563 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-9t2d5" Oct 30 01:32:55.465386 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:55.465345 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:55.723432 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:55.723399 6572 generic.go:356] "Generic (PLEG): container finished" podID="1f713498-db56-4752-b3f6-d98cb67f3dd2" containerID="64392540aaa61c122acfc1af0353c2b6543ebca152a4c2c415cb362f17196755" exitCode=0 Oct 30 01:32:55.723860 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:55.723483 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff85dbf6-mfl4s" event={"ID":"1f713498-db56-4752-b3f6-d98cb67f3dd2","Type":"ContainerDied","Data":"64392540aaa61c122acfc1af0353c2b6543ebca152a4c2c415cb362f17196755"} Oct 30 01:32:56.887592 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:56.887562 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff85dbf6-mfl4s" Oct 30 01:32:56.949480 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:56.949444 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f713498-db56-4752-b3f6-d98cb67f3dd2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f713498-db56-4752-b3f6-d98cb67f3dd2-maas-api-openshift-ca\") pod \"1f713498-db56-4752-b3f6-d98cb67f3dd2\" (UID: \"1f713498-db56-4752-b3f6-d98cb67f3dd2\") " Oct 30 01:32:56.949684 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:56.949526 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f713498-db56-4752-b3f6-d98cb67f3dd2-kube-api-access-bgff9\" (UniqueName: \"kubernetes.io/projected/1f713498-db56-4752-b3f6-d98cb67f3dd2-kube-api-access-bgff9\") pod \"1f713498-db56-4752-b3f6-d98cb67f3dd2\" (UID: \"1f713498-db56-4752-b3f6-d98cb67f3dd2\") " Oct 30 01:32:56.949684 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:56.949606 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f713498-db56-4752-b3f6-d98cb67f3dd2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f713498-db56-4752-b3f6-d98cb67f3dd2-maas-api-xks-ca\") pod \"1f713498-db56-4752-b3f6-d98cb67f3dd2\" (UID: \"1f713498-db56-4752-b3f6-d98cb67f3dd2\") " Oct 30 01:32:56.949956 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:56.949928 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f713498-db56-4752-b3f6-d98cb67f3dd2-maas-api-openshift-ca" pod "1f713498-db56-4752-b3f6-d98cb67f3dd2" (UID: "1f713498-db56-4752-b3f6-d98cb67f3dd2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:32:56.950269 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:56.950246 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f713498-db56-4752-b3f6-d98cb67f3dd2-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:56.951912 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:56.951882 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f713498-db56-4752-b3f6-d98cb67f3dd2-maas-api-xks-ca" pod "1f713498-db56-4752-b3f6-d98cb67f3dd2" (UID: "1f713498-db56-4752-b3f6-d98cb67f3dd2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:32:56.952024 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:56.951996 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f713498-db56-4752-b3f6-d98cb67f3dd2-kube-api-access-bgff9" pod "1f713498-db56-4752-b3f6-d98cb67f3dd2" (UID: "1f713498-db56-4752-b3f6-d98cb67f3dd2"). InnerVolumeSpecName "kube-api-access-bgff9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:57.054108 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.054007 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgff9\" (UniqueName: \"kubernetes.io/projected/1f713498-db56-4752-b3f6-d98cb67f3dd2-kube-api-access-bgff9\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:57.054108 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.054042 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f713498-db56-4752-b3f6-d98cb67f3dd2-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:32:57.079004 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.078960 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:57.386883 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.386802 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x"] Oct 30 01:32:57.387392 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.387378 6572 cpu_manager.go:414] "containerMap: removing container" podUID="1f713498-db56-4752-b3f6-d98cb67f3dd2" containerName="revoke-keys" Oct 30 01:32:57.387434 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.387394 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f713498-db56-4752-b3f6-d98cb67f3dd2" containerName="revoke-keys" Oct 30 01:32:57.387520 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.387511 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f713498-db56-4752-b3f6-d98cb67f3dd2" containerName="revoke-keys" Oct 30 01:32:57.391332 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.391310 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" Oct 30 01:32:57.407080 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.407045 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd"] Oct 30 01:32:57.413036 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.413003 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" Oct 30 01:32:57.415948 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.415920 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x"] Oct 30 01:32:57.435709 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.435676 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd"] Oct 30 01:32:57.464610 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.463093 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-credential-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-xq97x\" (UID: \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" Oct 30 01:32:57.464610 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.463155 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-istio-podinfo\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-xq97x\" (UID: \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" Oct 30 01:32:57.464610 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.463267 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-istio-token\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-xq97x\" (UID: \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" Oct 30 01:32:57.464610 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.463336 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46929775-3a67-4bd4-8387-52bab2ebdc0e-workload-certs\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-whkbd\" (UID: \"46929775-3a67-4bd4-8387-52bab2ebdc0e\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" Oct 30 01:32:57.464610 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.463380 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-workload-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-xq97x\" (UID: \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" Oct 30 01:32:57.464610 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.463419 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46929775-3a67-4bd4-8387-52bab2ebdc0e-istio-podinfo\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-whkbd\" (UID: \"46929775-3a67-4bd4-8387-52bab2ebdc0e\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" Oct 30 01:32:57.464610 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.463506 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46929775-3a67-4bd4-8387-52bab2ebdc0e-istio-data\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-whkbd\" (UID: \"46929775-3a67-4bd4-8387-52bab2ebdc0e\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" Oct 30 01:32:57.464610 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.463535 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46929775-3a67-4bd4-8387-52bab2ebdc0e-istio-token\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-whkbd\" (UID: \"46929775-3a67-4bd4-8387-52bab2ebdc0e\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" Oct 30 01:32:57.464610 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.463575 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p5ff\" (UniqueName: \"kubernetes.io/projected/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-kube-api-access-4p5ff\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-xq97x\" (UID: \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" Oct 30 01:32:57.464610 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.463695 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46929775-3a67-4bd4-8387-52bab2ebdc0e-workload-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-whkbd\" (UID: \"46929775-3a67-4bd4-8387-52bab2ebdc0e\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" Oct 30 01:32:57.464610 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.463758 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-istio-envoy\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-xq97x\" (UID: \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" Oct 30 01:32:57.464610 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.463799 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46929775-3a67-4bd4-8387-52bab2ebdc0e-credential-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-whkbd\" (UID: \"46929775-3a67-4bd4-8387-52bab2ebdc0e\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" Oct 30 01:32:57.464610 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.463876 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmwt4\" (UniqueName: \"kubernetes.io/projected/46929775-3a67-4bd4-8387-52bab2ebdc0e-kube-api-access-cmwt4\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-whkbd\" (UID: \"46929775-3a67-4bd4-8387-52bab2ebdc0e\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" Oct 30 01:32:57.464610 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.464092 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-istio-data\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-xq97x\" (UID: \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" Oct 30 01:32:57.464610 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.464133 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46929775-3a67-4bd4-8387-52bab2ebdc0e-istiod-ca-cert\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-whkbd\" (UID: \"46929775-3a67-4bd4-8387-52bab2ebdc0e\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" Oct 30 01:32:57.464610 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.464191 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-workload-certs\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-xq97x\" (UID: \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" Oct 30 01:32:57.464610 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.464229 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46929775-3a67-4bd4-8387-52bab2ebdc0e-istio-envoy\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-whkbd\" (UID: \"46929775-3a67-4bd4-8387-52bab2ebdc0e\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" Oct 30 01:32:57.464610 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.464261 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-istiod-ca-cert\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-xq97x\" (UID: \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" Oct 30 01:32:57.565994 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.565959 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-credential-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-xq97x\" (UID: \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" Oct 30 01:32:57.565994 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.565998 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-istio-podinfo\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-xq97x\" (UID: \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" Oct 30 01:32:57.566262 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.566025 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-istio-token\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-xq97x\" (UID: \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" Oct 30 01:32:57.566262 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.566062 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46929775-3a67-4bd4-8387-52bab2ebdc0e-workload-certs\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-whkbd\" (UID: \"46929775-3a67-4bd4-8387-52bab2ebdc0e\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" Oct 30 01:32:57.566262 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.566084 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-workload-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-xq97x\" (UID: \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" Oct 30 01:32:57.566262 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.566112 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46929775-3a67-4bd4-8387-52bab2ebdc0e-istio-podinfo\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-whkbd\" (UID: \"46929775-3a67-4bd4-8387-52bab2ebdc0e\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" Oct 30 01:32:57.566262 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.566241 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46929775-3a67-4bd4-8387-52bab2ebdc0e-istio-data\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-whkbd\" (UID: \"46929775-3a67-4bd4-8387-52bab2ebdc0e\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" Oct 30 01:32:57.566542 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.566274 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46929775-3a67-4bd4-8387-52bab2ebdc0e-istio-token\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-whkbd\" (UID: \"46929775-3a67-4bd4-8387-52bab2ebdc0e\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" Oct 30 01:32:57.566542 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.566302 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p5ff\" (UniqueName: \"kubernetes.io/projected/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-kube-api-access-4p5ff\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-xq97x\" (UID: \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" Oct 30 01:32:57.566542 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.566360 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46929775-3a67-4bd4-8387-52bab2ebdc0e-workload-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-whkbd\" (UID: \"46929775-3a67-4bd4-8387-52bab2ebdc0e\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" Oct 30 01:32:57.566542 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.566395 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-istio-envoy\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-xq97x\" (UID: \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" Oct 30 01:32:57.566542 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.566424 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46929775-3a67-4bd4-8387-52bab2ebdc0e-credential-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-whkbd\" (UID: \"46929775-3a67-4bd4-8387-52bab2ebdc0e\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" Oct 30 01:32:57.566542 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.566448 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmwt4\" (UniqueName: \"kubernetes.io/projected/46929775-3a67-4bd4-8387-52bab2ebdc0e-kube-api-access-cmwt4\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-whkbd\" (UID: \"46929775-3a67-4bd4-8387-52bab2ebdc0e\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" Oct 30 01:32:57.566542 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.566492 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-istio-data\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-xq97x\" (UID: \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" Oct 30 01:32:57.566542 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.566520 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46929775-3a67-4bd4-8387-52bab2ebdc0e-istiod-ca-cert\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-whkbd\" (UID: \"46929775-3a67-4bd4-8387-52bab2ebdc0e\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" Oct 30 01:32:57.566967 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.566549 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46929775-3a67-4bd4-8387-52bab2ebdc0e-workload-certs\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-whkbd\" (UID: \"46929775-3a67-4bd4-8387-52bab2ebdc0e\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" Oct 30 01:32:57.566967 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.566578 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-workload-certs\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-xq97x\" (UID: \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" Oct 30 01:32:57.566967 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.566577 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-credential-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-xq97x\" (UID: \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" Oct 30 01:32:57.566967 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.566630 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46929775-3a67-4bd4-8387-52bab2ebdc0e-istio-envoy\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-whkbd\" (UID: \"46929775-3a67-4bd4-8387-52bab2ebdc0e\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" Oct 30 01:32:57.566967 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.566659 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-istiod-ca-cert\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-xq97x\" (UID: \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" Oct 30 01:32:57.566967 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.566901 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46929775-3a67-4bd4-8387-52bab2ebdc0e-credential-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-whkbd\" (UID: \"46929775-3a67-4bd4-8387-52bab2ebdc0e\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" Oct 30 01:32:57.567348 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.567229 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-workload-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-xq97x\" (UID: \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" Oct 30 01:32:57.567445 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.567418 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-istio-data\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-xq97x\" (UID: \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" Oct 30 01:32:57.567521 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.567483 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46929775-3a67-4bd4-8387-52bab2ebdc0e-workload-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-whkbd\" (UID: \"46929775-3a67-4bd4-8387-52bab2ebdc0e\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" Oct 30 01:32:57.567718 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.567667 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-workload-certs\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-xq97x\" (UID: \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" Oct 30 01:32:57.567970 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.567947 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-istiod-ca-cert\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-xq97x\" (UID: \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" Oct 30 01:32:57.568070 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.567969 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46929775-3a67-4bd4-8387-52bab2ebdc0e-istiod-ca-cert\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-whkbd\" (UID: \"46929775-3a67-4bd4-8387-52bab2ebdc0e\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" Oct 30 01:32:57.568230 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.568200 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46929775-3a67-4bd4-8387-52bab2ebdc0e-istio-data\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-whkbd\" (UID: \"46929775-3a67-4bd4-8387-52bab2ebdc0e\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" Oct 30 01:32:57.568803 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.568781 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-istio-envoy\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-xq97x\" (UID: \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" Oct 30 01:32:57.569043 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.569023 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-istio-podinfo\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-xq97x\" (UID: \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" Oct 30 01:32:57.569148 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.569125 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46929775-3a67-4bd4-8387-52bab2ebdc0e-istio-podinfo\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-whkbd\" (UID: \"46929775-3a67-4bd4-8387-52bab2ebdc0e\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" Oct 30 01:32:57.569295 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.569276 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46929775-3a67-4bd4-8387-52bab2ebdc0e-istio-envoy\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-whkbd\" (UID: \"46929775-3a67-4bd4-8387-52bab2ebdc0e\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" Oct 30 01:32:57.583557 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.583521 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p5ff\" (UniqueName: \"kubernetes.io/projected/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-kube-api-access-4p5ff\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-xq97x\" (UID: \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" Oct 30 01:32:57.583667 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.583626 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmwt4\" (UniqueName: \"kubernetes.io/projected/46929775-3a67-4bd4-8387-52bab2ebdc0e-kube-api-access-cmwt4\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-whkbd\" (UID: \"46929775-3a67-4bd4-8387-52bab2ebdc0e\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" Oct 30 01:32:57.583711 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.583688 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-istio-token\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-xq97x\" (UID: \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" Oct 30 01:32:57.583798 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.583780 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46929775-3a67-4bd4-8387-52bab2ebdc0e-istio-token\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-whkbd\" (UID: \"46929775-3a67-4bd4-8387-52bab2ebdc0e\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" Oct 30 01:32:57.710851 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.710811 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" Oct 30 01:32:57.737306 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.737274 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" Oct 30 01:32:57.745378 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.742475 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff85dbf6-mfl4s" event={"ID":"1f713498-db56-4752-b3f6-d98cb67f3dd2","Type":"ContainerDied","Data":"3708d280e42298b78ece4521db95553be7b67737846666af5a23597a53210c52"} Oct 30 01:32:57.745378 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.742532 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3708d280e42298b78ece4521db95553be7b67737846666af5a23597a53210c52" Oct 30 01:32:57.745378 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:57.742682 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff85dbf6-mfl4s" Oct 30 01:32:58.029312 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:32:58.029285 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96ae6bcd_2343_4c65_b4db_f7a5106f51f2.slice/crio-fe39239b0984f2c2fc8032ff5de2a403fe87d2784dbe2dc07c2578ce9232cfdf WatchSource:0}: Error finding container fe39239b0984f2c2fc8032ff5de2a403fe87d2784dbe2dc07c2578ce9232cfdf: Status 404 returned error can't find the container with id fe39239b0984f2c2fc8032ff5de2a403fe87d2784dbe2dc07c2578ce9232cfdf Oct 30 01:32:58.030250 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:58.030226 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x"] Oct 30 01:32:58.032014 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:58.031978 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:32:58.032120 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:58.032059 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:32:58.032120 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:58.032104 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:32:58.082734 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:58.082704 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd"] Oct 30 01:32:58.082967 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:32:58.082942 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46929775_3a67_4bd4_8387_52bab2ebdc0e.slice/crio-f4cb6d6c0031de2eda37b667f39d667f5601a2d2200e029ea6baed28d95d8e92 WatchSource:0}: Error finding container f4cb6d6c0031de2eda37b667f39d667f5601a2d2200e029ea6baed28d95d8e92: Status 404 returned error can't find the container with id f4cb6d6c0031de2eda37b667f39d667f5601a2d2200e029ea6baed28d95d8e92 Oct 30 01:32:58.085596 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:58.085568 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:32:58.085678 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:58.085635 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:32:58.085678 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:58.085665 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:32:58.774472 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:58.774437 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" event={"ID":"46929775-3a67-4bd4-8387-52bab2ebdc0e","Type":"ContainerStarted","Data":"f10b67b9e2539aad5f526f85f46e89707c690efc2eb79e9d28c08d7bdcbd28e3"} Oct 30 01:32:58.774683 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:58.774490 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" event={"ID":"46929775-3a67-4bd4-8387-52bab2ebdc0e","Type":"ContainerStarted","Data":"f4cb6d6c0031de2eda37b667f39d667f5601a2d2200e029ea6baed28d95d8e92"} Oct 30 01:32:58.777069 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:58.777036 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" event={"ID":"96ae6bcd-2343-4c65-b4db-f7a5106f51f2","Type":"ContainerStarted","Data":"b1102e030bb072072f9910974ad6760d1da4f18647dcf5e9a37cc49083430c55"} Oct 30 01:32:58.777260 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:58.777243 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" event={"ID":"96ae6bcd-2343-4c65-b4db-f7a5106f51f2","Type":"ContainerStarted","Data":"fe39239b0984f2c2fc8032ff5de2a403fe87d2784dbe2dc07c2578ce9232cfdf"} Oct 30 01:32:58.868773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:58.865510 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" podStartSLOduration=1.865489736 podStartE2EDuration="1.865489736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:58.825039655 +0000 UTC m=+904.401465060" watchObservedRunningTime="2026-10-30 01:32:58.865489736 +0000 UTC m=+904.441915141" Oct 30 01:32:58.873122 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:58.873070 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" podStartSLOduration=1.873054367 podStartE2EDuration="1.873054367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:58.865013761 +0000 UTC m=+904.441439170" watchObservedRunningTime="2026-10-30 01:32:58.873054367 +0000 UTC m=+904.449479782" Oct 30 01:32:58.992020 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:58.991860 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:59.721772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:59.713981 6572 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-xq97x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 01:32:59.721772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:59.714050 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" podUID="96ae6bcd-2343-4c65-b4db-f7a5106f51f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 01:32:59.721772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:59.714240 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" Oct 30 01:32:59.756703 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:59.756657 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" Oct 30 01:32:59.756945 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:59.756839 6572 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-whkbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 01:32:59.756945 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:59.756895 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" podUID="46929775-3a67-4bd4-8387-52bab2ebdc0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 01:32:59.761838 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:59.761806 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f50466c-zj7p4"] Oct 30 01:32:59.767539 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:59.767517 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f50466c-zj7p4" Oct 30 01:32:59.778529 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:59.778481 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f50466c-zj7p4"] Oct 30 01:32:59.781645 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:59.781605 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:32:59.781645 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:59.781633 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:32:59.781883 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:59.781660 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:32:59.781883 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:59.781740 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:32:59.901395 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:59.901353 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aac7cb3b-87f6-44ef-9fc3-fa595bcb1437-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f50466c-zj7p4\" (UID: \"aac7cb3b-87f6-44ef-9fc3-fa595bcb1437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f50466c-zj7p4" Oct 30 01:32:59.902246 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:59.901633 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fbq2\" (UniqueName: \"kubernetes.io/projected/aac7cb3b-87f6-44ef-9fc3-fa595bcb1437-kube-api-access-7fbq2\") pod \"maas-api-revoke-sub-6f50466c-zj7p4\" (UID: \"aac7cb3b-87f6-44ef-9fc3-fa595bcb1437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f50466c-zj7p4" Oct 30 01:32:59.902246 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:32:59.901716 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aac7cb3b-87f6-44ef-9fc3-fa595bcb1437-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f50466c-zj7p4\" (UID: \"aac7cb3b-87f6-44ef-9fc3-fa595bcb1437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f50466c-zj7p4" Oct 30 01:33:00.003361 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:00.003265 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aac7cb3b-87f6-44ef-9fc3-fa595bcb1437-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f50466c-zj7p4\" (UID: \"aac7cb3b-87f6-44ef-9fc3-fa595bcb1437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f50466c-zj7p4" Oct 30 01:33:00.003510 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:00.003370 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aac7cb3b-87f6-44ef-9fc3-fa595bcb1437-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f50466c-zj7p4\" (UID: \"aac7cb3b-87f6-44ef-9fc3-fa595bcb1437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f50466c-zj7p4" Oct 30 01:33:00.003568 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:00.003550 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fbq2\" (UniqueName: \"kubernetes.io/projected/aac7cb3b-87f6-44ef-9fc3-fa595bcb1437-kube-api-access-7fbq2\") pod \"maas-api-revoke-sub-6f50466c-zj7p4\" (UID: \"aac7cb3b-87f6-44ef-9fc3-fa595bcb1437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f50466c-zj7p4" Oct 30 01:33:00.004330 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:00.004301 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aac7cb3b-87f6-44ef-9fc3-fa595bcb1437-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f50466c-zj7p4\" (UID: \"aac7cb3b-87f6-44ef-9fc3-fa595bcb1437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f50466c-zj7p4" Oct 30 01:33:00.017774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:00.005806 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aac7cb3b-87f6-44ef-9fc3-fa595bcb1437-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f50466c-zj7p4\" (UID: \"aac7cb3b-87f6-44ef-9fc3-fa595bcb1437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f50466c-zj7p4" Oct 30 01:33:00.038775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:00.031556 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fbq2\" (UniqueName: \"kubernetes.io/projected/aac7cb3b-87f6-44ef-9fc3-fa595bcb1437-kube-api-access-7fbq2\") pod \"maas-api-revoke-sub-6f50466c-zj7p4\" (UID: \"aac7cb3b-87f6-44ef-9fc3-fa595bcb1437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f50466c-zj7p4" Oct 30 01:33:00.086586 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:00.086546 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f50466c-zj7p4" Oct 30 01:33:00.458772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:00.451168 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f50466c-zj7p4"] Oct 30 01:33:00.466162 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:33:00.466117 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaac7cb3b_87f6_44ef_9fc3_fa595bcb1437.slice/crio-e54e7a2f2f1ef1bb77ff17933a149501ae8954b5165bc1c83b9c014350e7a8aa WatchSource:0}: Error finding container e54e7a2f2f1ef1bb77ff17933a149501ae8954b5165bc1c83b9c014350e7a8aa: Status 404 returned error can't find the container with id e54e7a2f2f1ef1bb77ff17933a149501ae8954b5165bc1c83b9c014350e7a8aa Oct 30 01:33:00.713862 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:00.712188 6572 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-xq97x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 01:33:00.713862 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:00.712252 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" podUID="96ae6bcd-2343-4c65-b4db-f7a5106f51f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 01:33:00.739221 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:00.739151 6572 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-whkbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 01:33:00.739810 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:00.739223 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" podUID="46929775-3a67-4bd4-8387-52bab2ebdc0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 01:33:00.797990 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:00.797954 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f50466c-zj7p4" event={"ID":"aac7cb3b-87f6-44ef-9fc3-fa595bcb1437","Type":"ContainerStarted","Data":"e54e7a2f2f1ef1bb77ff17933a149501ae8954b5165bc1c83b9c014350e7a8aa"} Oct 30 01:33:01.016772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:01.012162 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:01.716772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:01.714257 6572 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-xq97x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 01:33:01.716772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:01.714333 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" podUID="96ae6bcd-2343-4c65-b4db-f7a5106f51f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 01:33:01.738209 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:01.738088 6572 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-whkbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 01:33:01.738209 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:01.738150 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" podUID="46929775-3a67-4bd4-8387-52bab2ebdc0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 01:33:01.872765 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:01.872656 6572 generic.go:356] "Generic (PLEG): container finished" podID="aac7cb3b-87f6-44ef-9fc3-fa595bcb1437" containerID="492b5814d0de317ef16afc0ef49b9ee188677c7897e6daad482b32818ac22594" exitCode=0 Oct 30 01:33:01.873126 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:01.872764 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f50466c-zj7p4" event={"ID":"aac7cb3b-87f6-44ef-9fc3-fa595bcb1437","Type":"ContainerDied","Data":"492b5814d0de317ef16afc0ef49b9ee188677c7897e6daad482b32818ac22594"} Oct 30 01:33:02.713521 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:02.713485 6572 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-xq97x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 01:33:02.713841 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:02.713806 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" podUID="96ae6bcd-2343-4c65-b4db-f7a5106f51f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 01:33:02.743289 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:02.743250 6572 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-whkbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 01:33:02.743702 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:02.743669 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" podUID="46929775-3a67-4bd4-8387-52bab2ebdc0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 01:33:03.083674 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:03.083635 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:03.189595 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:03.189564 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f50466c-zj7p4" Oct 30 01:33:03.291005 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:03.290913 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aac7cb3b-87f6-44ef-9fc3-fa595bcb1437-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aac7cb3b-87f6-44ef-9fc3-fa595bcb1437-maas-api-xks-ca\") pod \"aac7cb3b-87f6-44ef-9fc3-fa595bcb1437\" (UID: \"aac7cb3b-87f6-44ef-9fc3-fa595bcb1437\") " Oct 30 01:33:03.291171 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:03.291124 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aac7cb3b-87f6-44ef-9fc3-fa595bcb1437-kube-api-access-7fbq2\" (UniqueName: \"kubernetes.io/projected/aac7cb3b-87f6-44ef-9fc3-fa595bcb1437-kube-api-access-7fbq2\") pod \"aac7cb3b-87f6-44ef-9fc3-fa595bcb1437\" (UID: \"aac7cb3b-87f6-44ef-9fc3-fa595bcb1437\") " Oct 30 01:33:03.291219 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:03.291177 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aac7cb3b-87f6-44ef-9fc3-fa595bcb1437-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aac7cb3b-87f6-44ef-9fc3-fa595bcb1437-maas-api-openshift-ca\") pod \"aac7cb3b-87f6-44ef-9fc3-fa595bcb1437\" (UID: \"aac7cb3b-87f6-44ef-9fc3-fa595bcb1437\") " Oct 30 01:33:03.291560 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:03.291532 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aac7cb3b-87f6-44ef-9fc3-fa595bcb1437-maas-api-openshift-ca" pod "aac7cb3b-87f6-44ef-9fc3-fa595bcb1437" (UID: "aac7cb3b-87f6-44ef-9fc3-fa595bcb1437"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:33:03.293101 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:03.293074 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aac7cb3b-87f6-44ef-9fc3-fa595bcb1437-maas-api-xks-ca" pod "aac7cb3b-87f6-44ef-9fc3-fa595bcb1437" (UID: "aac7cb3b-87f6-44ef-9fc3-fa595bcb1437"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:33:03.293279 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:03.293258 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac7cb3b-87f6-44ef-9fc3-fa595bcb1437-kube-api-access-7fbq2" pod "aac7cb3b-87f6-44ef-9fc3-fa595bcb1437" (UID: "aac7cb3b-87f6-44ef-9fc3-fa595bcb1437"). InnerVolumeSpecName "kube-api-access-7fbq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:33:03.392858 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:03.392822 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fbq2\" (UniqueName: \"kubernetes.io/projected/aac7cb3b-87f6-44ef-9fc3-fa595bcb1437-kube-api-access-7fbq2\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:33:03.392858 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:03.392850 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aac7cb3b-87f6-44ef-9fc3-fa595bcb1437-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:33:03.392858 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:03.392860 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aac7cb3b-87f6-44ef-9fc3-fa595bcb1437-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:33:03.643868 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:03.643380 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:03.713779 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:03.712325 6572 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-xq97x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 01:33:03.713779 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:03.712406 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" podUID="96ae6bcd-2343-4c65-b4db-f7a5106f51f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 01:33:03.763771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:03.763576 6572 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-whkbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 01:33:03.763771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:03.763650 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" podUID="46929775-3a67-4bd4-8387-52bab2ebdc0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 01:33:03.863853 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:03.863806 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:03.896507 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:03.896415 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f50466c-zj7p4" Oct 30 01:33:03.896507 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:03.896436 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f50466c-zj7p4" event={"ID":"aac7cb3b-87f6-44ef-9fc3-fa595bcb1437","Type":"ContainerDied","Data":"e54e7a2f2f1ef1bb77ff17933a149501ae8954b5165bc1c83b9c014350e7a8aa"} Oct 30 01:33:03.896507 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:03.896480 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e54e7a2f2f1ef1bb77ff17933a149501ae8954b5165bc1c83b9c014350e7a8aa" Oct 30 01:33:04.082200 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:04.082167 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:04.711738 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:04.711697 6572 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-xq97x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 01:33:04.712295 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:04.711797 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" podUID="96ae6bcd-2343-4c65-b4db-f7a5106f51f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 01:33:04.738429 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:04.738382 6572 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-whkbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 01:33:04.738641 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:04.738458 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" podUID="46929775-3a67-4bd4-8387-52bab2ebdc0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 01:33:05.074835 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:05.072896 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:05.266431 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:05.266389 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:05.716801 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:05.716764 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" Oct 30 01:33:05.717324 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:05.717174 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" Oct 30 01:33:05.717903 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:05.717881 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" Oct 30 01:33:05.760430 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:05.760391 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" Oct 30 01:33:05.760812 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:05.760781 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" Oct 30 01:33:05.762660 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:05.762636 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" Oct 30 01:33:06.217362 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:06.217327 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:06.535200 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:06.535104 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:06.804731 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:06.804644 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbf1ce8b-7ljnq"] Oct 30 01:33:06.805304 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:06.805285 6572 cpu_manager.go:414] "containerMap: removing container" podUID="aac7cb3b-87f6-44ef-9fc3-fa595bcb1437" containerName="revoke-keys" Oct 30 01:33:06.805364 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:06.805305 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aac7cb3b-87f6-44ef-9fc3-fa595bcb1437" containerName="revoke-keys" Oct 30 01:33:06.805518 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:06.805502 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="aac7cb3b-87f6-44ef-9fc3-fa595bcb1437" containerName="revoke-keys" Oct 30 01:33:06.809397 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:06.809379 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf1ce8b-7ljnq" Oct 30 01:33:06.833070 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:06.833039 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:33:06.833300 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:06.833118 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:33:06.833411 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:06.833190 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:33:06.833872 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:06.833846 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:33:06.840094 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:06.840061 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbf1ce8b-7ljnq"] Oct 30 01:33:06.840858 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:06.840831 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83968442-b868-4896-84c1-415c42cfbbcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbf1ce8b-7ljnq\" (UID: \"83968442-b868-4896-84c1-415c42cfbbcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf1ce8b-7ljnq" Oct 30 01:33:06.841043 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:06.841024 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf5c9\" (UniqueName: \"kubernetes.io/projected/83968442-b868-4896-84c1-415c42cfbbcc-kube-api-access-qf5c9\") pod \"maas-api-revoke-sub-bbf1ce8b-7ljnq\" (UID: \"83968442-b868-4896-84c1-415c42cfbbcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf1ce8b-7ljnq" Oct 30 01:33:06.841111 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:06.841096 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83968442-b868-4896-84c1-415c42cfbbcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbf1ce8b-7ljnq\" (UID: \"83968442-b868-4896-84c1-415c42cfbbcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf1ce8b-7ljnq" Oct 30 01:33:06.942864 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:06.942819 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83968442-b868-4896-84c1-415c42cfbbcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbf1ce8b-7ljnq\" (UID: \"83968442-b868-4896-84c1-415c42cfbbcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf1ce8b-7ljnq" Oct 30 01:33:06.943118 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:06.942950 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf5c9\" (UniqueName: \"kubernetes.io/projected/83968442-b868-4896-84c1-415c42cfbbcc-kube-api-access-qf5c9\") pod \"maas-api-revoke-sub-bbf1ce8b-7ljnq\" (UID: \"83968442-b868-4896-84c1-415c42cfbbcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf1ce8b-7ljnq" Oct 30 01:33:06.943118 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:06.942994 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83968442-b868-4896-84c1-415c42cfbbcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbf1ce8b-7ljnq\" (UID: \"83968442-b868-4896-84c1-415c42cfbbcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf1ce8b-7ljnq" Oct 30 01:33:06.943613 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:06.943588 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83968442-b868-4896-84c1-415c42cfbbcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbf1ce8b-7ljnq\" (UID: \"83968442-b868-4896-84c1-415c42cfbbcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf1ce8b-7ljnq" Oct 30 01:33:06.945454 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:06.945427 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83968442-b868-4896-84c1-415c42cfbbcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbf1ce8b-7ljnq\" (UID: \"83968442-b868-4896-84c1-415c42cfbbcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf1ce8b-7ljnq" Oct 30 01:33:06.961170 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:06.961127 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf5c9\" (UniqueName: \"kubernetes.io/projected/83968442-b868-4896-84c1-415c42cfbbcc-kube-api-access-qf5c9\") pod \"maas-api-revoke-sub-bbf1ce8b-7ljnq\" (UID: \"83968442-b868-4896-84c1-415c42cfbbcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf1ce8b-7ljnq" Oct 30 01:33:06.963495 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:06.963148 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:07.124855 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:07.124727 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf1ce8b-7ljnq" Oct 30 01:33:07.423358 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:33:07.423330 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83968442_b868_4896_84c1_415c42cfbbcc.slice/crio-dfe532b7fd5483e07b5abe70f337c641b91275b23524af2348790e930aed444f WatchSource:0}: Error finding container dfe532b7fd5483e07b5abe70f337c641b91275b23524af2348790e930aed444f: Status 404 returned error can't find the container with id dfe532b7fd5483e07b5abe70f337c641b91275b23524af2348790e930aed444f Oct 30 01:33:07.423502 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:07.423475 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbf1ce8b-7ljnq"] Oct 30 01:33:07.942543 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:07.942500 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf1ce8b-7ljnq" event={"ID":"83968442-b868-4896-84c1-415c42cfbbcc","Type":"ContainerStarted","Data":"25817e4f480731621ce956631f9f3fac978346fd715c7bf062ec3d77c6f9add1"} Oct 30 01:33:07.942940 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:07.942560 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf1ce8b-7ljnq" event={"ID":"83968442-b868-4896-84c1-415c42cfbbcc","Type":"ContainerStarted","Data":"dfe532b7fd5483e07b5abe70f337c641b91275b23524af2348790e930aed444f"} Oct 30 01:33:07.978103 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:07.978008 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf1ce8b-7ljnq" podStartSLOduration=1.977991366 podStartE2EDuration="1.977991366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:07.974962625 +0000 UTC m=+913.551388030" watchObservedRunningTime="2026-10-30 01:33:07.977991366 +0000 UTC m=+913.554416768" Oct 30 01:33:08.678075 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:08.678034 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:08.951815 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:08.951706 6572 generic.go:356] "Generic (PLEG): container finished" podID="83968442-b868-4896-84c1-415c42cfbbcc" containerID="25817e4f480731621ce956631f9f3fac978346fd715c7bf062ec3d77c6f9add1" exitCode=0 Oct 30 01:33:08.952199 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:08.951802 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf1ce8b-7ljnq" event={"ID":"83968442-b868-4896-84c1-415c42cfbbcc","Type":"ContainerDied","Data":"25817e4f480731621ce956631f9f3fac978346fd715c7bf062ec3d77c6f9add1"} Oct 30 01:33:10.171234 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:10.171209 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf1ce8b-7ljnq" Oct 30 01:33:10.238864 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:10.238775 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a09d277a-stkvg"] Oct 30 01:33:10.239542 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:10.239514 6572 cpu_manager.go:414] "containerMap: removing container" podUID="83968442-b868-4896-84c1-415c42cfbbcc" containerName="revoke-keys" Oct 30 01:33:10.239542 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:10.239541 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83968442-b868-4896-84c1-415c42cfbbcc" containerName="revoke-keys" Oct 30 01:33:10.239810 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:10.239792 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="83968442-b868-4896-84c1-415c42cfbbcc" containerName="revoke-keys" Oct 30 01:33:10.244021 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:10.243997 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a09d277a-stkvg" Oct 30 01:33:10.263026 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:10.263000 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a09d277a-stkvg"] Oct 30 01:33:10.297677 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:10.297640 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83968442-b868-4896-84c1-415c42cfbbcc-kube-api-access-qf5c9\" (UniqueName: \"kubernetes.io/projected/83968442-b868-4896-84c1-415c42cfbbcc-kube-api-access-qf5c9\") pod \"83968442-b868-4896-84c1-415c42cfbbcc\" (UID: \"83968442-b868-4896-84c1-415c42cfbbcc\") " Oct 30 01:33:10.297865 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:10.297730 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83968442-b868-4896-84c1-415c42cfbbcc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83968442-b868-4896-84c1-415c42cfbbcc-maas-api-openshift-ca\") pod \"83968442-b868-4896-84c1-415c42cfbbcc\" (UID: \"83968442-b868-4896-84c1-415c42cfbbcc\") " Oct 30 01:33:10.297947 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:10.297908 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83968442-b868-4896-84c1-415c42cfbbcc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83968442-b868-4896-84c1-415c42cfbbcc-maas-api-xks-ca\") pod \"83968442-b868-4896-84c1-415c42cfbbcc\" (UID: \"83968442-b868-4896-84c1-415c42cfbbcc\") " Oct 30 01:33:10.298200 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:10.298169 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83968442-b868-4896-84c1-415c42cfbbcc-maas-api-openshift-ca" pod "83968442-b868-4896-84c1-415c42cfbbcc" (UID: "83968442-b868-4896-84c1-415c42cfbbcc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:33:10.298808 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:10.298694 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eee5e778-212f-432e-8598-d27a840394e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a09d277a-stkvg\" (UID: \"eee5e778-212f-432e-8598-d27a840394e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a09d277a-stkvg" Oct 30 01:33:10.298912 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:10.298880 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c49bp\" (UniqueName: \"kubernetes.io/projected/eee5e778-212f-432e-8598-d27a840394e9-kube-api-access-c49bp\") pod \"maas-api-revoke-sub-a09d277a-stkvg\" (UID: \"eee5e778-212f-432e-8598-d27a840394e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a09d277a-stkvg" Oct 30 01:33:10.299054 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:10.299034 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eee5e778-212f-432e-8598-d27a840394e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a09d277a-stkvg\" (UID: \"eee5e778-212f-432e-8598-d27a840394e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a09d277a-stkvg" Oct 30 01:33:10.299297 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:10.299279 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83968442-b868-4896-84c1-415c42cfbbcc-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:33:10.300383 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:10.300342 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83968442-b868-4896-84c1-415c42cfbbcc-kube-api-access-qf5c9" pod "83968442-b868-4896-84c1-415c42cfbbcc" (UID: "83968442-b868-4896-84c1-415c42cfbbcc"). InnerVolumeSpecName "kube-api-access-qf5c9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:33:10.301860 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:10.301832 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83968442-b868-4896-84c1-415c42cfbbcc-maas-api-xks-ca" pod "83968442-b868-4896-84c1-415c42cfbbcc" (UID: "83968442-b868-4896-84c1-415c42cfbbcc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:33:10.404682 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:10.401013 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eee5e778-212f-432e-8598-d27a840394e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a09d277a-stkvg\" (UID: \"eee5e778-212f-432e-8598-d27a840394e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a09d277a-stkvg" Oct 30 01:33:10.404682 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:10.401164 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c49bp\" (UniqueName: \"kubernetes.io/projected/eee5e778-212f-432e-8598-d27a840394e9-kube-api-access-c49bp\") pod \"maas-api-revoke-sub-a09d277a-stkvg\" (UID: \"eee5e778-212f-432e-8598-d27a840394e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a09d277a-stkvg" Oct 30 01:33:10.404682 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:10.401284 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eee5e778-212f-432e-8598-d27a840394e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a09d277a-stkvg\" (UID: \"eee5e778-212f-432e-8598-d27a840394e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a09d277a-stkvg" Oct 30 01:33:10.404682 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:10.401520 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83968442-b868-4896-84c1-415c42cfbbcc-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:33:10.404682 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:10.401537 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf5c9\" (UniqueName: \"kubernetes.io/projected/83968442-b868-4896-84c1-415c42cfbbcc-kube-api-access-qf5c9\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:33:10.404682 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:10.404003 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eee5e778-212f-432e-8598-d27a840394e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a09d277a-stkvg\" (UID: \"eee5e778-212f-432e-8598-d27a840394e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a09d277a-stkvg" Oct 30 01:33:10.408384 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:10.408348 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eee5e778-212f-432e-8598-d27a840394e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a09d277a-stkvg\" (UID: \"eee5e778-212f-432e-8598-d27a840394e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a09d277a-stkvg" Oct 30 01:33:10.428510 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:10.428467 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c49bp\" (UniqueName: \"kubernetes.io/projected/eee5e778-212f-432e-8598-d27a840394e9-kube-api-access-c49bp\") pod \"maas-api-revoke-sub-a09d277a-stkvg\" (UID: \"eee5e778-212f-432e-8598-d27a840394e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a09d277a-stkvg" Oct 30 01:33:10.462208 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:10.462169 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:10.562513 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:10.562477 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a09d277a-stkvg" Oct 30 01:33:10.863898 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:33:10.863869 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeee5e778_212f_432e_8598_d27a840394e9.slice/crio-90d64fe329151a49036a452bf6cbb700f4410010c6945fd54b9154fbbe0704d0 WatchSource:0}: Error finding container 90d64fe329151a49036a452bf6cbb700f4410010c6945fd54b9154fbbe0704d0: Status 404 returned error can't find the container with id 90d64fe329151a49036a452bf6cbb700f4410010c6945fd54b9154fbbe0704d0 Oct 30 01:33:10.864065 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:10.864042 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a09d277a-stkvg"] Oct 30 01:33:10.972672 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:10.972636 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a09d277a-stkvg" event={"ID":"eee5e778-212f-432e-8598-d27a840394e9","Type":"ContainerStarted","Data":"90d64fe329151a49036a452bf6cbb700f4410010c6945fd54b9154fbbe0704d0"} Oct 30 01:33:10.974244 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:10.974220 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf1ce8b-7ljnq" Oct 30 01:33:10.974381 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:10.974216 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf1ce8b-7ljnq" event={"ID":"83968442-b868-4896-84c1-415c42cfbbcc","Type":"ContainerDied","Data":"dfe532b7fd5483e07b5abe70f337c641b91275b23524af2348790e930aed444f"} Oct 30 01:33:10.974381 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:10.974349 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfe532b7fd5483e07b5abe70f337c641b91275b23524af2348790e930aed444f" Oct 30 01:33:11.439169 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:11.439126 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c84c4f53-rmhlb"] Oct 30 01:33:11.443724 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:11.443694 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c84c4f53-rmhlb" Oct 30 01:33:11.463384 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:11.463341 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c84c4f53-rmhlb"] Oct 30 01:33:11.521427 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:11.521391 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b45c8424-ddb7-4018-9bc4-cf7986fe93b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c84c4f53-rmhlb\" (UID: \"b45c8424-ddb7-4018-9bc4-cf7986fe93b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c84c4f53-rmhlb" Oct 30 01:33:11.521640 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:11.521541 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b45c8424-ddb7-4018-9bc4-cf7986fe93b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c84c4f53-rmhlb\" (UID: \"b45c8424-ddb7-4018-9bc4-cf7986fe93b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c84c4f53-rmhlb" Oct 30 01:33:11.521640 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:11.521589 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fd7b\" (UniqueName: \"kubernetes.io/projected/b45c8424-ddb7-4018-9bc4-cf7986fe93b5-kube-api-access-6fd7b\") pod \"maas-api-revoke-sub-c84c4f53-rmhlb\" (UID: \"b45c8424-ddb7-4018-9bc4-cf7986fe93b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c84c4f53-rmhlb" Oct 30 01:33:11.622648 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:11.622609 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b45c8424-ddb7-4018-9bc4-cf7986fe93b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c84c4f53-rmhlb\" (UID: \"b45c8424-ddb7-4018-9bc4-cf7986fe93b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c84c4f53-rmhlb" Oct 30 01:33:11.622878 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:11.622706 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b45c8424-ddb7-4018-9bc4-cf7986fe93b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c84c4f53-rmhlb\" (UID: \"b45c8424-ddb7-4018-9bc4-cf7986fe93b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c84c4f53-rmhlb" Oct 30 01:33:11.622878 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:11.622740 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fd7b\" (UniqueName: \"kubernetes.io/projected/b45c8424-ddb7-4018-9bc4-cf7986fe93b5-kube-api-access-6fd7b\") pod \"maas-api-revoke-sub-c84c4f53-rmhlb\" (UID: \"b45c8424-ddb7-4018-9bc4-cf7986fe93b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c84c4f53-rmhlb" Oct 30 01:33:11.623522 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:11.623499 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b45c8424-ddb7-4018-9bc4-cf7986fe93b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c84c4f53-rmhlb\" (UID: \"b45c8424-ddb7-4018-9bc4-cf7986fe93b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c84c4f53-rmhlb" Oct 30 01:33:11.625090 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:11.625066 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b45c8424-ddb7-4018-9bc4-cf7986fe93b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c84c4f53-rmhlb\" (UID: \"b45c8424-ddb7-4018-9bc4-cf7986fe93b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c84c4f53-rmhlb" Oct 30 01:33:11.643998 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:11.643966 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fd7b\" (UniqueName: \"kubernetes.io/projected/b45c8424-ddb7-4018-9bc4-cf7986fe93b5-kube-api-access-6fd7b\") pod \"maas-api-revoke-sub-c84c4f53-rmhlb\" (UID: \"b45c8424-ddb7-4018-9bc4-cf7986fe93b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c84c4f53-rmhlb" Oct 30 01:33:11.762043 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:11.761991 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c84c4f53-rmhlb" Oct 30 01:33:11.984224 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:11.984137 6572 generic.go:356] "Generic (PLEG): container finished" podID="eee5e778-212f-432e-8598-d27a840394e9" containerID="714330df0c20aa3bfc7b40e2a4078e9aa8b4e2a5fd215957b901f4e2faf10feb" exitCode=0 Oct 30 01:33:11.984381 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:11.984221 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a09d277a-stkvg" event={"ID":"eee5e778-212f-432e-8598-d27a840394e9","Type":"ContainerDied","Data":"714330df0c20aa3bfc7b40e2a4078e9aa8b4e2a5fd215957b901f4e2faf10feb"} Oct 30 01:33:12.108475 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:12.086485 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c84c4f53-rmhlb"] Oct 30 01:33:12.322849 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:12.319555 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:13.043325 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:13.043289 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c84c4f53-rmhlb" event={"ID":"b45c8424-ddb7-4018-9bc4-cf7986fe93b5","Type":"ContainerStarted","Data":"eddd24cf5e63f63d05db0be8ffc9b9ca70e40000914b1c8b3bfd12c8b09b59f4"} Oct 30 01:33:13.043968 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:13.043927 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c84c4f53-rmhlb" event={"ID":"b45c8424-ddb7-4018-9bc4-cf7986fe93b5","Type":"ContainerStarted","Data":"b870c7bddc4611b39b4089b425901aa4496155287c3eaf1370d106707b1136a1"} Oct 30 01:33:13.114004 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:13.113939 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c84c4f53-rmhlb" podStartSLOduration=2.113917585 podStartE2EDuration="2.113917585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:13.101167306 +0000 UTC m=+918.677592708" watchObservedRunningTime="2026-10-30 01:33:13.113917585 +0000 UTC m=+918.690342987" Oct 30 01:33:13.554783 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:13.548138 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" podUID="0ee3f8e3-9b55-4ce5-afaf-86780a26391e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:13.826920 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:13.826861 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" podUID="fac3ca55-64af-42d9-90b1-0e821b893282" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:13.898970 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:13.827653 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" podUID="e76339df-9659-4bea-80a4-5f18c4a73fd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:13.899552 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:13.899505 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" podUID="9a56da7a-9b92-40e2-b08b-43cf591b5221" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 01:33:13.926225 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:13.841293 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" podUID="cb20484a-51e2-4581-a629-e92d4234784e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:13.969774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:13.968860 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" podUID="317505ef-238c-458e-a74e-1198806b94d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:14.239773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:14.227015 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c84c4f53-rmhlb" event={"ID":"b45c8424-ddb7-4018-9bc4-cf7986fe93b5","Type":"ContainerDied","Data":"eddd24cf5e63f63d05db0be8ffc9b9ca70e40000914b1c8b3bfd12c8b09b59f4"} Oct 30 01:33:14.239773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:14.236277 6572 generic.go:356] "Generic (PLEG): container finished" podID="b45c8424-ddb7-4018-9bc4-cf7986fe93b5" containerID="eddd24cf5e63f63d05db0be8ffc9b9ca70e40000914b1c8b3bfd12c8b09b59f4" exitCode=0 Oct 30 01:33:14.341923 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:14.339135 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:33:14.341923 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:14.339188 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" podUID="eb45f4d6-fea9-4316-97c7-0fa7911073cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:14.341923 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:14.340597 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a09d277a-stkvg" Oct 30 01:33:14.398772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:14.397363 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eee5e778-212f-432e-8598-d27a840394e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eee5e778-212f-432e-8598-d27a840394e9-maas-api-openshift-ca\") pod \"eee5e778-212f-432e-8598-d27a840394e9\" (UID: \"eee5e778-212f-432e-8598-d27a840394e9\") " Oct 30 01:33:14.398772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:14.397415 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eee5e778-212f-432e-8598-d27a840394e9-kube-api-access-c49bp\" (UniqueName: \"kubernetes.io/projected/eee5e778-212f-432e-8598-d27a840394e9-kube-api-access-c49bp\") pod \"eee5e778-212f-432e-8598-d27a840394e9\" (UID: \"eee5e778-212f-432e-8598-d27a840394e9\") " Oct 30 01:33:14.398772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:14.397517 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eee5e778-212f-432e-8598-d27a840394e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eee5e778-212f-432e-8598-d27a840394e9-maas-api-xks-ca\") pod \"eee5e778-212f-432e-8598-d27a840394e9\" (UID: \"eee5e778-212f-432e-8598-d27a840394e9\") " Oct 30 01:33:14.422773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:14.404476 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eee5e778-212f-432e-8598-d27a840394e9-maas-api-openshift-ca" pod "eee5e778-212f-432e-8598-d27a840394e9" (UID: "eee5e778-212f-432e-8598-d27a840394e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:33:14.422773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:14.406059 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eee5e778-212f-432e-8598-d27a840394e9-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:33:14.476824 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:14.467968 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eee5e778-212f-432e-8598-d27a840394e9-kube-api-access-c49bp" pod "eee5e778-212f-432e-8598-d27a840394e9" (UID: "eee5e778-212f-432e-8598-d27a840394e9"). InnerVolumeSpecName "kube-api-access-c49bp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:33:14.637833 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:14.586113 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c49bp\" (UniqueName: \"kubernetes.io/projected/eee5e778-212f-432e-8598-d27a840394e9-kube-api-access-c49bp\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:33:14.637833 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:14.589516 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" podUID="0ee3f8e3-9b55-4ce5-afaf-86780a26391e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:14.844049 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:14.828492 6572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" podUID="9a56da7a-9b92-40e2-b08b-43cf591b5221" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:14.844049 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:14.842219 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" Oct 30 01:33:14.904427 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:14.904377 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" podUID="e76339df-9659-4bea-80a4-5f18c4a73fd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 30 01:33:14.912474 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:14.906561 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" Oct 30 01:33:14.912474 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:14.904668 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" podUID="fac3ca55-64af-42d9-90b1-0e821b893282" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 01:33:14.934000 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:14.914816 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" Oct 30 01:33:14.934000 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:14.929833 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" podUID="cb20484a-51e2-4581-a629-e92d4234784e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 01:33:14.934000 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:14.930292 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" Oct 30 01:33:14.934000 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:14.931211 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" podUID="9a56da7a-9b92-40e2-b08b-43cf591b5221" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:14.934000 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:14.931482 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" Oct 30 01:33:15.000371 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:15.000267 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" podUID="317505ef-238c-458e-a74e-1198806b94d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:15.018964 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:15.001803 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" Oct 30 01:33:15.071215 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:15.052339 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eee5e778-212f-432e-8598-d27a840394e9-maas-api-xks-ca" pod "eee5e778-212f-432e-8598-d27a840394e9" (UID: "eee5e778-212f-432e-8598-d27a840394e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:33:15.405798 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:15.395967 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eee5e778-212f-432e-8598-d27a840394e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eee5e778-212f-432e-8598-d27a840394e9-maas-api-xks-ca\") pod \"eee5e778-212f-432e-8598-d27a840394e9\" (UID: \"eee5e778-212f-432e-8598-d27a840394e9\") " Oct 30 01:33:15.405798 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:33:15.396284 6572 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/eee5e778-212f-432e-8598-d27a840394e9/volumes/kubernetes.io~secret/maas-api-xks-ca Oct 30 01:33:15.405798 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:15.396304 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eee5e778-212f-432e-8598-d27a840394e9-maas-api-xks-ca" pod "eee5e778-212f-432e-8598-d27a840394e9" (UID: "eee5e778-212f-432e-8598-d27a840394e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:33:15.601810 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:15.588034 6572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" podUID="cb20484a-51e2-4581-a629-e92d4234784e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": context deadline exceeded" Oct 30 01:33:15.662984 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:15.601904 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" podUID="0ee3f8e3-9b55-4ce5-afaf-86780a26391e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:15.662984 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:15.613192 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eee5e778-212f-432e-8598-d27a840394e9-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:33:15.728389 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:15.663797 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:15.757335 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:33:15.755966 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:33:15.925065 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:15.924966 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" podUID="e76339df-9659-4bea-80a4-5f18c4a73fd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:15.952151 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:15.942535 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" podUID="9a56da7a-9b92-40e2-b08b-43cf591b5221" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 01:33:15.961773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:15.942573 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" podUID="cb20484a-51e2-4581-a629-e92d4234784e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 01:33:16.004108 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:15.942621 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" podUID="fac3ca55-64af-42d9-90b1-0e821b893282" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:16.004797 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:16.004738 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" podUID="317505ef-238c-458e-a74e-1198806b94d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:16.025512 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:16.025481 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a09d277a-stkvg" Oct 30 01:33:16.050281 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:16.044274 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a09d277a-stkvg" event={"ID":"eee5e778-212f-432e-8598-d27a840394e9","Type":"ContainerDied","Data":"90d64fe329151a49036a452bf6cbb700f4410010c6945fd54b9154fbbe0704d0"} Oct 30 01:33:16.050281 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:16.044334 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90d64fe329151a49036a452bf6cbb700f4410010c6945fd54b9154fbbe0704d0" Oct 30 01:33:16.465784 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:16.457416 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" Oct 30 01:33:16.479880 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:16.479856 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" Oct 30 01:33:16.781389 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:16.781218 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:16.919892 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:16.919864 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" Oct 30 01:33:16.920424 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:16.920404 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" Oct 30 01:33:16.921423 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:16.921407 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" Oct 30 01:33:16.973072 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:16.971911 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" podUID="cb20484a-51e2-4581-a629-e92d4234784e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:16.995271 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:16.991696 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" Oct 30 01:33:17.431444 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:17.428392 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:17.475139 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:17.475112 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c84c4f53-rmhlb" Oct 30 01:33:17.493857 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:17.493822 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b45c8424-ddb7-4018-9bc4-cf7986fe93b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b45c8424-ddb7-4018-9bc4-cf7986fe93b5-maas-api-xks-ca\") pod \"b45c8424-ddb7-4018-9bc4-cf7986fe93b5\" (UID: \"b45c8424-ddb7-4018-9bc4-cf7986fe93b5\") " Oct 30 01:33:17.494048 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:17.494026 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b45c8424-ddb7-4018-9bc4-cf7986fe93b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b45c8424-ddb7-4018-9bc4-cf7986fe93b5-maas-api-openshift-ca\") pod \"b45c8424-ddb7-4018-9bc4-cf7986fe93b5\" (UID: \"b45c8424-ddb7-4018-9bc4-cf7986fe93b5\") " Oct 30 01:33:17.494229 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:17.494212 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b45c8424-ddb7-4018-9bc4-cf7986fe93b5-kube-api-access-6fd7b\" (UniqueName: \"kubernetes.io/projected/b45c8424-ddb7-4018-9bc4-cf7986fe93b5-kube-api-access-6fd7b\") pod \"b45c8424-ddb7-4018-9bc4-cf7986fe93b5\" (UID: \"b45c8424-ddb7-4018-9bc4-cf7986fe93b5\") " Oct 30 01:33:17.495553 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:17.495525 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b45c8424-ddb7-4018-9bc4-cf7986fe93b5-maas-api-openshift-ca" pod "b45c8424-ddb7-4018-9bc4-cf7986fe93b5" (UID: "b45c8424-ddb7-4018-9bc4-cf7986fe93b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:33:17.501170 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:17.500926 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b45c8424-ddb7-4018-9bc4-cf7986fe93b5-kube-api-access-6fd7b" pod "b45c8424-ddb7-4018-9bc4-cf7986fe93b5" (UID: "b45c8424-ddb7-4018-9bc4-cf7986fe93b5"). InnerVolumeSpecName "kube-api-access-6fd7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:33:17.551319 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:17.551259 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b45c8424-ddb7-4018-9bc4-cf7986fe93b5-maas-api-xks-ca" pod "b45c8424-ddb7-4018-9bc4-cf7986fe93b5" (UID: "b45c8424-ddb7-4018-9bc4-cf7986fe93b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:33:17.596980 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:17.596879 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fd7b\" (UniqueName: \"kubernetes.io/projected/b45c8424-ddb7-4018-9bc4-cf7986fe93b5-kube-api-access-6fd7b\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:33:17.596980 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:17.596915 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b45c8424-ddb7-4018-9bc4-cf7986fe93b5-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:33:17.596980 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:17.596931 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b45c8424-ddb7-4018-9bc4-cf7986fe93b5-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:33:18.127771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:18.127700 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c84c4f53-rmhlb" event={"ID":"b45c8424-ddb7-4018-9bc4-cf7986fe93b5","Type":"ContainerDied","Data":"b870c7bddc4611b39b4089b425901aa4496155287c3eaf1370d106707b1136a1"} Oct 30 01:33:18.128038 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:18.128008 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b870c7bddc4611b39b4089b425901aa4496155287c3eaf1370d106707b1136a1" Oct 30 01:33:18.128251 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:18.128237 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c84c4f53-rmhlb" Oct 30 01:33:18.885415 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:18.885378 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:20.632615 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:20.632579 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:22.348825 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:22.348740 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:22.730420 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:22.730380 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:23.006916 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.006824 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f"] Oct 30 01:33:23.007603 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.007580 6572 cpu_manager.go:414] "containerMap: removing container" podUID="eee5e778-212f-432e-8598-d27a840394e9" containerName="revoke-keys" Oct 30 01:33:23.007793 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.007776 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eee5e778-212f-432e-8598-d27a840394e9" containerName="revoke-keys" Oct 30 01:33:23.007939 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.007925 6572 cpu_manager.go:414] "containerMap: removing container" podUID="b45c8424-ddb7-4018-9bc4-cf7986fe93b5" containerName="revoke-keys" Oct 30 01:33:23.008075 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.008051 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b45c8424-ddb7-4018-9bc4-cf7986fe93b5" containerName="revoke-keys" Oct 30 01:33:23.008376 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.008359 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="eee5e778-212f-432e-8598-d27a840394e9" containerName="revoke-keys" Oct 30 01:33:23.008376 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.008377 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="b45c8424-ddb7-4018-9bc4-cf7986fe93b5" containerName="revoke-keys" Oct 30 01:33:23.020495 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.014726 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:33:23.040964 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.040921 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ec2345-openshift-default-dockercfg-h8ssh\"" Oct 30 01:33:23.045612 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.045581 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f"] Oct 30 01:33:23.102501 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.102464 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/159d5541-9dbd-4b27-bdb2-019f09868d42-workload-certs\") pod \"e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f\" (UID: \"159d5541-9dbd-4b27-bdb2-019f09868d42\") " pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:33:23.102501 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.102499 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/159d5541-9dbd-4b27-bdb2-019f09868d42-istiod-ca-cert\") pod \"e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f\" (UID: \"159d5541-9dbd-4b27-bdb2-019f09868d42\") " pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:33:23.102741 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.102523 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/159d5541-9dbd-4b27-bdb2-019f09868d42-credential-socket\") pod \"e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f\" (UID: \"159d5541-9dbd-4b27-bdb2-019f09868d42\") " pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:33:23.102741 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.102575 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xkf4\" (UniqueName: \"kubernetes.io/projected/159d5541-9dbd-4b27-bdb2-019f09868d42-kube-api-access-7xkf4\") pod \"e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f\" (UID: \"159d5541-9dbd-4b27-bdb2-019f09868d42\") " pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:33:23.102861 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.102800 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/159d5541-9dbd-4b27-bdb2-019f09868d42-istio-envoy\") pod \"e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f\" (UID: \"159d5541-9dbd-4b27-bdb2-019f09868d42\") " pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:33:23.102861 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.102826 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/159d5541-9dbd-4b27-bdb2-019f09868d42-istio-token\") pod \"e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f\" (UID: \"159d5541-9dbd-4b27-bdb2-019f09868d42\") " pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:33:23.102861 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.102845 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/159d5541-9dbd-4b27-bdb2-019f09868d42-istio-podinfo\") pod \"e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f\" (UID: \"159d5541-9dbd-4b27-bdb2-019f09868d42\") " pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:33:23.103045 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.102862 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/159d5541-9dbd-4b27-bdb2-019f09868d42-istio-data\") pod \"e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f\" (UID: \"159d5541-9dbd-4b27-bdb2-019f09868d42\") " pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:33:23.103083 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.103044 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/159d5541-9dbd-4b27-bdb2-019f09868d42-workload-socket\") pod \"e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f\" (UID: \"159d5541-9dbd-4b27-bdb2-019f09868d42\") " pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:33:23.160665 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.160626 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:23.214237 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.214196 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/159d5541-9dbd-4b27-bdb2-019f09868d42-istio-envoy\") pod \"e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f\" (UID: \"159d5541-9dbd-4b27-bdb2-019f09868d42\") " pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:33:23.218374 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.218346 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/159d5541-9dbd-4b27-bdb2-019f09868d42-istio-token\") pod \"e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f\" (UID: \"159d5541-9dbd-4b27-bdb2-019f09868d42\") " pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:33:23.220066 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.220043 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/159d5541-9dbd-4b27-bdb2-019f09868d42-istio-podinfo\") pod \"e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f\" (UID: \"159d5541-9dbd-4b27-bdb2-019f09868d42\") " pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:33:23.220259 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.220232 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/159d5541-9dbd-4b27-bdb2-019f09868d42-istio-data\") pod \"e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f\" (UID: \"159d5541-9dbd-4b27-bdb2-019f09868d42\") " pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:33:23.220487 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.220447 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/159d5541-9dbd-4b27-bdb2-019f09868d42-workload-socket\") pod \"e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f\" (UID: \"159d5541-9dbd-4b27-bdb2-019f09868d42\") " pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:33:23.220679 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.220662 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/159d5541-9dbd-4b27-bdb2-019f09868d42-workload-certs\") pod \"e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f\" (UID: \"159d5541-9dbd-4b27-bdb2-019f09868d42\") " pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:33:23.220840 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.220824 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/159d5541-9dbd-4b27-bdb2-019f09868d42-istiod-ca-cert\") pod \"e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f\" (UID: \"159d5541-9dbd-4b27-bdb2-019f09868d42\") " pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:33:23.220972 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.220951 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/159d5541-9dbd-4b27-bdb2-019f09868d42-credential-socket\") pod \"e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f\" (UID: \"159d5541-9dbd-4b27-bdb2-019f09868d42\") " pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:33:23.221082 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.221067 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xkf4\" (UniqueName: \"kubernetes.io/projected/159d5541-9dbd-4b27-bdb2-019f09868d42-kube-api-access-7xkf4\") pod \"e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f\" (UID: \"159d5541-9dbd-4b27-bdb2-019f09868d42\") " pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:33:23.225124 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.223858 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/159d5541-9dbd-4b27-bdb2-019f09868d42-istio-data\") pod \"e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f\" (UID: \"159d5541-9dbd-4b27-bdb2-019f09868d42\") " pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:33:23.225124 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.224501 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/159d5541-9dbd-4b27-bdb2-019f09868d42-istiod-ca-cert\") pod \"e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f\" (UID: \"159d5541-9dbd-4b27-bdb2-019f09868d42\") " pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:33:23.225124 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.224808 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/159d5541-9dbd-4b27-bdb2-019f09868d42-workload-socket\") pod \"e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f\" (UID: \"159d5541-9dbd-4b27-bdb2-019f09868d42\") " pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:33:23.225124 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.225057 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/159d5541-9dbd-4b27-bdb2-019f09868d42-credential-socket\") pod \"e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f\" (UID: \"159d5541-9dbd-4b27-bdb2-019f09868d42\") " pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:33:23.225700 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.225677 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/159d5541-9dbd-4b27-bdb2-019f09868d42-workload-certs\") pod \"e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f\" (UID: \"159d5541-9dbd-4b27-bdb2-019f09868d42\") " pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:33:23.244248 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.244180 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/159d5541-9dbd-4b27-bdb2-019f09868d42-istio-token\") pod \"e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f\" (UID: \"159d5541-9dbd-4b27-bdb2-019f09868d42\") " pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:33:23.256524 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.256483 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/159d5541-9dbd-4b27-bdb2-019f09868d42-istio-podinfo\") pod \"e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f\" (UID: \"159d5541-9dbd-4b27-bdb2-019f09868d42\") " pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:33:23.258089 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.258021 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/159d5541-9dbd-4b27-bdb2-019f09868d42-istio-envoy\") pod \"e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f\" (UID: \"159d5541-9dbd-4b27-bdb2-019f09868d42\") " pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:33:23.258905 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.258854 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xkf4\" (UniqueName: \"kubernetes.io/projected/159d5541-9dbd-4b27-bdb2-019f09868d42-kube-api-access-7xkf4\") pod \"e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f\" (UID: \"159d5541-9dbd-4b27-bdb2-019f09868d42\") " pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:33:23.355306 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.355269 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:23.365857 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.365832 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:33:23.735433 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.735402 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f"] Oct 30 01:33:23.736259 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:33:23.736229 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod159d5541_9dbd_4b27_bdb2_019f09868d42.slice/crio-119ecc1d72ef7600693186d95140aeb855c1bf3f309b9be910d601bb22ecc8a9 WatchSource:0}: Error finding container 119ecc1d72ef7600693186d95140aeb855c1bf3f309b9be910d601bb22ecc8a9: Status 404 returned error can't find the container with id 119ecc1d72ef7600693186d95140aeb855c1bf3f309b9be910d601bb22ecc8a9 Oct 30 01:33:23.739736 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.739689 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:33:23.739857 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.739833 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:33:23.739909 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.739890 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:33:23.821508 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:23.821477 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:24.193599 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:24.193569 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:24.241216 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:24.241166 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" event={"ID":"159d5541-9dbd-4b27-bdb2-019f09868d42","Type":"ContainerStarted","Data":"b5dce9d5632391bc915d87a50ab0845ff095c1a1c49bde3b0af1eed1daf81dda"} Oct 30 01:33:24.241395 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:24.241243 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" event={"ID":"159d5541-9dbd-4b27-bdb2-019f09868d42","Type":"ContainerStarted","Data":"119ecc1d72ef7600693186d95140aeb855c1bf3f309b9be910d601bb22ecc8a9"} Oct 30 01:33:24.266936 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:24.266904 6572 patch_prober.go:29] interesting pod/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:33:24.267100 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:24.266953 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" podUID="7578c814-103e-433a-b58e-a173bc4d3a97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:24.279925 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:24.279873 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" podStartSLOduration=2.279858825 podStartE2EDuration="2.279858825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:24.277135953 +0000 UTC m=+929.853561357" watchObservedRunningTime="2026-10-30 01:33:24.279858825 +0000 UTC m=+929.856284289" Oct 30 01:33:24.366258 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:24.366223 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:33:24.372324 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:24.372297 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:33:24.965771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:24.964790 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl"] Oct 30 01:33:24.988053 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:24.987975 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl"] Oct 30 01:33:24.988218 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:24.988104 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" Oct 30 01:33:25.060404 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.060326 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdpw5\" (UniqueName: \"kubernetes.io/projected/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-kube-api-access-zdpw5\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl\" (UID: \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" Oct 30 01:33:25.060404 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.060363 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-workload-certs\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl\" (UID: \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" Oct 30 01:33:25.060635 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.060446 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-istio-envoy\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl\" (UID: \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" Oct 30 01:33:25.060635 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.060518 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-istio-token\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl\" (UID: \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" Oct 30 01:33:25.060635 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.060585 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-credential-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl\" (UID: \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" Oct 30 01:33:25.060917 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.060708 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-workload-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl\" (UID: \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" Oct 30 01:33:25.060917 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.060862 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-istio-podinfo\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl\" (UID: \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" Oct 30 01:33:25.060917 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.060906 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-istiod-ca-cert\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl\" (UID: \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" Oct 30 01:33:25.061075 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.061027 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-istio-data\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl\" (UID: \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" Oct 30 01:33:25.162076 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.162039 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdpw5\" (UniqueName: \"kubernetes.io/projected/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-kube-api-access-zdpw5\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl\" (UID: \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" Oct 30 01:33:25.162076 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.162079 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-workload-certs\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl\" (UID: \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" Oct 30 01:33:25.162299 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.162210 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-istio-envoy\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl\" (UID: \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" Oct 30 01:33:25.162299 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.162247 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-istio-token\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl\" (UID: \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" Oct 30 01:33:25.162415 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.162312 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-credential-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl\" (UID: \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" Oct 30 01:33:25.162415 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.162401 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-workload-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl\" (UID: \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" Oct 30 01:33:25.162542 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.162480 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-istio-podinfo\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl\" (UID: \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" Oct 30 01:33:25.162542 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.162529 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-istiod-ca-cert\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl\" (UID: \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" Oct 30 01:33:25.162666 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.162602 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-workload-certs\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl\" (UID: \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" Oct 30 01:33:25.162666 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.162630 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-istio-data\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl\" (UID: \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" Oct 30 01:33:25.162817 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.162723 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-credential-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl\" (UID: \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" Oct 30 01:33:25.162817 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.162803 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-workload-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl\" (UID: \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" Oct 30 01:33:25.162988 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.162968 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-istio-data\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl\" (UID: \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" Oct 30 01:33:25.163247 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.163226 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-istiod-ca-cert\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl\" (UID: \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" Oct 30 01:33:25.164574 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.164552 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-istio-envoy\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl\" (UID: \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" Oct 30 01:33:25.164841 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.164825 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-istio-podinfo\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl\" (UID: \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" Oct 30 01:33:25.178680 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.178653 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-istio-token\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl\" (UID: \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" Oct 30 01:33:25.178787 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.178741 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdpw5\" (UniqueName: \"kubernetes.io/projected/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-kube-api-access-zdpw5\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl\" (UID: \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" Oct 30 01:33:25.250319 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.250289 6572 generic.go:356] "Generic (PLEG): container finished" podID="7578c814-103e-433a-b58e-a173bc4d3a97" containerID="e5327fe09cd3167ce229da0ac44ae5f511e694c63e0318f3f7439835e5508e60" exitCode=0 Oct 30 01:33:25.250458 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.250365 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" event={"ID":"7578c814-103e-433a-b58e-a173bc4d3a97","Type":"ContainerDied","Data":"e5327fe09cd3167ce229da0ac44ae5f511e694c63e0318f3f7439835e5508e60"} Oct 30 01:33:25.250856 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.250835 6572 scope.go:122] "RemoveContainer" containerID="e5327fe09cd3167ce229da0ac44ae5f511e694c63e0318f3f7439835e5508e60" Oct 30 01:33:25.250958 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.250878 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:33:25.251883 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.251761 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:33:25.252340 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.252313 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:33:25.252392 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.252379 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:33:25.252428 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.252412 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:33:25.316387 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.303833 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" Oct 30 01:33:25.649272 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.649240 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl"] Oct 30 01:33:25.668375 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.666327 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:33:25.668375 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.666461 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:33:25.668375 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:25.666554 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:33:26.124345 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:26.124307 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:26.259988 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:26.259950 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" event={"ID":"7578c814-103e-433a-b58e-a173bc4d3a97","Type":"ContainerStarted","Data":"e3bf7533c6e01bd409fc6b2ae1254ee1792e70af5f0d874b1e33aa61538a9ad1"} Oct 30 01:33:26.261611 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:26.261583 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" event={"ID":"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb","Type":"ContainerStarted","Data":"83597389fbfc1de2fabbee107ee6deda551155cde09f7936df199a20c6428634"} Oct 30 01:33:26.261741 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:26.261633 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" event={"ID":"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb","Type":"ContainerStarted","Data":"6747cb5448e33cd4e279c88635198c2706803b76f9f78423b596916626a2b475"} Oct 30 01:33:26.305503 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:26.305467 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" Oct 30 01:33:26.306454 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:26.306425 6572 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 30 01:33:26.306940 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:26.306676 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" podUID="e7422bb8-dea0-4478-b1c8-05d1f1df1dcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 30 01:33:26.345135 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:26.342181 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" podStartSLOduration=2.342163213 podStartE2EDuration="2.342163213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:26.341214921 +0000 UTC m=+931.917640325" watchObservedRunningTime="2026-10-30 01:33:26.342163213 +0000 UTC m=+931.918588616" Oct 30 01:33:26.856422 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:26.856331 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:27.061440 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:27.061405 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:27.159822 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:27.159677 6572 patch_prober.go:29] interesting pod/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 01:33:27.159822 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:27.159722 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" podUID="7578c814-103e-433a-b58e-a173bc4d3a97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 01:33:27.160020 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:27.159962 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:33:27.305235 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:27.304954 6572 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 30 01:33:27.305235 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:27.305018 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" podUID="e7422bb8-dea0-4478-b1c8-05d1f1df1dcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 30 01:33:27.471115 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:27.471031 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:27.915317 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:27.915288 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:28.159164 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:28.159129 6572 patch_prober.go:29] interesting pod/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 01:33:28.159369 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:28.159187 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" podUID="7578c814-103e-433a-b58e-a173bc4d3a97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 01:33:28.305068 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:28.304926 6572 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 30 01:33:28.305412 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:28.305351 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" podUID="e7422bb8-dea0-4478-b1c8-05d1f1df1dcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 30 01:33:28.353802 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:28.353761 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:29.158946 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.158901 6572 patch_prober.go:29] interesting pod/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 01:33:29.159459 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.158964 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" podUID="7578c814-103e-433a-b58e-a173bc4d3a97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 01:33:29.305658 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.304992 6572 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 30 01:33:29.305658 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.305069 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" podUID="e7422bb8-dea0-4478-b1c8-05d1f1df1dcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 30 01:33:29.360775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.359786 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:29.377801 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.377774 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ec2345-6f57b55f9d-sm5jj"] Oct 30 01:33:29.382440 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.382421 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ec2345-6f57b55f9d-sm5jj" Oct 30 01:33:29.405116 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.405078 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ec2345\"" Oct 30 01:33:29.414476 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.405737 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ec2345-dockercfg-v4pzh\"" Oct 30 01:33:29.478295 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.472634 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ec2345-6f57b55f9d-sm5jj"] Oct 30 01:33:29.482588 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.482559 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ec2345-55774dc5f5-4ghc6"] Oct 30 01:33:29.485975 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.485939 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svx52\" (UniqueName: \"kubernetes.io/projected/2f3be9f9-cc25-4601-9015-8a323abb7db1-kube-api-access-svx52\") pod \"payload-processing-e2e-api-a-ec2345-6f57b55f9d-sm5jj\" (UID: \"2f3be9f9-cc25-4601-9015-8a323abb7db1\") " pod="openshift-ingress/payload-processing-e2e-api-a-ec2345-6f57b55f9d-sm5jj" Oct 30 01:33:29.486120 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.486107 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f3be9f9-cc25-4601-9015-8a323abb7db1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ec2345-6f57b55f9d-sm5jj\" (UID: \"2f3be9f9-cc25-4601-9015-8a323abb7db1\") " pod="openshift-ingress/payload-processing-e2e-api-a-ec2345-6f57b55f9d-sm5jj" Oct 30 01:33:29.489231 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.489207 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec2345-55774dc5f5-4ghc6" Oct 30 01:33:29.489431 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.489403 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ec2345-55774dc5f5-4ghc6"] Oct 30 01:33:29.633769 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.623426 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f3be9f9-cc25-4601-9015-8a323abb7db1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ec2345-6f57b55f9d-sm5jj\" (UID: \"2f3be9f9-cc25-4601-9015-8a323abb7db1\") " pod="openshift-ingress/payload-processing-e2e-api-a-ec2345-6f57b55f9d-sm5jj" Oct 30 01:33:29.633769 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.623612 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svx52\" (UniqueName: \"kubernetes.io/projected/2f3be9f9-cc25-4601-9015-8a323abb7db1-kube-api-access-svx52\") pod \"payload-processing-e2e-api-a-ec2345-6f57b55f9d-sm5jj\" (UID: \"2f3be9f9-cc25-4601-9015-8a323abb7db1\") " pod="openshift-ingress/payload-processing-e2e-api-a-ec2345-6f57b55f9d-sm5jj" Oct 30 01:33:29.633769 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.625176 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f3be9f9-cc25-4601-9015-8a323abb7db1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ec2345-6f57b55f9d-sm5jj\" (UID: \"2f3be9f9-cc25-4601-9015-8a323abb7db1\") " pod="openshift-ingress/payload-processing-e2e-api-a-ec2345-6f57b55f9d-sm5jj" Oct 30 01:33:29.646778 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.639072 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:29.702565 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.702482 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svx52\" (UniqueName: \"kubernetes.io/projected/2f3be9f9-cc25-4601-9015-8a323abb7db1-kube-api-access-svx52\") pod \"payload-processing-e2e-api-a-ec2345-6f57b55f9d-sm5jj\" (UID: \"2f3be9f9-cc25-4601-9015-8a323abb7db1\") " pod="openshift-ingress/payload-processing-e2e-api-a-ec2345-6f57b55f9d-sm5jj" Oct 30 01:33:29.729812 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.729775 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9ln8\" (UniqueName: \"kubernetes.io/projected/ea7df11c-947f-49ee-9174-28d78fef15ca-kube-api-access-b9ln8\") pod \"payload-pre-processing-e2e-api-a-ec2345-55774dc5f5-4ghc6\" (UID: \"ea7df11c-947f-49ee-9174-28d78fef15ca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec2345-55774dc5f5-4ghc6" Oct 30 01:33:29.735585 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.730338 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea7df11c-947f-49ee-9174-28d78fef15ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ec2345-55774dc5f5-4ghc6\" (UID: \"ea7df11c-947f-49ee-9174-28d78fef15ca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec2345-55774dc5f5-4ghc6" Oct 30 01:33:29.849772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.849595 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea7df11c-947f-49ee-9174-28d78fef15ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ec2345-55774dc5f5-4ghc6\" (UID: \"ea7df11c-947f-49ee-9174-28d78fef15ca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec2345-55774dc5f5-4ghc6" Oct 30 01:33:29.849772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.849705 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9ln8\" (UniqueName: \"kubernetes.io/projected/ea7df11c-947f-49ee-9174-28d78fef15ca-kube-api-access-b9ln8\") pod \"payload-pre-processing-e2e-api-a-ec2345-55774dc5f5-4ghc6\" (UID: \"ea7df11c-947f-49ee-9174-28d78fef15ca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec2345-55774dc5f5-4ghc6" Oct 30 01:33:29.853770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.851170 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea7df11c-947f-49ee-9174-28d78fef15ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ec2345-55774dc5f5-4ghc6\" (UID: \"ea7df11c-947f-49ee-9174-28d78fef15ca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec2345-55774dc5f5-4ghc6" Oct 30 01:33:29.865988 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.865959 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb"] Oct 30 01:33:29.869584 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.869549 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9ln8\" (UniqueName: \"kubernetes.io/projected/ea7df11c-947f-49ee-9174-28d78fef15ca-kube-api-access-b9ln8\") pod \"payload-pre-processing-e2e-api-a-ec2345-55774dc5f5-4ghc6\" (UID: \"ea7df11c-947f-49ee-9174-28d78fef15ca\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec2345-55774dc5f5-4ghc6" Oct 30 01:33:29.872105 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.872087 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" Oct 30 01:33:29.891718 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.891693 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb"] Oct 30 01:33:29.964818 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.964706 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d4d67f-e18a-4152-be41-e1832bfd78ed-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nsfcb\" (UID: \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" Oct 30 01:33:29.964818 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.964782 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d4d67f-e18a-4152-be41-e1832bfd78ed-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nsfcb\" (UID: \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" Oct 30 01:33:29.965011 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.964828 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d4d67f-e18a-4152-be41-e1832bfd78ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nsfcb\" (UID: \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" Oct 30 01:33:29.965011 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.964885 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d4d67f-e18a-4152-be41-e1832bfd78ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nsfcb\" (UID: \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" Oct 30 01:33:29.965011 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.964942 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbf5x\" (UniqueName: \"kubernetes.io/projected/c1d4d67f-e18a-4152-be41-e1832bfd78ed-kube-api-access-cbf5x\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nsfcb\" (UID: \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" Oct 30 01:33:29.965111 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.965035 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d4d67f-e18a-4152-be41-e1832bfd78ed-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nsfcb\" (UID: \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" Oct 30 01:33:29.965111 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.965051 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d4d67f-e18a-4152-be41-e1832bfd78ed-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nsfcb\" (UID: \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" Oct 30 01:33:29.965111 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.965076 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d4d67f-e18a-4152-be41-e1832bfd78ed-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nsfcb\" (UID: \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" Oct 30 01:33:29.965111 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:29.965105 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d4d67f-e18a-4152-be41-e1832bfd78ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nsfcb\" (UID: \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" Oct 30 01:33:30.000543 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:30.000516 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ec2345-6f57b55f9d-sm5jj" Oct 30 01:33:30.071094 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:30.066455 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d4d67f-e18a-4152-be41-e1832bfd78ed-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nsfcb\" (UID: \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" Oct 30 01:33:30.071094 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:30.066497 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d4d67f-e18a-4152-be41-e1832bfd78ed-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nsfcb\" (UID: \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" Oct 30 01:33:30.071094 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:30.066540 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d4d67f-e18a-4152-be41-e1832bfd78ed-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nsfcb\" (UID: \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" Oct 30 01:33:30.071094 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:30.066588 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d4d67f-e18a-4152-be41-e1832bfd78ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nsfcb\" (UID: \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" Oct 30 01:33:30.071094 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:30.066717 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d4d67f-e18a-4152-be41-e1832bfd78ed-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nsfcb\" (UID: \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" Oct 30 01:33:30.071094 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:30.066772 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d4d67f-e18a-4152-be41-e1832bfd78ed-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nsfcb\" (UID: \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" Oct 30 01:33:30.071094 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:30.066803 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d4d67f-e18a-4152-be41-e1832bfd78ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nsfcb\" (UID: \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" Oct 30 01:33:30.071094 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:30.066838 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d4d67f-e18a-4152-be41-e1832bfd78ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nsfcb\" (UID: \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" Oct 30 01:33:30.071094 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:30.066882 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbf5x\" (UniqueName: \"kubernetes.io/projected/c1d4d67f-e18a-4152-be41-e1832bfd78ed-kube-api-access-cbf5x\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nsfcb\" (UID: \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" Oct 30 01:33:30.071094 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:30.067625 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d4d67f-e18a-4152-be41-e1832bfd78ed-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nsfcb\" (UID: \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" Oct 30 01:33:30.071094 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:30.068955 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d4d67f-e18a-4152-be41-e1832bfd78ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nsfcb\" (UID: \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" Oct 30 01:33:30.071094 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:30.070913 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d4d67f-e18a-4152-be41-e1832bfd78ed-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nsfcb\" (UID: \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" Oct 30 01:33:30.071810 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:30.071173 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d4d67f-e18a-4152-be41-e1832bfd78ed-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nsfcb\" (UID: \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" Oct 30 01:33:30.071810 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:30.071364 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d4d67f-e18a-4152-be41-e1832bfd78ed-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nsfcb\" (UID: \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" Oct 30 01:33:30.072940 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:30.072919 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d4d67f-e18a-4152-be41-e1832bfd78ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nsfcb\" (UID: \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" Oct 30 01:33:30.073994 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:30.073972 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d4d67f-e18a-4152-be41-e1832bfd78ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nsfcb\" (UID: \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" Oct 30 01:33:30.083478 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:30.083448 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbf5x\" (UniqueName: \"kubernetes.io/projected/c1d4d67f-e18a-4152-be41-e1832bfd78ed-kube-api-access-cbf5x\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nsfcb\" (UID: \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" Oct 30 01:33:30.084112 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:30.084091 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d4d67f-e18a-4152-be41-e1832bfd78ed-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-nsfcb\" (UID: \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" Oct 30 01:33:30.106195 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:30.105961 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec2345-55774dc5f5-4ghc6" Oct 30 01:33:30.159308 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:30.159073 6572 patch_prober.go:29] interesting pod/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 01:33:30.159308 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:30.159127 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" podUID="7578c814-103e-433a-b58e-a173bc4d3a97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 01:33:30.199694 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:30.199663 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" Oct 30 01:33:30.295215 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:30.295176 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:30.315769 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:30.304773 6572 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 30 01:33:30.315769 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:30.304825 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" podUID="e7422bb8-dea0-4478-b1c8-05d1f1df1dcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 30 01:33:30.404145 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:30.404097 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ec2345-6f57b55f9d-sm5jj"] Oct 30 01:33:30.667189 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:30.667166 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ec2345-55774dc5f5-4ghc6"] Oct 30 01:33:30.748094 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:30.748044 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb"] Oct 30 01:33:30.785981 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:30.785942 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:33:30.786067 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:30.786021 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:33:30.786067 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:30.786049 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:33:31.161162 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:31.161130 6572 patch_prober.go:29] interesting pod/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 01:33:31.161713 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:31.161682 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" podUID="7578c814-103e-433a-b58e-a173bc4d3a97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 01:33:31.304481 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:31.304446 6572 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 30 01:33:31.304679 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:31.304504 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" podUID="e7422bb8-dea0-4478-b1c8-05d1f1df1dcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 30 01:33:31.375769 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:31.375524 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" event={"ID":"c1d4d67f-e18a-4152-be41-e1832bfd78ed","Type":"ContainerStarted","Data":"be6a218ad2dced4589e2c8870ca85335b324ff7b660c70950ec65701e681be52"} Oct 30 01:33:31.375769 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:31.375582 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" event={"ID":"c1d4d67f-e18a-4152-be41-e1832bfd78ed","Type":"ContainerStarted","Data":"e9d1ea4d0ad4770670163b524e46e94d377dae48e936b3fdb617a02f044165d2"} Oct 30 01:33:31.378554 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:31.378518 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ec2345-6f57b55f9d-sm5jj" event={"ID":"2f3be9f9-cc25-4601-9015-8a323abb7db1","Type":"ContainerStarted","Data":"2fb9ac42dba5fed2c2bdfa9d3dfc0fd38619e33e961b8a4a234bfb5ec67c8980"} Oct 30 01:33:31.378700 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:31.378572 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ec2345-6f57b55f9d-sm5jj" event={"ID":"2f3be9f9-cc25-4601-9015-8a323abb7db1","Type":"ContainerStarted","Data":"1bf568ba52b4c0ab8445d975f61ad5fde9bce801763f916ffce8cc6a640ec07a"} Oct 30 01:33:31.378700 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:31.378679 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-ec2345-6f57b55f9d-sm5jj" Oct 30 01:33:31.380729 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:31.380693 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec2345-55774dc5f5-4ghc6" event={"ID":"ea7df11c-947f-49ee-9174-28d78fef15ca","Type":"ContainerStarted","Data":"a9c63d386ae6b6bad0ec42e36fbd1f691740c5fa8d862532c09b474abe9e2f20"} Oct 30 01:33:31.380882 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:31.380740 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec2345-55774dc5f5-4ghc6" event={"ID":"ea7df11c-947f-49ee-9174-28d78fef15ca","Type":"ContainerStarted","Data":"3ef76528016fa1221fc99b037eafcb4ef280eaaff20f25712cfd94709a11479a"} Oct 30 01:33:31.381165 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:31.381149 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec2345-55774dc5f5-4ghc6" Oct 30 01:33:31.434233 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:31.434114 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" podStartSLOduration=2.434092815 podStartE2EDuration="2.434092815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:31.421564261 +0000 UTC m=+936.997989679" watchObservedRunningTime="2026-10-30 01:33:31.434092815 +0000 UTC m=+937.010518218" Oct 30 01:33:31.464864 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:31.464794 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec2345-55774dc5f5-4ghc6" podStartSLOduration=2.464775231 podStartE2EDuration="2.464775231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:31.455822527 +0000 UTC m=+937.032247926" watchObservedRunningTime="2026-10-30 01:33:31.464775231 +0000 UTC m=+937.041200634" Oct 30 01:33:31.496008 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:31.495944 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-ec2345-6f57b55f9d-sm5jj" podStartSLOduration=2.495930497 podStartE2EDuration="2.495930497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:31.493734222 +0000 UTC m=+937.070159680" watchObservedRunningTime="2026-10-30 01:33:31.495930497 +0000 UTC m=+937.072355922" Oct 30 01:33:32.101626 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:32.101568 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:32.163464 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:32.163435 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:33:32.163961 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:32.163920 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:33:32.164563 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:32.164544 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:33:32.200442 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:32.200413 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" Oct 30 01:33:32.201890 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:32.201866 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-nsfcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 30 01:33:32.202006 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:32.201920 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" podUID="c1d4d67f-e18a-4152-be41-e1832bfd78ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 30 01:33:32.304181 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:32.304145 6572 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 30 01:33:32.304387 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:32.304200 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" podUID="e7422bb8-dea0-4478-b1c8-05d1f1df1dcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 30 01:33:33.200924 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:33.200886 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-nsfcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 30 01:33:33.201313 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:33.200953 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" podUID="c1d4d67f-e18a-4152-be41-e1832bfd78ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 30 01:33:33.664218 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:33.664188 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" Oct 30 01:33:33.664653 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:33.664622 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" Oct 30 01:33:33.665480 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:33.665461 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" Oct 30 01:33:33.854406 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:33.854374 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:34.200209 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:34.200178 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-nsfcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 30 01:33:34.200378 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:34.200226 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" podUID="c1d4d67f-e18a-4152-be41-e1832bfd78ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 30 01:33:34.548209 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:34.548132 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:35.200868 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:35.200836 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-nsfcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 30 01:33:35.201053 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:35.200895 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" podUID="c1d4d67f-e18a-4152-be41-e1832bfd78ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 30 01:33:35.636848 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:35.636283 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:36.200526 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:36.200494 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-nsfcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 30 01:33:36.200682 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:36.200551 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" podUID="c1d4d67f-e18a-4152-be41-e1832bfd78ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 30 01:33:37.209773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:37.209289 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-nsfcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 30 01:33:37.209773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:37.209360 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" podUID="c1d4d67f-e18a-4152-be41-e1832bfd78ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 30 01:33:37.449167 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:37.449131 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:38.208211 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:38.208166 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-nsfcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 30 01:33:38.208520 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:38.208487 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" podUID="c1d4d67f-e18a-4152-be41-e1832bfd78ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 30 01:33:38.814773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:38.813868 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" podUID="9a56da7a-9b92-40e2-b08b-43cf591b5221" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:38.961873 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:38.961825 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" podUID="e76339df-9659-4bea-80a4-5f18c4a73fd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:38.977579 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:38.975960 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" podUID="cb20484a-51e2-4581-a629-e92d4234784e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:39.005056 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:39.005019 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" podUID="fac3ca55-64af-42d9-90b1-0e821b893282" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 01:33:39.009844 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:39.009811 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" podUID="317505ef-238c-458e-a74e-1198806b94d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:39.200590 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:39.200554 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-nsfcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 30 01:33:39.200962 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:39.200910 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" podUID="c1d4d67f-e18a-4152-be41-e1832bfd78ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 30 01:33:39.428772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:39.418836 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" podUID="0ee3f8e3-9b55-4ce5-afaf-86780a26391e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:39.776770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:39.776559 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:39.965181 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:39.964939 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:40.200472 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:40.200443 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-nsfcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 30 01:33:40.200661 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:40.200490 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" podUID="c1d4d67f-e18a-4152-be41-e1832bfd78ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 30 01:33:41.775102 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:41.775064 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" Oct 30 01:33:41.776335 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:41.776313 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" Oct 30 01:33:41.777425 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:41.777401 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" Oct 30 01:33:42.393859 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:42.393829 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec2345-55774dc5f5-4ghc6" Oct 30 01:33:42.394029 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:42.393934 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-ec2345-6f57b55f9d-sm5jj" Oct 30 01:33:43.138972 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:43.138941 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:44.689959 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:33:44.689924 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:33:44.801694 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:44.801653 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:46.493839 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:46.493801 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:48.094762 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:48.094711 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:49.887546 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:49.887514 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:49.908317 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:49.908280 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d361c1d-dxsn9"] Oct 30 01:33:49.916907 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:49.916877 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d361c1d-dxsn9" Oct 30 01:33:49.936834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:49.936801 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:33:49.937113 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:49.937095 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:33:49.937468 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:49.937452 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:33:49.940785 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:49.939429 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:33:49.943154 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:49.943129 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d361c1d-dxsn9"] Oct 30 01:33:49.994263 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:49.994229 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c938ec61-ba37-43cc-84f1-00f917a24b6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d361c1d-dxsn9\" (UID: \"c938ec61-ba37-43cc-84f1-00f917a24b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d361c1d-dxsn9" Oct 30 01:33:49.994435 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:49.994280 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c938ec61-ba37-43cc-84f1-00f917a24b6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d361c1d-dxsn9\" (UID: \"c938ec61-ba37-43cc-84f1-00f917a24b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d361c1d-dxsn9" Oct 30 01:33:49.994526 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:49.994503 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbrms\" (UniqueName: \"kubernetes.io/projected/c938ec61-ba37-43cc-84f1-00f917a24b6d-kube-api-access-dbrms\") pod \"maas-api-revoke-sub-4d361c1d-dxsn9\" (UID: \"c938ec61-ba37-43cc-84f1-00f917a24b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d361c1d-dxsn9" Oct 30 01:33:50.096172 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:50.096139 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbrms\" (UniqueName: \"kubernetes.io/projected/c938ec61-ba37-43cc-84f1-00f917a24b6d-kube-api-access-dbrms\") pod \"maas-api-revoke-sub-4d361c1d-dxsn9\" (UID: \"c938ec61-ba37-43cc-84f1-00f917a24b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d361c1d-dxsn9" Oct 30 01:33:50.096351 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:50.096289 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c938ec61-ba37-43cc-84f1-00f917a24b6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d361c1d-dxsn9\" (UID: \"c938ec61-ba37-43cc-84f1-00f917a24b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d361c1d-dxsn9" Oct 30 01:33:50.096351 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:50.096328 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c938ec61-ba37-43cc-84f1-00f917a24b6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d361c1d-dxsn9\" (UID: \"c938ec61-ba37-43cc-84f1-00f917a24b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d361c1d-dxsn9" Oct 30 01:33:50.097024 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:50.097005 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c938ec61-ba37-43cc-84f1-00f917a24b6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d361c1d-dxsn9\" (UID: \"c938ec61-ba37-43cc-84f1-00f917a24b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d361c1d-dxsn9" Oct 30 01:33:50.098648 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:50.098629 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c938ec61-ba37-43cc-84f1-00f917a24b6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d361c1d-dxsn9\" (UID: \"c938ec61-ba37-43cc-84f1-00f917a24b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d361c1d-dxsn9" Oct 30 01:33:50.113886 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:50.113859 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbrms\" (UniqueName: \"kubernetes.io/projected/c938ec61-ba37-43cc-84f1-00f917a24b6d-kube-api-access-dbrms\") pod \"maas-api-revoke-sub-4d361c1d-dxsn9\" (UID: \"c938ec61-ba37-43cc-84f1-00f917a24b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d361c1d-dxsn9" Oct 30 01:33:50.241920 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:50.241877 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d361c1d-dxsn9" Oct 30 01:33:50.548626 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:33:50.548592 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc938ec61_ba37_43cc_84f1_00f917a24b6d.slice/crio-38dd43cbad1fd4873ee7901b2d9696b6487f88ae6b5b3be19f111c42ecb61692 WatchSource:0}: Error finding container 38dd43cbad1fd4873ee7901b2d9696b6487f88ae6b5b3be19f111c42ecb61692: Status 404 returned error can't find the container with id 38dd43cbad1fd4873ee7901b2d9696b6487f88ae6b5b3be19f111c42ecb61692 Oct 30 01:33:50.549233 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:50.549210 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d361c1d-dxsn9"] Oct 30 01:33:50.890712 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:50.890670 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d361c1d-dxsn9" event={"ID":"c938ec61-ba37-43cc-84f1-00f917a24b6d","Type":"ContainerStarted","Data":"750e99744759bf76a092ffe03f50598aee979e60b9e4794ec358881924c3f4fb"} Oct 30 01:33:50.891132 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:50.890736 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d361c1d-dxsn9" event={"ID":"c938ec61-ba37-43cc-84f1-00f917a24b6d","Type":"ContainerStarted","Data":"38dd43cbad1fd4873ee7901b2d9696b6487f88ae6b5b3be19f111c42ecb61692"} Oct 30 01:33:50.928453 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:50.928401 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d361c1d-dxsn9" podStartSLOduration=1.9283881219999999 podStartE2EDuration="1.928388122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:50.924863112 +0000 UTC m=+956.501288525" watchObservedRunningTime="2026-10-30 01:33:50.928388122 +0000 UTC m=+956.504813579" Oct 30 01:33:51.494106 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:51.494065 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:51.902873 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:51.902831 6572 generic.go:356] "Generic (PLEG): container finished" podID="c938ec61-ba37-43cc-84f1-00f917a24b6d" containerID="750e99744759bf76a092ffe03f50598aee979e60b9e4794ec358881924c3f4fb" exitCode=0 Oct 30 01:33:51.903416 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:51.902923 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d361c1d-dxsn9" event={"ID":"c938ec61-ba37-43cc-84f1-00f917a24b6d","Type":"ContainerDied","Data":"750e99744759bf76a092ffe03f50598aee979e60b9e4794ec358881924c3f4fb"} Oct 30 01:33:53.109241 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:53.109206 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d361c1d-dxsn9" Oct 30 01:33:53.136670 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:53.136635 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c938ec61-ba37-43cc-84f1-00f917a24b6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c938ec61-ba37-43cc-84f1-00f917a24b6d-maas-api-xks-ca\") pod \"c938ec61-ba37-43cc-84f1-00f917a24b6d\" (UID: \"c938ec61-ba37-43cc-84f1-00f917a24b6d\") " Oct 30 01:33:53.136670 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:53.136677 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c938ec61-ba37-43cc-84f1-00f917a24b6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c938ec61-ba37-43cc-84f1-00f917a24b6d-maas-api-openshift-ca\") pod \"c938ec61-ba37-43cc-84f1-00f917a24b6d\" (UID: \"c938ec61-ba37-43cc-84f1-00f917a24b6d\") " Oct 30 01:33:53.136921 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:53.136913 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c938ec61-ba37-43cc-84f1-00f917a24b6d-kube-api-access-dbrms\" (UniqueName: \"kubernetes.io/projected/c938ec61-ba37-43cc-84f1-00f917a24b6d-kube-api-access-dbrms\") pod \"c938ec61-ba37-43cc-84f1-00f917a24b6d\" (UID: \"c938ec61-ba37-43cc-84f1-00f917a24b6d\") " Oct 30 01:33:53.137195 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:53.137174 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c938ec61-ba37-43cc-84f1-00f917a24b6d-maas-api-openshift-ca" pod "c938ec61-ba37-43cc-84f1-00f917a24b6d" (UID: "c938ec61-ba37-43cc-84f1-00f917a24b6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:33:53.137426 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:53.137406 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c938ec61-ba37-43cc-84f1-00f917a24b6d-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:33:53.138878 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:53.138858 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c938ec61-ba37-43cc-84f1-00f917a24b6d-maas-api-xks-ca" pod "c938ec61-ba37-43cc-84f1-00f917a24b6d" (UID: "c938ec61-ba37-43cc-84f1-00f917a24b6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:33:53.138985 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:53.138968 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c938ec61-ba37-43cc-84f1-00f917a24b6d-kube-api-access-dbrms" pod "c938ec61-ba37-43cc-84f1-00f917a24b6d" (UID: "c938ec61-ba37-43cc-84f1-00f917a24b6d"). InnerVolumeSpecName "kube-api-access-dbrms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:33:53.140770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:53.140732 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:53.238870 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:53.238786 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c938ec61-ba37-43cc-84f1-00f917a24b6d-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:33:53.238870 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:53.238817 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbrms\" (UniqueName: \"kubernetes.io/projected/c938ec61-ba37-43cc-84f1-00f917a24b6d-kube-api-access-dbrms\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:33:53.922312 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:53.922281 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d361c1d-dxsn9" Oct 30 01:33:53.922503 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:53.922313 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d361c1d-dxsn9" event={"ID":"c938ec61-ba37-43cc-84f1-00f917a24b6d","Type":"ContainerDied","Data":"38dd43cbad1fd4873ee7901b2d9696b6487f88ae6b5b3be19f111c42ecb61692"} Oct 30 01:33:53.922503 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:53.922361 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38dd43cbad1fd4873ee7901b2d9696b6487f88ae6b5b3be19f111c42ecb61692" Oct 30 01:33:54.791651 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:54.791609 6572 scope.go:122] "RemoveContainer" containerID="8c4818b9baed1460a0445ab6ec68d4fa13de58ada6979c03276a778513a5a175" Oct 30 01:33:54.793117 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:54.793089 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:54.812809 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:54.812772 6572 scope.go:122] "RemoveContainer" containerID="1adfd64134181ed41656ccdbd97bbc1763487ae55b004fce8bbf2382dd2438c8" Oct 30 01:33:54.886159 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:54.886126 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngtfq"] Oct 30 01:33:54.886821 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:54.886805 6572 cpu_manager.go:414] "containerMap: removing container" podUID="c938ec61-ba37-43cc-84f1-00f917a24b6d" containerName="revoke-keys" Oct 30 01:33:54.886877 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:54.886824 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c938ec61-ba37-43cc-84f1-00f917a24b6d" containerName="revoke-keys" Oct 30 01:33:54.886969 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:54.886959 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="c938ec61-ba37-43cc-84f1-00f917a24b6d" containerName="revoke-keys" Oct 30 01:33:54.892078 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:54.892055 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngtfq" Oct 30 01:33:54.904932 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:54.904903 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 01:33:54.905087 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:54.904988 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 01:33:54.905087 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:54.905035 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 01:33:54.914078 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:54.914047 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-44rm5\"" Oct 30 01:33:54.920539 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:54.920505 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngtfq"] Oct 30 01:33:54.966961 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:54.966920 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh8hq\" (UniqueName: \"kubernetes.io/projected/4773cabd-39d7-4386-852c-ae771c56e7f7-kube-api-access-kh8hq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngtfq\" (UID: \"4773cabd-39d7-4386-852c-ae771c56e7f7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngtfq" Oct 30 01:33:54.966961 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:54.966958 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4773cabd-39d7-4386-852c-ae771c56e7f7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngtfq\" (UID: \"4773cabd-39d7-4386-852c-ae771c56e7f7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngtfq" Oct 30 01:33:54.967251 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:54.967081 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4773cabd-39d7-4386-852c-ae771c56e7f7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngtfq\" (UID: \"4773cabd-39d7-4386-852c-ae771c56e7f7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngtfq" Oct 30 01:33:54.967251 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:54.967114 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4773cabd-39d7-4386-852c-ae771c56e7f7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngtfq\" (UID: \"4773cabd-39d7-4386-852c-ae771c56e7f7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngtfq" Oct 30 01:33:54.967251 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:54.967206 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4773cabd-39d7-4386-852c-ae771c56e7f7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngtfq\" (UID: \"4773cabd-39d7-4386-852c-ae771c56e7f7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngtfq" Oct 30 01:33:54.967251 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:54.967236 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4773cabd-39d7-4386-852c-ae771c56e7f7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngtfq\" (UID: \"4773cabd-39d7-4386-852c-ae771c56e7f7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngtfq" Oct 30 01:33:54.967477 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:54.967283 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4773cabd-39d7-4386-852c-ae771c56e7f7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngtfq\" (UID: \"4773cabd-39d7-4386-852c-ae771c56e7f7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngtfq" Oct 30 01:33:55.068725 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:55.068607 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh8hq\" (UniqueName: \"kubernetes.io/projected/4773cabd-39d7-4386-852c-ae771c56e7f7-kube-api-access-kh8hq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngtfq\" (UID: \"4773cabd-39d7-4386-852c-ae771c56e7f7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngtfq" Oct 30 01:33:55.068725 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:55.068656 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4773cabd-39d7-4386-852c-ae771c56e7f7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngtfq\" (UID: \"4773cabd-39d7-4386-852c-ae771c56e7f7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngtfq" Oct 30 01:33:55.069016 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:55.068796 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4773cabd-39d7-4386-852c-ae771c56e7f7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngtfq\" (UID: \"4773cabd-39d7-4386-852c-ae771c56e7f7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngtfq" Oct 30 01:33:55.069016 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:55.068841 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4773cabd-39d7-4386-852c-ae771c56e7f7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngtfq\" (UID: \"4773cabd-39d7-4386-852c-ae771c56e7f7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngtfq" Oct 30 01:33:55.069016 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:55.068892 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4773cabd-39d7-4386-852c-ae771c56e7f7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngtfq\" (UID: \"4773cabd-39d7-4386-852c-ae771c56e7f7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngtfq" Oct 30 01:33:55.069016 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:55.068918 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4773cabd-39d7-4386-852c-ae771c56e7f7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngtfq\" (UID: \"4773cabd-39d7-4386-852c-ae771c56e7f7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngtfq" Oct 30 01:33:55.069016 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:55.068982 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4773cabd-39d7-4386-852c-ae771c56e7f7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngtfq\" (UID: \"4773cabd-39d7-4386-852c-ae771c56e7f7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngtfq" Oct 30 01:33:55.069631 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:55.069599 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4773cabd-39d7-4386-852c-ae771c56e7f7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngtfq\" (UID: \"4773cabd-39d7-4386-852c-ae771c56e7f7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngtfq" Oct 30 01:33:55.071356 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:55.071304 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4773cabd-39d7-4386-852c-ae771c56e7f7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngtfq\" (UID: \"4773cabd-39d7-4386-852c-ae771c56e7f7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngtfq" Oct 30 01:33:55.071465 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:55.071375 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4773cabd-39d7-4386-852c-ae771c56e7f7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngtfq\" (UID: \"4773cabd-39d7-4386-852c-ae771c56e7f7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngtfq" Oct 30 01:33:55.071566 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:55.071548 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4773cabd-39d7-4386-852c-ae771c56e7f7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngtfq\" (UID: \"4773cabd-39d7-4386-852c-ae771c56e7f7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngtfq" Oct 30 01:33:55.071665 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:55.071650 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4773cabd-39d7-4386-852c-ae771c56e7f7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngtfq\" (UID: \"4773cabd-39d7-4386-852c-ae771c56e7f7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngtfq" Oct 30 01:33:55.086057 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:55.086019 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh8hq\" (UniqueName: \"kubernetes.io/projected/4773cabd-39d7-4386-852c-ae771c56e7f7-kube-api-access-kh8hq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngtfq\" (UID: \"4773cabd-39d7-4386-852c-ae771c56e7f7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngtfq" Oct 30 01:33:55.086203 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:55.086185 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4773cabd-39d7-4386-852c-ae771c56e7f7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ngtfq\" (UID: \"4773cabd-39d7-4386-852c-ae771c56e7f7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngtfq" Oct 30 01:33:55.140529 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:55.140482 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:55.205840 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:55.205803 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngtfq" Oct 30 01:33:55.329108 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:55.329012 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:55.562473 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:33:55.562433 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4773cabd_39d7_4386_852c_ae771c56e7f7.slice/crio-e8573d14febeb02ab37acd9a56d43dbc7858897a8514bb8ce6d008d280592117 WatchSource:0}: Error finding container e8573d14febeb02ab37acd9a56d43dbc7858897a8514bb8ce6d008d280592117: Status 404 returned error can't find the container with id e8573d14febeb02ab37acd9a56d43dbc7858897a8514bb8ce6d008d280592117 Oct 30 01:33:55.564785 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:55.564741 6572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:33:55.567341 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:55.567315 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngtfq"] Oct 30 01:33:55.944541 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:55.944502 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngtfq" event={"ID":"4773cabd-39d7-4386-852c-ae771c56e7f7","Type":"ContainerStarted","Data":"e8573d14febeb02ab37acd9a56d43dbc7858897a8514bb8ce6d008d280592117"} Oct 30 01:33:56.673675 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:56.673639 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:57.073451 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:57.073358 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:57.767942 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:57.767907 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:58.243145 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:58.236773 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:58.316005 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:58.315969 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:33:58.316167 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:58.316035 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:33:58.469313 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:58.469277 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:58.651891 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:58.651779 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:58.976456 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:58.976423 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngtfq" event={"ID":"4773cabd-39d7-4386-852c-ae771c56e7f7","Type":"ContainerStarted","Data":"a4b0ef574be62e5fb72ff420ec243e551ac0080cca830afd96db7985c2a6e2b0"} Oct 30 01:33:58.976668 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:58.976544 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngtfq" Oct 30 01:33:59.030218 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:33:59.030151 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngtfq" podStartSLOduration=2.280386871 podStartE2EDuration="5.030131348s" totalImagesPullingTime="2.749744477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:59.023573202 +0000 UTC m=+964.599998621" watchObservedRunningTime="2026-10-30 01:33:59.030131348 +0000 UTC m=+964.606556750" Oct 30 01:34:00.082160 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:00.082125 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ngtfq" Oct 30 01:34:00.416966 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:00.416922 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9121606-wg84d"] Oct 30 01:34:00.426680 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:00.426641 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9121606-wg84d" Oct 30 01:34:00.440208 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:00.440181 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:34:00.441273 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:00.441256 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:34:00.441554 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:00.441532 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:34:00.441771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:00.441757 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:34:00.459620 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:00.459590 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9121606-wg84d"] Oct 30 01:34:00.575441 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:00.575375 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd6v6\" (UniqueName: \"kubernetes.io/projected/f9e2aef8-2ffc-4b17-8c7f-25265e2056e7-kube-api-access-jd6v6\") pod \"maas-api-revoke-sub-c9121606-wg84d\" (UID: \"f9e2aef8-2ffc-4b17-8c7f-25265e2056e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9121606-wg84d" Oct 30 01:34:00.575620 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:00.575462 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9e2aef8-2ffc-4b17-8c7f-25265e2056e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9121606-wg84d\" (UID: \"f9e2aef8-2ffc-4b17-8c7f-25265e2056e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9121606-wg84d" Oct 30 01:34:00.575849 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:00.575821 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9e2aef8-2ffc-4b17-8c7f-25265e2056e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9121606-wg84d\" (UID: \"f9e2aef8-2ffc-4b17-8c7f-25265e2056e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9121606-wg84d" Oct 30 01:34:00.678837 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:00.678708 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9e2aef8-2ffc-4b17-8c7f-25265e2056e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9121606-wg84d\" (UID: \"f9e2aef8-2ffc-4b17-8c7f-25265e2056e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9121606-wg84d" Oct 30 01:34:00.679714 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:00.679685 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd6v6\" (UniqueName: \"kubernetes.io/projected/f9e2aef8-2ffc-4b17-8c7f-25265e2056e7-kube-api-access-jd6v6\") pod \"maas-api-revoke-sub-c9121606-wg84d\" (UID: \"f9e2aef8-2ffc-4b17-8c7f-25265e2056e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9121606-wg84d" Oct 30 01:34:00.679942 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:00.679924 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9e2aef8-2ffc-4b17-8c7f-25265e2056e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9121606-wg84d\" (UID: \"f9e2aef8-2ffc-4b17-8c7f-25265e2056e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9121606-wg84d" Oct 30 01:34:00.683100 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:00.683069 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9e2aef8-2ffc-4b17-8c7f-25265e2056e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9121606-wg84d\" (UID: \"f9e2aef8-2ffc-4b17-8c7f-25265e2056e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9121606-wg84d" Oct 30 01:34:00.686914 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:00.685996 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9e2aef8-2ffc-4b17-8c7f-25265e2056e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9121606-wg84d\" (UID: \"f9e2aef8-2ffc-4b17-8c7f-25265e2056e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9121606-wg84d" Oct 30 01:34:00.707485 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:00.707447 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd6v6\" (UniqueName: \"kubernetes.io/projected/f9e2aef8-2ffc-4b17-8c7f-25265e2056e7-kube-api-access-jd6v6\") pod \"maas-api-revoke-sub-c9121606-wg84d\" (UID: \"f9e2aef8-2ffc-4b17-8c7f-25265e2056e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9121606-wg84d" Oct 30 01:34:00.722763 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:00.709155 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:00.758694 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:00.758366 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9121606-wg84d" Oct 30 01:34:01.150986 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:01.150886 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9121606-wg84d"] Oct 30 01:34:02.100662 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:02.100630 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9121606-wg84d" event={"ID":"f9e2aef8-2ffc-4b17-8c7f-25265e2056e7","Type":"ContainerStarted","Data":"21aac4872ccbb34e54c8c6ba970def3af1182ef36a8b61b6db67b8648422b01e"} Oct 30 01:34:02.100901 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:02.100681 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9121606-wg84d" event={"ID":"f9e2aef8-2ffc-4b17-8c7f-25265e2056e7","Type":"ContainerStarted","Data":"46736c169586f70834fdd7bd56d7ef88071d8c7b7ece9aaf640eb2013149ae57"} Oct 30 01:34:02.157666 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:02.157607 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9121606-wg84d" podStartSLOduration=2.157588954 podStartE2EDuration="2.157588954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:02.152067247 +0000 UTC m=+967.728492651" watchObservedRunningTime="2026-10-30 01:34:02.157588954 +0000 UTC m=+967.734014357" Oct 30 01:34:02.331785 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:02.322532 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:02.530775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:02.530364 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:02.719768 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:02.719659 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:03.134896 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:03.134862 6572 generic.go:356] "Generic (PLEG): container finished" podID="f9e2aef8-2ffc-4b17-8c7f-25265e2056e7" containerID="21aac4872ccbb34e54c8c6ba970def3af1182ef36a8b61b6db67b8648422b01e" exitCode=0 Oct 30 01:34:03.135085 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:03.134951 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9121606-wg84d" event={"ID":"f9e2aef8-2ffc-4b17-8c7f-25265e2056e7","Type":"ContainerDied","Data":"21aac4872ccbb34e54c8c6ba970def3af1182ef36a8b61b6db67b8648422b01e"} Oct 30 01:34:03.737871 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:03.737835 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:03.978996 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:03.978955 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:04.264112 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:04.263904 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:04.578023 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:04.577991 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9121606-wg84d" Oct 30 01:34:04.664917 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:04.664807 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:04.671240 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:04.671188 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9e2aef8-2ffc-4b17-8c7f-25265e2056e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9e2aef8-2ffc-4b17-8c7f-25265e2056e7-maas-api-openshift-ca\") pod \"f9e2aef8-2ffc-4b17-8c7f-25265e2056e7\" (UID: \"f9e2aef8-2ffc-4b17-8c7f-25265e2056e7\") " Oct 30 01:34:04.671840 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:04.671802 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9e2aef8-2ffc-4b17-8c7f-25265e2056e7-kube-api-access-jd6v6\" (UniqueName: \"kubernetes.io/projected/f9e2aef8-2ffc-4b17-8c7f-25265e2056e7-kube-api-access-jd6v6\") pod \"f9e2aef8-2ffc-4b17-8c7f-25265e2056e7\" (UID: \"f9e2aef8-2ffc-4b17-8c7f-25265e2056e7\") " Oct 30 01:34:04.672027 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:04.672008 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9e2aef8-2ffc-4b17-8c7f-25265e2056e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9e2aef8-2ffc-4b17-8c7f-25265e2056e7-maas-api-xks-ca\") pod \"f9e2aef8-2ffc-4b17-8c7f-25265e2056e7\" (UID: \"f9e2aef8-2ffc-4b17-8c7f-25265e2056e7\") " Oct 30 01:34:04.673398 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:04.673170 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9e2aef8-2ffc-4b17-8c7f-25265e2056e7-maas-api-openshift-ca" pod "f9e2aef8-2ffc-4b17-8c7f-25265e2056e7" (UID: "f9e2aef8-2ffc-4b17-8c7f-25265e2056e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:34:04.677803 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:04.677156 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9e2aef8-2ffc-4b17-8c7f-25265e2056e7-maas-api-xks-ca" pod "f9e2aef8-2ffc-4b17-8c7f-25265e2056e7" (UID: "f9e2aef8-2ffc-4b17-8c7f-25265e2056e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:34:04.678854 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:04.678826 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9e2aef8-2ffc-4b17-8c7f-25265e2056e7-kube-api-access-jd6v6" pod "f9e2aef8-2ffc-4b17-8c7f-25265e2056e7" (UID: "f9e2aef8-2ffc-4b17-8c7f-25265e2056e7"). InnerVolumeSpecName "kube-api-access-jd6v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:34:04.774577 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:04.774543 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd6v6\" (UniqueName: \"kubernetes.io/projected/f9e2aef8-2ffc-4b17-8c7f-25265e2056e7-kube-api-access-jd6v6\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:34:04.774577 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:04.774573 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9e2aef8-2ffc-4b17-8c7f-25265e2056e7-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:34:04.774577 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:04.774584 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9e2aef8-2ffc-4b17-8c7f-25265e2056e7-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:34:05.292598 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:05.292569 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9121606-wg84d" Oct 30 01:34:05.292855 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:05.292602 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9121606-wg84d" event={"ID":"f9e2aef8-2ffc-4b17-8c7f-25265e2056e7","Type":"ContainerDied","Data":"46736c169586f70834fdd7bd56d7ef88071d8c7b7ece9aaf640eb2013149ae57"} Oct 30 01:34:05.292855 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:05.292648 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46736c169586f70834fdd7bd56d7ef88071d8c7b7ece9aaf640eb2013149ae57" Oct 30 01:34:06.291818 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:06.291778 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:07.908817 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:07.908735 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:09.557769 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:09.554313 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:10.252448 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:10.252369 6572 patch_prober.go:29] interesting pod/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 01:34:10.252448 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:10.252434 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" podUID="159d5541-9dbd-4b27-bdb2-019f09868d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 01:34:10.340609 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:10.340519 6572 generic.go:356] "Generic (PLEG): container finished" podID="159d5541-9dbd-4b27-bdb2-019f09868d42" containerID="b5dce9d5632391bc915d87a50ab0845ff095c1a1c49bde3b0af1eed1daf81dda" exitCode=0 Oct 30 01:34:10.340609 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:10.340590 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" event={"ID":"159d5541-9dbd-4b27-bdb2-019f09868d42","Type":"ContainerDied","Data":"b5dce9d5632391bc915d87a50ab0845ff095c1a1c49bde3b0af1eed1daf81dda"} Oct 30 01:34:10.341089 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:10.341075 6572 scope.go:122] "RemoveContainer" containerID="b5dce9d5632391bc915d87a50ab0845ff095c1a1c49bde3b0af1eed1daf81dda" Oct 30 01:34:10.342605 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:10.342576 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:34:10.342688 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:10.342638 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:34:10.342688 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:10.342670 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:34:11.204287 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:11.204251 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:11.350702 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:11.350663 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" event={"ID":"159d5541-9dbd-4b27-bdb2-019f09868d42","Type":"ContainerStarted","Data":"7be1a04c78aaf7b5e6957925129d4fe9f77e8426a05980c508e7c4b645df4c81"} Oct 30 01:34:11.366694 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:11.366658 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:34:11.366944 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:11.366919 6572 patch_prober.go:29] interesting pod/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 01:34:11.367014 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:11.366976 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" podUID="159d5541-9dbd-4b27-bdb2-019f09868d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 01:34:12.366531 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:12.366496 6572 patch_prober.go:29] interesting pod/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 01:34:12.366983 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:12.366549 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" podUID="159d5541-9dbd-4b27-bdb2-019f09868d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 01:34:12.485606 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:12.485572 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:12.882771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:12.873559 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:13.070708 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:13.070672 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:13.258038 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:13.257981 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:13.366409 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:13.366376 6572 patch_prober.go:29] interesting pod/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 01:34:13.366684 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:13.366654 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" podUID="159d5541-9dbd-4b27-bdb2-019f09868d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 01:34:14.366879 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:14.366844 6572 patch_prober.go:29] interesting pod/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 01:34:14.367349 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:14.366931 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" podUID="159d5541-9dbd-4b27-bdb2-019f09868d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 01:34:14.694672 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:34:14.694635 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:34:14.722771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:14.719294 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:15.366740 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:15.366698 6572 patch_prober.go:29] interesting pod/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 01:34:15.366964 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:15.366795 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" podUID="159d5541-9dbd-4b27-bdb2-019f09868d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 01:34:16.095894 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:16.095862 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:16.366972 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:16.366422 6572 patch_prober.go:29] interesting pod/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 01:34:16.366972 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:16.366475 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" podUID="159d5541-9dbd-4b27-bdb2-019f09868d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 01:34:16.442175 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:16.442129 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:16.635085 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:16.634966 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:17.371338 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:17.371308 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:34:17.371812 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:17.371790 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:34:17.372697 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:17.372673 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:34:17.795527 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:17.795498 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:18.088473 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.088195 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:18.350206 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.350103 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9"] Oct 30 01:34:18.350970 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.350943 6572 cpu_manager.go:414] "containerMap: removing container" podUID="f9e2aef8-2ffc-4b17-8c7f-25265e2056e7" containerName="revoke-keys" Oct 30 01:34:18.350970 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.350969 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9e2aef8-2ffc-4b17-8c7f-25265e2056e7" containerName="revoke-keys" Oct 30 01:34:18.351525 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.351237 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9e2aef8-2ffc-4b17-8c7f-25265e2056e7" containerName="revoke-keys" Oct 30 01:34:18.356401 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.356378 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" Oct 30 01:34:18.380790 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.380717 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9"] Oct 30 01:34:18.386387 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.386361 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf"] Oct 30 01:34:18.394038 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.393981 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" Oct 30 01:34:18.417152 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.417125 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf"] Oct 30 01:34:18.515218 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.515184 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm5bk\" (UniqueName: \"kubernetes.io/projected/e9764434-6c1a-4889-a18a-6b714f6404fe-kube-api-access-fm5bk\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf\" (UID: \"e9764434-6c1a-4889-a18a-6b714f6404fe\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" Oct 30 01:34:18.515218 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.515222 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/780b98b5-6658-4a59-83cd-ae57c033bde9-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9\" (UID: \"780b98b5-6658-4a59-83cd-ae57c033bde9\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" Oct 30 01:34:18.515473 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.515245 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9764434-6c1a-4889-a18a-6b714f6404fe-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf\" (UID: \"e9764434-6c1a-4889-a18a-6b714f6404fe\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" Oct 30 01:34:18.515473 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.515309 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9764434-6c1a-4889-a18a-6b714f6404fe-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf\" (UID: \"e9764434-6c1a-4889-a18a-6b714f6404fe\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" Oct 30 01:34:18.515473 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.515391 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9764434-6c1a-4889-a18a-6b714f6404fe-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf\" (UID: \"e9764434-6c1a-4889-a18a-6b714f6404fe\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" Oct 30 01:34:18.515473 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.515424 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9764434-6c1a-4889-a18a-6b714f6404fe-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf\" (UID: \"e9764434-6c1a-4889-a18a-6b714f6404fe\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" Oct 30 01:34:18.515609 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.515521 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9764434-6c1a-4889-a18a-6b714f6404fe-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf\" (UID: \"e9764434-6c1a-4889-a18a-6b714f6404fe\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" Oct 30 01:34:18.515609 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.515591 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/780b98b5-6658-4a59-83cd-ae57c033bde9-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9\" (UID: \"780b98b5-6658-4a59-83cd-ae57c033bde9\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" Oct 30 01:34:18.515673 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.515631 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9764434-6c1a-4889-a18a-6b714f6404fe-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf\" (UID: \"e9764434-6c1a-4889-a18a-6b714f6404fe\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" Oct 30 01:34:18.515673 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.515659 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/780b98b5-6658-4a59-83cd-ae57c033bde9-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9\" (UID: \"780b98b5-6658-4a59-83cd-ae57c033bde9\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" Oct 30 01:34:18.515831 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.515812 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjsjt\" (UniqueName: \"kubernetes.io/projected/780b98b5-6658-4a59-83cd-ae57c033bde9-kube-api-access-tjsjt\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9\" (UID: \"780b98b5-6658-4a59-83cd-ae57c033bde9\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" Oct 30 01:34:18.515899 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.515860 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/780b98b5-6658-4a59-83cd-ae57c033bde9-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9\" (UID: \"780b98b5-6658-4a59-83cd-ae57c033bde9\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" Oct 30 01:34:18.515975 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.515960 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9764434-6c1a-4889-a18a-6b714f6404fe-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf\" (UID: \"e9764434-6c1a-4889-a18a-6b714f6404fe\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" Oct 30 01:34:18.516033 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.516008 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/780b98b5-6658-4a59-83cd-ae57c033bde9-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9\" (UID: \"780b98b5-6658-4a59-83cd-ae57c033bde9\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" Oct 30 01:34:18.516106 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.516093 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/780b98b5-6658-4a59-83cd-ae57c033bde9-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9\" (UID: \"780b98b5-6658-4a59-83cd-ae57c033bde9\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" Oct 30 01:34:18.516165 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.516122 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/780b98b5-6658-4a59-83cd-ae57c033bde9-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9\" (UID: \"780b98b5-6658-4a59-83cd-ae57c033bde9\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" Oct 30 01:34:18.516221 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.516172 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9764434-6c1a-4889-a18a-6b714f6404fe-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf\" (UID: \"e9764434-6c1a-4889-a18a-6b714f6404fe\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" Oct 30 01:34:18.516221 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.516213 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/780b98b5-6658-4a59-83cd-ae57c033bde9-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9\" (UID: \"780b98b5-6658-4a59-83cd-ae57c033bde9\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" Oct 30 01:34:18.617304 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.617214 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm5bk\" (UniqueName: \"kubernetes.io/projected/e9764434-6c1a-4889-a18a-6b714f6404fe-kube-api-access-fm5bk\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf\" (UID: \"e9764434-6c1a-4889-a18a-6b714f6404fe\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" Oct 30 01:34:18.617567 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.617530 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/780b98b5-6658-4a59-83cd-ae57c033bde9-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9\" (UID: \"780b98b5-6658-4a59-83cd-ae57c033bde9\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" Oct 30 01:34:18.617709 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.617575 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9764434-6c1a-4889-a18a-6b714f6404fe-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf\" (UID: \"e9764434-6c1a-4889-a18a-6b714f6404fe\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" Oct 30 01:34:18.617709 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.617617 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9764434-6c1a-4889-a18a-6b714f6404fe-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf\" (UID: \"e9764434-6c1a-4889-a18a-6b714f6404fe\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" Oct 30 01:34:18.617709 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.617670 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9764434-6c1a-4889-a18a-6b714f6404fe-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf\" (UID: \"e9764434-6c1a-4889-a18a-6b714f6404fe\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" Oct 30 01:34:18.618019 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.617737 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9764434-6c1a-4889-a18a-6b714f6404fe-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf\" (UID: \"e9764434-6c1a-4889-a18a-6b714f6404fe\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" Oct 30 01:34:18.618229 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.618051 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9764434-6c1a-4889-a18a-6b714f6404fe-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf\" (UID: \"e9764434-6c1a-4889-a18a-6b714f6404fe\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" Oct 30 01:34:18.618229 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.618159 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/780b98b5-6658-4a59-83cd-ae57c033bde9-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9\" (UID: \"780b98b5-6658-4a59-83cd-ae57c033bde9\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" Oct 30 01:34:18.618380 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.618227 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9764434-6c1a-4889-a18a-6b714f6404fe-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf\" (UID: \"e9764434-6c1a-4889-a18a-6b714f6404fe\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" Oct 30 01:34:18.618380 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.618257 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9764434-6c1a-4889-a18a-6b714f6404fe-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf\" (UID: \"e9764434-6c1a-4889-a18a-6b714f6404fe\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" Oct 30 01:34:18.618380 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.618287 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9764434-6c1a-4889-a18a-6b714f6404fe-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf\" (UID: \"e9764434-6c1a-4889-a18a-6b714f6404fe\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" Oct 30 01:34:18.618380 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.618316 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/780b98b5-6658-4a59-83cd-ae57c033bde9-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9\" (UID: \"780b98b5-6658-4a59-83cd-ae57c033bde9\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" Oct 30 01:34:18.618608 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.618385 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjsjt\" (UniqueName: \"kubernetes.io/projected/780b98b5-6658-4a59-83cd-ae57c033bde9-kube-api-access-tjsjt\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9\" (UID: \"780b98b5-6658-4a59-83cd-ae57c033bde9\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" Oct 30 01:34:18.618608 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.618446 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/780b98b5-6658-4a59-83cd-ae57c033bde9-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9\" (UID: \"780b98b5-6658-4a59-83cd-ae57c033bde9\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" Oct 30 01:34:18.618608 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.618536 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9764434-6c1a-4889-a18a-6b714f6404fe-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf\" (UID: \"e9764434-6c1a-4889-a18a-6b714f6404fe\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" Oct 30 01:34:18.618819 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.618590 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/780b98b5-6658-4a59-83cd-ae57c033bde9-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9\" (UID: \"780b98b5-6658-4a59-83cd-ae57c033bde9\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" Oct 30 01:34:18.618819 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.618688 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/780b98b5-6658-4a59-83cd-ae57c033bde9-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9\" (UID: \"780b98b5-6658-4a59-83cd-ae57c033bde9\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" Oct 30 01:34:18.618819 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.618717 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/780b98b5-6658-4a59-83cd-ae57c033bde9-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9\" (UID: \"780b98b5-6658-4a59-83cd-ae57c033bde9\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" Oct 30 01:34:18.618967 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.618905 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9764434-6c1a-4889-a18a-6b714f6404fe-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf\" (UID: \"e9764434-6c1a-4889-a18a-6b714f6404fe\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" Oct 30 01:34:18.619004 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.618963 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/780b98b5-6658-4a59-83cd-ae57c033bde9-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9\" (UID: \"780b98b5-6658-4a59-83cd-ae57c033bde9\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" Oct 30 01:34:18.619053 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.619034 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9764434-6c1a-4889-a18a-6b714f6404fe-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf\" (UID: \"e9764434-6c1a-4889-a18a-6b714f6404fe\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" Oct 30 01:34:18.620234 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.619609 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/780b98b5-6658-4a59-83cd-ae57c033bde9-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9\" (UID: \"780b98b5-6658-4a59-83cd-ae57c033bde9\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" Oct 30 01:34:18.620234 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.619681 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/780b98b5-6658-4a59-83cd-ae57c033bde9-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9\" (UID: \"780b98b5-6658-4a59-83cd-ae57c033bde9\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" Oct 30 01:34:18.620234 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.619761 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9764434-6c1a-4889-a18a-6b714f6404fe-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf\" (UID: \"e9764434-6c1a-4889-a18a-6b714f6404fe\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" Oct 30 01:34:18.620234 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.617816 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:18.620234 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.619950 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/780b98b5-6658-4a59-83cd-ae57c033bde9-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9\" (UID: \"780b98b5-6658-4a59-83cd-ae57c033bde9\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" Oct 30 01:34:18.622480 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.620525 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/780b98b5-6658-4a59-83cd-ae57c033bde9-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9\" (UID: \"780b98b5-6658-4a59-83cd-ae57c033bde9\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" Oct 30 01:34:18.622480 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.620608 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9764434-6c1a-4889-a18a-6b714f6404fe-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf\" (UID: \"e9764434-6c1a-4889-a18a-6b714f6404fe\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" Oct 30 01:34:18.622480 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.621086 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/780b98b5-6658-4a59-83cd-ae57c033bde9-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9\" (UID: \"780b98b5-6658-4a59-83cd-ae57c033bde9\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" Oct 30 01:34:18.623042 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.623018 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9764434-6c1a-4889-a18a-6b714f6404fe-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf\" (UID: \"e9764434-6c1a-4889-a18a-6b714f6404fe\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" Oct 30 01:34:18.623618 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.623579 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9764434-6c1a-4889-a18a-6b714f6404fe-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf\" (UID: \"e9764434-6c1a-4889-a18a-6b714f6404fe\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" Oct 30 01:34:18.624244 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.624188 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/780b98b5-6658-4a59-83cd-ae57c033bde9-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9\" (UID: \"780b98b5-6658-4a59-83cd-ae57c033bde9\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" Oct 30 01:34:18.624718 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.624678 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/780b98b5-6658-4a59-83cd-ae57c033bde9-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9\" (UID: \"780b98b5-6658-4a59-83cd-ae57c033bde9\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" Oct 30 01:34:18.637624 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.637523 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/780b98b5-6658-4a59-83cd-ae57c033bde9-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9\" (UID: \"780b98b5-6658-4a59-83cd-ae57c033bde9\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" Oct 30 01:34:18.637928 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.635664 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm5bk\" (UniqueName: \"kubernetes.io/projected/e9764434-6c1a-4889-a18a-6b714f6404fe-kube-api-access-fm5bk\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf\" (UID: \"e9764434-6c1a-4889-a18a-6b714f6404fe\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" Oct 30 01:34:18.638259 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.638235 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9764434-6c1a-4889-a18a-6b714f6404fe-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf\" (UID: \"e9764434-6c1a-4889-a18a-6b714f6404fe\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" Oct 30 01:34:18.640402 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.640371 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjsjt\" (UniqueName: \"kubernetes.io/projected/780b98b5-6658-4a59-83cd-ae57c033bde9-kube-api-access-tjsjt\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9\" (UID: \"780b98b5-6658-4a59-83cd-ae57c033bde9\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" Oct 30 01:34:18.674771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.674706 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" Oct 30 01:34:18.726126 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:18.726099 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" Oct 30 01:34:19.005338 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:34:19.005304 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod780b98b5_6658_4a59_83cd_ae57c033bde9.slice/crio-3ba3ed720d85298493bb9df76f308f038cab9de909ad15e17a33569686ef433e WatchSource:0}: Error finding container 3ba3ed720d85298493bb9df76f308f038cab9de909ad15e17a33569686ef433e: Status 404 returned error can't find the container with id 3ba3ed720d85298493bb9df76f308f038cab9de909ad15e17a33569686ef433e Oct 30 01:34:19.006556 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:19.006534 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9"] Oct 30 01:34:19.008642 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:19.008599 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:34:19.008796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:19.008677 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:34:19.008796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:19.008718 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:34:19.070086 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:19.070055 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf"] Oct 30 01:34:19.070985 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:34:19.070961 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9764434_6c1a_4889_a18a_6b714f6404fe.slice/crio-288e48deef67fbe5cc8957833e17569f3a1bfd3fd966fff6edfe96ad59cf2a0d WatchSource:0}: Error finding container 288e48deef67fbe5cc8957833e17569f3a1bfd3fd966fff6edfe96ad59cf2a0d: Status 404 returned error can't find the container with id 288e48deef67fbe5cc8957833e17569f3a1bfd3fd966fff6edfe96ad59cf2a0d Oct 30 01:34:19.074043 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:19.073999 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:34:19.074150 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:19.074083 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:34:19.074150 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:19.074116 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:34:19.472127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:19.472084 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" event={"ID":"780b98b5-6658-4a59-83cd-ae57c033bde9","Type":"ContainerStarted","Data":"a4e5d9d81321b10b45cc71be49e95a704a8a75e97577e3462a1166693faaad86"} Oct 30 01:34:19.472618 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:19.472182 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" event={"ID":"780b98b5-6658-4a59-83cd-ae57c033bde9","Type":"ContainerStarted","Data":"3ba3ed720d85298493bb9df76f308f038cab9de909ad15e17a33569686ef433e"} Oct 30 01:34:19.474283 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:19.474251 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" event={"ID":"e9764434-6c1a-4889-a18a-6b714f6404fe","Type":"ContainerStarted","Data":"65d381682ac70a70717bd9e4d16e6baad97ce02fe955f91f819328c24c93cc91"} Oct 30 01:34:19.474415 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:19.474308 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" event={"ID":"e9764434-6c1a-4889-a18a-6b714f6404fe","Type":"ContainerStarted","Data":"288e48deef67fbe5cc8957833e17569f3a1bfd3fd966fff6edfe96ad59cf2a0d"} Oct 30 01:34:19.523085 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:19.523032 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" podStartSLOduration=1.523017323 podStartE2EDuration="1.523017323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:19.50862115 +0000 UTC m=+985.085046590" watchObservedRunningTime="2026-10-30 01:34:19.523017323 +0000 UTC m=+985.099442724" Oct 30 01:34:19.550568 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:19.550511 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" podStartSLOduration=1.550495159 podStartE2EDuration="1.550495159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:19.543901417 +0000 UTC m=+985.120326830" watchObservedRunningTime="2026-10-30 01:34:19.550495159 +0000 UTC m=+985.126920561" Oct 30 01:34:19.676771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:19.676472 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" Oct 30 01:34:19.731827 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:19.731005 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" Oct 30 01:34:20.396814 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:20.396781 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:20.677093 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:20.677063 6572 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 30 01:34:20.677485 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:20.677120 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" podUID="780b98b5-6658-4a59-83cd-ae57c033bde9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 30 01:34:20.729777 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:20.729636 6572 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 01:34:20.729777 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:20.729700 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" podUID="e9764434-6c1a-4889-a18a-6b714f6404fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 01:34:21.675627 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:21.675580 6572 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 30 01:34:21.675828 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:21.675659 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" podUID="780b98b5-6658-4a59-83cd-ae57c033bde9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 30 01:34:21.735196 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:21.735154 6572 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 01:34:21.735849 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:21.735812 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" podUID="e9764434-6c1a-4889-a18a-6b714f6404fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 01:34:22.375769 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:22.367684 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:22.675971 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:22.675933 6572 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 30 01:34:22.676272 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:22.676238 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" podUID="780b98b5-6658-4a59-83cd-ae57c033bde9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 30 01:34:22.732785 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:22.730103 6572 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 01:34:22.732785 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:22.730205 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" podUID="e9764434-6c1a-4889-a18a-6b714f6404fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 01:34:23.675815 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:23.675779 6572 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 30 01:34:23.676241 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:23.675831 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" podUID="780b98b5-6658-4a59-83cd-ae57c033bde9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 30 01:34:23.727132 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:23.727094 6572 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 01:34:23.727402 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:23.727370 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" podUID="e9764434-6c1a-4889-a18a-6b714f6404fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 01:34:24.320773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:24.318381 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:24.675131 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:24.675096 6572 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 30 01:34:24.675349 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:24.675150 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" podUID="780b98b5-6658-4a59-83cd-ae57c033bde9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 30 01:34:24.733769 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:24.733126 6572 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 01:34:24.733769 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:24.733183 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" podUID="e9764434-6c1a-4889-a18a-6b714f6404fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 01:34:25.675025 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:25.674988 6572 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 30 01:34:25.675203 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:25.675052 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" podUID="780b98b5-6658-4a59-83cd-ae57c033bde9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 30 01:34:25.727111 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:25.727078 6572 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 01:34:25.727267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:25.727137 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" podUID="e9764434-6c1a-4889-a18a-6b714f6404fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 01:34:26.098945 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:26.098860 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:27.715972 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:27.715931 6572 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:34:27.716602 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:27.716555 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" podUID="780b98b5-6658-4a59-83cd-ae57c033bde9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:34:27.768003 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:27.767955 6572 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:34:27.768222 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:27.768031 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" podUID="e9764434-6c1a-4889-a18a-6b714f6404fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:34:27.968084 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:27.967988 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:28.286305 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:28.286216 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" Oct 30 01:34:28.286665 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:28.286645 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" Oct 30 01:34:28.287716 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:28.287681 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" Oct 30 01:34:28.310860 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:28.310829 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" Oct 30 01:34:28.311226 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:28.311200 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" Oct 30 01:34:28.312060 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:28.312034 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" Oct 30 01:34:29.708438 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:29.708400 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:31.335851 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:31.335809 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-f9pk6"] Oct 30 01:34:31.341781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:31.341664 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-f9pk6" Oct 30 01:34:31.348814 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:31.348779 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-f9pk6"] Oct 30 01:34:31.355117 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:31.355085 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:34:31.355290 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:31.355154 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:34:31.364989 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:31.364955 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:34:31.365302 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:31.365268 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:34:31.437010 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:31.436976 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7gd7\" (UniqueName: \"kubernetes.io/projected/d5763345-21f8-48b3-880b-d36fd04c9634-kube-api-access-p7gd7\") pod \"maas-api-revoke-keys-e2e-mt-278fe7d5-f9pk6\" (UID: \"d5763345-21f8-48b3-880b-d36fd04c9634\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-f9pk6" Oct 30 01:34:31.437186 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:31.437095 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5763345-21f8-48b3-880b-d36fd04c9634-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-278fe7d5-f9pk6\" (UID: \"d5763345-21f8-48b3-880b-d36fd04c9634\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-f9pk6" Oct 30 01:34:31.437186 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:31.437178 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5763345-21f8-48b3-880b-d36fd04c9634-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-278fe7d5-f9pk6\" (UID: \"d5763345-21f8-48b3-880b-d36fd04c9634\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-f9pk6" Oct 30 01:34:31.538714 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:31.538682 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7gd7\" (UniqueName: \"kubernetes.io/projected/d5763345-21f8-48b3-880b-d36fd04c9634-kube-api-access-p7gd7\") pod \"maas-api-revoke-keys-e2e-mt-278fe7d5-f9pk6\" (UID: \"d5763345-21f8-48b3-880b-d36fd04c9634\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-f9pk6" Oct 30 01:34:31.538926 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:31.538743 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5763345-21f8-48b3-880b-d36fd04c9634-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-278fe7d5-f9pk6\" (UID: \"d5763345-21f8-48b3-880b-d36fd04c9634\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-f9pk6" Oct 30 01:34:31.538926 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:31.538820 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5763345-21f8-48b3-880b-d36fd04c9634-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-278fe7d5-f9pk6\" (UID: \"d5763345-21f8-48b3-880b-d36fd04c9634\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-f9pk6" Oct 30 01:34:31.539712 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:31.539679 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5763345-21f8-48b3-880b-d36fd04c9634-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-278fe7d5-f9pk6\" (UID: \"d5763345-21f8-48b3-880b-d36fd04c9634\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-f9pk6" Oct 30 01:34:31.542005 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:31.541639 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5763345-21f8-48b3-880b-d36fd04c9634-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-278fe7d5-f9pk6\" (UID: \"d5763345-21f8-48b3-880b-d36fd04c9634\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-f9pk6" Oct 30 01:34:31.564673 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:31.562086 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7gd7\" (UniqueName: \"kubernetes.io/projected/d5763345-21f8-48b3-880b-d36fd04c9634-kube-api-access-p7gd7\") pod \"maas-api-revoke-keys-e2e-mt-278fe7d5-f9pk6\" (UID: \"d5763345-21f8-48b3-880b-d36fd04c9634\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-f9pk6" Oct 30 01:34:31.607367 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:31.607250 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:31.662144 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:31.662115 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-f9pk6" Oct 30 01:34:31.967046 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:34:31.967010 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5763345_21f8_48b3_880b_d36fd04c9634.slice/crio-7cc6804138431660d12d9405ea811d7bb0c5d5d28dcdfbee53fa84f72b7b0a2b WatchSource:0}: Error finding container 7cc6804138431660d12d9405ea811d7bb0c5d5d28dcdfbee53fa84f72b7b0a2b: Status 404 returned error can't find the container with id 7cc6804138431660d12d9405ea811d7bb0c5d5d28dcdfbee53fa84f72b7b0a2b Oct 30 01:34:31.969297 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:31.969272 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-f9pk6"] Oct 30 01:34:32.650776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:32.650650 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-f9pk6" event={"ID":"d5763345-21f8-48b3-880b-d36fd04c9634","Type":"ContainerStarted","Data":"2cd0d5c12766664f2414990d2b10bae1b3f0eac35bffee23b37cb74fa5d4a1f2"} Oct 30 01:34:32.650776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:32.650712 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-f9pk6" event={"ID":"d5763345-21f8-48b3-880b-d36fd04c9634","Type":"ContainerStarted","Data":"7cc6804138431660d12d9405ea811d7bb0c5d5d28dcdfbee53fa84f72b7b0a2b"} Oct 30 01:34:32.688980 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:32.688910 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-f9pk6" podStartSLOduration=1.688889327 podStartE2EDuration="1.688889327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:32.68481238 +0000 UTC m=+998.261237778" watchObservedRunningTime="2026-10-30 01:34:32.688889327 +0000 UTC m=+998.265314728" Oct 30 01:34:33.446515 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:33.446425 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:33.636406 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:33.636367 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:33.660993 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:33.660957 6572 generic.go:356] "Generic (PLEG): container finished" podID="d5763345-21f8-48b3-880b-d36fd04c9634" containerID="2cd0d5c12766664f2414990d2b10bae1b3f0eac35bffee23b37cb74fa5d4a1f2" exitCode=0 Oct 30 01:34:33.661438 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:33.661047 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-f9pk6" event={"ID":"d5763345-21f8-48b3-880b-d36fd04c9634","Type":"ContainerDied","Data":"2cd0d5c12766664f2414990d2b10bae1b3f0eac35bffee23b37cb74fa5d4a1f2"} Oct 30 01:34:33.831727 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:33.831634 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:34.852013 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:34.851982 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-f9pk6" Oct 30 01:34:34.916946 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:34.916909 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5763345-21f8-48b3-880b-d36fd04c9634-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5763345-21f8-48b3-880b-d36fd04c9634-maas-api-openshift-ca\") pod \"d5763345-21f8-48b3-880b-d36fd04c9634\" (UID: \"d5763345-21f8-48b3-880b-d36fd04c9634\") " Oct 30 01:34:34.917143 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:34.917031 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5763345-21f8-48b3-880b-d36fd04c9634-kube-api-access-p7gd7\" (UniqueName: \"kubernetes.io/projected/d5763345-21f8-48b3-880b-d36fd04c9634-kube-api-access-p7gd7\") pod \"d5763345-21f8-48b3-880b-d36fd04c9634\" (UID: \"d5763345-21f8-48b3-880b-d36fd04c9634\") " Oct 30 01:34:34.917244 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:34.917228 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5763345-21f8-48b3-880b-d36fd04c9634-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5763345-21f8-48b3-880b-d36fd04c9634-maas-api-xks-ca\") pod \"d5763345-21f8-48b3-880b-d36fd04c9634\" (UID: \"d5763345-21f8-48b3-880b-d36fd04c9634\") " Oct 30 01:34:34.917375 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:34.917346 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5763345-21f8-48b3-880b-d36fd04c9634-maas-api-openshift-ca" pod "d5763345-21f8-48b3-880b-d36fd04c9634" (UID: "d5763345-21f8-48b3-880b-d36fd04c9634"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:34:34.917885 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:34.917863 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5763345-21f8-48b3-880b-d36fd04c9634-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:34:34.919626 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:34.919562 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5763345-21f8-48b3-880b-d36fd04c9634-maas-api-xks-ca" pod "d5763345-21f8-48b3-880b-d36fd04c9634" (UID: "d5763345-21f8-48b3-880b-d36fd04c9634"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:34:34.920147 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:34.920118 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5763345-21f8-48b3-880b-d36fd04c9634-kube-api-access-p7gd7" pod "d5763345-21f8-48b3-880b-d36fd04c9634" (UID: "d5763345-21f8-48b3-880b-d36fd04c9634"). InnerVolumeSpecName "kube-api-access-p7gd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:34:35.019268 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:35.019233 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7gd7\" (UniqueName: \"kubernetes.io/projected/d5763345-21f8-48b3-880b-d36fd04c9634-kube-api-access-p7gd7\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:34:35.019268 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:35.019266 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5763345-21f8-48b3-880b-d36fd04c9634-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:34:35.263338 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:35.263236 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:35.679415 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:35.679383 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-f9pk6" Oct 30 01:34:35.679598 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:35.679427 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-f9pk6" event={"ID":"d5763345-21f8-48b3-880b-d36fd04c9634","Type":"ContainerDied","Data":"7cc6804138431660d12d9405ea811d7bb0c5d5d28dcdfbee53fa84f72b7b0a2b"} Oct 30 01:34:35.679598 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:35.679481 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cc6804138431660d12d9405ea811d7bb0c5d5d28dcdfbee53fa84f72b7b0a2b" Oct 30 01:34:36.925759 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:36.925706 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:37.713998 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:37.713957 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:38.800018 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:38.799985 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:39.968470 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:39.968432 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec2345-6lcns"] Oct 30 01:34:39.969212 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:39.969196 6572 cpu_manager.go:414] "containerMap: removing container" podUID="d5763345-21f8-48b3-880b-d36fd04c9634" containerName="revoke-keys" Oct 30 01:34:39.969260 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:39.969214 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5763345-21f8-48b3-880b-d36fd04c9634" containerName="revoke-keys" Oct 30 01:34:39.969329 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:39.969319 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5763345-21f8-48b3-880b-d36fd04c9634" containerName="revoke-keys" Oct 30 01:34:39.973202 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:39.973179 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec2345-6lcns" Oct 30 01:34:39.990409 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:39.990378 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec2345-6lcns"] Oct 30 01:34:39.994047 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:39.994004 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:34:39.994047 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:39.994041 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:34:39.994228 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:39.994040 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:34:39.995163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:39.995141 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:34:40.092479 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:40.092446 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ec2345-6lcns\" (UID: \"bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec2345-6lcns" Oct 30 01:34:40.092479 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:40.092484 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ec2345-6lcns\" (UID: \"bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec2345-6lcns" Oct 30 01:34:40.092732 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:40.092706 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8rxb\" (UniqueName: \"kubernetes.io/projected/bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed-kube-api-access-r8rxb\") pod \"maas-api-revoke-keys-e2e-api-a-ec2345-6lcns\" (UID: \"bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec2345-6lcns" Oct 30 01:34:40.193557 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:40.193516 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ec2345-6lcns\" (UID: \"bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec2345-6lcns" Oct 30 01:34:40.193557 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:40.193560 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ec2345-6lcns\" (UID: \"bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec2345-6lcns" Oct 30 01:34:40.193866 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:40.193729 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8rxb\" (UniqueName: \"kubernetes.io/projected/bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed-kube-api-access-r8rxb\") pod \"maas-api-revoke-keys-e2e-api-a-ec2345-6lcns\" (UID: \"bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec2345-6lcns" Oct 30 01:34:40.194251 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:40.194229 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ec2345-6lcns\" (UID: \"bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec2345-6lcns" Oct 30 01:34:40.196043 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:40.196024 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ec2345-6lcns\" (UID: \"bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec2345-6lcns" Oct 30 01:34:40.211172 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:40.211139 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8rxb\" (UniqueName: \"kubernetes.io/projected/bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed-kube-api-access-r8rxb\") pod \"maas-api-revoke-keys-e2e-api-a-ec2345-6lcns\" (UID: \"bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec2345-6lcns" Oct 30 01:34:40.291855 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:40.291732 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec2345-6lcns" Oct 30 01:34:40.535229 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:40.535190 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:40.599280 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:34:40.599246 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc65cdaf_3bd6_494b_8a9d_68ea6373c8ed.slice/crio-d21bdfc1528af5bf8fad1afbc881082452cda059cdeef4640e70e12d0e61c843 WatchSource:0}: Error finding container d21bdfc1528af5bf8fad1afbc881082452cda059cdeef4640e70e12d0e61c843: Status 404 returned error can't find the container with id d21bdfc1528af5bf8fad1afbc881082452cda059cdeef4640e70e12d0e61c843 Oct 30 01:34:40.599784 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:40.599740 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec2345-6lcns"] Oct 30 01:34:40.730815 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:40.730776 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec2345-6lcns" event={"ID":"bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed","Type":"ContainerStarted","Data":"d21bdfc1528af5bf8fad1afbc881082452cda059cdeef4640e70e12d0e61c843"} Oct 30 01:34:41.742311 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:41.742277 6572 generic.go:356] "Generic (PLEG): container finished" podID="bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed" containerID="6ebf2030e9184d67ebe2950149cb181e6f31529567072bcce6b2134d69432ac2" exitCode=0 Oct 30 01:34:41.742920 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:41.742364 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec2345-6lcns" event={"ID":"bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed","Type":"ContainerDied","Data":"6ebf2030e9184d67ebe2950149cb181e6f31529567072bcce6b2134d69432ac2"} Oct 30 01:34:41.742920 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:41.742433 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-9t2d5"] Oct 30 01:34:41.946520 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:41.946473 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04cef553-42l96"] Oct 30 01:34:41.952438 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:41.952407 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cef553-42l96" Oct 30 01:34:41.967166 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:41.967126 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04cef553-42l96"] Oct 30 01:34:41.969149 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:41.969111 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:42.123315 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:42.123280 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79828b67-08a2-4d32-9dc7-5ecf367791a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04cef553-42l96\" (UID: \"79828b67-08a2-4d32-9dc7-5ecf367791a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cef553-42l96" Oct 30 01:34:42.123511 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:42.123352 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79828b67-08a2-4d32-9dc7-5ecf367791a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04cef553-42l96\" (UID: \"79828b67-08a2-4d32-9dc7-5ecf367791a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cef553-42l96" Oct 30 01:34:42.123559 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:42.123498 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rssxn\" (UniqueName: \"kubernetes.io/projected/79828b67-08a2-4d32-9dc7-5ecf367791a9-kube-api-access-rssxn\") pod \"maas-api-revoke-sub-04cef553-42l96\" (UID: \"79828b67-08a2-4d32-9dc7-5ecf367791a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cef553-42l96" Oct 30 01:34:42.157273 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:42.157233 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:42.225301 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:42.225214 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79828b67-08a2-4d32-9dc7-5ecf367791a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04cef553-42l96\" (UID: \"79828b67-08a2-4d32-9dc7-5ecf367791a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cef553-42l96" Oct 30 01:34:42.225301 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:42.225283 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79828b67-08a2-4d32-9dc7-5ecf367791a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04cef553-42l96\" (UID: \"79828b67-08a2-4d32-9dc7-5ecf367791a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cef553-42l96" Oct 30 01:34:42.225554 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:42.225440 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rssxn\" (UniqueName: \"kubernetes.io/projected/79828b67-08a2-4d32-9dc7-5ecf367791a9-kube-api-access-rssxn\") pod \"maas-api-revoke-sub-04cef553-42l96\" (UID: \"79828b67-08a2-4d32-9dc7-5ecf367791a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cef553-42l96" Oct 30 01:34:42.226003 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:42.225982 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79828b67-08a2-4d32-9dc7-5ecf367791a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04cef553-42l96\" (UID: \"79828b67-08a2-4d32-9dc7-5ecf367791a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cef553-42l96" Oct 30 01:34:42.227789 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:42.227730 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79828b67-08a2-4d32-9dc7-5ecf367791a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04cef553-42l96\" (UID: \"79828b67-08a2-4d32-9dc7-5ecf367791a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cef553-42l96" Oct 30 01:34:42.242070 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:42.242045 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rssxn\" (UniqueName: \"kubernetes.io/projected/79828b67-08a2-4d32-9dc7-5ecf367791a9-kube-api-access-rssxn\") pod \"maas-api-revoke-sub-04cef553-42l96\" (UID: \"79828b67-08a2-4d32-9dc7-5ecf367791a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cef553-42l96" Oct 30 01:34:42.274041 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:42.274007 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cef553-42l96" Oct 30 01:34:42.525836 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:42.514933 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:42.783459 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:34:42.783425 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79828b67_08a2_4d32_9dc7_5ecf367791a9.slice/crio-a42e3761b263c1c0f46a2cfb3f28e2a44ba13766c17cbea7221cbe7b16568967 WatchSource:0}: Error finding container a42e3761b263c1c0f46a2cfb3f28e2a44ba13766c17cbea7221cbe7b16568967: Status 404 returned error can't find the container with id a42e3761b263c1c0f46a2cfb3f28e2a44ba13766c17cbea7221cbe7b16568967 Oct 30 01:34:42.783459 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:42.783440 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04cef553-42l96"] Oct 30 01:34:42.895422 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:42.895385 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec2345-6lcns" Oct 30 01:34:42.956875 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:42.956836 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed-maas-api-xks-ca\") pod \"bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed\" (UID: \"bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed\") " Oct 30 01:34:42.957088 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:42.957032 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed-kube-api-access-r8rxb\" (UniqueName: \"kubernetes.io/projected/bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed-kube-api-access-r8rxb\") pod \"bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed\" (UID: \"bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed\") " Oct 30 01:34:42.957169 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:42.957156 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed-maas-api-openshift-ca\") pod \"bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed\" (UID: \"bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed\") " Oct 30 01:34:42.957738 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:42.957705 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed-maas-api-openshift-ca" pod "bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed" (UID: "bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:34:42.959919 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:42.959874 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed-maas-api-xks-ca" pod "bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed" (UID: "bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:34:42.960158 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:42.960132 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed-kube-api-access-r8rxb" pod "bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed" (UID: "bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed"). InnerVolumeSpecName "kube-api-access-r8rxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:34:43.059636 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:43.059547 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8rxb\" (UniqueName: \"kubernetes.io/projected/bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed-kube-api-access-r8rxb\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:34:43.059636 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:43.059579 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:34:43.059636 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:43.059589 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:34:43.764674 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:43.764647 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec2345-6lcns" Oct 30 01:34:43.764915 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:43.764685 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec2345-6lcns" event={"ID":"bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed","Type":"ContainerDied","Data":"d21bdfc1528af5bf8fad1afbc881082452cda059cdeef4640e70e12d0e61c843"} Oct 30 01:34:43.764915 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:43.764773 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d21bdfc1528af5bf8fad1afbc881082452cda059cdeef4640e70e12d0e61c843" Oct 30 01:34:43.766653 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:43.766621 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cef553-42l96" event={"ID":"79828b67-08a2-4d32-9dc7-5ecf367791a9","Type":"ContainerStarted","Data":"c8c5ff6ac86ef6f7a5a645beb5b799b973284153af0e62e5470172dd14991fa1"} Oct 30 01:34:43.766824 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:43.766670 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cef553-42l96" event={"ID":"79828b67-08a2-4d32-9dc7-5ecf367791a9","Type":"ContainerStarted","Data":"a42e3761b263c1c0f46a2cfb3f28e2a44ba13766c17cbea7221cbe7b16568967"} Oct 30 01:34:43.806888 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:43.806835 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cef553-42l96" podStartSLOduration=2.8068202859999998 podStartE2EDuration="2.806820286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:43.800576984 +0000 UTC m=+1009.377002386" watchObservedRunningTime="2026-10-30 01:34:43.806820286 +0000 UTC m=+1009.383245688" Oct 30 01:34:44.418324 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:44.418284 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:44.689678 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:34:44.689587 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:34:44.776849 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:44.776813 6572 generic.go:356] "Generic (PLEG): container finished" podID="79828b67-08a2-4d32-9dc7-5ecf367791a9" containerID="c8c5ff6ac86ef6f7a5a645beb5b799b973284153af0e62e5470172dd14991fa1" exitCode=0 Oct 30 01:34:44.777009 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:44.776907 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cef553-42l96" event={"ID":"79828b67-08a2-4d32-9dc7-5ecf367791a9","Type":"ContainerDied","Data":"c8c5ff6ac86ef6f7a5a645beb5b799b973284153af0e62e5470172dd14991fa1"} Oct 30 01:34:45.034977 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:45.034890 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be8d0c4f-l2kt9"] Oct 30 01:34:45.040524 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:45.040165 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be8d0c4f-l2kt9"] Oct 30 01:34:45.957952 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:45.957926 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cef553-42l96" Oct 30 01:34:46.037146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:46.037046 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79828b67-08a2-4d32-9dc7-5ecf367791a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79828b67-08a2-4d32-9dc7-5ecf367791a9-maas-api-xks-ca\") pod \"79828b67-08a2-4d32-9dc7-5ecf367791a9\" (UID: \"79828b67-08a2-4d32-9dc7-5ecf367791a9\") " Oct 30 01:34:46.037626 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:46.037151 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79828b67-08a2-4d32-9dc7-5ecf367791a9-kube-api-access-rssxn\" (UniqueName: \"kubernetes.io/projected/79828b67-08a2-4d32-9dc7-5ecf367791a9-kube-api-access-rssxn\") pod \"79828b67-08a2-4d32-9dc7-5ecf367791a9\" (UID: \"79828b67-08a2-4d32-9dc7-5ecf367791a9\") " Oct 30 01:34:46.037626 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:46.037358 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79828b67-08a2-4d32-9dc7-5ecf367791a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79828b67-08a2-4d32-9dc7-5ecf367791a9-maas-api-openshift-ca\") pod \"79828b67-08a2-4d32-9dc7-5ecf367791a9\" (UID: \"79828b67-08a2-4d32-9dc7-5ecf367791a9\") " Oct 30 01:34:46.037766 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:46.037719 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79828b67-08a2-4d32-9dc7-5ecf367791a9-maas-api-openshift-ca" pod "79828b67-08a2-4d32-9dc7-5ecf367791a9" (UID: "79828b67-08a2-4d32-9dc7-5ecf367791a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:34:46.038146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:46.038120 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79828b67-08a2-4d32-9dc7-5ecf367791a9-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:34:46.042827 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:46.042797 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79828b67-08a2-4d32-9dc7-5ecf367791a9-maas-api-xks-ca" pod "79828b67-08a2-4d32-9dc7-5ecf367791a9" (UID: "79828b67-08a2-4d32-9dc7-5ecf367791a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:34:46.043895 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:46.043867 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79828b67-08a2-4d32-9dc7-5ecf367791a9-kube-api-access-rssxn" pod "79828b67-08a2-4d32-9dc7-5ecf367791a9" (UID: "79828b67-08a2-4d32-9dc7-5ecf367791a9"). InnerVolumeSpecName "kube-api-access-rssxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:34:46.139444 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:46.139408 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79828b67-08a2-4d32-9dc7-5ecf367791a9-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:34:46.139444 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:46.139443 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rssxn\" (UniqueName: \"kubernetes.io/projected/79828b67-08a2-4d32-9dc7-5ecf367791a9-kube-api-access-rssxn\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:34:46.151025 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:46.150910 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:46.543626 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:46.543589 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82531f0a-89ec-4156-889a-af4b9b949bfd" path="/var/lib/kubelet/pods/82531f0a-89ec-4156-889a-af4b9b949bfd/volumes" Oct 30 01:34:46.743090 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:46.743038 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-9t2d5" podUID="485b6169-3cf8-4111-be91-8050f32439d7" containerName="payload-processing" containerID="cri-o://04eb97acfe1994c72454bfaf1078733f44e22f8338a8c0786ca58417d21fce3b" gracePeriod=25 Oct 30 01:34:46.797180 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:46.797100 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cef553-42l96" Oct 30 01:34:46.797180 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:46.797122 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cef553-42l96" event={"ID":"79828b67-08a2-4d32-9dc7-5ecf367791a9","Type":"ContainerDied","Data":"a42e3761b263c1c0f46a2cfb3f28e2a44ba13766c17cbea7221cbe7b16568967"} Oct 30 01:34:46.797180 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:46.797170 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a42e3761b263c1c0f46a2cfb3f28e2a44ba13766c17cbea7221cbe7b16568967" Oct 30 01:34:47.923238 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:47.923194 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:47.947698 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:47.947666 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebdd4dab-hnc55"] Oct 30 01:34:47.948442 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:47.948415 6572 cpu_manager.go:414] "containerMap: removing container" podUID="bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed" containerName="revoke-keys" Oct 30 01:34:47.948442 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:47.948437 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed" containerName="revoke-keys" Oct 30 01:34:47.948622 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:47.948459 6572 cpu_manager.go:414] "containerMap: removing container" podUID="79828b67-08a2-4d32-9dc7-5ecf367791a9" containerName="revoke-keys" Oct 30 01:34:47.948622 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:47.948464 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79828b67-08a2-4d32-9dc7-5ecf367791a9" containerName="revoke-keys" Oct 30 01:34:47.948622 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:47.948569 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed" containerName="revoke-keys" Oct 30 01:34:47.948622 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:47.948581 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="79828b67-08a2-4d32-9dc7-5ecf367791a9" containerName="revoke-keys" Oct 30 01:34:47.953429 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:47.953403 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebdd4dab-hnc55" Oct 30 01:34:47.975905 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:47.975301 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebdd4dab-hnc55"] Oct 30 01:34:48.000858 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:48.000824 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:34:48.001067 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:48.000824 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:34:48.001457 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:48.001407 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:34:48.001604 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:48.001543 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:34:48.067859 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:48.067816 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d14081a-d8ef-49af-8792-26f6956b6141-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebdd4dab-hnc55\" (UID: \"4d14081a-d8ef-49af-8792-26f6956b6141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebdd4dab-hnc55" Oct 30 01:34:48.068127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:48.068100 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d14081a-d8ef-49af-8792-26f6956b6141-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebdd4dab-hnc55\" (UID: \"4d14081a-d8ef-49af-8792-26f6956b6141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebdd4dab-hnc55" Oct 30 01:34:48.068177 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:48.068143 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm67n\" (UniqueName: \"kubernetes.io/projected/4d14081a-d8ef-49af-8792-26f6956b6141-kube-api-access-rm67n\") pod \"maas-api-revoke-sub-ebdd4dab-hnc55\" (UID: \"4d14081a-d8ef-49af-8792-26f6956b6141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebdd4dab-hnc55" Oct 30 01:34:48.169332 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:48.169285 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d14081a-d8ef-49af-8792-26f6956b6141-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebdd4dab-hnc55\" (UID: \"4d14081a-d8ef-49af-8792-26f6956b6141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebdd4dab-hnc55" Oct 30 01:34:48.169332 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:48.169337 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm67n\" (UniqueName: \"kubernetes.io/projected/4d14081a-d8ef-49af-8792-26f6956b6141-kube-api-access-rm67n\") pod \"maas-api-revoke-sub-ebdd4dab-hnc55\" (UID: \"4d14081a-d8ef-49af-8792-26f6956b6141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebdd4dab-hnc55" Oct 30 01:34:48.169552 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:48.169393 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d14081a-d8ef-49af-8792-26f6956b6141-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebdd4dab-hnc55\" (UID: \"4d14081a-d8ef-49af-8792-26f6956b6141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebdd4dab-hnc55" Oct 30 01:34:48.170131 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:48.170105 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d14081a-d8ef-49af-8792-26f6956b6141-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebdd4dab-hnc55\" (UID: \"4d14081a-d8ef-49af-8792-26f6956b6141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebdd4dab-hnc55" Oct 30 01:34:48.171794 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:48.171770 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d14081a-d8ef-49af-8792-26f6956b6141-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebdd4dab-hnc55\" (UID: \"4d14081a-d8ef-49af-8792-26f6956b6141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebdd4dab-hnc55" Oct 30 01:34:48.186068 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:48.185994 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm67n\" (UniqueName: \"kubernetes.io/projected/4d14081a-d8ef-49af-8792-26f6956b6141-kube-api-access-rm67n\") pod \"maas-api-revoke-sub-ebdd4dab-hnc55\" (UID: \"4d14081a-d8ef-49af-8792-26f6956b6141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebdd4dab-hnc55" Oct 30 01:34:48.288978 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:48.288928 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebdd4dab-hnc55" Oct 30 01:34:48.600129 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:48.599958 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebdd4dab-hnc55"] Oct 30 01:34:48.606399 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:34:48.606372 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d14081a_d8ef_49af_8792_26f6956b6141.slice/crio-71bb7b00c911ecdf0e662041efc47a3ad97d261ff13e19f43aac4f32542a0b8f WatchSource:0}: Error finding container 71bb7b00c911ecdf0e662041efc47a3ad97d261ff13e19f43aac4f32542a0b8f: Status 404 returned error can't find the container with id 71bb7b00c911ecdf0e662041efc47a3ad97d261ff13e19f43aac4f32542a0b8f Oct 30 01:34:48.816328 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:48.816287 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebdd4dab-hnc55" event={"ID":"4d14081a-d8ef-49af-8792-26f6956b6141","Type":"ContainerStarted","Data":"71bb7b00c911ecdf0e662041efc47a3ad97d261ff13e19f43aac4f32542a0b8f"} Oct 30 01:34:49.697113 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:49.697021 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:49.826728 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:49.826696 6572 generic.go:356] "Generic (PLEG): container finished" podID="4d14081a-d8ef-49af-8792-26f6956b6141" containerID="176a7d6064b2dba41e33018b0bfe33ce4d6e03e4aa6284d57e4f8dbdbca2b4a3" exitCode=0 Oct 30 01:34:49.826949 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:49.826800 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebdd4dab-hnc55" event={"ID":"4d14081a-d8ef-49af-8792-26f6956b6141","Type":"ContainerDied","Data":"176a7d6064b2dba41e33018b0bfe33ce4d6e03e4aa6284d57e4f8dbdbca2b4a3"} Oct 30 01:34:50.384081 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:50.384042 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ec2345-6f57b55f9d-sm5jj"] Oct 30 01:34:50.412293 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:50.412259 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ec2345-55774dc5f5-4ghc6"] Oct 30 01:34:50.602265 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:50.602220 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:50.796521 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:50.796472 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:51.008923 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:51.008897 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebdd4dab-hnc55" Oct 30 01:34:51.114358 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:51.111491 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d14081a-d8ef-49af-8792-26f6956b6141-kube-api-access-rm67n\" (UniqueName: \"kubernetes.io/projected/4d14081a-d8ef-49af-8792-26f6956b6141-kube-api-access-rm67n\") pod \"4d14081a-d8ef-49af-8792-26f6956b6141\" (UID: \"4d14081a-d8ef-49af-8792-26f6956b6141\") " Oct 30 01:34:51.114358 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:51.111598 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d14081a-d8ef-49af-8792-26f6956b6141-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d14081a-d8ef-49af-8792-26f6956b6141-maas-api-openshift-ca\") pod \"4d14081a-d8ef-49af-8792-26f6956b6141\" (UID: \"4d14081a-d8ef-49af-8792-26f6956b6141\") " Oct 30 01:34:51.114358 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:51.111694 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d14081a-d8ef-49af-8792-26f6956b6141-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d14081a-d8ef-49af-8792-26f6956b6141-maas-api-xks-ca\") pod \"4d14081a-d8ef-49af-8792-26f6956b6141\" (UID: \"4d14081a-d8ef-49af-8792-26f6956b6141\") " Oct 30 01:34:51.136801 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:51.134264 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d14081a-d8ef-49af-8792-26f6956b6141-maas-api-openshift-ca" pod "4d14081a-d8ef-49af-8792-26f6956b6141" (UID: "4d14081a-d8ef-49af-8792-26f6956b6141"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:34:51.157802 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:51.156318 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d14081a-d8ef-49af-8792-26f6956b6141-kube-api-access-rm67n" pod "4d14081a-d8ef-49af-8792-26f6956b6141" (UID: "4d14081a-d8ef-49af-8792-26f6956b6141"). InnerVolumeSpecName "kube-api-access-rm67n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:34:51.217770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:51.214184 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm67n\" (UniqueName: \"kubernetes.io/projected/4d14081a-d8ef-49af-8792-26f6956b6141-kube-api-access-rm67n\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:34:51.217770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:51.214221 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d14081a-d8ef-49af-8792-26f6956b6141-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:34:51.257772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:51.247058 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d14081a-d8ef-49af-8792-26f6956b6141-maas-api-xks-ca" pod "4d14081a-d8ef-49af-8792-26f6956b6141" (UID: "4d14081a-d8ef-49af-8792-26f6956b6141"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:34:51.315965 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:51.315914 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d14081a-d8ef-49af-8792-26f6956b6141-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:34:51.447767 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:51.446707 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:51.845065 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:51.844978 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebdd4dab-hnc55" Oct 30 01:34:51.845065 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:51.845014 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebdd4dab-hnc55" event={"ID":"4d14081a-d8ef-49af-8792-26f6956b6141","Type":"ContainerDied","Data":"71bb7b00c911ecdf0e662041efc47a3ad97d261ff13e19f43aac4f32542a0b8f"} Oct 30 01:34:51.845065 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:51.845061 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71bb7b00c911ecdf0e662041efc47a3ad97d261ff13e19f43aac4f32542a0b8f" Oct 30 01:34:53.222868 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:53.222834 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:54.491541 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:54.491512 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:54.687343 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:54.687310 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:54.870392 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:54.870283 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:55.295229 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:55.295193 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-9t2d5" podUID="485b6169-3cf8-4111-be91-8050f32439d7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:34:55.384792 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:55.384696 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-ec2345-6f57b55f9d-sm5jj" podUID="2f3be9f9-cc25-4601-9015-8a323abb7db1" containerName="payload-processing" containerID="cri-o://2fb9ac42dba5fed2c2bdfa9d3dfc0fd38619e33e961b8a4a234bfb5ec67c8980" gracePeriod=25 Oct 30 01:34:55.413137 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:55.413099 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec2345-55774dc5f5-4ghc6" podUID="ea7df11c-947f-49ee-9174-28d78fef15ca" containerName="payload-pre-processing" containerID="cri-o://a9c63d386ae6b6bad0ec42e36fbd1f691740c5fa8d862532c09b474abe9e2f20" gracePeriod=25 Oct 30 01:34:55.697076 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:55.697044 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ec2345-6f57b55f9d-sm5jj" Oct 30 01:34:55.751656 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:55.751626 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec2345-55774dc5f5-4ghc6" Oct 30 01:34:55.782210 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:55.782178 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f3be9f9-cc25-4601-9015-8a323abb7db1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f3be9f9-cc25-4601-9015-8a323abb7db1-plugins-config-volume\") pod \"2f3be9f9-cc25-4601-9015-8a323abb7db1\" (UID: \"2f3be9f9-cc25-4601-9015-8a323abb7db1\") " Oct 30 01:34:55.782417 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:55.782344 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f3be9f9-cc25-4601-9015-8a323abb7db1-kube-api-access-svx52\" (UniqueName: \"kubernetes.io/projected/2f3be9f9-cc25-4601-9015-8a323abb7db1-kube-api-access-svx52\") pod \"2f3be9f9-cc25-4601-9015-8a323abb7db1\" (UID: \"2f3be9f9-cc25-4601-9015-8a323abb7db1\") " Oct 30 01:34:55.783522 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:55.783488 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f3be9f9-cc25-4601-9015-8a323abb7db1-plugins-config-volume" pod "2f3be9f9-cc25-4601-9015-8a323abb7db1" (UID: "2f3be9f9-cc25-4601-9015-8a323abb7db1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:34:55.785446 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:55.785415 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f3be9f9-cc25-4601-9015-8a323abb7db1-kube-api-access-svx52" pod "2f3be9f9-cc25-4601-9015-8a323abb7db1" (UID: "2f3be9f9-cc25-4601-9015-8a323abb7db1"). InnerVolumeSpecName "kube-api-access-svx52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:34:55.883635 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:55.883553 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea7df11c-947f-49ee-9174-28d78fef15ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea7df11c-947f-49ee-9174-28d78fef15ca-plugins-config-volume\") pod \"ea7df11c-947f-49ee-9174-28d78fef15ca\" (UID: \"ea7df11c-947f-49ee-9174-28d78fef15ca\") " Oct 30 01:34:55.883822 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:55.883774 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea7df11c-947f-49ee-9174-28d78fef15ca-kube-api-access-b9ln8\" (UniqueName: \"kubernetes.io/projected/ea7df11c-947f-49ee-9174-28d78fef15ca-kube-api-access-b9ln8\") pod \"ea7df11c-947f-49ee-9174-28d78fef15ca\" (UID: \"ea7df11c-947f-49ee-9174-28d78fef15ca\") " Oct 30 01:34:55.884020 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:55.883996 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea7df11c-947f-49ee-9174-28d78fef15ca-plugins-config-volume" pod "ea7df11c-947f-49ee-9174-28d78fef15ca" (UID: "ea7df11c-947f-49ee-9174-28d78fef15ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:34:55.884363 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:55.884336 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svx52\" (UniqueName: \"kubernetes.io/projected/2f3be9f9-cc25-4601-9015-8a323abb7db1-kube-api-access-svx52\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:34:55.884363 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:55.884360 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea7df11c-947f-49ee-9174-28d78fef15ca-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:34:55.884516 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:55.884374 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f3be9f9-cc25-4601-9015-8a323abb7db1-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:34:55.885731 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:55.885706 6572 generic.go:356] "Generic (PLEG): container finished" podID="2f3be9f9-cc25-4601-9015-8a323abb7db1" containerID="2fb9ac42dba5fed2c2bdfa9d3dfc0fd38619e33e961b8a4a234bfb5ec67c8980" exitCode=0 Oct 30 01:34:55.885833 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:55.885787 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ec2345-6f57b55f9d-sm5jj" Oct 30 01:34:55.885833 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:55.885794 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ec2345-6f57b55f9d-sm5jj" event={"ID":"2f3be9f9-cc25-4601-9015-8a323abb7db1","Type":"ContainerDied","Data":"2fb9ac42dba5fed2c2bdfa9d3dfc0fd38619e33e961b8a4a234bfb5ec67c8980"} Oct 30 01:34:55.885971 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:55.885860 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ec2345-6f57b55f9d-sm5jj" event={"ID":"2f3be9f9-cc25-4601-9015-8a323abb7db1","Type":"ContainerDied","Data":"1bf568ba52b4c0ab8445d975f61ad5fde9bce801763f916ffce8cc6a640ec07a"} Oct 30 01:34:55.885971 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:55.885897 6572 scope.go:122] "RemoveContainer" containerID="2fb9ac42dba5fed2c2bdfa9d3dfc0fd38619e33e961b8a4a234bfb5ec67c8980" Oct 30 01:34:55.886292 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:55.886270 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea7df11c-947f-49ee-9174-28d78fef15ca-kube-api-access-b9ln8" pod "ea7df11c-947f-49ee-9174-28d78fef15ca" (UID: "ea7df11c-947f-49ee-9174-28d78fef15ca"). InnerVolumeSpecName "kube-api-access-b9ln8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:34:55.887573 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:55.887545 6572 generic.go:356] "Generic (PLEG): container finished" podID="ea7df11c-947f-49ee-9174-28d78fef15ca" containerID="a9c63d386ae6b6bad0ec42e36fbd1f691740c5fa8d862532c09b474abe9e2f20" exitCode=0 Oct 30 01:34:55.887674 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:55.887580 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec2345-55774dc5f5-4ghc6" Oct 30 01:34:55.887674 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:55.887590 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec2345-55774dc5f5-4ghc6" event={"ID":"ea7df11c-947f-49ee-9174-28d78fef15ca","Type":"ContainerDied","Data":"a9c63d386ae6b6bad0ec42e36fbd1f691740c5fa8d862532c09b474abe9e2f20"} Oct 30 01:34:55.887674 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:55.887633 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ec2345-55774dc5f5-4ghc6" event={"ID":"ea7df11c-947f-49ee-9174-28d78fef15ca","Type":"ContainerDied","Data":"3ef76528016fa1221fc99b037eafcb4ef280eaaff20f25712cfd94709a11479a"} Oct 30 01:34:55.898947 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:55.898930 6572 scope.go:122] "RemoveContainer" containerID="2fb9ac42dba5fed2c2bdfa9d3dfc0fd38619e33e961b8a4a234bfb5ec67c8980" Oct 30 01:34:55.899208 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:34:55.899193 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fb9ac42dba5fed2c2bdfa9d3dfc0fd38619e33e961b8a4a234bfb5ec67c8980\": container with ID starting with 2fb9ac42dba5fed2c2bdfa9d3dfc0fd38619e33e961b8a4a234bfb5ec67c8980 not found: ID does not exist" containerID="2fb9ac42dba5fed2c2bdfa9d3dfc0fd38619e33e961b8a4a234bfb5ec67c8980" Oct 30 01:34:55.899249 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:55.899215 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fb9ac42dba5fed2c2bdfa9d3dfc0fd38619e33e961b8a4a234bfb5ec67c8980"} err="failed to get container status \"2fb9ac42dba5fed2c2bdfa9d3dfc0fd38619e33e961b8a4a234bfb5ec67c8980\": rpc error: code = NotFound desc = could not find container \"2fb9ac42dba5fed2c2bdfa9d3dfc0fd38619e33e961b8a4a234bfb5ec67c8980\": container with ID starting with 2fb9ac42dba5fed2c2bdfa9d3dfc0fd38619e33e961b8a4a234bfb5ec67c8980 not found: ID does not exist" Oct 30 01:34:55.899249 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:55.899230 6572 scope.go:122] "RemoveContainer" containerID="a9c63d386ae6b6bad0ec42e36fbd1f691740c5fa8d862532c09b474abe9e2f20" Oct 30 01:34:55.910165 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:55.910139 6572 scope.go:122] "RemoveContainer" containerID="a9c63d386ae6b6bad0ec42e36fbd1f691740c5fa8d862532c09b474abe9e2f20" Oct 30 01:34:55.910495 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:34:55.910475 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9c63d386ae6b6bad0ec42e36fbd1f691740c5fa8d862532c09b474abe9e2f20\": container with ID starting with a9c63d386ae6b6bad0ec42e36fbd1f691740c5fa8d862532c09b474abe9e2f20 not found: ID does not exist" containerID="a9c63d386ae6b6bad0ec42e36fbd1f691740c5fa8d862532c09b474abe9e2f20" Oct 30 01:34:55.910555 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:55.910503 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9c63d386ae6b6bad0ec42e36fbd1f691740c5fa8d862532c09b474abe9e2f20"} err="failed to get container status \"a9c63d386ae6b6bad0ec42e36fbd1f691740c5fa8d862532c09b474abe9e2f20\": rpc error: code = NotFound desc = could not find container \"a9c63d386ae6b6bad0ec42e36fbd1f691740c5fa8d862532c09b474abe9e2f20\": container with ID starting with a9c63d386ae6b6bad0ec42e36fbd1f691740c5fa8d862532c09b474abe9e2f20 not found: ID does not exist" Oct 30 01:34:55.935317 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:55.935192 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ec2345-6f57b55f9d-sm5jj"] Oct 30 01:34:55.940642 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:55.940610 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ec2345-6f57b55f9d-sm5jj"] Oct 30 01:34:55.966989 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:55.966958 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ec2345-55774dc5f5-4ghc6"] Oct 30 01:34:55.971835 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:55.971808 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ec2345-55774dc5f5-4ghc6"] Oct 30 01:34:55.986082 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:55.986052 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9ln8\" (UniqueName: \"kubernetes.io/projected/ea7df11c-947f-49ee-9174-28d78fef15ca-kube-api-access-b9ln8\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:34:56.542660 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:56.542627 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f3be9f9-cc25-4601-9015-8a323abb7db1" path="/var/lib/kubelet/pods/2f3be9f9-cc25-4601-9015-8a323abb7db1/volumes" Oct 30 01:34:56.543195 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:56.543177 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea7df11c-947f-49ee-9174-28d78fef15ca" path="/var/lib/kubelet/pods/ea7df11c-947f-49ee-9174-28d78fef15ca/volumes" Oct 30 01:34:56.624126 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:56.624096 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:56.812386 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:56.812352 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:57.503823 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:57.503784 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:58.033783 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:58.032789 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:58.080019 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:58.077514 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77309087-rpq8g"] Oct 30 01:34:58.080019 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:58.077576 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77309087-rpq8g"] Oct 30 01:34:58.437360 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:58.437327 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:58.545504 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:58.545461 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="560bdb6b-fe12-4c95-8db1-fdc29792f939" path="/var/lib/kubelet/pods/560bdb6b-fe12-4c95-8db1-fdc29792f939/volumes" Oct 30 01:34:58.625162 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:58.625132 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:58.826827 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:58.825884 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:59.011978 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:59.011906 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:59.199828 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:59.199789 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:59.573914 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:34:59.573702 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:00.028314 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:00.028277 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa3985f0-lbpwz"] Oct 30 01:35:00.037315 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:00.037289 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa3985f0-lbpwz"] Oct 30 01:35:00.545156 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:00.545114 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117" path="/var/lib/kubelet/pods/93ebd1c7-2b98-49ad-8c4a-f0e3f37d3117/volumes" Oct 30 01:35:00.989821 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:00.989782 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:02.299728 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:02.299694 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:02.502444 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:02.502412 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:02.791924 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:02.791890 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:03.276122 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:03.276058 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:03.481014 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:03.480956 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:03.680623 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:03.680591 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:03.833302 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:03.833262 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-f9pk6"] Oct 30 01:35:03.841561 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:03.841535 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-f9pk6"] Oct 30 01:35:03.896391 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:03.896355 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:04.543363 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:04.543334 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5763345-21f8-48b3-880b-d36fd04c9634" path="/var/lib/kubelet/pods/d5763345-21f8-48b3-880b-d36fd04c9634/volumes" Oct 30 01:35:04.590731 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:04.590701 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:04.827884 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:04.825377 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:05.295213 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:05.295178 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-9t2d5" podUID="485b6169-3cf8-4111-be91-8050f32439d7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:35:05.801141 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:05.801102 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:06.071873 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:06.071779 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:06.408308 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:06.408220 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:06.429340 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:06.429309 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-288c8f-6c4f86c884-vhn9v"] Oct 30 01:35:06.430031 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:06.430013 6572 cpu_manager.go:414] "containerMap: removing container" podUID="2f3be9f9-cc25-4601-9015-8a323abb7db1" containerName="payload-processing" Oct 30 01:35:06.430084 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:06.430033 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f3be9f9-cc25-4601-9015-8a323abb7db1" containerName="payload-processing" Oct 30 01:35:06.430084 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:06.430043 6572 cpu_manager.go:414] "containerMap: removing container" podUID="ea7df11c-947f-49ee-9174-28d78fef15ca" containerName="payload-pre-processing" Oct 30 01:35:06.430084 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:06.430048 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea7df11c-947f-49ee-9174-28d78fef15ca" containerName="payload-pre-processing" Oct 30 01:35:06.430084 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:06.430063 6572 cpu_manager.go:414] "containerMap: removing container" podUID="4d14081a-d8ef-49af-8792-26f6956b6141" containerName="revoke-keys" Oct 30 01:35:06.430084 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:06.430068 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d14081a-d8ef-49af-8792-26f6956b6141" containerName="revoke-keys" Oct 30 01:35:06.430239 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:06.430202 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f3be9f9-cc25-4601-9015-8a323abb7db1" containerName="payload-processing" Oct 30 01:35:06.430239 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:06.430209 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea7df11c-947f-49ee-9174-28d78fef15ca" containerName="payload-pre-processing" Oct 30 01:35:06.430239 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:06.430216 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d14081a-d8ef-49af-8792-26f6956b6141" containerName="revoke-keys" Oct 30 01:35:06.436256 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:06.436232 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-288c8f-6c4f86c884-vhn9v" Oct 30 01:35:06.456547 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:06.456515 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-288c8f-6c4f86c884-vhn9v"] Oct 30 01:35:06.460026 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:06.460002 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-288c8f-dockercfg-7d4c5\"" Oct 30 01:35:06.507976 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:06.507946 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlpr7\" (UniqueName: \"kubernetes.io/projected/14e7706d-95c7-465c-a80d-9041790b9d9d-kube-api-access-xlpr7\") pod \"payload-processing-e2e-api-a-288c8f-6c4f86c884-vhn9v\" (UID: \"14e7706d-95c7-465c-a80d-9041790b9d9d\") " pod="openshift-ingress/payload-processing-e2e-api-a-288c8f-6c4f86c884-vhn9v" Oct 30 01:35:06.508135 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:06.508077 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e7706d-95c7-465c-a80d-9041790b9d9d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-288c8f-6c4f86c884-vhn9v\" (UID: \"14e7706d-95c7-465c-a80d-9041790b9d9d\") " pod="openshift-ingress/payload-processing-e2e-api-a-288c8f-6c4f86c884-vhn9v" Oct 30 01:35:06.534494 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:06.534459 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-288c8f\"" Oct 30 01:35:06.620773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:06.609364 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlpr7\" (UniqueName: \"kubernetes.io/projected/14e7706d-95c7-465c-a80d-9041790b9d9d-kube-api-access-xlpr7\") pod \"payload-processing-e2e-api-a-288c8f-6c4f86c884-vhn9v\" (UID: \"14e7706d-95c7-465c-a80d-9041790b9d9d\") " pod="openshift-ingress/payload-processing-e2e-api-a-288c8f-6c4f86c884-vhn9v" Oct 30 01:35:06.620773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:06.609541 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e7706d-95c7-465c-a80d-9041790b9d9d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-288c8f-6c4f86c884-vhn9v\" (UID: \"14e7706d-95c7-465c-a80d-9041790b9d9d\") " pod="openshift-ingress/payload-processing-e2e-api-a-288c8f-6c4f86c884-vhn9v" Oct 30 01:35:06.620773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:06.610619 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e7706d-95c7-465c-a80d-9041790b9d9d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-288c8f-6c4f86c884-vhn9v\" (UID: \"14e7706d-95c7-465c-a80d-9041790b9d9d\") " pod="openshift-ingress/payload-processing-e2e-api-a-288c8f-6c4f86c884-vhn9v" Oct 30 01:35:06.627773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:06.624287 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:06.646813 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:06.645824 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlpr7\" (UniqueName: \"kubernetes.io/projected/14e7706d-95c7-465c-a80d-9041790b9d9d-kube-api-access-xlpr7\") pod \"payload-processing-e2e-api-a-288c8f-6c4f86c884-vhn9v\" (UID: \"14e7706d-95c7-465c-a80d-9041790b9d9d\") " pod="openshift-ingress/payload-processing-e2e-api-a-288c8f-6c4f86c884-vhn9v" Oct 30 01:35:06.756659 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:06.756624 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-288c8f-6c4f86c884-vhn9v" Oct 30 01:35:07.029601 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:07.029572 6572 generic.go:356] "Generic (PLEG): container finished" podID="485b6169-3cf8-4111-be91-8050f32439d7" containerID="04eb97acfe1994c72454bfaf1078733f44e22f8338a8c0786ca58417d21fce3b" exitCode=0 Oct 30 01:35:07.029956 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:07.029646 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-9t2d5" event={"ID":"485b6169-3cf8-4111-be91-8050f32439d7","Type":"ContainerDied","Data":"04eb97acfe1994c72454bfaf1078733f44e22f8338a8c0786ca58417d21fce3b"} Oct 30 01:35:07.029956 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:07.029738 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-9t2d5" event={"ID":"485b6169-3cf8-4111-be91-8050f32439d7","Type":"ContainerDied","Data":"88e5fc10db768257ec6d64b468eb13182e9d15eebc735308cb33c10bd193cc59"} Oct 30 01:35:07.029956 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:07.029774 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88e5fc10db768257ec6d64b468eb13182e9d15eebc735308cb33c10bd193cc59" Oct 30 01:35:07.039984 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:07.039962 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-9t2d5" Oct 30 01:35:07.068901 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:35:07.068876 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e7706d_95c7_465c_a80d_9041790b9d9d.slice/crio-a5a6c1a3b17bd2b8b7cc035f78bd7c6ce30d3f103bc99382c453599d00342bd2 WatchSource:0}: Error finding container a5a6c1a3b17bd2b8b7cc035f78bd7c6ce30d3f103bc99382c453599d00342bd2: Status 404 returned error can't find the container with id a5a6c1a3b17bd2b8b7cc035f78bd7c6ce30d3f103bc99382c453599d00342bd2 Oct 30 01:35:07.069236 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:07.069217 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-288c8f-6c4f86c884-vhn9v"] Oct 30 01:35:07.118924 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:07.118894 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/485b6169-3cf8-4111-be91-8050f32439d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/485b6169-3cf8-4111-be91-8050f32439d7-plugins-config-volume\") pod \"485b6169-3cf8-4111-be91-8050f32439d7\" (UID: \"485b6169-3cf8-4111-be91-8050f32439d7\") " Oct 30 01:35:07.119107 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:07.119088 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/485b6169-3cf8-4111-be91-8050f32439d7-kube-api-access-7g9nb\" (UniqueName: \"kubernetes.io/projected/485b6169-3cf8-4111-be91-8050f32439d7-kube-api-access-7g9nb\") pod \"485b6169-3cf8-4111-be91-8050f32439d7\" (UID: \"485b6169-3cf8-4111-be91-8050f32439d7\") " Oct 30 01:35:07.119256 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:07.119232 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/485b6169-3cf8-4111-be91-8050f32439d7-plugins-config-volume" pod "485b6169-3cf8-4111-be91-8050f32439d7" (UID: "485b6169-3cf8-4111-be91-8050f32439d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:35:07.119711 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:07.119687 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/485b6169-3cf8-4111-be91-8050f32439d7-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:35:07.121177 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:07.121148 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/485b6169-3cf8-4111-be91-8050f32439d7-kube-api-access-7g9nb" pod "485b6169-3cf8-4111-be91-8050f32439d7" (UID: "485b6169-3cf8-4111-be91-8050f32439d7"). InnerVolumeSpecName "kube-api-access-7g9nb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:35:07.221541 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:07.221504 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g9nb\" (UniqueName: \"kubernetes.io/projected/485b6169-3cf8-4111-be91-8050f32439d7-kube-api-access-7g9nb\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:35:07.492732 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:07.492687 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:08.039991 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:08.039954 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-288c8f-6c4f86c884-vhn9v" event={"ID":"14e7706d-95c7-465c-a80d-9041790b9d9d","Type":"ContainerStarted","Data":"24593ee4bf2445baa5d15aad3c2099ee11398ace9c68fd13489ff90720f23596"} Oct 30 01:35:08.039991 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:08.039979 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-9t2d5" Oct 30 01:35:08.040535 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:08.040005 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-288c8f-6c4f86c884-vhn9v" event={"ID":"14e7706d-95c7-465c-a80d-9041790b9d9d","Type":"ContainerStarted","Data":"a5a6c1a3b17bd2b8b7cc035f78bd7c6ce30d3f103bc99382c453599d00342bd2"} Oct 30 01:35:08.040535 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:08.040073 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-288c8f-6c4f86c884-vhn9v" Oct 30 01:35:08.075369 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:08.075322 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-288c8f-6c4f86c884-vhn9v" podStartSLOduration=2.075309241 podStartE2EDuration="2.075309241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:35:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:08.073676669 +0000 UTC m=+1033.650102070" watchObservedRunningTime="2026-10-30 01:35:08.075309241 +0000 UTC m=+1033.651734644" Oct 30 01:35:08.118476 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:08.118441 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-9t2d5"] Oct 30 01:35:08.123469 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:08.123444 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-9t2d5"] Oct 30 01:35:08.543534 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:08.543497 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="485b6169-3cf8-4111-be91-8050f32439d7" path="/var/lib/kubelet/pods/485b6169-3cf8-4111-be91-8050f32439d7/volumes" Oct 30 01:35:08.756719 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:08.756675 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:09.362321 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:09.362289 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:11.583774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:11.578047 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:12.826770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:12.822954 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-nsfcb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:35:12.826770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:12.823040 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" podUID="c1d4d67f-e18a-4152-be41-e1832bfd78ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:35:13.465357 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:13.465309 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:14.029442 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:14.029397 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-136a9875-pcfzv"] Oct 30 01:35:14.038251 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:14.038228 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-136a9875-pcfzv"] Oct 30 01:35:14.545100 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:14.545072 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ed2e616-52c0-4575-8a58-27c122e24b31" path="/var/lib/kubelet/pods/5ed2e616-52c0-4575-8a58-27c122e24b31/volumes" Oct 30 01:35:14.688370 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:35:14.688331 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:35:14.984319 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:14.984291 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:15.692578 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:15.692523 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:15.915074 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:15.915040 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:16.098595 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:16.098501 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:16.337477 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:16.337441 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:16.418731 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:16.418702 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec2345-6lcns"] Oct 30 01:35:16.429969 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:16.429938 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ec2345-6lcns"] Oct 30 01:35:16.539763 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:16.539716 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:16.546846 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:16.546811 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed" path="/var/lib/kubelet/pods/bc65cdaf-3bd6-494b-8a9d-68ea6373c8ed/volumes" Oct 30 01:35:16.735413 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:16.735329 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:16.861799 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:16.861760 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-bqmkb"] Oct 30 01:35:16.863052 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:16.863025 6572 cpu_manager.go:414] "containerMap: removing container" podUID="485b6169-3cf8-4111-be91-8050f32439d7" containerName="payload-processing" Oct 30 01:35:16.863304 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:16.863285 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="485b6169-3cf8-4111-be91-8050f32439d7" containerName="payload-processing" Oct 30 01:35:16.863790 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:16.863771 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="485b6169-3cf8-4111-be91-8050f32439d7" containerName="payload-processing" Oct 30 01:35:16.918776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:16.885068 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-bqmkb" Oct 30 01:35:16.919152 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:16.916824 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-bqmkb"] Oct 30 01:35:16.954978 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:16.946554 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-278fe7d5-dockercfg-rdtnj\"" Oct 30 01:35:16.954978 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:16.947304 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-278fe7d5\"" Oct 30 01:35:16.954978 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:16.952265 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b78b9533-d059-4206-993f-a98c40696290-plugins-config-volume\") pod \"payload-processing-e2e-mt-278fe7d5-7b956c9596-bqmkb\" (UID: \"b78b9533-d059-4206-993f-a98c40696290\") " pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-bqmkb" Oct 30 01:35:16.954978 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:16.952338 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldxlf\" (UniqueName: \"kubernetes.io/projected/b78b9533-d059-4206-993f-a98c40696290-kube-api-access-ldxlf\") pod \"payload-processing-e2e-mt-278fe7d5-7b956c9596-bqmkb\" (UID: \"b78b9533-d059-4206-993f-a98c40696290\") " pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-bqmkb" Oct 30 01:35:17.046562 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:17.046475 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:17.060815 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:17.055199 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b78b9533-d059-4206-993f-a98c40696290-plugins-config-volume\") pod \"payload-processing-e2e-mt-278fe7d5-7b956c9596-bqmkb\" (UID: \"b78b9533-d059-4206-993f-a98c40696290\") " pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-bqmkb" Oct 30 01:35:17.060815 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:17.055284 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldxlf\" (UniqueName: \"kubernetes.io/projected/b78b9533-d059-4206-993f-a98c40696290-kube-api-access-ldxlf\") pod \"payload-processing-e2e-mt-278fe7d5-7b956c9596-bqmkb\" (UID: \"b78b9533-d059-4206-993f-a98c40696290\") " pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-bqmkb" Oct 30 01:35:17.060815 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:17.057152 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b78b9533-d059-4206-993f-a98c40696290-plugins-config-volume\") pod \"payload-processing-e2e-mt-278fe7d5-7b956c9596-bqmkb\" (UID: \"b78b9533-d059-4206-993f-a98c40696290\") " pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-bqmkb" Oct 30 01:35:17.081799 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:17.081765 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldxlf\" (UniqueName: \"kubernetes.io/projected/b78b9533-d059-4206-993f-a98c40696290-kube-api-access-ldxlf\") pod \"payload-processing-e2e-mt-278fe7d5-7b956c9596-bqmkb\" (UID: \"b78b9533-d059-4206-993f-a98c40696290\") " pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-bqmkb" Oct 30 01:35:17.250886 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:17.250851 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:17.346961 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:17.346872 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-bqmkb" Oct 30 01:35:17.656733 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:17.656694 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-bqmkb"] Oct 30 01:35:17.656901 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:35:17.656834 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb78b9533_d059_4206_993f_a98c40696290.slice/crio-903ea1d8e4f90e01a8863317eab813894b3baa06716ca840a7c4d448d9356d05 WatchSource:0}: Error finding container 903ea1d8e4f90e01a8863317eab813894b3baa06716ca840a7c4d448d9356d05: Status 404 returned error can't find the container with id 903ea1d8e4f90e01a8863317eab813894b3baa06716ca840a7c4d448d9356d05 Oct 30 01:35:18.076564 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:18.076522 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:18.159736 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:18.159693 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-bqmkb" event={"ID":"b78b9533-d059-4206-993f-a98c40696290","Type":"ContainerStarted","Data":"182cd9061833a404a4cdc1fa54f27189b3dbf7e69d9c84662b1e03a80f9b246e"} Oct 30 01:35:18.159970 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:18.159812 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-bqmkb" event={"ID":"b78b9533-d059-4206-993f-a98c40696290","Type":"ContainerStarted","Data":"903ea1d8e4f90e01a8863317eab813894b3baa06716ca840a7c4d448d9356d05"} Oct 30 01:35:18.160660 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:18.160615 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-bqmkb" Oct 30 01:35:18.199777 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:18.199701 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-bqmkb" podStartSLOduration=2.199682467 podStartE2EDuration="2.199682467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:18.194504566 +0000 UTC m=+1043.770929973" watchObservedRunningTime="2026-10-30 01:35:18.199682467 +0000 UTC m=+1043.776107871" Oct 30 01:35:18.260628 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:18.260594 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:18.447648 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:18.447592 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:19.052294 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:19.052258 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-288c8f-6c4f86c884-vhn9v" Oct 30 01:35:20.032252 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:20.032218 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:21.717698 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:21.717656 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:21.904358 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:21.904316 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:23.413672 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:23.413639 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:23.631167 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:23.631132 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:23.657198 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:23.657163 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f"] Oct 30 01:35:23.830425 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:23.830337 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:24.033772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:24.033725 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:24.194510 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:24.194473 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4d38e5-bb6tg"] Oct 30 01:35:24.199477 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:24.199427 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4d38e5-bb6tg" Oct 30 01:35:24.219622 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:24.216976 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" podUID="159d5541-9dbd-4b27-bdb2-019f09868d42" containerName="istio-proxy" containerID="cri-o://7be1a04c78aaf7b5e6957925129d4fe9f77e8426a05980c508e7c4b645df4c81" gracePeriod=30 Oct 30 01:35:24.219622 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:24.218467 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:35:24.219622 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:24.218586 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:35:24.219622 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:24.219125 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4d38e5-bb6tg"] Oct 30 01:35:24.219622 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:24.219244 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:35:24.220999 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:24.220970 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:35:24.322312 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:24.322275 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73e94d8b-541f-437c-bfad-85131aacbf7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4d38e5-bb6tg\" (UID: \"73e94d8b-541f-437c-bfad-85131aacbf7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4d38e5-bb6tg" Oct 30 01:35:24.322481 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:24.322409 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r6rn\" (UniqueName: \"kubernetes.io/projected/73e94d8b-541f-437c-bfad-85131aacbf7b-kube-api-access-7r6rn\") pod \"maas-api-revoke-keys-e2e-api-b-4d38e5-bb6tg\" (UID: \"73e94d8b-541f-437c-bfad-85131aacbf7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4d38e5-bb6tg" Oct 30 01:35:24.322481 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:24.322449 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73e94d8b-541f-437c-bfad-85131aacbf7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4d38e5-bb6tg\" (UID: \"73e94d8b-541f-437c-bfad-85131aacbf7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4d38e5-bb6tg" Oct 30 01:35:24.424797 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:24.424677 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73e94d8b-541f-437c-bfad-85131aacbf7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4d38e5-bb6tg\" (UID: \"73e94d8b-541f-437c-bfad-85131aacbf7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4d38e5-bb6tg" Oct 30 01:35:24.425280 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:24.424833 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r6rn\" (UniqueName: \"kubernetes.io/projected/73e94d8b-541f-437c-bfad-85131aacbf7b-kube-api-access-7r6rn\") pod \"maas-api-revoke-keys-e2e-api-b-4d38e5-bb6tg\" (UID: \"73e94d8b-541f-437c-bfad-85131aacbf7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4d38e5-bb6tg" Oct 30 01:35:24.425280 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:24.424884 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73e94d8b-541f-437c-bfad-85131aacbf7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4d38e5-bb6tg\" (UID: \"73e94d8b-541f-437c-bfad-85131aacbf7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4d38e5-bb6tg" Oct 30 01:35:24.425611 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:24.425569 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73e94d8b-541f-437c-bfad-85131aacbf7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4d38e5-bb6tg\" (UID: \"73e94d8b-541f-437c-bfad-85131aacbf7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4d38e5-bb6tg" Oct 30 01:35:24.428052 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:24.428027 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73e94d8b-541f-437c-bfad-85131aacbf7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4d38e5-bb6tg\" (UID: \"73e94d8b-541f-437c-bfad-85131aacbf7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4d38e5-bb6tg" Oct 30 01:35:24.452604 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:24.452517 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r6rn\" (UniqueName: \"kubernetes.io/projected/73e94d8b-541f-437c-bfad-85131aacbf7b-kube-api-access-7r6rn\") pod \"maas-api-revoke-keys-e2e-api-b-4d38e5-bb6tg\" (UID: \"73e94d8b-541f-437c-bfad-85131aacbf7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4d38e5-bb6tg" Oct 30 01:35:24.526547 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:24.526506 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4d38e5-bb6tg" Oct 30 01:35:24.838814 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:24.838783 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4d38e5-bb6tg"] Oct 30 01:35:24.841103 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:35:24.841047 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e94d8b_541f_437c_bfad_85131aacbf7b.slice/crio-8581b4c9fcbecf60def0df30e18023021138a2639d152b98ed0b6b0b6f3c1c57 WatchSource:0}: Error finding container 8581b4c9fcbecf60def0df30e18023021138a2639d152b98ed0b6b0b6f3c1c57: Status 404 returned error can't find the container with id 8581b4c9fcbecf60def0df30e18023021138a2639d152b98ed0b6b0b6f3c1c57 Oct 30 01:35:25.226282 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:25.226245 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4d38e5-bb6tg" event={"ID":"73e94d8b-541f-437c-bfad-85131aacbf7b","Type":"ContainerStarted","Data":"85e6280a559adae7f53d83ae19e1c621f6a6c9d00ba6192a22b737f332e1cdb3"} Oct 30 01:35:25.226453 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:25.226313 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4d38e5-bb6tg" event={"ID":"73e94d8b-541f-437c-bfad-85131aacbf7b","Type":"ContainerStarted","Data":"8581b4c9fcbecf60def0df30e18023021138a2639d152b98ed0b6b0b6f3c1c57"} Oct 30 01:35:25.261837 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:25.261771 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4d38e5-bb6tg" podStartSLOduration=1.26173221 podStartE2EDuration="1.26173221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:35:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:25.260105663 +0000 UTC m=+1050.836531067" watchObservedRunningTime="2026-10-30 01:35:25.26173221 +0000 UTC m=+1050.838157613" Oct 30 01:35:25.604118 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:25.604087 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:26.237334 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:26.237300 6572 generic.go:356] "Generic (PLEG): container finished" podID="73e94d8b-541f-437c-bfad-85131aacbf7b" containerID="85e6280a559adae7f53d83ae19e1c621f6a6c9d00ba6192a22b737f332e1cdb3" exitCode=0 Oct 30 01:35:26.237518 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:26.237370 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4d38e5-bb6tg" event={"ID":"73e94d8b-541f-437c-bfad-85131aacbf7b","Type":"ContainerDied","Data":"85e6280a559adae7f53d83ae19e1c621f6a6c9d00ba6192a22b737f332e1cdb3"} Oct 30 01:35:27.290834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:27.290735 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:27.448360 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:27.448327 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4d38e5-bb6tg" Oct 30 01:35:27.571645 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:27.571561 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73e94d8b-541f-437c-bfad-85131aacbf7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73e94d8b-541f-437c-bfad-85131aacbf7b-maas-api-openshift-ca\") pod \"73e94d8b-541f-437c-bfad-85131aacbf7b\" (UID: \"73e94d8b-541f-437c-bfad-85131aacbf7b\") " Oct 30 01:35:27.571645 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:27.571593 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73e94d8b-541f-437c-bfad-85131aacbf7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73e94d8b-541f-437c-bfad-85131aacbf7b-maas-api-xks-ca\") pod \"73e94d8b-541f-437c-bfad-85131aacbf7b\" (UID: \"73e94d8b-541f-437c-bfad-85131aacbf7b\") " Oct 30 01:35:27.571924 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:27.571651 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73e94d8b-541f-437c-bfad-85131aacbf7b-kube-api-access-7r6rn\" (UniqueName: \"kubernetes.io/projected/73e94d8b-541f-437c-bfad-85131aacbf7b-kube-api-access-7r6rn\") pod \"73e94d8b-541f-437c-bfad-85131aacbf7b\" (UID: \"73e94d8b-541f-437c-bfad-85131aacbf7b\") " Oct 30 01:35:27.572040 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:27.572015 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73e94d8b-541f-437c-bfad-85131aacbf7b-maas-api-openshift-ca" pod "73e94d8b-541f-437c-bfad-85131aacbf7b" (UID: "73e94d8b-541f-437c-bfad-85131aacbf7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:35:27.572384 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:27.572365 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73e94d8b-541f-437c-bfad-85131aacbf7b-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:35:27.573776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:27.573727 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73e94d8b-541f-437c-bfad-85131aacbf7b-maas-api-xks-ca" pod "73e94d8b-541f-437c-bfad-85131aacbf7b" (UID: "73e94d8b-541f-437c-bfad-85131aacbf7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:35:27.573898 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:27.573880 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e94d8b-541f-437c-bfad-85131aacbf7b-kube-api-access-7r6rn" pod "73e94d8b-541f-437c-bfad-85131aacbf7b" (UID: "73e94d8b-541f-437c-bfad-85131aacbf7b"). InnerVolumeSpecName "kube-api-access-7r6rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:35:27.674435 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:27.674396 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73e94d8b-541f-437c-bfad-85131aacbf7b-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:35:27.674435 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:27.674432 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r6rn\" (UniqueName: \"kubernetes.io/projected/73e94d8b-541f-437c-bfad-85131aacbf7b-kube-api-access-7r6rn\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:35:28.036237 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:28.036206 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c63a0579-j2qk9"] Oct 30 01:35:28.040889 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:28.040859 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c63a0579-j2qk9"] Oct 30 01:35:28.264437 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:28.264404 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4d38e5-bb6tg" Oct 30 01:35:28.264437 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:28.264416 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4d38e5-bb6tg" event={"ID":"73e94d8b-541f-437c-bfad-85131aacbf7b","Type":"ContainerDied","Data":"8581b4c9fcbecf60def0df30e18023021138a2639d152b98ed0b6b0b6f3c1c57"} Oct 30 01:35:28.264645 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:28.264484 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8581b4c9fcbecf60def0df30e18023021138a2639d152b98ed0b6b0b6f3c1c57" Oct 30 01:35:28.542758 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:28.542700 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac7d7752-da8b-4385-9f5b-48f401708d87" path="/var/lib/kubelet/pods/ac7d7752-da8b-4385-9f5b-48f401708d87/volumes" Oct 30 01:35:28.977196 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:28.977162 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:29.171977 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:29.171945 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-bqmkb" Oct 30 01:35:29.506424 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:29.506350 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:35:29.600489 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:29.600446 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/159d5541-9dbd-4b27-bdb2-019f09868d42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/159d5541-9dbd-4b27-bdb2-019f09868d42-workload-socket\") pod \"159d5541-9dbd-4b27-bdb2-019f09868d42\" (UID: \"159d5541-9dbd-4b27-bdb2-019f09868d42\") " Oct 30 01:35:29.601007 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:29.600618 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/159d5541-9dbd-4b27-bdb2-019f09868d42-kube-api-access-7xkf4\" (UniqueName: \"kubernetes.io/projected/159d5541-9dbd-4b27-bdb2-019f09868d42-kube-api-access-7xkf4\") pod \"159d5541-9dbd-4b27-bdb2-019f09868d42\" (UID: \"159d5541-9dbd-4b27-bdb2-019f09868d42\") " Oct 30 01:35:29.601007 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:29.600685 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/159d5541-9dbd-4b27-bdb2-019f09868d42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/159d5541-9dbd-4b27-bdb2-019f09868d42-workload-certs\") pod \"159d5541-9dbd-4b27-bdb2-019f09868d42\" (UID: \"159d5541-9dbd-4b27-bdb2-019f09868d42\") " Oct 30 01:35:29.601007 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:29.600732 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159d5541-9dbd-4b27-bdb2-019f09868d42-workload-socket" pod "159d5541-9dbd-4b27-bdb2-019f09868d42" (UID: "159d5541-9dbd-4b27-bdb2-019f09868d42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:35:29.601007 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:29.600793 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/159d5541-9dbd-4b27-bdb2-019f09868d42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/159d5541-9dbd-4b27-bdb2-019f09868d42-istio-data\") pod \"159d5541-9dbd-4b27-bdb2-019f09868d42\" (UID: \"159d5541-9dbd-4b27-bdb2-019f09868d42\") " Oct 30 01:35:29.601007 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:29.600946 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159d5541-9dbd-4b27-bdb2-019f09868d42-workload-certs" pod "159d5541-9dbd-4b27-bdb2-019f09868d42" (UID: "159d5541-9dbd-4b27-bdb2-019f09868d42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:35:29.601007 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:29.600964 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/159d5541-9dbd-4b27-bdb2-019f09868d42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/159d5541-9dbd-4b27-bdb2-019f09868d42-istio-envoy\") pod \"159d5541-9dbd-4b27-bdb2-019f09868d42\" (UID: \"159d5541-9dbd-4b27-bdb2-019f09868d42\") " Oct 30 01:35:29.601304 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:29.601021 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/159d5541-9dbd-4b27-bdb2-019f09868d42-istio-token\" (UniqueName: \"kubernetes.io/projected/159d5541-9dbd-4b27-bdb2-019f09868d42-istio-token\") pod \"159d5541-9dbd-4b27-bdb2-019f09868d42\" (UID: \"159d5541-9dbd-4b27-bdb2-019f09868d42\") " Oct 30 01:35:29.601304 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:29.601067 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159d5541-9dbd-4b27-bdb2-019f09868d42-istio-data" pod "159d5541-9dbd-4b27-bdb2-019f09868d42" (UID: "159d5541-9dbd-4b27-bdb2-019f09868d42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:35:29.601304 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:29.601082 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/159d5541-9dbd-4b27-bdb2-019f09868d42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/159d5541-9dbd-4b27-bdb2-019f09868d42-istio-podinfo\") pod \"159d5541-9dbd-4b27-bdb2-019f09868d42\" (UID: \"159d5541-9dbd-4b27-bdb2-019f09868d42\") " Oct 30 01:35:29.601304 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:29.601127 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/159d5541-9dbd-4b27-bdb2-019f09868d42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/159d5541-9dbd-4b27-bdb2-019f09868d42-credential-socket\") pod \"159d5541-9dbd-4b27-bdb2-019f09868d42\" (UID: \"159d5541-9dbd-4b27-bdb2-019f09868d42\") " Oct 30 01:35:29.601304 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:29.601188 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/159d5541-9dbd-4b27-bdb2-019f09868d42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/159d5541-9dbd-4b27-bdb2-019f09868d42-istiod-ca-cert\") pod \"159d5541-9dbd-4b27-bdb2-019f09868d42\" (UID: \"159d5541-9dbd-4b27-bdb2-019f09868d42\") " Oct 30 01:35:29.601909 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:29.601879 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/159d5541-9dbd-4b27-bdb2-019f09868d42-istiod-ca-cert" pod "159d5541-9dbd-4b27-bdb2-019f09868d42" (UID: "159d5541-9dbd-4b27-bdb2-019f09868d42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:35:29.602038 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:29.601964 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/159d5541-9dbd-4b27-bdb2-019f09868d42-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:35:29.602038 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:29.601988 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/159d5541-9dbd-4b27-bdb2-019f09868d42-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:35:29.602038 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:29.602004 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/159d5541-9dbd-4b27-bdb2-019f09868d42-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:35:29.602555 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:29.602528 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159d5541-9dbd-4b27-bdb2-019f09868d42-credential-socket" pod "159d5541-9dbd-4b27-bdb2-019f09868d42" (UID: "159d5541-9dbd-4b27-bdb2-019f09868d42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:35:29.603421 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:29.603395 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/159d5541-9dbd-4b27-bdb2-019f09868d42-istio-token" pod "159d5541-9dbd-4b27-bdb2-019f09868d42" (UID: "159d5541-9dbd-4b27-bdb2-019f09868d42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:35:29.603514 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:29.603430 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/159d5541-9dbd-4b27-bdb2-019f09868d42-kube-api-access-7xkf4" pod "159d5541-9dbd-4b27-bdb2-019f09868d42" (UID: "159d5541-9dbd-4b27-bdb2-019f09868d42"). InnerVolumeSpecName "kube-api-access-7xkf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:35:29.603514 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:29.603471 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159d5541-9dbd-4b27-bdb2-019f09868d42-istio-envoy" pod "159d5541-9dbd-4b27-bdb2-019f09868d42" (UID: "159d5541-9dbd-4b27-bdb2-019f09868d42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:35:29.603716 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:29.603694 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/159d5541-9dbd-4b27-bdb2-019f09868d42-istio-podinfo" pod "159d5541-9dbd-4b27-bdb2-019f09868d42" (UID: "159d5541-9dbd-4b27-bdb2-019f09868d42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:35:29.703456 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:29.703426 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xkf4\" (UniqueName: \"kubernetes.io/projected/159d5541-9dbd-4b27-bdb2-019f09868d42-kube-api-access-7xkf4\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:35:29.703456 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:29.703452 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/159d5541-9dbd-4b27-bdb2-019f09868d42-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:35:29.703456 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:29.703462 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/159d5541-9dbd-4b27-bdb2-019f09868d42-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:35:29.703678 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:29.703471 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/159d5541-9dbd-4b27-bdb2-019f09868d42-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:35:29.703678 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:29.703480 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/159d5541-9dbd-4b27-bdb2-019f09868d42-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:35:29.703678 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:29.703488 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/159d5541-9dbd-4b27-bdb2-019f09868d42-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:35:30.283003 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:30.282969 6572 generic.go:356] "Generic (PLEG): container finished" podID="159d5541-9dbd-4b27-bdb2-019f09868d42" containerID="7be1a04c78aaf7b5e6957925129d4fe9f77e8426a05980c508e7c4b645df4c81" exitCode=0 Oct 30 01:35:30.283215 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:30.283018 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" Oct 30 01:35:30.283215 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:30.283057 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" event={"ID":"159d5541-9dbd-4b27-bdb2-019f09868d42","Type":"ContainerDied","Data":"7be1a04c78aaf7b5e6957925129d4fe9f77e8426a05980c508e7c4b645df4c81"} Oct 30 01:35:30.283215 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:30.283121 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f" event={"ID":"159d5541-9dbd-4b27-bdb2-019f09868d42","Type":"ContainerDied","Data":"119ecc1d72ef7600693186d95140aeb855c1bf3f309b9be910d601bb22ecc8a9"} Oct 30 01:35:30.283215 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:30.283150 6572 scope.go:122] "RemoveContainer" containerID="7be1a04c78aaf7b5e6957925129d4fe9f77e8426a05980c508e7c4b645df4c81" Oct 30 01:35:30.295377 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:30.295356 6572 scope.go:122] "RemoveContainer" containerID="b5dce9d5632391bc915d87a50ab0845ff095c1a1c49bde3b0af1eed1daf81dda" Oct 30 01:35:30.305604 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:30.305585 6572 scope.go:122] "RemoveContainer" containerID="7be1a04c78aaf7b5e6957925129d4fe9f77e8426a05980c508e7c4b645df4c81" Oct 30 01:35:30.305944 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:35:30.305917 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7be1a04c78aaf7b5e6957925129d4fe9f77e8426a05980c508e7c4b645df4c81\": container with ID starting with 7be1a04c78aaf7b5e6957925129d4fe9f77e8426a05980c508e7c4b645df4c81 not found: ID does not exist" containerID="7be1a04c78aaf7b5e6957925129d4fe9f77e8426a05980c508e7c4b645df4c81" Oct 30 01:35:30.306012 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:30.305955 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7be1a04c78aaf7b5e6957925129d4fe9f77e8426a05980c508e7c4b645df4c81"} err="failed to get container status \"7be1a04c78aaf7b5e6957925129d4fe9f77e8426a05980c508e7c4b645df4c81\": rpc error: code = NotFound desc = could not find container \"7be1a04c78aaf7b5e6957925129d4fe9f77e8426a05980c508e7c4b645df4c81\": container with ID starting with 7be1a04c78aaf7b5e6957925129d4fe9f77e8426a05980c508e7c4b645df4c81 not found: ID does not exist" Oct 30 01:35:30.306012 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:30.305974 6572 scope.go:122] "RemoveContainer" containerID="b5dce9d5632391bc915d87a50ab0845ff095c1a1c49bde3b0af1eed1daf81dda" Oct 30 01:35:30.306238 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:35:30.306222 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5dce9d5632391bc915d87a50ab0845ff095c1a1c49bde3b0af1eed1daf81dda\": container with ID starting with b5dce9d5632391bc915d87a50ab0845ff095c1a1c49bde3b0af1eed1daf81dda not found: ID does not exist" containerID="b5dce9d5632391bc915d87a50ab0845ff095c1a1c49bde3b0af1eed1daf81dda" Oct 30 01:35:30.306296 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:30.306241 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5dce9d5632391bc915d87a50ab0845ff095c1a1c49bde3b0af1eed1daf81dda"} err="failed to get container status \"b5dce9d5632391bc915d87a50ab0845ff095c1a1c49bde3b0af1eed1daf81dda\": rpc error: code = NotFound desc = could not find container \"b5dce9d5632391bc915d87a50ab0845ff095c1a1c49bde3b0af1eed1daf81dda\": container with ID starting with b5dce9d5632391bc915d87a50ab0845ff095c1a1c49bde3b0af1eed1daf81dda not found: ID does not exist" Oct 30 01:35:30.347195 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:30.347164 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f"] Oct 30 01:35:30.356656 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:30.356629 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ec2345-openshift-default-5b77f944fc-d5s4f"] Oct 30 01:35:30.543719 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:30.543647 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="159d5541-9dbd-4b27-bdb2-019f09868d42" path="/var/lib/kubelet/pods/159d5541-9dbd-4b27-bdb2-019f09868d42/volumes" Oct 30 01:35:30.672170 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:30.672132 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:32.378181 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:32.378141 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:34.002320 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:34.002289 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:34.906439 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:34.906406 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:35.129593 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:35.129549 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:36.090319 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:36.090274 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:37.849947 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:37.849910 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:39.562961 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:39.562914 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:41.258340 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:41.258303 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:42.910448 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:42.910387 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:43.103553 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:43.103512 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:43.908163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:43.908115 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:44.193004 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:44.192965 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a06822-9bt6s"] Oct 30 01:35:44.193637 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:44.193616 6572 cpu_manager.go:414] "containerMap: removing container" podUID="159d5541-9dbd-4b27-bdb2-019f09868d42" containerName="istio-proxy" Oct 30 01:35:44.193637 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:44.193633 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="159d5541-9dbd-4b27-bdb2-019f09868d42" containerName="istio-proxy" Oct 30 01:35:44.193637 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:44.193643 6572 cpu_manager.go:414] "containerMap: removing container" podUID="73e94d8b-541f-437c-bfad-85131aacbf7b" containerName="revoke-keys" Oct 30 01:35:44.193899 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:44.193649 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73e94d8b-541f-437c-bfad-85131aacbf7b" containerName="revoke-keys" Oct 30 01:35:44.193899 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:44.193662 6572 cpu_manager.go:414] "containerMap: removing container" podUID="159d5541-9dbd-4b27-bdb2-019f09868d42" containerName="istio-proxy" Oct 30 01:35:44.193899 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:44.193668 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="159d5541-9dbd-4b27-bdb2-019f09868d42" containerName="istio-proxy" Oct 30 01:35:44.193899 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:44.193837 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="159d5541-9dbd-4b27-bdb2-019f09868d42" containerName="istio-proxy" Oct 30 01:35:44.193899 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:44.193846 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="73e94d8b-541f-437c-bfad-85131aacbf7b" containerName="revoke-keys" Oct 30 01:35:44.193899 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:44.193853 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="159d5541-9dbd-4b27-bdb2-019f09868d42" containerName="istio-proxy" Oct 30 01:35:44.200418 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:44.200395 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a06822-9bt6s" Oct 30 01:35:44.213894 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:44.213862 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a06822-9bt6s"] Oct 30 01:35:44.217712 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:44.217675 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:35:44.217712 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:44.217704 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:35:44.217948 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:44.217695 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:35:44.222008 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:44.221982 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:35:44.323456 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:44.323419 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b3ece1b-d5c6-469d-995d-6df9054f3d98-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a06822-9bt6s\" (UID: \"5b3ece1b-d5c6-469d-995d-6df9054f3d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a06822-9bt6s" Oct 30 01:35:44.323639 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:44.323553 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppj8t\" (UniqueName: \"kubernetes.io/projected/5b3ece1b-d5c6-469d-995d-6df9054f3d98-kube-api-access-ppj8t\") pod \"maas-api-revoke-keys-e2e-api-a-a06822-9bt6s\" (UID: \"5b3ece1b-d5c6-469d-995d-6df9054f3d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a06822-9bt6s" Oct 30 01:35:44.323639 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:44.323586 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b3ece1b-d5c6-469d-995d-6df9054f3d98-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a06822-9bt6s\" (UID: \"5b3ece1b-d5c6-469d-995d-6df9054f3d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a06822-9bt6s" Oct 30 01:35:44.425380 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:44.425324 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppj8t\" (UniqueName: \"kubernetes.io/projected/5b3ece1b-d5c6-469d-995d-6df9054f3d98-kube-api-access-ppj8t\") pod \"maas-api-revoke-keys-e2e-api-a-a06822-9bt6s\" (UID: \"5b3ece1b-d5c6-469d-995d-6df9054f3d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a06822-9bt6s" Oct 30 01:35:44.425380 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:44.425384 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b3ece1b-d5c6-469d-995d-6df9054f3d98-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a06822-9bt6s\" (UID: \"5b3ece1b-d5c6-469d-995d-6df9054f3d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a06822-9bt6s" Oct 30 01:35:44.426090 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:44.425579 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b3ece1b-d5c6-469d-995d-6df9054f3d98-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a06822-9bt6s\" (UID: \"5b3ece1b-d5c6-469d-995d-6df9054f3d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a06822-9bt6s" Oct 30 01:35:44.426324 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:44.426298 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b3ece1b-d5c6-469d-995d-6df9054f3d98-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a06822-9bt6s\" (UID: \"5b3ece1b-d5c6-469d-995d-6df9054f3d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a06822-9bt6s" Oct 30 01:35:44.428003 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:44.427976 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b3ece1b-d5c6-469d-995d-6df9054f3d98-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a06822-9bt6s\" (UID: \"5b3ece1b-d5c6-469d-995d-6df9054f3d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a06822-9bt6s" Oct 30 01:35:44.445231 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:44.445128 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppj8t\" (UniqueName: \"kubernetes.io/projected/5b3ece1b-d5c6-469d-995d-6df9054f3d98-kube-api-access-ppj8t\") pod \"maas-api-revoke-keys-e2e-api-a-a06822-9bt6s\" (UID: \"5b3ece1b-d5c6-469d-995d-6df9054f3d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a06822-9bt6s" Oct 30 01:35:44.517922 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:44.517888 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a06822-9bt6s" Oct 30 01:35:44.610475 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:44.610438 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:44.727225 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:35:44.727073 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:35:44.946838 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:35:44.945221 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b3ece1b_d5c6_469d_995d_6df9054f3d98.slice/crio-fb1dad951c0aeae668002cb437ffc63059a1e4f6495059a9768f377e2a4e56ac WatchSource:0}: Error finding container fb1dad951c0aeae668002cb437ffc63059a1e4f6495059a9768f377e2a4e56ac: Status 404 returned error can't find the container with id fb1dad951c0aeae668002cb437ffc63059a1e4f6495059a9768f377e2a4e56ac Oct 30 01:35:44.946838 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:44.946432 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a06822-9bt6s"] Oct 30 01:35:45.089398 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:45.089357 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:45.292935 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:45.292826 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:45.463247 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:45.463207 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a06822-9bt6s" event={"ID":"5b3ece1b-d5c6-469d-995d-6df9054f3d98","Type":"ContainerStarted","Data":"e0c789a00bb3ceddc35b1b1bb1947ccdd094099ec19ae7be6d03afde7d9328f5"} Oct 30 01:35:45.463429 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:45.463270 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a06822-9bt6s" event={"ID":"5b3ece1b-d5c6-469d-995d-6df9054f3d98","Type":"ContainerStarted","Data":"fb1dad951c0aeae668002cb437ffc63059a1e4f6495059a9768f377e2a4e56ac"} Oct 30 01:35:45.501095 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:45.500462 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a06822-9bt6s" podStartSLOduration=1.500439243 podStartE2EDuration="1.500439243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:35:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:45.495805624 +0000 UTC m=+1071.072231028" watchObservedRunningTime="2026-10-30 01:35:45.500439243 +0000 UTC m=+1071.076864646" Oct 30 01:35:45.566099 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:45.566014 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:46.473758 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:46.473724 6572 generic.go:356] "Generic (PLEG): container finished" podID="5b3ece1b-d5c6-469d-995d-6df9054f3d98" containerID="e0c789a00bb3ceddc35b1b1bb1947ccdd094099ec19ae7be6d03afde7d9328f5" exitCode=0 Oct 30 01:35:46.474156 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:46.473801 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a06822-9bt6s" event={"ID":"5b3ece1b-d5c6-469d-995d-6df9054f3d98","Type":"ContainerDied","Data":"e0c789a00bb3ceddc35b1b1bb1947ccdd094099ec19ae7be6d03afde7d9328f5"} Oct 30 01:35:46.982492 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:46.982457 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:47.653988 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:47.653952 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a06822-9bt6s" Oct 30 01:35:47.784658 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:47.784553 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b3ece1b-d5c6-469d-995d-6df9054f3d98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b3ece1b-d5c6-469d-995d-6df9054f3d98-maas-api-xks-ca\") pod \"5b3ece1b-d5c6-469d-995d-6df9054f3d98\" (UID: \"5b3ece1b-d5c6-469d-995d-6df9054f3d98\") " Oct 30 01:35:47.784901 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:47.784796 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b3ece1b-d5c6-469d-995d-6df9054f3d98-kube-api-access-ppj8t\" (UniqueName: \"kubernetes.io/projected/5b3ece1b-d5c6-469d-995d-6df9054f3d98-kube-api-access-ppj8t\") pod \"5b3ece1b-d5c6-469d-995d-6df9054f3d98\" (UID: \"5b3ece1b-d5c6-469d-995d-6df9054f3d98\") " Oct 30 01:35:47.784983 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:47.784920 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b3ece1b-d5c6-469d-995d-6df9054f3d98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b3ece1b-d5c6-469d-995d-6df9054f3d98-maas-api-openshift-ca\") pod \"5b3ece1b-d5c6-469d-995d-6df9054f3d98\" (UID: \"5b3ece1b-d5c6-469d-995d-6df9054f3d98\") " Oct 30 01:35:47.785313 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:47.785288 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b3ece1b-d5c6-469d-995d-6df9054f3d98-maas-api-openshift-ca" pod "5b3ece1b-d5c6-469d-995d-6df9054f3d98" (UID: "5b3ece1b-d5c6-469d-995d-6df9054f3d98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:35:47.785463 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:47.785450 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b3ece1b-d5c6-469d-995d-6df9054f3d98-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:35:47.786895 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:47.786868 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b3ece1b-d5c6-469d-995d-6df9054f3d98-kube-api-access-ppj8t" pod "5b3ece1b-d5c6-469d-995d-6df9054f3d98" (UID: "5b3ece1b-d5c6-469d-995d-6df9054f3d98"). InnerVolumeSpecName "kube-api-access-ppj8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:35:47.786985 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:47.786889 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b3ece1b-d5c6-469d-995d-6df9054f3d98-maas-api-xks-ca" pod "5b3ece1b-d5c6-469d-995d-6df9054f3d98" (UID: "5b3ece1b-d5c6-469d-995d-6df9054f3d98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:35:47.887300 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:47.887257 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppj8t\" (UniqueName: \"kubernetes.io/projected/5b3ece1b-d5c6-469d-995d-6df9054f3d98-kube-api-access-ppj8t\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:35:47.887300 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:47.887296 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b3ece1b-d5c6-469d-995d-6df9054f3d98-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:35:48.495518 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:48.495489 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a06822-9bt6s" Oct 30 01:35:48.495518 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:48.495503 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a06822-9bt6s" event={"ID":"5b3ece1b-d5c6-469d-995d-6df9054f3d98","Type":"ContainerDied","Data":"fb1dad951c0aeae668002cb437ffc63059a1e4f6495059a9768f377e2a4e56ac"} Oct 30 01:35:48.495786 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:48.495549 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb1dad951c0aeae668002cb437ffc63059a1e4f6495059a9768f377e2a4e56ac" Oct 30 01:35:48.698524 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:48.698487 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:48.999186 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:48.999138 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:49.190243 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:49.190203 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:49.378580 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:49.378486 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:49.573033 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:49.573000 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:50.670870 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:50.670833 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:51.211904 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.211716 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:51.225919 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.225881 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-xznhf"] Oct 30 01:35:51.226583 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.226569 6572 cpu_manager.go:414] "containerMap: removing container" podUID="5b3ece1b-d5c6-469d-995d-6df9054f3d98" containerName="revoke-keys" Oct 30 01:35:51.226632 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.226585 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b3ece1b-d5c6-469d-995d-6df9054f3d98" containerName="revoke-keys" Oct 30 01:35:51.226705 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.226694 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b3ece1b-d5c6-469d-995d-6df9054f3d98" containerName="revoke-keys" Oct 30 01:35:51.230741 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.230716 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-xznhf" Oct 30 01:35:51.248984 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.248950 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-xznhf"] Oct 30 01:35:51.252283 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.251982 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:35:51.252283 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.252129 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:35:51.252283 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.252148 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:35:51.252975 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.252951 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:35:51.304223 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.304188 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-666b0b-5c7f78c4d7-dzbzn"] Oct 30 01:35:51.308826 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.308799 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-666b0b-5c7f78c4d7-dzbzn" Oct 30 01:35:51.330036 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.330003 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-666b0b-5c7f78c4d7-dzbzn"] Oct 30 01:35:51.333672 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.333632 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc2qc\" (UniqueName: \"kubernetes.io/projected/4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279-kube-api-access-zc2qc\") pod \"maas-api-revoke-keys-e2e-mt-278fe7d5-xznhf\" (UID: \"4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-xznhf" Oct 30 01:35:51.333895 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.333761 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-278fe7d5-xznhf\" (UID: \"4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-xznhf" Oct 30 01:35:51.333973 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.333906 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-278fe7d5-xznhf\" (UID: \"4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-xznhf" Oct 30 01:35:51.344054 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.344007 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-666b0b-dockercfg-x8cvk\"" Oct 30 01:35:51.344234 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.344016 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-666b0b\"" Oct 30 01:35:51.384550 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.384510 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-666b0b-76df5449dd-qhpm8"] Oct 30 01:35:51.389643 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.389613 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-666b0b-76df5449dd-qhpm8" Oct 30 01:35:51.416506 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.416338 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-666b0b-76df5449dd-qhpm8"] Oct 30 01:35:51.433798 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.429355 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:51.436148 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.436104 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c51d899-6cd9-45f1-8bcb-44588117aa25-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-666b0b-5c7f78c4d7-dzbzn\" (UID: \"7c51d899-6cd9-45f1-8bcb-44588117aa25\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-666b0b-5c7f78c4d7-dzbzn" Oct 30 01:35:51.436323 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.436202 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc2qc\" (UniqueName: \"kubernetes.io/projected/4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279-kube-api-access-zc2qc\") pod \"maas-api-revoke-keys-e2e-mt-278fe7d5-xznhf\" (UID: \"4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-xznhf" Oct 30 01:35:51.436323 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.436266 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-278fe7d5-xznhf\" (UID: \"4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-xznhf" Oct 30 01:35:51.436434 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.436352 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84r8v\" (UniqueName: \"kubernetes.io/projected/7c51d899-6cd9-45f1-8bcb-44588117aa25-kube-api-access-84r8v\") pod \"payload-pre-processing-e2e-api-b-666b0b-5c7f78c4d7-dzbzn\" (UID: \"7c51d899-6cd9-45f1-8bcb-44588117aa25\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-666b0b-5c7f78c4d7-dzbzn" Oct 30 01:35:51.436434 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.436420 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-278fe7d5-xznhf\" (UID: \"4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-xznhf" Oct 30 01:35:51.438369 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.438339 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-278fe7d5-xznhf\" (UID: \"4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-xznhf" Oct 30 01:35:51.451517 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.447224 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-278fe7d5-xznhf\" (UID: \"4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-xznhf" Oct 30 01:35:51.463874 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.463786 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc2qc\" (UniqueName: \"kubernetes.io/projected/4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279-kube-api-access-zc2qc\") pod \"maas-api-revoke-keys-e2e-mt-278fe7d5-xznhf\" (UID: \"4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-xznhf" Oct 30 01:35:51.541799 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.538718 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c51d899-6cd9-45f1-8bcb-44588117aa25-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-666b0b-5c7f78c4d7-dzbzn\" (UID: \"7c51d899-6cd9-45f1-8bcb-44588117aa25\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-666b0b-5c7f78c4d7-dzbzn" Oct 30 01:35:51.541799 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.538808 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a697892a-1733-4bdb-879c-c9df9482954e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-666b0b-76df5449dd-qhpm8\" (UID: \"a697892a-1733-4bdb-879c-c9df9482954e\") " pod="openshift-ingress/payload-processing-e2e-api-b-666b0b-76df5449dd-qhpm8" Oct 30 01:35:51.541799 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.538892 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84r8v\" (UniqueName: \"kubernetes.io/projected/7c51d899-6cd9-45f1-8bcb-44588117aa25-kube-api-access-84r8v\") pod \"payload-pre-processing-e2e-api-b-666b0b-5c7f78c4d7-dzbzn\" (UID: \"7c51d899-6cd9-45f1-8bcb-44588117aa25\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-666b0b-5c7f78c4d7-dzbzn" Oct 30 01:35:51.541799 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.538964 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v26b8\" (UniqueName: \"kubernetes.io/projected/a697892a-1733-4bdb-879c-c9df9482954e-kube-api-access-v26b8\") pod \"payload-processing-e2e-api-b-666b0b-76df5449dd-qhpm8\" (UID: \"a697892a-1733-4bdb-879c-c9df9482954e\") " pod="openshift-ingress/payload-processing-e2e-api-b-666b0b-76df5449dd-qhpm8" Oct 30 01:35:51.541799 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.539989 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c51d899-6cd9-45f1-8bcb-44588117aa25-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-666b0b-5c7f78c4d7-dzbzn\" (UID: \"7c51d899-6cd9-45f1-8bcb-44588117aa25\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-666b0b-5c7f78c4d7-dzbzn" Oct 30 01:35:51.548903 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.548866 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-xznhf" Oct 30 01:35:51.590519 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.590479 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84r8v\" (UniqueName: \"kubernetes.io/projected/7c51d899-6cd9-45f1-8bcb-44588117aa25-kube-api-access-84r8v\") pod \"payload-pre-processing-e2e-api-b-666b0b-5c7f78c4d7-dzbzn\" (UID: \"7c51d899-6cd9-45f1-8bcb-44588117aa25\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-666b0b-5c7f78c4d7-dzbzn" Oct 30 01:35:51.626081 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.626035 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-666b0b-5c7f78c4d7-dzbzn" Oct 30 01:35:51.640255 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.640211 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a697892a-1733-4bdb-879c-c9df9482954e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-666b0b-76df5449dd-qhpm8\" (UID: \"a697892a-1733-4bdb-879c-c9df9482954e\") " pod="openshift-ingress/payload-processing-e2e-api-b-666b0b-76df5449dd-qhpm8" Oct 30 01:35:51.640456 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.640416 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v26b8\" (UniqueName: \"kubernetes.io/projected/a697892a-1733-4bdb-879c-c9df9482954e-kube-api-access-v26b8\") pod \"payload-processing-e2e-api-b-666b0b-76df5449dd-qhpm8\" (UID: \"a697892a-1733-4bdb-879c-c9df9482954e\") " pod="openshift-ingress/payload-processing-e2e-api-b-666b0b-76df5449dd-qhpm8" Oct 30 01:35:51.647956 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.646734 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a697892a-1733-4bdb-879c-c9df9482954e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-666b0b-76df5449dd-qhpm8\" (UID: \"a697892a-1733-4bdb-879c-c9df9482954e\") " pod="openshift-ingress/payload-processing-e2e-api-b-666b0b-76df5449dd-qhpm8" Oct 30 01:35:51.669797 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.664422 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v26b8\" (UniqueName: \"kubernetes.io/projected/a697892a-1733-4bdb-879c-c9df9482954e-kube-api-access-v26b8\") pod \"payload-processing-e2e-api-b-666b0b-76df5449dd-qhpm8\" (UID: \"a697892a-1733-4bdb-879c-c9df9482954e\") " pod="openshift-ingress/payload-processing-e2e-api-b-666b0b-76df5449dd-qhpm8" Oct 30 01:35:51.742855 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.731021 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-666b0b-76df5449dd-qhpm8" Oct 30 01:35:51.892108 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:51.892068 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-xznhf"] Oct 30 01:35:52.049565 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:52.049535 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-666b0b-5c7f78c4d7-dzbzn"] Oct 30 01:35:52.108854 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:35:52.108813 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c51d899_6cd9_45f1_8bcb_44588117aa25.slice/crio-40c1009cbf4a15ccb8ac57fa967afcb1253e09e25ede0f55c2b7afc33f312bb7 WatchSource:0}: Error finding container 40c1009cbf4a15ccb8ac57fa967afcb1253e09e25ede0f55c2b7afc33f312bb7: Status 404 returned error can't find the container with id 40c1009cbf4a15ccb8ac57fa967afcb1253e09e25ede0f55c2b7afc33f312bb7 Oct 30 01:35:52.142587 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:52.142553 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-666b0b-76df5449dd-qhpm8"] Oct 30 01:35:52.145099 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:35:52.145070 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda697892a_1733_4bdb_879c_c9df9482954e.slice/crio-2a68734bb5f8ee395dc49d9f786b4f11472002600121d9c19a6e6bf34cb3c895 WatchSource:0}: Error finding container 2a68734bb5f8ee395dc49d9f786b4f11472002600121d9c19a6e6bf34cb3c895: Status 404 returned error can't find the container with id 2a68734bb5f8ee395dc49d9f786b4f11472002600121d9c19a6e6bf34cb3c895 Oct 30 01:35:52.506155 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:52.506109 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:52.554948 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:52.554902 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-666b0b-76df5449dd-qhpm8" event={"ID":"a697892a-1733-4bdb-879c-c9df9482954e","Type":"ContainerStarted","Data":"d8455bfcf4939c8865df883a47b22929d06534c922fe5d9868d9b14e02532332"} Oct 30 01:35:52.555126 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:52.554975 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-666b0b-76df5449dd-qhpm8" event={"ID":"a697892a-1733-4bdb-879c-c9df9482954e","Type":"ContainerStarted","Data":"2a68734bb5f8ee395dc49d9f786b4f11472002600121d9c19a6e6bf34cb3c895"} Oct 30 01:35:52.555126 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:52.555008 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-666b0b-76df5449dd-qhpm8" Oct 30 01:35:52.557101 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:52.557073 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-xznhf" event={"ID":"4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279","Type":"ContainerStarted","Data":"5668577a077c96238c98a58812de34e2bf8ca0f0520c77e4c162bda5e68a0581"} Oct 30 01:35:52.557211 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:52.557124 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-xznhf" event={"ID":"4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279","Type":"ContainerStarted","Data":"8ef6b3390ac453f7df9f02a3cba60832ebc683ff924afcebbb6b8b46509b0a3f"} Oct 30 01:35:52.558810 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:52.558783 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-666b0b-5c7f78c4d7-dzbzn" event={"ID":"7c51d899-6cd9-45f1-8bcb-44588117aa25","Type":"ContainerStarted","Data":"3a74c0ce781c92b095c4e871720483d961d07f61934e9b2f6f1c0328809c0b61"} Oct 30 01:35:52.558916 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:52.558832 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-666b0b-5c7f78c4d7-dzbzn" event={"ID":"7c51d899-6cd9-45f1-8bcb-44588117aa25","Type":"ContainerStarted","Data":"40c1009cbf4a15ccb8ac57fa967afcb1253e09e25ede0f55c2b7afc33f312bb7"} Oct 30 01:35:52.558916 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:52.558883 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-666b0b-5c7f78c4d7-dzbzn" Oct 30 01:35:52.597301 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:52.597183 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-666b0b-76df5449dd-qhpm8" podStartSLOduration=1.59716612 podStartE2EDuration="1.59716612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:35:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:52.58973006 +0000 UTC m=+1078.166155475" watchObservedRunningTime="2026-10-30 01:35:52.59716612 +0000 UTC m=+1078.173591516" Oct 30 01:35:52.626351 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:52.626171 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-xznhf" podStartSLOduration=1.626150143 podStartE2EDuration="1.626150143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:35:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:52.62326992 +0000 UTC m=+1078.199695322" watchObservedRunningTime="2026-10-30 01:35:52.626150143 +0000 UTC m=+1078.202575547" Oct 30 01:35:52.662199 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:52.662134 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-666b0b-5c7f78c4d7-dzbzn" podStartSLOduration=1.6621169820000001 podStartE2EDuration="1.662116982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:35:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:52.656955581 +0000 UTC m=+1078.233380983" watchObservedRunningTime="2026-10-30 01:35:52.662116982 +0000 UTC m=+1078.238542381" Oct 30 01:35:53.569221 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:53.569182 6572 generic.go:356] "Generic (PLEG): container finished" podID="4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279" containerID="5668577a077c96238c98a58812de34e2bf8ca0f0520c77e4c162bda5e68a0581" exitCode=0 Oct 30 01:35:53.569780 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:53.569282 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-xznhf" event={"ID":"4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279","Type":"ContainerDied","Data":"5668577a077c96238c98a58812de34e2bf8ca0f0520c77e4c162bda5e68a0581"} Oct 30 01:35:54.192364 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:54.192332 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:54.759309 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:54.759275 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-xznhf" Oct 30 01:35:54.812571 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:54.812538 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279-maas-api-openshift-ca\") pod \"4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279\" (UID: \"4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279\") " Oct 30 01:35:54.812797 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:54.812624 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279-maas-api-xks-ca\") pod \"4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279\" (UID: \"4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279\") " Oct 30 01:35:54.812797 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:54.812715 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279-kube-api-access-zc2qc\" (UniqueName: \"kubernetes.io/projected/4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279-kube-api-access-zc2qc\") pod \"4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279\" (UID: \"4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279\") " Oct 30 01:35:54.812995 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:54.812966 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279-maas-api-openshift-ca" pod "4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279" (UID: "4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:35:54.813429 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:54.813385 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:35:54.814970 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:54.814943 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279-maas-api-xks-ca" pod "4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279" (UID: "4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:35:54.815140 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:54.815121 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279-kube-api-access-zc2qc" pod "4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279" (UID: "4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279"). InnerVolumeSpecName "kube-api-access-zc2qc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:35:54.869394 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:54.869300 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:54.881965 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:54.881778 6572 scope.go:122] "RemoveContainer" containerID="e21ae05e3857f20b05057742525deef06bd80555abda7e95b3e99dc094339290" Oct 30 01:35:54.898909 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:54.898884 6572 scope.go:122] "RemoveContainer" containerID="6647e5c506db41b246cde5d447e565d9e42cc82734252386e4e9defc3658bbec" Oct 30 01:35:54.915507 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:54.915473 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:35:54.915672 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:54.915511 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc2qc\" (UniqueName: \"kubernetes.io/projected/4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279-kube-api-access-zc2qc\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:35:55.057385 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:55.057349 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:55.587956 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:55.587927 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-xznhf" Oct 30 01:35:55.588118 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:55.587953 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-xznhf" event={"ID":"4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279","Type":"ContainerDied","Data":"8ef6b3390ac453f7df9f02a3cba60832ebc683ff924afcebbb6b8b46509b0a3f"} Oct 30 01:35:55.588118 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:55.588003 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ef6b3390ac453f7df9f02a3cba60832ebc683ff924afcebbb6b8b46509b0a3f" Oct 30 01:35:55.919193 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:55.919162 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:57.606152 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:57.606112 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:59.383826 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:35:59.383771 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:01.080457 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:01.080418 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:01.640421 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:01.640390 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-bqmkb"] Oct 30 01:36:01.868553 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:01.868515 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:02.058662 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:02.058624 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:02.881322 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:02.881291 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:03.573490 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:03.573412 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-666b0b-5c7f78c4d7-dzbzn" Oct 30 01:36:03.573490 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:03.573483 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-666b0b-76df5449dd-qhpm8" Oct 30 01:36:03.973261 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:03.973217 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:04.693718 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:04.693678 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:04.872008 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:04.871973 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:06.204000 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:06.203965 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:06.641092 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:06.640988 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-bqmkb" podUID="b78b9533-d059-4206-993f-a98c40696290" containerName="payload-processing" containerID="cri-o://182cd9061833a404a4cdc1fa54f27189b3dbf7e69d9c84662b1e03a80f9b246e" gracePeriod=25 Oct 30 01:36:06.849949 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:06.849918 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:06.916524 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:06.916490 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4d38e5-bb6tg"] Oct 30 01:36:06.922363 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:06.922332 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4d38e5-bb6tg"] Oct 30 01:36:06.960467 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:06.960439 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-bqmkb" Oct 30 01:36:07.050013 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:07.049977 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:07.119197 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:07.119161 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b78b9533-d059-4206-993f-a98c40696290-kube-api-access-ldxlf\" (UniqueName: \"kubernetes.io/projected/b78b9533-d059-4206-993f-a98c40696290-kube-api-access-ldxlf\") pod \"b78b9533-d059-4206-993f-a98c40696290\" (UID: \"b78b9533-d059-4206-993f-a98c40696290\") " Oct 30 01:36:07.119381 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:07.119370 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b78b9533-d059-4206-993f-a98c40696290-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b78b9533-d059-4206-993f-a98c40696290-plugins-config-volume\") pod \"b78b9533-d059-4206-993f-a98c40696290\" (UID: \"b78b9533-d059-4206-993f-a98c40696290\") " Oct 30 01:36:07.119853 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:07.119818 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b78b9533-d059-4206-993f-a98c40696290-plugins-config-volume" pod "b78b9533-d059-4206-993f-a98c40696290" (UID: "b78b9533-d059-4206-993f-a98c40696290"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:36:07.120016 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:07.119990 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b78b9533-d059-4206-993f-a98c40696290-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:36:07.121542 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:07.121522 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b78b9533-d059-4206-993f-a98c40696290-kube-api-access-ldxlf" pod "b78b9533-d059-4206-993f-a98c40696290" (UID: "b78b9533-d059-4206-993f-a98c40696290"). InnerVolumeSpecName "kube-api-access-ldxlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:36:07.221220 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:07.221141 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldxlf\" (UniqueName: \"kubernetes.io/projected/b78b9533-d059-4206-993f-a98c40696290-kube-api-access-ldxlf\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:36:07.241158 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:07.241107 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:07.508929 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:07.508848 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:07.730232 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:07.730191 6572 generic.go:356] "Generic (PLEG): container finished" podID="b78b9533-d059-4206-993f-a98c40696290" containerID="182cd9061833a404a4cdc1fa54f27189b3dbf7e69d9c84662b1e03a80f9b246e" exitCode=0 Oct 30 01:36:07.730413 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:07.730256 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-bqmkb" Oct 30 01:36:07.730413 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:07.730271 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-bqmkb" event={"ID":"b78b9533-d059-4206-993f-a98c40696290","Type":"ContainerDied","Data":"182cd9061833a404a4cdc1fa54f27189b3dbf7e69d9c84662b1e03a80f9b246e"} Oct 30 01:36:07.730413 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:07.730336 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-bqmkb" event={"ID":"b78b9533-d059-4206-993f-a98c40696290","Type":"ContainerDied","Data":"903ea1d8e4f90e01a8863317eab813894b3baa06716ca840a7c4d448d9356d05"} Oct 30 01:36:07.730413 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:07.730363 6572 scope.go:122] "RemoveContainer" containerID="182cd9061833a404a4cdc1fa54f27189b3dbf7e69d9c84662b1e03a80f9b246e" Oct 30 01:36:07.748486 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:07.748443 6572 scope.go:122] "RemoveContainer" containerID="182cd9061833a404a4cdc1fa54f27189b3dbf7e69d9c84662b1e03a80f9b246e" Oct 30 01:36:07.749043 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:36:07.749015 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"182cd9061833a404a4cdc1fa54f27189b3dbf7e69d9c84662b1e03a80f9b246e\": container with ID starting with 182cd9061833a404a4cdc1fa54f27189b3dbf7e69d9c84662b1e03a80f9b246e not found: ID does not exist" containerID="182cd9061833a404a4cdc1fa54f27189b3dbf7e69d9c84662b1e03a80f9b246e" Oct 30 01:36:07.749163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:07.749050 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"182cd9061833a404a4cdc1fa54f27189b3dbf7e69d9c84662b1e03a80f9b246e"} err="failed to get container status \"182cd9061833a404a4cdc1fa54f27189b3dbf7e69d9c84662b1e03a80f9b246e\": rpc error: code = NotFound desc = could not find container \"182cd9061833a404a4cdc1fa54f27189b3dbf7e69d9c84662b1e03a80f9b246e\": container with ID starting with 182cd9061833a404a4cdc1fa54f27189b3dbf7e69d9c84662b1e03a80f9b246e not found: ID does not exist" Oct 30 01:36:07.777347 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:07.777246 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-bqmkb"] Oct 30 01:36:07.786821 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:07.786781 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-278fe7d5-7b956c9596-bqmkb"] Oct 30 01:36:08.543194 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:08.543157 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73e94d8b-541f-437c-bfad-85131aacbf7b" path="/var/lib/kubelet/pods/73e94d8b-541f-437c-bfad-85131aacbf7b/volumes" Oct 30 01:36:08.543603 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:08.543568 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b78b9533-d059-4206-993f-a98c40696290" path="/var/lib/kubelet/pods/b78b9533-d059-4206-993f-a98c40696290/volumes" Oct 30 01:36:08.892025 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:08.891927 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:10.578472 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:10.578408 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:12.186976 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:12.186920 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:13.505854 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:13.505805 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:13.692564 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:13.692516 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:13.904474 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:13.904376 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:14.238995 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:14.238959 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:14.689131 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:36:14.689079 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:36:15.592875 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:15.592840 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:17.193415 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:17.193377 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:18.638638 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:18.638578 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:19.204646 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:19.204613 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:19.273656 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:19.273617 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-xznhf"] Oct 30 01:36:19.284622 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:19.284582 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-278fe7d5-xznhf"] Oct 30 01:36:19.397577 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:19.397536 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:19.856823 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:19.854090 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:20.543490 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:20.543455 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279" path="/var/lib/kubelet/pods/4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279/volumes" Oct 30 01:36:21.000548 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:21.000503 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:21.568556 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:21.568522 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:21.724874 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:21.724834 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a06822-9bt6s"] Oct 30 01:36:21.731355 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:21.731322 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a06822-9bt6s"] Oct 30 01:36:21.759167 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:21.759125 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:21.953293 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:21.953237 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:22.272715 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:22.272625 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:22.545643 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:22.545561 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b3ece1b-d5c6-469d-995d-6df9054f3d98" path="/var/lib/kubelet/pods/5b3ece1b-d5c6-469d-995d-6df9054f3d98/volumes" Oct 30 01:36:23.623935 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:23.623895 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:24.785359 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:24.785323 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:24.995192 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:24.995141 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:25.382084 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:25.382056 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:25.603643 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:25.603611 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288c8f-8mxtw"] Oct 30 01:36:25.604296 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:25.604280 6572 cpu_manager.go:414] "containerMap: removing container" podUID="b78b9533-d059-4206-993f-a98c40696290" containerName="payload-processing" Oct 30 01:36:25.604345 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:25.604298 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b78b9533-d059-4206-993f-a98c40696290" containerName="payload-processing" Oct 30 01:36:25.604345 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:25.604322 6572 cpu_manager.go:414] "containerMap: removing container" podUID="4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279" containerName="revoke-keys" Oct 30 01:36:25.604345 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:25.604329 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279" containerName="revoke-keys" Oct 30 01:36:25.604459 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:25.604450 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="b78b9533-d059-4206-993f-a98c40696290" containerName="payload-processing" Oct 30 01:36:25.604494 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:25.604464 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f839f8c-87c3-4cd0-8fe9-7fbaf4b12279" containerName="revoke-keys" Oct 30 01:36:25.609798 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:25.609741 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288c8f-8mxtw" Oct 30 01:36:25.624848 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:25.624806 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288c8f-8mxtw"] Oct 30 01:36:25.628461 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:25.628432 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:36:25.628461 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:25.628430 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:36:25.628677 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:25.628526 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:36:25.634418 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:25.634343 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:36:25.646506 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:25.646467 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efbb5209-6cd6-4b1b-b458-737cedc0299e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-288c8f-8mxtw\" (UID: \"efbb5209-6cd6-4b1b-b458-737cedc0299e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288c8f-8mxtw" Oct 30 01:36:25.646687 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:25.646555 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efbb5209-6cd6-4b1b-b458-737cedc0299e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-288c8f-8mxtw\" (UID: \"efbb5209-6cd6-4b1b-b458-737cedc0299e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288c8f-8mxtw" Oct 30 01:36:25.646687 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:25.646663 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqd2k\" (UniqueName: \"kubernetes.io/projected/efbb5209-6cd6-4b1b-b458-737cedc0299e-kube-api-access-tqd2k\") pod \"maas-api-revoke-keys-e2e-api-a-288c8f-8mxtw\" (UID: \"efbb5209-6cd6-4b1b-b458-737cedc0299e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288c8f-8mxtw" Oct 30 01:36:25.747733 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:25.747696 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqd2k\" (UniqueName: \"kubernetes.io/projected/efbb5209-6cd6-4b1b-b458-737cedc0299e-kube-api-access-tqd2k\") pod \"maas-api-revoke-keys-e2e-api-a-288c8f-8mxtw\" (UID: \"efbb5209-6cd6-4b1b-b458-737cedc0299e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288c8f-8mxtw" Oct 30 01:36:25.747939 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:25.747914 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efbb5209-6cd6-4b1b-b458-737cedc0299e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-288c8f-8mxtw\" (UID: \"efbb5209-6cd6-4b1b-b458-737cedc0299e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288c8f-8mxtw" Oct 30 01:36:25.747990 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:25.747960 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efbb5209-6cd6-4b1b-b458-737cedc0299e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-288c8f-8mxtw\" (UID: \"efbb5209-6cd6-4b1b-b458-737cedc0299e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288c8f-8mxtw" Oct 30 01:36:25.748580 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:25.748550 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efbb5209-6cd6-4b1b-b458-737cedc0299e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-288c8f-8mxtw\" (UID: \"efbb5209-6cd6-4b1b-b458-737cedc0299e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288c8f-8mxtw" Oct 30 01:36:25.750414 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:25.750390 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efbb5209-6cd6-4b1b-b458-737cedc0299e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-288c8f-8mxtw\" (UID: \"efbb5209-6cd6-4b1b-b458-737cedc0299e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288c8f-8mxtw" Oct 30 01:36:25.764420 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:25.764389 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqd2k\" (UniqueName: \"kubernetes.io/projected/efbb5209-6cd6-4b1b-b458-737cedc0299e-kube-api-access-tqd2k\") pod \"maas-api-revoke-keys-e2e-api-a-288c8f-8mxtw\" (UID: \"efbb5209-6cd6-4b1b-b458-737cedc0299e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288c8f-8mxtw" Oct 30 01:36:25.925907 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:25.925870 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288c8f-8mxtw" Oct 30 01:36:26.227720 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:36:26.227683 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefbb5209_6cd6_4b1b_b458_737cedc0299e.slice/crio-c94d094e108aa747ab548154f0a613ec3ffb8778ffe80c4cb90123465839c785 WatchSource:0}: Error finding container c94d094e108aa747ab548154f0a613ec3ffb8778ffe80c4cb90123465839c785: Status 404 returned error can't find the container with id c94d094e108aa747ab548154f0a613ec3ffb8778ffe80c4cb90123465839c785 Oct 30 01:36:26.229394 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:26.229372 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288c8f-8mxtw"] Oct 30 01:36:26.949881 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:26.949841 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288c8f-8mxtw" event={"ID":"efbb5209-6cd6-4b1b-b458-737cedc0299e","Type":"ContainerStarted","Data":"9cb8fac76441e5ac79431475acb6101dc64ec9f22432800a78e7ebba6d83bad2"} Oct 30 01:36:26.950301 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:26.949903 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288c8f-8mxtw" event={"ID":"efbb5209-6cd6-4b1b-b458-737cedc0299e","Type":"ContainerStarted","Data":"c94d094e108aa747ab548154f0a613ec3ffb8778ffe80c4cb90123465839c785"} Oct 30 01:36:26.987258 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:26.987191 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288c8f-8mxtw" podStartSLOduration=1.987171117 podStartE2EDuration="1.987171117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:36:26.986569465 +0000 UTC m=+1112.562994873" watchObservedRunningTime="2026-10-30 01:36:26.987171117 +0000 UTC m=+1112.563596519" Oct 30 01:36:27.098718 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:27.098681 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:27.961726 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:27.961690 6572 generic.go:356] "Generic (PLEG): container finished" podID="efbb5209-6cd6-4b1b-b458-737cedc0299e" containerID="9cb8fac76441e5ac79431475acb6101dc64ec9f22432800a78e7ebba6d83bad2" exitCode=0 Oct 30 01:36:27.962171 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:27.961781 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288c8f-8mxtw" event={"ID":"efbb5209-6cd6-4b1b-b458-737cedc0299e","Type":"ContainerDied","Data":"9cb8fac76441e5ac79431475acb6101dc64ec9f22432800a78e7ebba6d83bad2"} Oct 30 01:36:28.327292 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:28.327185 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:28.523703 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:28.523649 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:28.707999 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:28.707959 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:28.893690 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:28.893654 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:29.140233 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:29.140195 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:29.157426 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:29.157397 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288c8f-8mxtw" Oct 30 01:36:29.174343 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:29.174309 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97c137-kxwxx"] Oct 30 01:36:29.175037 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:29.175017 6572 cpu_manager.go:414] "containerMap: removing container" podUID="efbb5209-6cd6-4b1b-b458-737cedc0299e" containerName="revoke-keys" Oct 30 01:36:29.175087 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:29.175038 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efbb5209-6cd6-4b1b-b458-737cedc0299e" containerName="revoke-keys" Oct 30 01:36:29.175148 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:29.175137 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="efbb5209-6cd6-4b1b-b458-737cedc0299e" containerName="revoke-keys" Oct 30 01:36:29.178989 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:29.178971 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97c137-kxwxx" Oct 30 01:36:29.197532 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:29.197501 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97c137-kxwxx"] Oct 30 01:36:29.212865 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:29.210870 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efbb5209-6cd6-4b1b-b458-737cedc0299e-kube-api-access-tqd2k\" (UniqueName: \"kubernetes.io/projected/efbb5209-6cd6-4b1b-b458-737cedc0299e-kube-api-access-tqd2k\") pod \"efbb5209-6cd6-4b1b-b458-737cedc0299e\" (UID: \"efbb5209-6cd6-4b1b-b458-737cedc0299e\") " Oct 30 01:36:29.212865 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:29.210948 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efbb5209-6cd6-4b1b-b458-737cedc0299e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efbb5209-6cd6-4b1b-b458-737cedc0299e-maas-api-openshift-ca\") pod \"efbb5209-6cd6-4b1b-b458-737cedc0299e\" (UID: \"efbb5209-6cd6-4b1b-b458-737cedc0299e\") " Oct 30 01:36:29.212865 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:29.211066 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efbb5209-6cd6-4b1b-b458-737cedc0299e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efbb5209-6cd6-4b1b-b458-737cedc0299e-maas-api-xks-ca\") pod \"efbb5209-6cd6-4b1b-b458-737cedc0299e\" (UID: \"efbb5209-6cd6-4b1b-b458-737cedc0299e\") " Oct 30 01:36:29.214384 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:29.214349 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efbb5209-6cd6-4b1b-b458-737cedc0299e-maas-api-openshift-ca" pod "efbb5209-6cd6-4b1b-b458-737cedc0299e" (UID: "efbb5209-6cd6-4b1b-b458-737cedc0299e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:36:29.225070 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:29.225005 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efbb5209-6cd6-4b1b-b458-737cedc0299e-maas-api-xks-ca" pod "efbb5209-6cd6-4b1b-b458-737cedc0299e" (UID: "efbb5209-6cd6-4b1b-b458-737cedc0299e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:36:29.225197 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:29.225108 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efbb5209-6cd6-4b1b-b458-737cedc0299e-kube-api-access-tqd2k" pod "efbb5209-6cd6-4b1b-b458-737cedc0299e" (UID: "efbb5209-6cd6-4b1b-b458-737cedc0299e"). InnerVolumeSpecName "kube-api-access-tqd2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:36:29.312992 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:29.312952 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8bc6\" (UniqueName: \"kubernetes.io/projected/1c280d5a-8b6c-4361-a1ca-69232d068f08-kube-api-access-n8bc6\") pod \"maas-api-revoke-keys-e2e-api-b-97c137-kxwxx\" (UID: \"1c280d5a-8b6c-4361-a1ca-69232d068f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97c137-kxwxx" Oct 30 01:36:29.313192 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:29.313074 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c280d5a-8b6c-4361-a1ca-69232d068f08-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-97c137-kxwxx\" (UID: \"1c280d5a-8b6c-4361-a1ca-69232d068f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97c137-kxwxx" Oct 30 01:36:29.313192 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:29.313126 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c280d5a-8b6c-4361-a1ca-69232d068f08-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-97c137-kxwxx\" (UID: \"1c280d5a-8b6c-4361-a1ca-69232d068f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97c137-kxwxx" Oct 30 01:36:29.313382 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:29.313362 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqd2k\" (UniqueName: \"kubernetes.io/projected/efbb5209-6cd6-4b1b-b458-737cedc0299e-kube-api-access-tqd2k\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:36:29.313422 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:29.313386 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efbb5209-6cd6-4b1b-b458-737cedc0299e-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:36:29.313422 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:29.313396 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efbb5209-6cd6-4b1b-b458-737cedc0299e-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:36:29.414929 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:29.414888 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8bc6\" (UniqueName: \"kubernetes.io/projected/1c280d5a-8b6c-4361-a1ca-69232d068f08-kube-api-access-n8bc6\") pod \"maas-api-revoke-keys-e2e-api-b-97c137-kxwxx\" (UID: \"1c280d5a-8b6c-4361-a1ca-69232d068f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97c137-kxwxx" Oct 30 01:36:29.415138 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:29.414991 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c280d5a-8b6c-4361-a1ca-69232d068f08-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-97c137-kxwxx\" (UID: \"1c280d5a-8b6c-4361-a1ca-69232d068f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97c137-kxwxx" Oct 30 01:36:29.415138 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:29.415036 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c280d5a-8b6c-4361-a1ca-69232d068f08-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-97c137-kxwxx\" (UID: \"1c280d5a-8b6c-4361-a1ca-69232d068f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97c137-kxwxx" Oct 30 01:36:29.415680 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:29.415660 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c280d5a-8b6c-4361-a1ca-69232d068f08-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-97c137-kxwxx\" (UID: \"1c280d5a-8b6c-4361-a1ca-69232d068f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97c137-kxwxx" Oct 30 01:36:29.417846 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:29.417820 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c280d5a-8b6c-4361-a1ca-69232d068f08-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-97c137-kxwxx\" (UID: \"1c280d5a-8b6c-4361-a1ca-69232d068f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97c137-kxwxx" Oct 30 01:36:29.435552 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:29.435514 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8bc6\" (UniqueName: \"kubernetes.io/projected/1c280d5a-8b6c-4361-a1ca-69232d068f08-kube-api-access-n8bc6\") pod \"maas-api-revoke-keys-e2e-api-b-97c137-kxwxx\" (UID: \"1c280d5a-8b6c-4361-a1ca-69232d068f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97c137-kxwxx" Oct 30 01:36:29.493841 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:29.493732 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97c137-kxwxx" Oct 30 01:36:29.703771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:29.694773 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:29.829097 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:36:29.829057 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c280d5a_8b6c_4361_a1ca_69232d068f08.slice/crio-ae833c6e0d386b80500860b6df8e7f31c1a683d6cd2ab41e3b8b6868ec39ecb1 WatchSource:0}: Error finding container ae833c6e0d386b80500860b6df8e7f31c1a683d6cd2ab41e3b8b6868ec39ecb1: Status 404 returned error can't find the container with id ae833c6e0d386b80500860b6df8e7f31c1a683d6cd2ab41e3b8b6868ec39ecb1 Oct 30 01:36:29.829473 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:29.829453 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97c137-kxwxx"] Oct 30 01:36:29.982016 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:29.981975 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97c137-kxwxx" event={"ID":"1c280d5a-8b6c-4361-a1ca-69232d068f08","Type":"ContainerStarted","Data":"ae833c6e0d386b80500860b6df8e7f31c1a683d6cd2ab41e3b8b6868ec39ecb1"} Oct 30 01:36:29.984408 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:29.984385 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288c8f-8mxtw" Oct 30 01:36:29.984547 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:29.984408 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288c8f-8mxtw" event={"ID":"efbb5209-6cd6-4b1b-b458-737cedc0299e","Type":"ContainerDied","Data":"c94d094e108aa747ab548154f0a613ec3ffb8778ffe80c4cb90123465839c785"} Oct 30 01:36:29.984547 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:29.984447 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c94d094e108aa747ab548154f0a613ec3ffb8778ffe80c4cb90123465839c785" Oct 30 01:36:30.285392 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:30.285356 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:30.995324 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:30.995225 6572 generic.go:356] "Generic (PLEG): container finished" podID="1c280d5a-8b6c-4361-a1ca-69232d068f08" containerID="fa17973dd266f07f6fbe84cee9209f1b8126a6c7dc0a1b173465fa5995d007a6" exitCode=0 Oct 30 01:36:30.995324 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:30.995309 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97c137-kxwxx" event={"ID":"1c280d5a-8b6c-4361-a1ca-69232d068f08","Type":"ContainerDied","Data":"fa17973dd266f07f6fbe84cee9209f1b8126a6c7dc0a1b173465fa5995d007a6"} Oct 30 01:36:31.146970 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:31.146883 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:32.180947 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:32.180917 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97c137-kxwxx" Oct 30 01:36:32.300446 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:32.300367 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c280d5a-8b6c-4361-a1ca-69232d068f08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c280d5a-8b6c-4361-a1ca-69232d068f08-maas-api-openshift-ca\") pod \"1c280d5a-8b6c-4361-a1ca-69232d068f08\" (UID: \"1c280d5a-8b6c-4361-a1ca-69232d068f08\") " Oct 30 01:36:32.300601 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:32.300531 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c280d5a-8b6c-4361-a1ca-69232d068f08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c280d5a-8b6c-4361-a1ca-69232d068f08-maas-api-xks-ca\") pod \"1c280d5a-8b6c-4361-a1ca-69232d068f08\" (UID: \"1c280d5a-8b6c-4361-a1ca-69232d068f08\") " Oct 30 01:36:32.300601 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:32.300556 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c280d5a-8b6c-4361-a1ca-69232d068f08-kube-api-access-n8bc6\" (UniqueName: \"kubernetes.io/projected/1c280d5a-8b6c-4361-a1ca-69232d068f08-kube-api-access-n8bc6\") pod \"1c280d5a-8b6c-4361-a1ca-69232d068f08\" (UID: \"1c280d5a-8b6c-4361-a1ca-69232d068f08\") " Oct 30 01:36:32.300824 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:32.300799 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c280d5a-8b6c-4361-a1ca-69232d068f08-maas-api-openshift-ca" pod "1c280d5a-8b6c-4361-a1ca-69232d068f08" (UID: "1c280d5a-8b6c-4361-a1ca-69232d068f08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:36:32.301174 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:32.301154 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c280d5a-8b6c-4361-a1ca-69232d068f08-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:36:32.302861 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:32.302833 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c280d5a-8b6c-4361-a1ca-69232d068f08-kube-api-access-n8bc6" pod "1c280d5a-8b6c-4361-a1ca-69232d068f08" (UID: "1c280d5a-8b6c-4361-a1ca-69232d068f08"). InnerVolumeSpecName "kube-api-access-n8bc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:36:32.302969 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:32.302866 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c280d5a-8b6c-4361-a1ca-69232d068f08-maas-api-xks-ca" pod "1c280d5a-8b6c-4361-a1ca-69232d068f08" (UID: "1c280d5a-8b6c-4361-a1ca-69232d068f08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:36:32.402470 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:32.402432 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c280d5a-8b6c-4361-a1ca-69232d068f08-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:36:32.402470 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:32.402467 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8bc6\" (UniqueName: \"kubernetes.io/projected/1c280d5a-8b6c-4361-a1ca-69232d068f08-kube-api-access-n8bc6\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:36:32.997404 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:32.997370 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:33.018379 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:33.018339 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97c137-kxwxx" event={"ID":"1c280d5a-8b6c-4361-a1ca-69232d068f08","Type":"ContainerDied","Data":"ae833c6e0d386b80500860b6df8e7f31c1a683d6cd2ab41e3b8b6868ec39ecb1"} Oct 30 01:36:33.018379 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:33.018361 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97c137-kxwxx" Oct 30 01:36:33.018603 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:33.018392 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae833c6e0d386b80500860b6df8e7f31c1a683d6cd2ab41e3b8b6868ec39ecb1" Oct 30 01:36:33.194403 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:33.194366 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:34.663530 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:34.663491 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:34.695783 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:34.695714 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f09934-9d4445b84-wcp9v"] Oct 30 01:36:34.696376 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:34.696362 6572 cpu_manager.go:414] "containerMap: removing container" podUID="1c280d5a-8b6c-4361-a1ca-69232d068f08" containerName="revoke-keys" Oct 30 01:36:34.696423 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:34.696377 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c280d5a-8b6c-4361-a1ca-69232d068f08" containerName="revoke-keys" Oct 30 01:36:34.696522 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:34.696511 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c280d5a-8b6c-4361-a1ca-69232d068f08" containerName="revoke-keys" Oct 30 01:36:34.700573 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:34.700555 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f09934-9d4445b84-wcp9v" Oct 30 01:36:34.717976 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:34.717949 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f09934-9d4445b84-wcp9v"] Oct 30 01:36:34.721864 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:34.721838 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f09934\"" Oct 30 01:36:34.722714 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:34.722684 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f09934-dockercfg-kfxb7\"" Oct 30 01:36:34.862770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:34.860692 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf5fd44-9bd6-48fe-badd-c1643822e48b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f09934-9d4445b84-wcp9v\" (UID: \"7bf5fd44-9bd6-48fe-badd-c1643822e48b\") " pod="openshift-ingress/payload-processing-e2e-api-a-f09934-9d4445b84-wcp9v" Oct 30 01:36:34.862770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:34.860885 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jksbd\" (UniqueName: \"kubernetes.io/projected/7bf5fd44-9bd6-48fe-badd-c1643822e48b-kube-api-access-jksbd\") pod \"payload-processing-e2e-api-a-f09934-9d4445b84-wcp9v\" (UID: \"7bf5fd44-9bd6-48fe-badd-c1643822e48b\") " pod="openshift-ingress/payload-processing-e2e-api-a-f09934-9d4445b84-wcp9v" Oct 30 01:36:34.893495 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:34.881058 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:34.964828 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:34.964127 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jksbd\" (UniqueName: \"kubernetes.io/projected/7bf5fd44-9bd6-48fe-badd-c1643822e48b-kube-api-access-jksbd\") pod \"payload-processing-e2e-api-a-f09934-9d4445b84-wcp9v\" (UID: \"7bf5fd44-9bd6-48fe-badd-c1643822e48b\") " pod="openshift-ingress/payload-processing-e2e-api-a-f09934-9d4445b84-wcp9v" Oct 30 01:36:34.964828 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:34.964386 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf5fd44-9bd6-48fe-badd-c1643822e48b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f09934-9d4445b84-wcp9v\" (UID: \"7bf5fd44-9bd6-48fe-badd-c1643822e48b\") " pod="openshift-ingress/payload-processing-e2e-api-a-f09934-9d4445b84-wcp9v" Oct 30 01:36:34.972813 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:34.965428 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf5fd44-9bd6-48fe-badd-c1643822e48b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f09934-9d4445b84-wcp9v\" (UID: \"7bf5fd44-9bd6-48fe-badd-c1643822e48b\") " pod="openshift-ingress/payload-processing-e2e-api-a-f09934-9d4445b84-wcp9v" Oct 30 01:36:35.001973 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:35.001929 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jksbd\" (UniqueName: \"kubernetes.io/projected/7bf5fd44-9bd6-48fe-badd-c1643822e48b-kube-api-access-jksbd\") pod \"payload-processing-e2e-api-a-f09934-9d4445b84-wcp9v\" (UID: \"7bf5fd44-9bd6-48fe-badd-c1643822e48b\") " pod="openshift-ingress/payload-processing-e2e-api-a-f09934-9d4445b84-wcp9v" Oct 30 01:36:35.016043 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:35.016002 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f09934-9d4445b84-wcp9v" Oct 30 01:36:35.111413 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:35.111259 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:35.359828 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:35.359792 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f09934-9d4445b84-wcp9v"] Oct 30 01:36:35.360582 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:36:35.360553 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf5fd44_9bd6_48fe_badd_c1643822e48b.slice/crio-f6ffa6479a7e501f764fe9d4b0d928e2bc7b2d0dbb3a4b609b52b77171e1bd57 WatchSource:0}: Error finding container f6ffa6479a7e501f764fe9d4b0d928e2bc7b2d0dbb3a4b609b52b77171e1bd57: Status 404 returned error can't find the container with id f6ffa6479a7e501f764fe9d4b0d928e2bc7b2d0dbb3a4b609b52b77171e1bd57 Oct 30 01:36:36.053311 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:36.053274 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f09934-9d4445b84-wcp9v" event={"ID":"7bf5fd44-9bd6-48fe-badd-c1643822e48b","Type":"ContainerStarted","Data":"78a54f05417b52aea05df39aaec43512fe3e61889e3930b82e4d41ad58e1f119"} Oct 30 01:36:36.053311 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:36.053334 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f09934-9d4445b84-wcp9v" event={"ID":"7bf5fd44-9bd6-48fe-badd-c1643822e48b","Type":"ContainerStarted","Data":"f6ffa6479a7e501f764fe9d4b0d928e2bc7b2d0dbb3a4b609b52b77171e1bd57"} Oct 30 01:36:36.053971 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:36.053532 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f09934-9d4445b84-wcp9v" Oct 30 01:36:36.089278 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:36.089214 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f09934-9d4445b84-wcp9v" podStartSLOduration=2.089196947 podStartE2EDuration="2.089196947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:36:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:36:36.086572652 +0000 UTC m=+1121.662998055" watchObservedRunningTime="2026-10-30 01:36:36.089196947 +0000 UTC m=+1121.665622350" Oct 30 01:36:36.251622 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:36.251582 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-288c8f-6c4f86c884-vhn9v"] Oct 30 01:36:36.502306 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:36.502276 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:36.685483 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:36.685443 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:36.881986 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:36.881891 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:38.599545 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:38.599504 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:39.812869 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:39.812836 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:39.993907 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:39.993873 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:40.303662 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:40.297790 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:41.252314 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:41.252271 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-288c8f-6c4f86c884-vhn9v" podUID="14e7706d-95c7-465c-a80d-9041790b9d9d" containerName="payload-processing" containerID="cri-o://24593ee4bf2445baa5d15aad3c2099ee11398ace9c68fd13489ff90720f23596" gracePeriod=25 Oct 30 01:36:41.532378 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:41.532355 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-288c8f-6c4f86c884-vhn9v" Oct 30 01:36:41.674170 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:41.674138 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14e7706d-95c7-465c-a80d-9041790b9d9d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e7706d-95c7-465c-a80d-9041790b9d9d-plugins-config-volume\") pod \"14e7706d-95c7-465c-a80d-9041790b9d9d\" (UID: \"14e7706d-95c7-465c-a80d-9041790b9d9d\") " Oct 30 01:36:41.674432 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:41.674413 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14e7706d-95c7-465c-a80d-9041790b9d9d-kube-api-access-xlpr7\" (UniqueName: \"kubernetes.io/projected/14e7706d-95c7-465c-a80d-9041790b9d9d-kube-api-access-xlpr7\") pod \"14e7706d-95c7-465c-a80d-9041790b9d9d\" (UID: \"14e7706d-95c7-465c-a80d-9041790b9d9d\") " Oct 30 01:36:41.674507 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:41.674491 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14e7706d-95c7-465c-a80d-9041790b9d9d-plugins-config-volume" pod "14e7706d-95c7-465c-a80d-9041790b9d9d" (UID: "14e7706d-95c7-465c-a80d-9041790b9d9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:36:41.674981 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:41.674962 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e7706d-95c7-465c-a80d-9041790b9d9d-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:36:41.676590 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:41.676567 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14e7706d-95c7-465c-a80d-9041790b9d9d-kube-api-access-xlpr7" pod "14e7706d-95c7-465c-a80d-9041790b9d9d" (UID: "14e7706d-95c7-465c-a80d-9041790b9d9d"). InnerVolumeSpecName "kube-api-access-xlpr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:36:41.776700 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:41.776609 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlpr7\" (UniqueName: \"kubernetes.io/projected/14e7706d-95c7-465c-a80d-9041790b9d9d-kube-api-access-xlpr7\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:36:42.017047 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:42.016784 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:42.132382 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:42.132288 6572 generic.go:356] "Generic (PLEG): container finished" podID="14e7706d-95c7-465c-a80d-9041790b9d9d" containerID="24593ee4bf2445baa5d15aad3c2099ee11398ace9c68fd13489ff90720f23596" exitCode=0 Oct 30 01:36:42.132382 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:42.132355 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-288c8f-6c4f86c884-vhn9v" Oct 30 01:36:42.132623 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:42.132381 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-288c8f-6c4f86c884-vhn9v" event={"ID":"14e7706d-95c7-465c-a80d-9041790b9d9d","Type":"ContainerDied","Data":"24593ee4bf2445baa5d15aad3c2099ee11398ace9c68fd13489ff90720f23596"} Oct 30 01:36:42.132623 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:42.132491 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-288c8f-6c4f86c884-vhn9v" event={"ID":"14e7706d-95c7-465c-a80d-9041790b9d9d","Type":"ContainerDied","Data":"a5a6c1a3b17bd2b8b7cc035f78bd7c6ce30d3f103bc99382c453599d00342bd2"} Oct 30 01:36:42.132623 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:42.132523 6572 scope.go:122] "RemoveContainer" containerID="24593ee4bf2445baa5d15aad3c2099ee11398ace9c68fd13489ff90720f23596" Oct 30 01:36:42.144816 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:42.144794 6572 scope.go:122] "RemoveContainer" containerID="24593ee4bf2445baa5d15aad3c2099ee11398ace9c68fd13489ff90720f23596" Oct 30 01:36:42.145163 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:36:42.145143 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24593ee4bf2445baa5d15aad3c2099ee11398ace9c68fd13489ff90720f23596\": container with ID starting with 24593ee4bf2445baa5d15aad3c2099ee11398ace9c68fd13489ff90720f23596 not found: ID does not exist" containerID="24593ee4bf2445baa5d15aad3c2099ee11398ace9c68fd13489ff90720f23596" Oct 30 01:36:42.145242 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:42.145176 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24593ee4bf2445baa5d15aad3c2099ee11398ace9c68fd13489ff90720f23596"} err="failed to get container status \"24593ee4bf2445baa5d15aad3c2099ee11398ace9c68fd13489ff90720f23596\": rpc error: code = NotFound desc = could not find container \"24593ee4bf2445baa5d15aad3c2099ee11398ace9c68fd13489ff90720f23596\": container with ID starting with 24593ee4bf2445baa5d15aad3c2099ee11398ace9c68fd13489ff90720f23596 not found: ID does not exist" Oct 30 01:36:42.179394 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:42.179344 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-288c8f-6c4f86c884-vhn9v"] Oct 30 01:36:42.188593 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:42.188560 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-288c8f-6c4f86c884-vhn9v"] Oct 30 01:36:42.543277 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:42.543241 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14e7706d-95c7-465c-a80d-9041790b9d9d" path="/var/lib/kubelet/pods/14e7706d-95c7-465c-a80d-9041790b9d9d/volumes" Oct 30 01:36:43.696046 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:43.696010 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:44.028298 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:44.028210 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16725ed7-t895t"] Oct 30 01:36:44.037519 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:44.037481 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16725ed7-t895t"] Oct 30 01:36:44.544615 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:44.544586 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7d28ee6-0e8a-45dc-91ca-73c70f96576c" path="/var/lib/kubelet/pods/f7d28ee6-0e8a-45dc-91ca-73c70f96576c/volumes" Oct 30 01:36:44.695347 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:36:44.695308 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:36:45.359109 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:45.359077 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:46.956125 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:46.956070 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:47.093779 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:47.093712 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f09934-9d4445b84-wcp9v" Oct 30 01:36:47.163767 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:47.163712 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:48.699040 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:48.698986 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:50.472817 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:50.472775 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:52.128424 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:52.128129 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:53.852130 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:53.852082 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:55.038455 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:55.038418 6572 scope.go:122] "RemoveContainer" containerID="c8a4817dcbd10d466d116b4d163c58211d0f2dc69266635e88c47dad0183ca31" Oct 30 01:36:55.096103 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:55.096072 6572 scope.go:122] "RemoveContainer" containerID="812b77051b7d7b7b876fa5c00ae709bf8dece87733feaf4b178c442eb56b480b" Oct 30 01:36:55.111105 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:55.110177 6572 scope.go:122] "RemoveContainer" containerID="5ca99ecaf118e30faea18a8dd53d6b7afdb935ef1651a6156d679d0ca990e5ab" Oct 30 01:36:55.507852 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:55.507816 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:57.325773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:57.319306 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:59.077541 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:59.077501 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3eb7deca-jlslv"] Oct 30 01:36:59.087000 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:59.086971 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3eb7deca-jlslv"] Oct 30 01:36:59.179533 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:36:59.179496 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:00.545399 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:00.545364 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e4f9c14-e607-49c0-88ae-804ab307b074" path="/var/lib/kubelet/pods/5e4f9c14-e607-49c0-88ae-804ab307b074/volumes" Oct 30 01:37:01.045079 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:01.045044 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:01.846397 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:01.846358 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:02.302945 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:02.302915 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:03.033730 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:03.033695 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f69974da-gpnw2"] Oct 30 01:37:03.040876 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:03.040843 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f69974da-gpnw2"] Oct 30 01:37:03.049863 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:03.049829 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:04.096277 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:04.096234 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:04.554649 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:04.554608 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70404cb5-1c78-4fdb-9dcb-c06667b44754" path="/var/lib/kubelet/pods/70404cb5-1c78-4fdb-9dcb-c06667b44754/volumes" Oct 30 01:37:04.816294 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:04.816191 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:06.208322 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:06.208274 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:06.788621 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:06.788577 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:06.957476 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:06.957444 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97c137-kxwxx"] Oct 30 01:37:06.966727 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:06.966695 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97c137-kxwxx"] Oct 30 01:37:06.975710 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:06.975678 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:07.032148 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:07.032107 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7b538df-bq9bk"] Oct 30 01:37:07.038479 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:07.038446 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7b538df-bq9bk"] Oct 30 01:37:07.171528 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:07.171493 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:07.513285 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:07.513200 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:08.504886 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:08.504849 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:08.543578 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:08.543539 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c280d5a-8b6c-4361-a1ca-69232d068f08" path="/var/lib/kubelet/pods/1c280d5a-8b6c-4361-a1ca-69232d068f08/volumes" Oct 30 01:37:08.544035 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:08.544019 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3598ded5-aba2-40f9-a3e5-36ad98429c05" path="/var/lib/kubelet/pods/3598ded5-aba2-40f9-a3e5-36ad98429c05/volumes" Oct 30 01:37:08.936519 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:08.936481 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:09.140949 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:09.140916 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:09.264584 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:09.264489 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc"] Oct 30 01:37:09.264836 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:09.264799 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" podUID="7578c814-103e-433a-b58e-a173bc4d3a97" containerName="istio-proxy" containerID="cri-o://e3bf7533c6e01bd409fc6b2ae1254ee1792e70af5f0d874b1e33aa61538a9ad1" gracePeriod=30 Oct 30 01:37:09.353277 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:09.353237 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:09.543743 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:09.543644 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:09.985769 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:09.985716 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:10.467268 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:10.467234 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:10.785727 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:10.785625 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:11.453621 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:11.453579 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:11.638769 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:11.638722 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:11.656940 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:11.656901 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288c8f-8mxtw"] Oct 30 01:37:11.665685 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:11.665648 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-288c8f-8mxtw"] Oct 30 01:37:11.902847 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:11.902732 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:12.028079 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:12.028041 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-833c3c85-9stwn"] Oct 30 01:37:12.036940 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:12.036905 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-833c3c85-9stwn"] Oct 30 01:37:12.174936 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:12.174903 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:12.543180 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:12.543098 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f869d5c-1aca-49c0-ae2e-bdb13e847d5f" path="/var/lib/kubelet/pods/4f869d5c-1aca-49c0-ae2e-bdb13e847d5f/volumes" Oct 30 01:37:12.543616 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:12.543602 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efbb5209-6cd6-4b1b-b458-737cedc0299e" path="/var/lib/kubelet/pods/efbb5209-6cd6-4b1b-b458-737cedc0299e/volumes" Oct 30 01:37:13.053694 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:13.053653 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61b3fc92-2wcqv"] Oct 30 01:37:13.063217 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:13.063180 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61b3fc92-2wcqv"] Oct 30 01:37:13.725494 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:13.725456 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:13.981458 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:13.981369 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:14.188298 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:14.187499 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:14.390211 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:14.390180 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:14.536179 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:14.536096 6572 generic.go:356] "Generic (PLEG): container finished" podID="7578c814-103e-433a-b58e-a173bc4d3a97" containerID="e3bf7533c6e01bd409fc6b2ae1254ee1792e70af5f0d874b1e33aa61538a9ad1" exitCode=0 Oct 30 01:37:14.545173 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:14.545137 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87f831d2-1c73-4a89-a171-b5059272e385" path="/var/lib/kubelet/pods/87f831d2-1c73-4a89-a171-b5059272e385/volumes" Oct 30 01:37:14.545506 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:14.545485 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" event={"ID":"7578c814-103e-433a-b58e-a173bc4d3a97","Type":"ContainerDied","Data":"e3bf7533c6e01bd409fc6b2ae1254ee1792e70af5f0d874b1e33aa61538a9ad1"} Oct 30 01:37:14.545563 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:14.545550 6572 scope.go:122] "RemoveContainer" containerID="e5327fe09cd3167ce229da0ac44ae5f511e694c63e0318f3f7439835e5508e60" Oct 30 01:37:14.572383 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:14.572357 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:37:14.640511 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:14.640474 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7578c814-103e-433a-b58e-a173bc4d3a97-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7578c814-103e-433a-b58e-a173bc4d3a97-workload-certs\") pod \"7578c814-103e-433a-b58e-a173bc4d3a97\" (UID: \"7578c814-103e-433a-b58e-a173bc4d3a97\") " Oct 30 01:37:14.640757 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:14.640524 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7578c814-103e-433a-b58e-a173bc4d3a97-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7578c814-103e-433a-b58e-a173bc4d3a97-istio-data\") pod \"7578c814-103e-433a-b58e-a173bc4d3a97\" (UID: \"7578c814-103e-433a-b58e-a173bc4d3a97\") " Oct 30 01:37:14.640757 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:14.640659 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7578c814-103e-433a-b58e-a173bc4d3a97-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7578c814-103e-433a-b58e-a173bc4d3a97-istio-podinfo\") pod \"7578c814-103e-433a-b58e-a173bc4d3a97\" (UID: \"7578c814-103e-433a-b58e-a173bc4d3a97\") " Oct 30 01:37:14.640757 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:14.640700 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7578c814-103e-433a-b58e-a173bc4d3a97-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7578c814-103e-433a-b58e-a173bc4d3a97-workload-socket\") pod \"7578c814-103e-433a-b58e-a173bc4d3a97\" (UID: \"7578c814-103e-433a-b58e-a173bc4d3a97\") " Oct 30 01:37:14.640950 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:14.640922 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7578c814-103e-433a-b58e-a173bc4d3a97-workload-certs" pod "7578c814-103e-433a-b58e-a173bc4d3a97" (UID: "7578c814-103e-433a-b58e-a173bc4d3a97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:37:14.641045 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:14.641023 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7578c814-103e-433a-b58e-a173bc4d3a97-workload-socket" pod "7578c814-103e-433a-b58e-a173bc4d3a97" (UID: "7578c814-103e-433a-b58e-a173bc4d3a97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:37:14.641112 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:14.641027 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7578c814-103e-433a-b58e-a173bc4d3a97-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7578c814-103e-433a-b58e-a173bc4d3a97-istio-envoy\") pod \"7578c814-103e-433a-b58e-a173bc4d3a97\" (UID: \"7578c814-103e-433a-b58e-a173bc4d3a97\") " Oct 30 01:37:14.641112 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:14.641097 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7578c814-103e-433a-b58e-a173bc4d3a97-kube-api-access-9tbnc\" (UniqueName: \"kubernetes.io/projected/7578c814-103e-433a-b58e-a173bc4d3a97-kube-api-access-9tbnc\") pod \"7578c814-103e-433a-b58e-a173bc4d3a97\" (UID: \"7578c814-103e-433a-b58e-a173bc4d3a97\") " Oct 30 01:37:14.641235 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:14.641136 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7578c814-103e-433a-b58e-a173bc4d3a97-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7578c814-103e-433a-b58e-a173bc4d3a97-istiod-ca-cert\") pod \"7578c814-103e-433a-b58e-a173bc4d3a97\" (UID: \"7578c814-103e-433a-b58e-a173bc4d3a97\") " Oct 30 01:37:14.641235 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:14.641199 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7578c814-103e-433a-b58e-a173bc4d3a97-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7578c814-103e-433a-b58e-a173bc4d3a97-credential-socket\") pod \"7578c814-103e-433a-b58e-a173bc4d3a97\" (UID: \"7578c814-103e-433a-b58e-a173bc4d3a97\") " Oct 30 01:37:14.641390 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:14.641260 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7578c814-103e-433a-b58e-a173bc4d3a97-istio-token\" (UniqueName: \"kubernetes.io/projected/7578c814-103e-433a-b58e-a173bc4d3a97-istio-token\") pod \"7578c814-103e-433a-b58e-a173bc4d3a97\" (UID: \"7578c814-103e-433a-b58e-a173bc4d3a97\") " Oct 30 01:37:14.641951 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:14.641782 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7578c814-103e-433a-b58e-a173bc4d3a97-istio-data" pod "7578c814-103e-433a-b58e-a173bc4d3a97" (UID: "7578c814-103e-433a-b58e-a173bc4d3a97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:37:14.642213 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:14.642191 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7578c814-103e-433a-b58e-a173bc4d3a97-istiod-ca-cert" pod "7578c814-103e-433a-b58e-a173bc4d3a97" (UID: "7578c814-103e-433a-b58e-a173bc4d3a97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:37:14.642288 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:14.642219 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7578c814-103e-433a-b58e-a173bc4d3a97-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:37:14.642288 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:14.642242 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7578c814-103e-433a-b58e-a173bc4d3a97-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:37:14.642288 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:14.642258 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7578c814-103e-433a-b58e-a173bc4d3a97-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:37:14.642656 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:14.642627 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7578c814-103e-433a-b58e-a173bc4d3a97-credential-socket" pod "7578c814-103e-433a-b58e-a173bc4d3a97" (UID: "7578c814-103e-433a-b58e-a173bc4d3a97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:37:14.644599 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:14.644558 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7578c814-103e-433a-b58e-a173bc4d3a97-kube-api-access-9tbnc" pod "7578c814-103e-433a-b58e-a173bc4d3a97" (UID: "7578c814-103e-433a-b58e-a173bc4d3a97"). InnerVolumeSpecName "kube-api-access-9tbnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:37:14.644689 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:14.644646 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7578c814-103e-433a-b58e-a173bc4d3a97-istio-envoy" pod "7578c814-103e-433a-b58e-a173bc4d3a97" (UID: "7578c814-103e-433a-b58e-a173bc4d3a97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:37:14.644973 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:14.644952 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7578c814-103e-433a-b58e-a173bc4d3a97-istio-podinfo" pod "7578c814-103e-433a-b58e-a173bc4d3a97" (UID: "7578c814-103e-433a-b58e-a173bc4d3a97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:37:14.653643 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:14.653603 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7578c814-103e-433a-b58e-a173bc4d3a97-istio-token" pod "7578c814-103e-433a-b58e-a173bc4d3a97" (UID: "7578c814-103e-433a-b58e-a173bc4d3a97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:37:14.695375 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:37:14.695320 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:37:14.743660 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:14.743606 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7578c814-103e-433a-b58e-a173bc4d3a97-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:37:14.743660 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:14.743652 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7578c814-103e-433a-b58e-a173bc4d3a97-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:37:14.743660 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:14.743675 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tbnc\" (UniqueName: \"kubernetes.io/projected/7578c814-103e-433a-b58e-a173bc4d3a97-kube-api-access-9tbnc\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:37:14.744036 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:14.743685 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7578c814-103e-433a-b58e-a173bc4d3a97-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:37:14.744036 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:14.743697 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7578c814-103e-433a-b58e-a173bc4d3a97-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:37:14.744036 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:14.743705 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7578c814-103e-433a-b58e-a173bc4d3a97-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:37:15.546489 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:15.546437 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" event={"ID":"7578c814-103e-433a-b58e-a173bc4d3a97","Type":"ContainerDied","Data":"c1cbc306eacd8ccd9ca8fac254b2099e37685eae9d6a312ab52afd58921f294c"} Oct 30 01:37:15.546995 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:15.546498 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc" Oct 30 01:37:15.546995 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:15.546520 6572 scope.go:122] "RemoveContainer" containerID="e3bf7533c6e01bd409fc6b2ae1254ee1792e70af5f0d874b1e33aa61538a9ad1" Oct 30 01:37:15.629165 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:15.629125 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc"] Oct 30 01:37:15.633544 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:15.633510 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-4zknc"] Oct 30 01:37:15.784183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:15.784143 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:15.949022 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:15.948985 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-18560b-ffb64bd8c-f4c2s"] Oct 30 01:37:15.949735 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:15.949716 6572 cpu_manager.go:414] "containerMap: removing container" podUID="7578c814-103e-433a-b58e-a173bc4d3a97" containerName="istio-proxy" Oct 30 01:37:15.949804 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:15.949736 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7578c814-103e-433a-b58e-a173bc4d3a97" containerName="istio-proxy" Oct 30 01:37:15.949804 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:15.949780 6572 cpu_manager.go:414] "containerMap: removing container" podUID="14e7706d-95c7-465c-a80d-9041790b9d9d" containerName="payload-processing" Oct 30 01:37:15.949804 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:15.949786 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14e7706d-95c7-465c-a80d-9041790b9d9d" containerName="payload-processing" Oct 30 01:37:15.949899 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:15.949804 6572 cpu_manager.go:414] "containerMap: removing container" podUID="7578c814-103e-433a-b58e-a173bc4d3a97" containerName="istio-proxy" Oct 30 01:37:15.949899 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:15.949811 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7578c814-103e-433a-b58e-a173bc4d3a97" containerName="istio-proxy" Oct 30 01:37:15.949962 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:15.949927 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="14e7706d-95c7-465c-a80d-9041790b9d9d" containerName="payload-processing" Oct 30 01:37:15.949962 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:15.949940 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="7578c814-103e-433a-b58e-a173bc4d3a97" containerName="istio-proxy" Oct 30 01:37:15.955144 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:15.955116 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18560b-ffb64bd8c-f4c2s" Oct 30 01:37:15.970666 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:15.970634 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-18560b-ffb64bd8c-f4c2s"] Oct 30 01:37:15.978301 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:15.978269 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-18560b\"" Oct 30 01:37:16.009776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:16.002562 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:16.047617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:16.046080 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-18560b-dockercfg-swvfv\"" Oct 30 01:37:16.134775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:16.129686 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-18560b-ffb64bd8c-f4c2s\" (UID: \"24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-18560b-ffb64bd8c-f4c2s" Oct 30 01:37:16.134775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:16.130890 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fsgs\" (UniqueName: \"kubernetes.io/projected/24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad-kube-api-access-6fsgs\") pod \"payload-pre-processing-e2e-api-b-18560b-ffb64bd8c-f4c2s\" (UID: \"24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-18560b-ffb64bd8c-f4c2s" Oct 30 01:37:16.232860 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:16.232739 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fsgs\" (UniqueName: \"kubernetes.io/projected/24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad-kube-api-access-6fsgs\") pod \"payload-pre-processing-e2e-api-b-18560b-ffb64bd8c-f4c2s\" (UID: \"24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-18560b-ffb64bd8c-f4c2s" Oct 30 01:37:16.233079 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:16.232933 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-18560b-ffb64bd8c-f4c2s\" (UID: \"24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-18560b-ffb64bd8c-f4c2s" Oct 30 01:37:16.233732 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:16.233708 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-18560b-ffb64bd8c-f4c2s\" (UID: \"24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-18560b-ffb64bd8c-f4c2s" Oct 30 01:37:16.251416 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:16.251381 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fsgs\" (UniqueName: \"kubernetes.io/projected/24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad-kube-api-access-6fsgs\") pod \"payload-pre-processing-e2e-api-b-18560b-ffb64bd8c-f4c2s\" (UID: \"24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-18560b-ffb64bd8c-f4c2s" Oct 30 01:37:16.272235 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:16.272202 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18560b-ffb64bd8c-f4c2s" Oct 30 01:37:16.556287 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:16.556198 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7578c814-103e-433a-b58e-a173bc4d3a97" path="/var/lib/kubelet/pods/7578c814-103e-433a-b58e-a173bc4d3a97/volumes" Oct 30 01:37:16.577697 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:16.577662 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-18560b-ffb64bd8c-f4c2s"] Oct 30 01:37:16.578439 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:37:16.578414 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a9c2bd_b4e6_4f34_ae3e_bdc63569b8ad.slice/crio-ffcd02f156ba0284d95ec2a9081ee8b3f4b779515738c42c01ee0d1de22e5d1c WatchSource:0}: Error finding container ffcd02f156ba0284d95ec2a9081ee8b3f4b779515738c42c01ee0d1de22e5d1c: Status 404 returned error can't find the container with id ffcd02f156ba0284d95ec2a9081ee8b3f4b779515738c42c01ee0d1de22e5d1c Oct 30 01:37:17.530821 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:17.529242 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:17.577155 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:17.577111 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18560b-ffb64bd8c-f4c2s" event={"ID":"24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad","Type":"ContainerStarted","Data":"72820a42e18e739e86733d7f830c029e5f970fad23d2fd846c9c7d90bde50d76"} Oct 30 01:37:17.577714 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:17.577214 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18560b-ffb64bd8c-f4c2s" event={"ID":"24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad","Type":"ContainerStarted","Data":"ffcd02f156ba0284d95ec2a9081ee8b3f4b779515738c42c01ee0d1de22e5d1c"} Oct 30 01:37:17.577714 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:17.577519 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18560b-ffb64bd8c-f4c2s" Oct 30 01:37:17.615607 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:17.615536 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18560b-ffb64bd8c-f4c2s" podStartSLOduration=2.615514241 podStartE2EDuration="2.615514241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:37:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:37:17.611206875 +0000 UTC m=+1163.187632277" watchObservedRunningTime="2026-10-30 01:37:17.615514241 +0000 UTC m=+1163.191939644" Oct 30 01:37:18.153280 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:18.153242 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:18.391237 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:18.391201 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:18.595773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:18.595342 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:19.126498 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:19.126458 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-666b0b-pkkws"] Oct 30 01:37:19.127261 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:19.127246 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="7578c814-103e-433a-b58e-a173bc4d3a97" containerName="istio-proxy" Oct 30 01:37:19.130632 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:19.130612 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-666b0b-pkkws" Oct 30 01:37:19.147835 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:19.147801 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-666b0b-pkkws"] Oct 30 01:37:19.152308 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:19.152264 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:37:19.152308 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:19.152288 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:37:19.152509 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:19.152305 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:37:19.152509 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:19.152312 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:37:19.284871 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:19.284825 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6d7aa69-c87d-4f3b-83a9-9afb5940118d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-666b0b-pkkws\" (UID: \"f6d7aa69-c87d-4f3b-83a9-9afb5940118d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-666b0b-pkkws" Oct 30 01:37:19.285132 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:19.285114 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6d7aa69-c87d-4f3b-83a9-9afb5940118d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-666b0b-pkkws\" (UID: \"f6d7aa69-c87d-4f3b-83a9-9afb5940118d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-666b0b-pkkws" Oct 30 01:37:19.285227 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:19.285181 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfcxl\" (UniqueName: \"kubernetes.io/projected/f6d7aa69-c87d-4f3b-83a9-9afb5940118d-kube-api-access-nfcxl\") pod \"maas-api-revoke-keys-e2e-api-b-666b0b-pkkws\" (UID: \"f6d7aa69-c87d-4f3b-83a9-9afb5940118d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-666b0b-pkkws" Oct 30 01:37:19.386549 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:19.386448 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfcxl\" (UniqueName: \"kubernetes.io/projected/f6d7aa69-c87d-4f3b-83a9-9afb5940118d-kube-api-access-nfcxl\") pod \"maas-api-revoke-keys-e2e-api-b-666b0b-pkkws\" (UID: \"f6d7aa69-c87d-4f3b-83a9-9afb5940118d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-666b0b-pkkws" Oct 30 01:37:19.386549 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:19.386525 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6d7aa69-c87d-4f3b-83a9-9afb5940118d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-666b0b-pkkws\" (UID: \"f6d7aa69-c87d-4f3b-83a9-9afb5940118d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-666b0b-pkkws" Oct 30 01:37:19.386835 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:19.386816 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6d7aa69-c87d-4f3b-83a9-9afb5940118d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-666b0b-pkkws\" (UID: \"f6d7aa69-c87d-4f3b-83a9-9afb5940118d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-666b0b-pkkws" Oct 30 01:37:19.387271 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:19.387249 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6d7aa69-c87d-4f3b-83a9-9afb5940118d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-666b0b-pkkws\" (UID: \"f6d7aa69-c87d-4f3b-83a9-9afb5940118d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-666b0b-pkkws" Oct 30 01:37:19.393802 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:19.389858 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6d7aa69-c87d-4f3b-83a9-9afb5940118d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-666b0b-pkkws\" (UID: \"f6d7aa69-c87d-4f3b-83a9-9afb5940118d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-666b0b-pkkws" Oct 30 01:37:19.393802 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:19.393607 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:19.405834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:19.405800 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfcxl\" (UniqueName: \"kubernetes.io/projected/f6d7aa69-c87d-4f3b-83a9-9afb5940118d-kube-api-access-nfcxl\") pod \"maas-api-revoke-keys-e2e-api-b-666b0b-pkkws\" (UID: \"f6d7aa69-c87d-4f3b-83a9-9afb5940118d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-666b0b-pkkws" Oct 30 01:37:19.446199 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:19.446164 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-666b0b-pkkws" Oct 30 01:37:19.759838 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:37:19.759742 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6d7aa69_c87d_4f3b_83a9_9afb5940118d.slice/crio-1154578a89583b7eb1f9bdb02d5ba51a256f6114ef3ee08dc0c88bb088eaca71 WatchSource:0}: Error finding container 1154578a89583b7eb1f9bdb02d5ba51a256f6114ef3ee08dc0c88bb088eaca71: Status 404 returned error can't find the container with id 1154578a89583b7eb1f9bdb02d5ba51a256f6114ef3ee08dc0c88bb088eaca71 Oct 30 01:37:19.760298 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:19.760271 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-666b0b-pkkws"] Oct 30 01:37:20.028621 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:20.028592 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0388b014-p6zjm"] Oct 30 01:37:20.032848 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:20.032821 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0388b014-p6zjm"] Oct 30 01:37:20.544099 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:20.544060 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af93de6e-815c-4103-9437-3a5a45afecaf" path="/var/lib/kubelet/pods/af93de6e-815c-4103-9437-3a5a45afecaf/volumes" Oct 30 01:37:20.641976 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:20.641937 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-666b0b-pkkws" event={"ID":"f6d7aa69-c87d-4f3b-83a9-9afb5940118d","Type":"ContainerStarted","Data":"fc1daa2108128a3dd14abde3c5ba71bd5c9d118e2ce116c823381e8b8c8400f7"} Oct 30 01:37:20.642128 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:20.641994 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-666b0b-pkkws" event={"ID":"f6d7aa69-c87d-4f3b-83a9-9afb5940118d","Type":"ContainerStarted","Data":"1154578a89583b7eb1f9bdb02d5ba51a256f6114ef3ee08dc0c88bb088eaca71"} Oct 30 01:37:20.680326 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:20.680269 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-666b0b-pkkws" podStartSLOduration=1.680252437 podStartE2EDuration="1.680252437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:37:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:37:20.676361072 +0000 UTC m=+1166.252786475" watchObservedRunningTime="2026-10-30 01:37:20.680252437 +0000 UTC m=+1166.256677883" Oct 30 01:37:21.211685 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:21.211646 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:21.653493 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:21.653387 6572 generic.go:356] "Generic (PLEG): container finished" podID="f6d7aa69-c87d-4f3b-83a9-9afb5940118d" containerID="fc1daa2108128a3dd14abde3c5ba71bd5c9d118e2ce116c823381e8b8c8400f7" exitCode=0 Oct 30 01:37:21.653493 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:21.653472 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-666b0b-pkkws" event={"ID":"f6d7aa69-c87d-4f3b-83a9-9afb5940118d","Type":"ContainerDied","Data":"fc1daa2108128a3dd14abde3c5ba71bd5c9d118e2ce116c823381e8b8c8400f7"} Oct 30 01:37:22.832381 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:22.832344 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-666b0b-pkkws" Oct 30 01:37:22.836480 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:22.836445 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6d7aa69-c87d-4f3b-83a9-9afb5940118d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6d7aa69-c87d-4f3b-83a9-9afb5940118d-maas-api-openshift-ca\") pod \"f6d7aa69-c87d-4f3b-83a9-9afb5940118d\" (UID: \"f6d7aa69-c87d-4f3b-83a9-9afb5940118d\") " Oct 30 01:37:22.836684 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:22.836605 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6d7aa69-c87d-4f3b-83a9-9afb5940118d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6d7aa69-c87d-4f3b-83a9-9afb5940118d-maas-api-xks-ca\") pod \"f6d7aa69-c87d-4f3b-83a9-9afb5940118d\" (UID: \"f6d7aa69-c87d-4f3b-83a9-9afb5940118d\") " Oct 30 01:37:22.836684 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:22.836657 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6d7aa69-c87d-4f3b-83a9-9afb5940118d-kube-api-access-nfcxl\" (UniqueName: \"kubernetes.io/projected/f6d7aa69-c87d-4f3b-83a9-9afb5940118d-kube-api-access-nfcxl\") pod \"f6d7aa69-c87d-4f3b-83a9-9afb5940118d\" (UID: \"f6d7aa69-c87d-4f3b-83a9-9afb5940118d\") " Oct 30 01:37:22.836856 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:22.836822 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6d7aa69-c87d-4f3b-83a9-9afb5940118d-maas-api-openshift-ca" pod "f6d7aa69-c87d-4f3b-83a9-9afb5940118d" (UID: "f6d7aa69-c87d-4f3b-83a9-9afb5940118d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:37:22.837268 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:22.837244 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6d7aa69-c87d-4f3b-83a9-9afb5940118d-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:37:22.838832 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:22.838797 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6d7aa69-c87d-4f3b-83a9-9afb5940118d-maas-api-xks-ca" pod "f6d7aa69-c87d-4f3b-83a9-9afb5940118d" (UID: "f6d7aa69-c87d-4f3b-83a9-9afb5940118d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:37:22.839202 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:22.839185 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6d7aa69-c87d-4f3b-83a9-9afb5940118d-kube-api-access-nfcxl" pod "f6d7aa69-c87d-4f3b-83a9-9afb5940118d" (UID: "f6d7aa69-c87d-4f3b-83a9-9afb5940118d"). InnerVolumeSpecName "kube-api-access-nfcxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:37:22.903719 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:22.903657 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:22.938365 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:22.938263 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6d7aa69-c87d-4f3b-83a9-9afb5940118d-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:37:22.938365 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:22.938292 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfcxl\" (UniqueName: \"kubernetes.io/projected/f6d7aa69-c87d-4f3b-83a9-9afb5940118d-kube-api-access-nfcxl\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:37:23.671931 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:23.671896 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-666b0b-pkkws" Oct 30 01:37:23.672128 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:23.671924 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-666b0b-pkkws" event={"ID":"f6d7aa69-c87d-4f3b-83a9-9afb5940118d","Type":"ContainerDied","Data":"1154578a89583b7eb1f9bdb02d5ba51a256f6114ef3ee08dc0c88bb088eaca71"} Oct 30 01:37:23.672128 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:23.671979 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1154578a89583b7eb1f9bdb02d5ba51a256f6114ef3ee08dc0c88bb088eaca71" Oct 30 01:37:24.692985 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:24.692915 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:26.836206 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:26.836171 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:28.037775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:28.037725 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fea92145-wmbl9"] Oct 30 01:37:28.047250 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:28.047214 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fea92145-wmbl9"] Oct 30 01:37:28.543815 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:28.543775 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4" path="/var/lib/kubelet/pods/bbf8dc66-e1f8-41d4-9d9f-a113beaf2bb4/volumes" Oct 30 01:37:28.546202 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:28.546164 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:28.627931 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:28.627900 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18560b-ffb64bd8c-f4c2s" Oct 30 01:37:28.730810 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:28.730772 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:29.026787 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:29.026713 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0324c2b-6gpz8"] Oct 30 01:37:29.036983 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:29.036927 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0324c2b-6gpz8"] Oct 30 01:37:29.706317 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:29.706276 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-666b0b-76df5449dd-qhpm8"] Oct 30 01:37:29.737688 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:29.737640 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-666b0b-5c7f78c4d7-dzbzn"] Oct 30 01:37:29.926286 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:29.926250 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:30.105794 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:30.105641 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:30.285277 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:30.285234 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:30.470905 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:30.470868 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:30.546222 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:30.546183 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0c5e0cf-8408-4a9f-b63f-521614b65232" path="/var/lib/kubelet/pods/d0c5e0cf-8408-4a9f-b63f-521614b65232/volumes" Oct 30 01:37:31.155016 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:31.154538 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:31.370553 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:31.370514 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:31.619290 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:31.619255 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28af3258-qjrpd"] Oct 30 01:37:31.620038 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:31.620019 6572 cpu_manager.go:414] "containerMap: removing container" podUID="f6d7aa69-c87d-4f3b-83a9-9afb5940118d" containerName="revoke-keys" Oct 30 01:37:31.620038 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:31.620037 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6d7aa69-c87d-4f3b-83a9-9afb5940118d" containerName="revoke-keys" Oct 30 01:37:31.620158 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:31.620139 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6d7aa69-c87d-4f3b-83a9-9afb5940118d" containerName="revoke-keys" Oct 30 01:37:31.625250 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:31.625226 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28af3258-qjrpd" Oct 30 01:37:31.641441 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:31.641395 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28af3258-qjrpd"] Oct 30 01:37:31.645183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:31.645142 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:37:31.645183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:31.645176 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:37:31.646810 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:31.646785 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:37:31.646932 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:31.646889 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:37:31.737887 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:31.737848 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/456ee785-1d45-499c-810d-68121a7a2f2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28af3258-qjrpd\" (UID: \"456ee785-1d45-499c-810d-68121a7a2f2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28af3258-qjrpd" Oct 30 01:37:31.738074 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:31.737906 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5q5l\" (UniqueName: \"kubernetes.io/projected/456ee785-1d45-499c-810d-68121a7a2f2d-kube-api-access-d5q5l\") pod \"maas-api-revoke-sub-28af3258-qjrpd\" (UID: \"456ee785-1d45-499c-810d-68121a7a2f2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28af3258-qjrpd" Oct 30 01:37:31.738115 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:31.738065 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/456ee785-1d45-499c-810d-68121a7a2f2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28af3258-qjrpd\" (UID: \"456ee785-1d45-499c-810d-68121a7a2f2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28af3258-qjrpd" Oct 30 01:37:31.841571 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:31.841522 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/456ee785-1d45-499c-810d-68121a7a2f2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28af3258-qjrpd\" (UID: \"456ee785-1d45-499c-810d-68121a7a2f2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28af3258-qjrpd" Oct 30 01:37:31.841806 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:31.841614 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5q5l\" (UniqueName: \"kubernetes.io/projected/456ee785-1d45-499c-810d-68121a7a2f2d-kube-api-access-d5q5l\") pod \"maas-api-revoke-sub-28af3258-qjrpd\" (UID: \"456ee785-1d45-499c-810d-68121a7a2f2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28af3258-qjrpd" Oct 30 01:37:31.841806 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:31.841712 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/456ee785-1d45-499c-810d-68121a7a2f2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28af3258-qjrpd\" (UID: \"456ee785-1d45-499c-810d-68121a7a2f2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28af3258-qjrpd" Oct 30 01:37:31.842836 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:31.842802 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/456ee785-1d45-499c-810d-68121a7a2f2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28af3258-qjrpd\" (UID: \"456ee785-1d45-499c-810d-68121a7a2f2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28af3258-qjrpd" Oct 30 01:37:31.845921 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:31.845891 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/456ee785-1d45-499c-810d-68121a7a2f2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28af3258-qjrpd\" (UID: \"456ee785-1d45-499c-810d-68121a7a2f2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28af3258-qjrpd" Oct 30 01:37:31.862418 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:31.862380 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5q5l\" (UniqueName: \"kubernetes.io/projected/456ee785-1d45-499c-810d-68121a7a2f2d-kube-api-access-d5q5l\") pod \"maas-api-revoke-sub-28af3258-qjrpd\" (UID: \"456ee785-1d45-499c-810d-68121a7a2f2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28af3258-qjrpd" Oct 30 01:37:31.943079 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:31.943031 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28af3258-qjrpd" Oct 30 01:37:32.091663 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:32.091073 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:32.251881 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:32.251844 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28af3258-qjrpd"] Oct 30 01:37:32.773005 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:32.772909 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28af3258-qjrpd" event={"ID":"456ee785-1d45-499c-810d-68121a7a2f2d","Type":"ContainerStarted","Data":"4463e3cde0d8502569c31d9039b3efa194350696c70e7e19f27e1e4ac07b690c"} Oct 30 01:37:32.773005 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:32.772965 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28af3258-qjrpd" event={"ID":"456ee785-1d45-499c-810d-68121a7a2f2d","Type":"ContainerStarted","Data":"5d74c3410531bdbbeb473db306bf263e3c35023c5e593a9300cfb6efdd0b2f03"} Oct 30 01:37:32.809464 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:32.809396 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28af3258-qjrpd" podStartSLOduration=1.809378334 podStartE2EDuration="1.809378334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:37:32.806426612 +0000 UTC m=+1178.382852022" watchObservedRunningTime="2026-10-30 01:37:32.809378334 +0000 UTC m=+1178.385803738" Oct 30 01:37:33.618475 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:33.618396 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:33.783303 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:33.783266 6572 generic.go:356] "Generic (PLEG): container finished" podID="456ee785-1d45-499c-810d-68121a7a2f2d" containerID="4463e3cde0d8502569c31d9039b3efa194350696c70e7e19f27e1e4ac07b690c" exitCode=0 Oct 30 01:37:33.783501 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:33.783359 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28af3258-qjrpd" event={"ID":"456ee785-1d45-499c-810d-68121a7a2f2d","Type":"ContainerDied","Data":"4463e3cde0d8502569c31d9039b3efa194350696c70e7e19f27e1e4ac07b690c"} Oct 30 01:37:33.831928 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:33.831889 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:34.108486 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:34.108405 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:34.307805 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:34.307763 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:34.707576 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:34.707526 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-666b0b-76df5449dd-qhpm8" podUID="a697892a-1733-4bdb-879c-c9df9482954e" containerName="payload-processing" containerID="cri-o://d8455bfcf4939c8865df883a47b22929d06534c922fe5d9868d9b14e02532332" gracePeriod=25 Oct 30 01:37:34.738660 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:34.738617 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-666b0b-5c7f78c4d7-dzbzn" podUID="7c51d899-6cd9-45f1-8bcb-44588117aa25" containerName="payload-pre-processing" containerID="cri-o://3a74c0ce781c92b095c4e871720483d961d07f61934e9b2f6f1c0328809c0b61" gracePeriod=25 Oct 30 01:37:34.976515 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:34.976432 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28af3258-qjrpd" Oct 30 01:37:34.995029 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:34.994571 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/456ee785-1d45-499c-810d-68121a7a2f2d-kube-api-access-d5q5l\" (UniqueName: \"kubernetes.io/projected/456ee785-1d45-499c-810d-68121a7a2f2d-kube-api-access-d5q5l\") pod \"456ee785-1d45-499c-810d-68121a7a2f2d\" (UID: \"456ee785-1d45-499c-810d-68121a7a2f2d\") " Oct 30 01:37:34.995508 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:34.995484 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/456ee785-1d45-499c-810d-68121a7a2f2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/456ee785-1d45-499c-810d-68121a7a2f2d-maas-api-xks-ca\") pod \"456ee785-1d45-499c-810d-68121a7a2f2d\" (UID: \"456ee785-1d45-499c-810d-68121a7a2f2d\") " Oct 30 01:37:34.995927 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:34.995887 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/456ee785-1d45-499c-810d-68121a7a2f2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/456ee785-1d45-499c-810d-68121a7a2f2d-maas-api-openshift-ca\") pod \"456ee785-1d45-499c-810d-68121a7a2f2d\" (UID: \"456ee785-1d45-499c-810d-68121a7a2f2d\") " Oct 30 01:37:34.996384 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:34.996361 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/456ee785-1d45-499c-810d-68121a7a2f2d-maas-api-openshift-ca" pod "456ee785-1d45-499c-810d-68121a7a2f2d" (UID: "456ee785-1d45-499c-810d-68121a7a2f2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:37:34.997123 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:34.997102 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/456ee785-1d45-499c-810d-68121a7a2f2d-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:37:34.998897 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:34.998868 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/456ee785-1d45-499c-810d-68121a7a2f2d-kube-api-access-d5q5l" pod "456ee785-1d45-499c-810d-68121a7a2f2d" (UID: "456ee785-1d45-499c-810d-68121a7a2f2d"). InnerVolumeSpecName "kube-api-access-d5q5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:37:34.999787 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:34.999734 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/456ee785-1d45-499c-810d-68121a7a2f2d-maas-api-xks-ca" pod "456ee785-1d45-499c-810d-68121a7a2f2d" (UID: "456ee785-1d45-499c-810d-68121a7a2f2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:37:35.039081 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.039051 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-666b0b-76df5449dd-qhpm8" Oct 30 01:37:35.098109 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.098080 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a697892a-1733-4bdb-879c-c9df9482954e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a697892a-1733-4bdb-879c-c9df9482954e-plugins-config-volume\") pod \"a697892a-1733-4bdb-879c-c9df9482954e\" (UID: \"a697892a-1733-4bdb-879c-c9df9482954e\") " Oct 30 01:37:35.098423 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.098397 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a697892a-1733-4bdb-879c-c9df9482954e-kube-api-access-v26b8\" (UniqueName: \"kubernetes.io/projected/a697892a-1733-4bdb-879c-c9df9482954e-kube-api-access-v26b8\") pod \"a697892a-1733-4bdb-879c-c9df9482954e\" (UID: \"a697892a-1733-4bdb-879c-c9df9482954e\") " Oct 30 01:37:35.098557 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.098520 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a697892a-1733-4bdb-879c-c9df9482954e-plugins-config-volume" pod "a697892a-1733-4bdb-879c-c9df9482954e" (UID: "a697892a-1733-4bdb-879c-c9df9482954e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:37:35.099017 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.098991 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a697892a-1733-4bdb-879c-c9df9482954e-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:37:35.099122 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.099022 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/456ee785-1d45-499c-810d-68121a7a2f2d-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:37:35.099122 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.099037 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5q5l\" (UniqueName: \"kubernetes.io/projected/456ee785-1d45-499c-810d-68121a7a2f2d-kube-api-access-d5q5l\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:37:35.101031 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.100997 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a697892a-1733-4bdb-879c-c9df9482954e-kube-api-access-v26b8" pod "a697892a-1733-4bdb-879c-c9df9482954e" (UID: "a697892a-1733-4bdb-879c-c9df9482954e"). InnerVolumeSpecName "kube-api-access-v26b8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:37:35.107001 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.106948 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-666b0b-5c7f78c4d7-dzbzn" Oct 30 01:37:35.199877 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.199835 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c51d899-6cd9-45f1-8bcb-44588117aa25-kube-api-access-84r8v\" (UniqueName: \"kubernetes.io/projected/7c51d899-6cd9-45f1-8bcb-44588117aa25-kube-api-access-84r8v\") pod \"7c51d899-6cd9-45f1-8bcb-44588117aa25\" (UID: \"7c51d899-6cd9-45f1-8bcb-44588117aa25\") " Oct 30 01:37:35.200059 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.200019 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c51d899-6cd9-45f1-8bcb-44588117aa25-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c51d899-6cd9-45f1-8bcb-44588117aa25-plugins-config-volume\") pod \"7c51d899-6cd9-45f1-8bcb-44588117aa25\" (UID: \"7c51d899-6cd9-45f1-8bcb-44588117aa25\") " Oct 30 01:37:35.200346 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.200312 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c51d899-6cd9-45f1-8bcb-44588117aa25-plugins-config-volume" pod "7c51d899-6cd9-45f1-8bcb-44588117aa25" (UID: "7c51d899-6cd9-45f1-8bcb-44588117aa25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:37:35.200517 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.200500 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c51d899-6cd9-45f1-8bcb-44588117aa25-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:37:35.200559 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.200522 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v26b8\" (UniqueName: \"kubernetes.io/projected/a697892a-1733-4bdb-879c-c9df9482954e-kube-api-access-v26b8\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:37:35.201996 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.201970 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c51d899-6cd9-45f1-8bcb-44588117aa25-kube-api-access-84r8v" pod "7c51d899-6cd9-45f1-8bcb-44588117aa25" (UID: "7c51d899-6cd9-45f1-8bcb-44588117aa25"). InnerVolumeSpecName "kube-api-access-84r8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:37:35.301699 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.301605 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84r8v\" (UniqueName: \"kubernetes.io/projected/7c51d899-6cd9-45f1-8bcb-44588117aa25-kube-api-access-84r8v\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:37:35.619472 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.619382 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:35.853794 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.853378 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:35.867773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.860302 6572 generic.go:356] "Generic (PLEG): container finished" podID="a697892a-1733-4bdb-879c-c9df9482954e" containerID="d8455bfcf4939c8865df883a47b22929d06534c922fe5d9868d9b14e02532332" exitCode=0 Oct 30 01:37:35.867773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.860540 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-666b0b-76df5449dd-qhpm8" event={"ID":"a697892a-1733-4bdb-879c-c9df9482954e","Type":"ContainerDied","Data":"d8455bfcf4939c8865df883a47b22929d06534c922fe5d9868d9b14e02532332"} Oct 30 01:37:35.867773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.860594 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-666b0b-76df5449dd-qhpm8" event={"ID":"a697892a-1733-4bdb-879c-c9df9482954e","Type":"ContainerDied","Data":"2a68734bb5f8ee395dc49d9f786b4f11472002600121d9c19a6e6bf34cb3c895"} Oct 30 01:37:35.867773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.860623 6572 scope.go:122] "RemoveContainer" containerID="d8455bfcf4939c8865df883a47b22929d06534c922fe5d9868d9b14e02532332" Oct 30 01:37:35.867773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.860833 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-666b0b-76df5449dd-qhpm8" Oct 30 01:37:35.867773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.864254 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28af3258-qjrpd" event={"ID":"456ee785-1d45-499c-810d-68121a7a2f2d","Type":"ContainerDied","Data":"5d74c3410531bdbbeb473db306bf263e3c35023c5e593a9300cfb6efdd0b2f03"} Oct 30 01:37:35.867773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.864304 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d74c3410531bdbbeb473db306bf263e3c35023c5e593a9300cfb6efdd0b2f03" Oct 30 01:37:35.867773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.864371 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28af3258-qjrpd" Oct 30 01:37:35.877396 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.877308 6572 generic.go:356] "Generic (PLEG): container finished" podID="7c51d899-6cd9-45f1-8bcb-44588117aa25" containerID="3a74c0ce781c92b095c4e871720483d961d07f61934e9b2f6f1c0328809c0b61" exitCode=0 Oct 30 01:37:35.877571 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.877476 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-666b0b-5c7f78c4d7-dzbzn" event={"ID":"7c51d899-6cd9-45f1-8bcb-44588117aa25","Type":"ContainerDied","Data":"3a74c0ce781c92b095c4e871720483d961d07f61934e9b2f6f1c0328809c0b61"} Oct 30 01:37:35.877571 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.877528 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-666b0b-5c7f78c4d7-dzbzn" event={"ID":"7c51d899-6cd9-45f1-8bcb-44588117aa25","Type":"ContainerDied","Data":"40c1009cbf4a15ccb8ac57fa967afcb1253e09e25ede0f55c2b7afc33f312bb7"} Oct 30 01:37:35.877704 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.877620 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-666b0b-5c7f78c4d7-dzbzn" Oct 30 01:37:35.898045 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.898018 6572 scope.go:122] "RemoveContainer" containerID="d8455bfcf4939c8865df883a47b22929d06534c922fe5d9868d9b14e02532332" Oct 30 01:37:35.898483 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:37:35.898447 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8455bfcf4939c8865df883a47b22929d06534c922fe5d9868d9b14e02532332\": container with ID starting with d8455bfcf4939c8865df883a47b22929d06534c922fe5d9868d9b14e02532332 not found: ID does not exist" containerID="d8455bfcf4939c8865df883a47b22929d06534c922fe5d9868d9b14e02532332" Oct 30 01:37:35.898643 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.898488 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8455bfcf4939c8865df883a47b22929d06534c922fe5d9868d9b14e02532332"} err="failed to get container status \"d8455bfcf4939c8865df883a47b22929d06534c922fe5d9868d9b14e02532332\": rpc error: code = NotFound desc = could not find container \"d8455bfcf4939c8865df883a47b22929d06534c922fe5d9868d9b14e02532332\": container with ID starting with d8455bfcf4939c8865df883a47b22929d06534c922fe5d9868d9b14e02532332 not found: ID does not exist" Oct 30 01:37:35.898643 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.898512 6572 scope.go:122] "RemoveContainer" containerID="3a74c0ce781c92b095c4e871720483d961d07f61934e9b2f6f1c0328809c0b61" Oct 30 01:37:35.933002 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.932949 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-666b0b-5c7f78c4d7-dzbzn"] Oct 30 01:37:35.938284 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.934628 6572 scope.go:122] "RemoveContainer" containerID="3a74c0ce781c92b095c4e871720483d961d07f61934e9b2f6f1c0328809c0b61" Oct 30 01:37:35.938284 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.938190 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-666b0b-5c7f78c4d7-dzbzn"] Oct 30 01:37:35.938284 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:37:35.938271 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a74c0ce781c92b095c4e871720483d961d07f61934e9b2f6f1c0328809c0b61\": container with ID starting with 3a74c0ce781c92b095c4e871720483d961d07f61934e9b2f6f1c0328809c0b61 not found: ID does not exist" containerID="3a74c0ce781c92b095c4e871720483d961d07f61934e9b2f6f1c0328809c0b61" Oct 30 01:37:35.938536 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.938306 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a74c0ce781c92b095c4e871720483d961d07f61934e9b2f6f1c0328809c0b61"} err="failed to get container status \"3a74c0ce781c92b095c4e871720483d961d07f61934e9b2f6f1c0328809c0b61\": rpc error: code = NotFound desc = could not find container \"3a74c0ce781c92b095c4e871720483d961d07f61934e9b2f6f1c0328809c0b61\": container with ID starting with 3a74c0ce781c92b095c4e871720483d961d07f61934e9b2f6f1c0328809c0b61 not found: ID does not exist" Oct 30 01:37:35.962219 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.962183 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-666b0b-76df5449dd-qhpm8"] Oct 30 01:37:35.967544 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:35.967513 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-666b0b-76df5449dd-qhpm8"] Oct 30 01:37:36.544789 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:36.544740 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c51d899-6cd9-45f1-8bcb-44588117aa25" path="/var/lib/kubelet/pods/7c51d899-6cd9-45f1-8bcb-44588117aa25/volumes" Oct 30 01:37:36.545181 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:36.545168 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a697892a-1733-4bdb-879c-c9df9482954e" path="/var/lib/kubelet/pods/a697892a-1733-4bdb-879c-c9df9482954e/volumes" Oct 30 01:37:37.296889 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:37.296856 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:37.336791 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:37.336738 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-799dc77c98-g76nh"] Oct 30 01:37:37.337404 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:37.337390 6572 cpu_manager.go:414] "containerMap: removing container" podUID="7c51d899-6cd9-45f1-8bcb-44588117aa25" containerName="payload-pre-processing" Oct 30 01:37:37.337446 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:37.337406 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c51d899-6cd9-45f1-8bcb-44588117aa25" containerName="payload-pre-processing" Oct 30 01:37:37.337446 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:37.337419 6572 cpu_manager.go:414] "containerMap: removing container" podUID="456ee785-1d45-499c-810d-68121a7a2f2d" containerName="revoke-keys" Oct 30 01:37:37.337446 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:37.337425 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="456ee785-1d45-499c-810d-68121a7a2f2d" containerName="revoke-keys" Oct 30 01:37:37.337543 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:37.337451 6572 cpu_manager.go:414] "containerMap: removing container" podUID="a697892a-1733-4bdb-879c-c9df9482954e" containerName="payload-processing" Oct 30 01:37:37.337543 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:37.337456 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a697892a-1733-4bdb-879c-c9df9482954e" containerName="payload-processing" Oct 30 01:37:37.337608 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:37.337579 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c51d899-6cd9-45f1-8bcb-44588117aa25" containerName="payload-pre-processing" Oct 30 01:37:37.337608 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:37.337592 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="456ee785-1d45-499c-810d-68121a7a2f2d" containerName="revoke-keys" Oct 30 01:37:37.337608 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:37.337601 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="a697892a-1733-4bdb-879c-c9df9482954e" containerName="payload-processing" Oct 30 01:37:37.342889 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:37.342867 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-799dc77c98-g76nh" Oct 30 01:37:37.365360 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:37.365326 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-799dc77c98-g76nh"] Oct 30 01:37:37.368784 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:37.368740 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d9ede0\"" Oct 30 01:37:37.368955 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:37.368867 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d9ede0-dockercfg-gc2xq\"" Oct 30 01:37:37.387515 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:37.387478 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-6fb94d5599-46prg"] Oct 30 01:37:37.393464 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:37.393400 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-6fb94d5599-46prg" Oct 30 01:37:37.412131 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:37.412093 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-6fb94d5599-46prg"] Oct 30 01:37:37.510452 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:37.510412 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1beef457-ba1a-437d-aa79-a625c5032b19-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d9ede0-799dc77c98-g76nh\" (UID: \"1beef457-ba1a-437d-aa79-a625c5032b19\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-799dc77c98-g76nh" Oct 30 01:37:37.522781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:37.510880 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzc7t\" (UniqueName: \"kubernetes.io/projected/1beef457-ba1a-437d-aa79-a625c5032b19-kube-api-access-jzc7t\") pod \"payload-processing-e2e-ipp-a-d9ede0-799dc77c98-g76nh\" (UID: \"1beef457-ba1a-437d-aa79-a625c5032b19\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-799dc77c98-g76nh" Oct 30 01:37:37.523150 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:37.520675 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:37.526359 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:37.526314 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8ce41ba-cb52-4d64-8b85-d420b3d96711-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d9ede0-6fb94d5599-46prg\" (UID: \"a8ce41ba-cb52-4d64-8b85-d420b3d96711\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-6fb94d5599-46prg" Oct 30 01:37:37.534514 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:37.534473 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bx44\" (UniqueName: \"kubernetes.io/projected/a8ce41ba-cb52-4d64-8b85-d420b3d96711-kube-api-access-4bx44\") pod \"payload-pre-processing-e2e-ipp-a-d9ede0-6fb94d5599-46prg\" (UID: \"a8ce41ba-cb52-4d64-8b85-d420b3d96711\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-6fb94d5599-46prg" Oct 30 01:37:37.636742 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:37.636634 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bx44\" (UniqueName: \"kubernetes.io/projected/a8ce41ba-cb52-4d64-8b85-d420b3d96711-kube-api-access-4bx44\") pod \"payload-pre-processing-e2e-ipp-a-d9ede0-6fb94d5599-46prg\" (UID: \"a8ce41ba-cb52-4d64-8b85-d420b3d96711\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-6fb94d5599-46prg" Oct 30 01:37:37.636936 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:37.636896 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1beef457-ba1a-437d-aa79-a625c5032b19-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d9ede0-799dc77c98-g76nh\" (UID: \"1beef457-ba1a-437d-aa79-a625c5032b19\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-799dc77c98-g76nh" Oct 30 01:37:37.637002 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:37.636987 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzc7t\" (UniqueName: \"kubernetes.io/projected/1beef457-ba1a-437d-aa79-a625c5032b19-kube-api-access-jzc7t\") pod \"payload-processing-e2e-ipp-a-d9ede0-799dc77c98-g76nh\" (UID: \"1beef457-ba1a-437d-aa79-a625c5032b19\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-799dc77c98-g76nh" Oct 30 01:37:37.637068 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:37.637051 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8ce41ba-cb52-4d64-8b85-d420b3d96711-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d9ede0-6fb94d5599-46prg\" (UID: \"a8ce41ba-cb52-4d64-8b85-d420b3d96711\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-6fb94d5599-46prg" Oct 30 01:37:37.637864 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:37.637834 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1beef457-ba1a-437d-aa79-a625c5032b19-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d9ede0-799dc77c98-g76nh\" (UID: \"1beef457-ba1a-437d-aa79-a625c5032b19\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-799dc77c98-g76nh" Oct 30 01:37:37.637954 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:37.637915 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8ce41ba-cb52-4d64-8b85-d420b3d96711-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d9ede0-6fb94d5599-46prg\" (UID: \"a8ce41ba-cb52-4d64-8b85-d420b3d96711\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-6fb94d5599-46prg" Oct 30 01:37:37.655979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:37.655947 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bx44\" (UniqueName: \"kubernetes.io/projected/a8ce41ba-cb52-4d64-8b85-d420b3d96711-kube-api-access-4bx44\") pod \"payload-pre-processing-e2e-ipp-a-d9ede0-6fb94d5599-46prg\" (UID: \"a8ce41ba-cb52-4d64-8b85-d420b3d96711\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-6fb94d5599-46prg" Oct 30 01:37:37.658554 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:37.658263 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzc7t\" (UniqueName: \"kubernetes.io/projected/1beef457-ba1a-437d-aa79-a625c5032b19-kube-api-access-jzc7t\") pod \"payload-processing-e2e-ipp-a-d9ede0-799dc77c98-g76nh\" (UID: \"1beef457-ba1a-437d-aa79-a625c5032b19\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-799dc77c98-g76nh" Oct 30 01:37:37.732217 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:37.732150 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-6fb94d5599-46prg" Oct 30 01:37:37.957861 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:37.957825 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-799dc77c98-g76nh" Oct 30 01:37:38.106406 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:38.106360 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-6fb94d5599-46prg"] Oct 30 01:37:38.109831 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:38.109695 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:38.269473 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:38.267671 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-799dc77c98-g76nh"] Oct 30 01:37:38.592294 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:38.592205 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:38.840742 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:38.840697 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:38.913170 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:38.913132 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-799dc77c98-g76nh" event={"ID":"1beef457-ba1a-437d-aa79-a625c5032b19","Type":"ContainerStarted","Data":"de09413faeed12daeae145c36d3d3a1fbcb1b282916b2dc56008cfdec6a6383d"} Oct 30 01:37:38.913375 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:38.913199 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-799dc77c98-g76nh" Oct 30 01:37:38.913375 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:38.913225 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-799dc77c98-g76nh" event={"ID":"1beef457-ba1a-437d-aa79-a625c5032b19","Type":"ContainerStarted","Data":"7c163fd74a0ddd39421463d71b541495ef3176edb0b2457f236f0737c8e9d24e"} Oct 30 01:37:38.915014 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:38.914988 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-6fb94d5599-46prg" event={"ID":"a8ce41ba-cb52-4d64-8b85-d420b3d96711","Type":"ContainerStarted","Data":"f19b5f03e03fed67f170b45a426f6761d15a8f183decef5917d198bfaa5e3fc8"} Oct 30 01:37:38.915131 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:38.915030 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-6fb94d5599-46prg" event={"ID":"a8ce41ba-cb52-4d64-8b85-d420b3d96711","Type":"ContainerStarted","Data":"e5fab0abd094e6ada0c34b9571e2c5c627d868a09fc188f5229d7a5cca475a90"} Oct 30 01:37:38.915131 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:38.915089 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-6fb94d5599-46prg" Oct 30 01:37:38.961942 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:38.961880 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-799dc77c98-g76nh" podStartSLOduration=1.961861226 podStartE2EDuration="1.961861226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:37:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:37:38.959479392 +0000 UTC m=+1184.535904790" watchObservedRunningTime="2026-10-30 01:37:38.961861226 +0000 UTC m=+1184.538286629" Oct 30 01:37:39.002200 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:39.002128 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-6fb94d5599-46prg" podStartSLOduration=2.002109766 podStartE2EDuration="2.002109766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:37:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:37:39.000008764 +0000 UTC m=+1184.576434189" watchObservedRunningTime="2026-10-30 01:37:39.002109766 +0000 UTC m=+1184.578535167" Oct 30 01:37:39.043875 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:39.043828 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c764d992-xhd7f"] Oct 30 01:37:39.048258 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:39.048224 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c764d992-xhd7f"] Oct 30 01:37:40.267448 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:40.267405 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:40.344600 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:40.344557 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7668f5d547-dxb9b"] Oct 30 01:37:40.348958 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:40.348933 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7668f5d547-dxb9b" Oct 30 01:37:40.371695 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:40.371662 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7668f5d547-dxb9b"] Oct 30 01:37:40.381872 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:40.381833 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a147d7d-b019-4960-9813-9cc1495d9901-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-19a5f1-7668f5d547-dxb9b\" (UID: \"1a147d7d-b019-4960-9813-9cc1495d9901\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7668f5d547-dxb9b" Oct 30 01:37:40.382041 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:40.382021 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq2f7\" (UniqueName: \"kubernetes.io/projected/1a147d7d-b019-4960-9813-9cc1495d9901-kube-api-access-jq2f7\") pod \"payload-pre-processing-e2e-ipp-a-19a5f1-7668f5d547-dxb9b\" (UID: \"1a147d7d-b019-4960-9813-9cc1495d9901\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7668f5d547-dxb9b" Oct 30 01:37:40.397585 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:40.397534 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-19a5f1-dockercfg-gdtlr\"" Oct 30 01:37:40.398232 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:40.398159 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-19a5f1\"" Oct 30 01:37:40.480245 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:40.467084 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:40.496584 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:40.496546 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a147d7d-b019-4960-9813-9cc1495d9901-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-19a5f1-7668f5d547-dxb9b\" (UID: \"1a147d7d-b019-4960-9813-9cc1495d9901\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7668f5d547-dxb9b" Oct 30 01:37:40.496834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:40.496726 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq2f7\" (UniqueName: \"kubernetes.io/projected/1a147d7d-b019-4960-9813-9cc1495d9901-kube-api-access-jq2f7\") pod \"payload-pre-processing-e2e-ipp-a-19a5f1-7668f5d547-dxb9b\" (UID: \"1a147d7d-b019-4960-9813-9cc1495d9901\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7668f5d547-dxb9b" Oct 30 01:37:40.498157 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:40.498098 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a147d7d-b019-4960-9813-9cc1495d9901-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-19a5f1-7668f5d547-dxb9b\" (UID: \"1a147d7d-b019-4960-9813-9cc1495d9901\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7668f5d547-dxb9b" Oct 30 01:37:40.554462 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:40.554372 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7" path="/var/lib/kubelet/pods/82e4a162-d1e0-4d0a-a2ef-b34d22f52ca7/volumes" Oct 30 01:37:40.581328 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:40.573416 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq2f7\" (UniqueName: \"kubernetes.io/projected/1a147d7d-b019-4960-9813-9cc1495d9901-kube-api-access-jq2f7\") pod \"payload-pre-processing-e2e-ipp-a-19a5f1-7668f5d547-dxb9b\" (UID: \"1a147d7d-b019-4960-9813-9cc1495d9901\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7668f5d547-dxb9b" Oct 30 01:37:40.664855 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:40.664812 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7668f5d547-dxb9b" Oct 30 01:37:40.976461 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:40.976423 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7668f5d547-dxb9b"] Oct 30 01:37:40.976574 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:37:40.976452 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a147d7d_b019_4960_9813_9cc1495d9901.slice/crio-df1f0932abb0700acb22269d32353a2e840325d41a03e6a219f98fb3b30b4069 WatchSource:0}: Error finding container df1f0932abb0700acb22269d32353a2e840325d41a03e6a219f98fb3b30b4069: Status 404 returned error can't find the container with id df1f0932abb0700acb22269d32353a2e840325d41a03e6a219f98fb3b30b4069 Oct 30 01:37:41.946068 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:41.946028 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7668f5d547-dxb9b" event={"ID":"1a147d7d-b019-4960-9813-9cc1495d9901","Type":"ContainerStarted","Data":"8ed19ac9d4a09bf20c4eee39e915f2b59f7e6f1f29d09cae6a9b25ae881c46b5"} Oct 30 01:37:41.946578 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:41.946098 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7668f5d547-dxb9b" event={"ID":"1a147d7d-b019-4960-9813-9cc1495d9901","Type":"ContainerStarted","Data":"df1f0932abb0700acb22269d32353a2e840325d41a03e6a219f98fb3b30b4069"} Oct 30 01:37:41.946578 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:41.946127 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7668f5d547-dxb9b" Oct 30 01:37:41.981899 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:41.981835 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7668f5d547-dxb9b" podStartSLOduration=1.981815559 podStartE2EDuration="1.981815559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:37:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:37:41.980251161 +0000 UTC m=+1187.556676564" watchObservedRunningTime="2026-10-30 01:37:41.981815559 +0000 UTC m=+1187.558240962" Oct 30 01:37:42.297680 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:42.297584 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:43.154161 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:43.154121 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:43.658119 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:43.658051 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:43.879183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:43.879142 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:44.328067 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:44.328033 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:44.688259 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:37:44.688219 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:37:45.027378 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:45.027268 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fb82088-mn4fn"] Oct 30 01:37:45.036960 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:45.036922 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fb82088-mn4fn"] Oct 30 01:37:46.119852 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:46.119815 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:46.543474 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:46.543435 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5" path="/var/lib/kubelet/pods/4107c0e6-8225-4b5d-9b47-5bcef0b5e7f5/volumes" Oct 30 01:37:47.034454 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:47.034416 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5af953b7-ctpc5"] Oct 30 01:37:47.044022 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:47.043986 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5af953b7-ctpc5"] Oct 30 01:37:47.579233 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:47.579196 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:47.774880 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:47.774847 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:48.166571 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:48.166519 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:48.543634 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:48.543550 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46b4ef76-9c38-436f-a193-dc11782fcfaf" path="/var/lib/kubelet/pods/46b4ef76-9c38-436f-a193-dc11782fcfaf/volumes" Oct 30 01:37:49.317635 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:49.317599 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:49.514986 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:49.514939 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-56668cb6-85kf8"] Oct 30 01:37:49.526422 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:49.526388 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-56668cb6-85kf8" Oct 30 01:37:49.544441 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:49.544407 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-56668cb6-85kf8"] Oct 30 01:37:49.546060 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:49.545398 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f7bdcc\"" Oct 30 01:37:49.588053 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:49.587963 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:49.594619 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:49.594585 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f7bdcc-dockercfg-bkj47\"" Oct 30 01:37:49.660732 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:49.660687 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f485bb6f-8dcb-40de-9661-d12993b82934-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f7bdcc-56668cb6-85kf8\" (UID: \"f485bb6f-8dcb-40de-9661-d12993b82934\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-56668cb6-85kf8" Oct 30 01:37:49.660978 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:49.660833 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk7dt\" (UniqueName: \"kubernetes.io/projected/f485bb6f-8dcb-40de-9661-d12993b82934-kube-api-access-dk7dt\") pod \"payload-processing-e2e-ipp-a-f7bdcc-56668cb6-85kf8\" (UID: \"f485bb6f-8dcb-40de-9661-d12993b82934\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-56668cb6-85kf8" Oct 30 01:37:49.762292 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:49.762253 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f485bb6f-8dcb-40de-9661-d12993b82934-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f7bdcc-56668cb6-85kf8\" (UID: \"f485bb6f-8dcb-40de-9661-d12993b82934\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-56668cb6-85kf8" Oct 30 01:37:49.762521 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:49.762420 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk7dt\" (UniqueName: \"kubernetes.io/projected/f485bb6f-8dcb-40de-9661-d12993b82934-kube-api-access-dk7dt\") pod \"payload-processing-e2e-ipp-a-f7bdcc-56668cb6-85kf8\" (UID: \"f485bb6f-8dcb-40de-9661-d12993b82934\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-56668cb6-85kf8" Oct 30 01:37:49.763102 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:49.763079 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f485bb6f-8dcb-40de-9661-d12993b82934-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f7bdcc-56668cb6-85kf8\" (UID: \"f485bb6f-8dcb-40de-9661-d12993b82934\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-56668cb6-85kf8" Oct 30 01:37:49.781362 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:49.781331 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk7dt\" (UniqueName: \"kubernetes.io/projected/f485bb6f-8dcb-40de-9661-d12993b82934-kube-api-access-dk7dt\") pod \"payload-processing-e2e-ipp-a-f7bdcc-56668cb6-85kf8\" (UID: \"f485bb6f-8dcb-40de-9661-d12993b82934\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-56668cb6-85kf8" Oct 30 01:37:49.783128 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:49.783098 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:49.894676 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:49.894587 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-56668cb6-85kf8" Oct 30 01:37:49.927351 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:49.927323 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-799dc77c98-g76nh" Oct 30 01:37:49.927510 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:49.927403 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-6fb94d5599-46prg" Oct 30 01:37:50.204031 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:37:50.203998 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf485bb6f_8dcb_40de_9661_d12993b82934.slice/crio-414d82200f5e0080c410769749bfe992ffd64c99d0dd38990c978ceb6c501db1 WatchSource:0}: Error finding container 414d82200f5e0080c410769749bfe992ffd64c99d0dd38990c978ceb6c501db1: Status 404 returned error can't find the container with id 414d82200f5e0080c410769749bfe992ffd64c99d0dd38990c978ceb6c501db1 Oct 30 01:37:50.205059 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:50.205033 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-56668cb6-85kf8"] Oct 30 01:37:50.323724 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:50.323693 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:50.902680 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:50.902643 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f09934-htnf2"] Oct 30 01:37:50.907311 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:50.907287 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f09934-htnf2" Oct 30 01:37:50.920586 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:50.920560 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:37:50.920586 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:50.920577 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:37:50.926259 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:50.926233 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f09934-htnf2"] Oct 30 01:37:50.941794 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:50.930261 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:37:50.941794 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:50.930287 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:37:50.987418 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:50.987378 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a618ddde-ca90-4180-a97e-8321e808f93a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f09934-htnf2\" (UID: \"a618ddde-ca90-4180-a97e-8321e808f93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f09934-htnf2" Oct 30 01:37:50.987574 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:50.987538 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8thw8\" (UniqueName: \"kubernetes.io/projected/a618ddde-ca90-4180-a97e-8321e808f93a-kube-api-access-8thw8\") pod \"maas-api-revoke-keys-e2e-api-a-f09934-htnf2\" (UID: \"a618ddde-ca90-4180-a97e-8321e808f93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f09934-htnf2" Oct 30 01:37:50.987660 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:50.987643 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a618ddde-ca90-4180-a97e-8321e808f93a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f09934-htnf2\" (UID: \"a618ddde-ca90-4180-a97e-8321e808f93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f09934-htnf2" Oct 30 01:37:51.051376 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:51.051339 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-56668cb6-85kf8" event={"ID":"f485bb6f-8dcb-40de-9661-d12993b82934","Type":"ContainerStarted","Data":"bebdfbcedcb36bc288516cadb785f552f906d6a228c614c619920db05025ee1c"} Oct 30 01:37:51.051565 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:51.051407 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-56668cb6-85kf8" event={"ID":"f485bb6f-8dcb-40de-9661-d12993b82934","Type":"ContainerStarted","Data":"414d82200f5e0080c410769749bfe992ffd64c99d0dd38990c978ceb6c501db1"} Oct 30 01:37:51.051565 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:51.051442 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-56668cb6-85kf8" Oct 30 01:37:51.089895 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:51.089853 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8thw8\" (UniqueName: \"kubernetes.io/projected/a618ddde-ca90-4180-a97e-8321e808f93a-kube-api-access-8thw8\") pod \"maas-api-revoke-keys-e2e-api-a-f09934-htnf2\" (UID: \"a618ddde-ca90-4180-a97e-8321e808f93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f09934-htnf2" Oct 30 01:37:51.090233 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:51.089980 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a618ddde-ca90-4180-a97e-8321e808f93a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f09934-htnf2\" (UID: \"a618ddde-ca90-4180-a97e-8321e808f93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f09934-htnf2" Oct 30 01:37:51.090233 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:51.090129 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a618ddde-ca90-4180-a97e-8321e808f93a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f09934-htnf2\" (UID: \"a618ddde-ca90-4180-a97e-8321e808f93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f09934-htnf2" Oct 30 01:37:51.091547 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:51.091520 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a618ddde-ca90-4180-a97e-8321e808f93a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f09934-htnf2\" (UID: \"a618ddde-ca90-4180-a97e-8321e808f93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f09934-htnf2" Oct 30 01:37:51.092118 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:51.092064 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-56668cb6-85kf8" podStartSLOduration=2.092048409 podStartE2EDuration="2.092048409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:37:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:37:51.089191186 +0000 UTC m=+1196.665616588" watchObservedRunningTime="2026-10-30 01:37:51.092048409 +0000 UTC m=+1196.668473812" Oct 30 01:37:51.094144 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:51.094119 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a618ddde-ca90-4180-a97e-8321e808f93a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f09934-htnf2\" (UID: \"a618ddde-ca90-4180-a97e-8321e808f93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f09934-htnf2" Oct 30 01:37:51.107589 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:51.107557 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8thw8\" (UniqueName: \"kubernetes.io/projected/a618ddde-ca90-4180-a97e-8321e808f93a-kube-api-access-8thw8\") pod \"maas-api-revoke-keys-e2e-api-a-f09934-htnf2\" (UID: \"a618ddde-ca90-4180-a97e-8321e808f93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f09934-htnf2" Oct 30 01:37:51.224147 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:51.224110 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f09934-htnf2" Oct 30 01:37:51.553215 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:51.553180 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f09934-htnf2"] Oct 30 01:37:51.553650 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:37:51.553629 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda618ddde_ca90_4180_a97e_8321e808f93a.slice/crio-df603d0c4523cfba9f20cbfa99109a92df939c6f5176d3fc1f66982c59cf1bdc WatchSource:0}: Error finding container df603d0c4523cfba9f20cbfa99109a92df939c6f5176d3fc1f66982c59cf1bdc: Status 404 returned error can't find the container with id df603d0c4523cfba9f20cbfa99109a92df939c6f5176d3fc1f66982c59cf1bdc Oct 30 01:37:52.062688 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:52.062643 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f09934-htnf2" event={"ID":"a618ddde-ca90-4180-a97e-8321e808f93a","Type":"ContainerStarted","Data":"061d6970c7b3241e6f5e4f7b40e92d03503d6299024fac1f41c55c73bd432ae4"} Oct 30 01:37:52.062889 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:52.062709 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f09934-htnf2" event={"ID":"a618ddde-ca90-4180-a97e-8321e808f93a","Type":"ContainerStarted","Data":"df603d0c4523cfba9f20cbfa99109a92df939c6f5176d3fc1f66982c59cf1bdc"} Oct 30 01:37:52.101597 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:52.101468 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f09934-htnf2" podStartSLOduration=2.101447211 podStartE2EDuration="2.101447211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:37:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:37:52.099585962 +0000 UTC m=+1197.676011364" watchObservedRunningTime="2026-10-30 01:37:52.101447211 +0000 UTC m=+1197.677872613" Oct 30 01:37:52.317198 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:52.317095 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:52.514014 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:52.513979 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:52.958673 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:52.958641 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7668f5d547-dxb9b" Oct 30 01:37:53.073313 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:53.073278 6572 generic.go:356] "Generic (PLEG): container finished" podID="a618ddde-ca90-4180-a97e-8321e808f93a" containerID="061d6970c7b3241e6f5e4f7b40e92d03503d6299024fac1f41c55c73bd432ae4" exitCode=0 Oct 30 01:37:53.073514 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:53.073365 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f09934-htnf2" event={"ID":"a618ddde-ca90-4180-a97e-8321e808f93a","Type":"ContainerDied","Data":"061d6970c7b3241e6f5e4f7b40e92d03503d6299024fac1f41c55c73bd432ae4"} Oct 30 01:37:54.247591 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:54.247563 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f09934-htnf2" Oct 30 01:37:54.251344 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:54.251320 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a618ddde-ca90-4180-a97e-8321e808f93a-kube-api-access-8thw8\" (UniqueName: \"kubernetes.io/projected/a618ddde-ca90-4180-a97e-8321e808f93a-kube-api-access-8thw8\") pod \"a618ddde-ca90-4180-a97e-8321e808f93a\" (UID: \"a618ddde-ca90-4180-a97e-8321e808f93a\") " Oct 30 01:37:54.251464 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:54.251382 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a618ddde-ca90-4180-a97e-8321e808f93a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a618ddde-ca90-4180-a97e-8321e808f93a-maas-api-xks-ca\") pod \"a618ddde-ca90-4180-a97e-8321e808f93a\" (UID: \"a618ddde-ca90-4180-a97e-8321e808f93a\") " Oct 30 01:37:54.251528 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:54.251482 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a618ddde-ca90-4180-a97e-8321e808f93a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a618ddde-ca90-4180-a97e-8321e808f93a-maas-api-openshift-ca\") pod \"a618ddde-ca90-4180-a97e-8321e808f93a\" (UID: \"a618ddde-ca90-4180-a97e-8321e808f93a\") " Oct 30 01:37:54.251875 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:54.251851 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a618ddde-ca90-4180-a97e-8321e808f93a-maas-api-openshift-ca" pod "a618ddde-ca90-4180-a97e-8321e808f93a" (UID: "a618ddde-ca90-4180-a97e-8321e808f93a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:37:54.252092 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:54.252077 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a618ddde-ca90-4180-a97e-8321e808f93a-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:37:54.253417 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:54.253392 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a618ddde-ca90-4180-a97e-8321e808f93a-maas-api-xks-ca" pod "a618ddde-ca90-4180-a97e-8321e808f93a" (UID: "a618ddde-ca90-4180-a97e-8321e808f93a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:37:54.253567 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:54.253468 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a618ddde-ca90-4180-a97e-8321e808f93a-kube-api-access-8thw8" pod "a618ddde-ca90-4180-a97e-8321e808f93a" (UID: "a618ddde-ca90-4180-a97e-8321e808f93a"). InnerVolumeSpecName "kube-api-access-8thw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:37:54.353716 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:54.353627 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8thw8\" (UniqueName: \"kubernetes.io/projected/a618ddde-ca90-4180-a97e-8321e808f93a-kube-api-access-8thw8\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:37:54.353716 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:54.353662 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a618ddde-ca90-4180-a97e-8321e808f93a-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:37:54.452263 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:54.452210 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:55.091460 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:55.091429 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f09934-htnf2" Oct 30 01:37:55.091460 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:55.091436 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f09934-htnf2" event={"ID":"a618ddde-ca90-4180-a97e-8321e808f93a","Type":"ContainerDied","Data":"df603d0c4523cfba9f20cbfa99109a92df939c6f5176d3fc1f66982c59cf1bdc"} Oct 30 01:37:55.091709 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:55.091492 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df603d0c4523cfba9f20cbfa99109a92df939c6f5176d3fc1f66982c59cf1bdc" Oct 30 01:37:55.195684 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:55.195647 6572 scope.go:122] "RemoveContainer" containerID="384025cc5788363ecf1be475709e853e7837e8321a76608bad2e2ab344c55a58" Oct 30 01:37:56.259400 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:56.259359 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:56.447810 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:56.447772 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:57.032705 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:57.032668 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff85dbf6-mfl4s"] Oct 30 01:37:57.042403 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:57.042369 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff85dbf6-mfl4s"] Oct 30 01:37:58.332087 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:58.332050 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:58.543370 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:58.543332 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f713498-db56-4752-b3f6-d98cb67f3dd2" path="/var/lib/kubelet/pods/1f713498-db56-4752-b3f6-d98cb67f3dd2/volumes" Oct 30 01:37:59.854449 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:37:59.854414 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:00.604404 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:00.604367 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:00.762476 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:00.762440 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-666b0b-pkkws"] Oct 30 01:38:00.771881 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:00.771845 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-666b0b-pkkws"] Oct 30 01:38:00.850761 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:00.850714 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:01.071492 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:01.071458 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:01.309366 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:01.309320 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:01.391407 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:01.391306 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f09934-9d4445b84-wcp9v"] Oct 30 01:38:01.526119 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:01.526079 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:01.725176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:01.725139 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:02.066935 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:02.066829 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-56668cb6-85kf8" Oct 30 01:38:02.543450 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:02.543410 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6d7aa69-c87d-4f3b-83a9-9afb5940118d" path="/var/lib/kubelet/pods/f6d7aa69-c87d-4f3b-83a9-9afb5940118d/volumes" Oct 30 01:38:03.040560 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:03.040503 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:04.044529 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:04.044479 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f50466c-zj7p4"] Oct 30 01:38:04.053225 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:04.053190 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f50466c-zj7p4"] Oct 30 01:38:04.210565 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:04.210525 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:04.435302 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:04.435264 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:04.544293 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:04.544260 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aac7cb3b-87f6-44ef-9fc3-fa595bcb1437" path="/var/lib/kubelet/pods/aac7cb3b-87f6-44ef-9fc3-fa595bcb1437/volumes" Oct 30 01:38:04.993043 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:04.993004 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:06.392271 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:06.392200 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f09934-9d4445b84-wcp9v" podUID="7bf5fd44-9bd6-48fe-badd-c1643822e48b" containerName="payload-processing" containerID="cri-o://78a54f05417b52aea05df39aaec43512fe3e61889e3930b82e4d41ad58e1f119" gracePeriod=25 Oct 30 01:38:06.667802 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:06.667639 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:06.718115 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:06.718083 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f09934-9d4445b84-wcp9v" Oct 30 01:38:06.844073 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:06.844032 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bf5fd44-9bd6-48fe-badd-c1643822e48b-kube-api-access-jksbd\" (UniqueName: \"kubernetes.io/projected/7bf5fd44-9bd6-48fe-badd-c1643822e48b-kube-api-access-jksbd\") pod \"7bf5fd44-9bd6-48fe-badd-c1643822e48b\" (UID: \"7bf5fd44-9bd6-48fe-badd-c1643822e48b\") " Oct 30 01:38:06.844263 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:06.844178 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bf5fd44-9bd6-48fe-badd-c1643822e48b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf5fd44-9bd6-48fe-badd-c1643822e48b-plugins-config-volume\") pod \"7bf5fd44-9bd6-48fe-badd-c1643822e48b\" (UID: \"7bf5fd44-9bd6-48fe-badd-c1643822e48b\") " Oct 30 01:38:06.844561 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:06.844535 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bf5fd44-9bd6-48fe-badd-c1643822e48b-plugins-config-volume" pod "7bf5fd44-9bd6-48fe-badd-c1643822e48b" (UID: "7bf5fd44-9bd6-48fe-badd-c1643822e48b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:06.844800 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:06.844773 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf5fd44-9bd6-48fe-badd-c1643822e48b-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:06.846357 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:06.846328 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bf5fd44-9bd6-48fe-badd-c1643822e48b-kube-api-access-jksbd" pod "7bf5fd44-9bd6-48fe-badd-c1643822e48b" (UID: "7bf5fd44-9bd6-48fe-badd-c1643822e48b"). InnerVolumeSpecName "kube-api-access-jksbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:06.946278 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:06.946178 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jksbd\" (UniqueName: \"kubernetes.io/projected/7bf5fd44-9bd6-48fe-badd-c1643822e48b-kube-api-access-jksbd\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:07.211567 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:07.211471 6572 generic.go:356] "Generic (PLEG): container finished" podID="7bf5fd44-9bd6-48fe-badd-c1643822e48b" containerID="78a54f05417b52aea05df39aaec43512fe3e61889e3930b82e4d41ad58e1f119" exitCode=0 Oct 30 01:38:07.211567 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:07.211541 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f09934-9d4445b84-wcp9v" Oct 30 01:38:07.211845 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:07.211583 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f09934-9d4445b84-wcp9v" event={"ID":"7bf5fd44-9bd6-48fe-badd-c1643822e48b","Type":"ContainerDied","Data":"78a54f05417b52aea05df39aaec43512fe3e61889e3930b82e4d41ad58e1f119"} Oct 30 01:38:07.211845 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:07.211654 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f09934-9d4445b84-wcp9v" event={"ID":"7bf5fd44-9bd6-48fe-badd-c1643822e48b","Type":"ContainerDied","Data":"f6ffa6479a7e501f764fe9d4b0d928e2bc7b2d0dbb3a4b609b52b77171e1bd57"} Oct 30 01:38:07.211845 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:07.211684 6572 scope.go:122] "RemoveContainer" containerID="78a54f05417b52aea05df39aaec43512fe3e61889e3930b82e4d41ad58e1f119" Oct 30 01:38:07.224551 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:07.224533 6572 scope.go:122] "RemoveContainer" containerID="78a54f05417b52aea05df39aaec43512fe3e61889e3930b82e4d41ad58e1f119" Oct 30 01:38:07.224941 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:38:07.224919 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78a54f05417b52aea05df39aaec43512fe3e61889e3930b82e4d41ad58e1f119\": container with ID starting with 78a54f05417b52aea05df39aaec43512fe3e61889e3930b82e4d41ad58e1f119 not found: ID does not exist" containerID="78a54f05417b52aea05df39aaec43512fe3e61889e3930b82e4d41ad58e1f119" Oct 30 01:38:07.225035 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:07.224950 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78a54f05417b52aea05df39aaec43512fe3e61889e3930b82e4d41ad58e1f119"} err="failed to get container status \"78a54f05417b52aea05df39aaec43512fe3e61889e3930b82e4d41ad58e1f119\": rpc error: code = NotFound desc = could not find container \"78a54f05417b52aea05df39aaec43512fe3e61889e3930b82e4d41ad58e1f119\": container with ID starting with 78a54f05417b52aea05df39aaec43512fe3e61889e3930b82e4d41ad58e1f119 not found: ID does not exist" Oct 30 01:38:07.259957 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:07.259924 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f09934-9d4445b84-wcp9v"] Oct 30 01:38:07.269447 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:07.269415 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f09934-9d4445b84-wcp9v"] Oct 30 01:38:07.500724 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:07.500623 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:07.721468 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:07.721371 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:07.981773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:07.975362 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:08.464228 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:08.464191 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:08.565957 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:08.565919 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bf5fd44-9bd6-48fe-badd-c1643822e48b" path="/var/lib/kubelet/pods/7bf5fd44-9bd6-48fe-badd-c1643822e48b/volumes" Oct 30 01:38:08.677557 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:08.677522 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:08.891051 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:08.890964 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:09.189997 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.189959 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7"] Oct 30 01:38:09.191040 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.191018 6572 cpu_manager.go:414] "containerMap: removing container" podUID="7bf5fd44-9bd6-48fe-badd-c1643822e48b" containerName="payload-processing" Oct 30 01:38:09.191113 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.191044 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bf5fd44-9bd6-48fe-badd-c1643822e48b" containerName="payload-processing" Oct 30 01:38:09.191113 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.191071 6572 cpu_manager.go:414] "containerMap: removing container" podUID="a618ddde-ca90-4180-a97e-8321e808f93a" containerName="revoke-keys" Oct 30 01:38:09.191113 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.191079 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a618ddde-ca90-4180-a97e-8321e808f93a" containerName="revoke-keys" Oct 30 01:38:09.191303 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.191289 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bf5fd44-9bd6-48fe-badd-c1643822e48b" containerName="payload-processing" Oct 30 01:38:09.191337 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.191311 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="a618ddde-ca90-4180-a97e-8321e808f93a" containerName="revoke-keys" Oct 30 01:38:09.197210 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.197174 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:38:09.218155 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.218128 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-6b3dbd-openshift-default-dockercfg-lbw9c\"" Oct 30 01:38:09.225989 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.225942 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7"] Oct 30 01:38:09.283176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.283137 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-istiod-ca-cert\") pod \"e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7\" (UID: \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\") " pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:38:09.283351 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.283187 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-istio-podinfo\") pod \"e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7\" (UID: \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\") " pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:38:09.283351 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.283265 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fp6t\" (UniqueName: \"kubernetes.io/projected/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-kube-api-access-9fp6t\") pod \"e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7\" (UID: \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\") " pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:38:09.283448 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.283381 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-workload-socket\") pod \"e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7\" (UID: \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\") " pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:38:09.283448 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.283420 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-istio-envoy\") pod \"e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7\" (UID: \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\") " pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:38:09.283604 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.283455 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-istio-data\") pod \"e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7\" (UID: \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\") " pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:38:09.283604 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.283559 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-workload-certs\") pod \"e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7\" (UID: \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\") " pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:38:09.283681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.283614 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-credential-socket\") pod \"e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7\" (UID: \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\") " pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:38:09.283681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.283659 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-istio-token\") pod \"e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7\" (UID: \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\") " pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:38:09.345801 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.345742 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:09.417774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.386727 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-workload-socket\") pod \"e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7\" (UID: \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\") " pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:38:09.417774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.386797 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-istio-envoy\") pod \"e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7\" (UID: \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\") " pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:38:09.417774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.386846 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-istio-data\") pod \"e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7\" (UID: \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\") " pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:38:09.417774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.386933 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-workload-certs\") pod \"e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7\" (UID: \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\") " pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:38:09.417774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.386991 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-credential-socket\") pod \"e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7\" (UID: \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\") " pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:38:09.417774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.391350 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-istio-envoy\") pod \"e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7\" (UID: \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\") " pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:38:09.417774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.391642 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-workload-socket\") pod \"e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7\" (UID: \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\") " pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:38:09.417774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.391688 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-istio-token\") pod \"e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7\" (UID: \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\") " pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:38:09.417774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.391880 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-istiod-ca-cert\") pod \"e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7\" (UID: \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\") " pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:38:09.417774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.391919 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-istio-podinfo\") pod \"e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7\" (UID: \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\") " pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:38:09.417774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.391952 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fp6t\" (UniqueName: \"kubernetes.io/projected/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-kube-api-access-9fp6t\") pod \"e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7\" (UID: \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\") " pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:38:09.417774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.393207 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-istiod-ca-cert\") pod \"e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7\" (UID: \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\") " pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:38:09.417774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.397464 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-credential-socket\") pod \"e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7\" (UID: \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\") " pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:38:09.417774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.397729 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-istio-data\") pod \"e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7\" (UID: \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\") " pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:38:09.417774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.398002 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-workload-certs\") pod \"e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7\" (UID: \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\") " pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:38:09.432487 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.432410 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-istio-podinfo\") pod \"e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7\" (UID: \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\") " pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:38:09.448874 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.448363 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fp6t\" (UniqueName: \"kubernetes.io/projected/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-kube-api-access-9fp6t\") pod \"e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7\" (UID: \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\") " pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:38:09.448874 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.448364 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-istio-token\") pod \"e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7\" (UID: \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\") " pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:38:09.523112 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.523076 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:38:09.949183 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:38:09.948285 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeced80d4_0bf9_4f4e_a25e_0448b3a2f70b.slice/crio-69af19c67252dd2733113e4d762d7625b8e4b1168c726a7e22c170e15e86de7f WatchSource:0}: Error finding container 69af19c67252dd2733113e4d762d7625b8e4b1168c726a7e22c170e15e86de7f: Status 404 returned error can't find the container with id 69af19c67252dd2733113e4d762d7625b8e4b1168c726a7e22c170e15e86de7f Oct 30 01:38:09.949183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.948414 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7"] Oct 30 01:38:09.962266 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.962230 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:38:09.962347 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.962330 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:38:09.962415 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.962370 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:38:09.973824 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:09.971851 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:10.285827 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:10.273398 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" event={"ID":"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b","Type":"ContainerStarted","Data":"6e137d476a869f00b916844a51918cba2c9b8f33aaccdbcba7b970bea4ff74c3"} Oct 30 01:38:10.285827 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:10.273458 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" event={"ID":"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b","Type":"ContainerStarted","Data":"69af19c67252dd2733113e4d762d7625b8e4b1168c726a7e22c170e15e86de7f"} Oct 30 01:38:10.381774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:10.377354 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" podStartSLOduration=1.377327636 podStartE2EDuration="1.377327636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:38:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:38:10.34909043 +0000 UTC m=+1215.925515835" watchObservedRunningTime="2026-10-30 01:38:10.377327636 +0000 UTC m=+1215.953753041" Oct 30 01:38:10.531362 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:10.531326 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:38:10.702480 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:10.702446 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:11.074836 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:11.067764 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbf1ce8b-7ljnq"] Oct 30 01:38:11.094772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:11.087319 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbf1ce8b-7ljnq"] Oct 30 01:38:11.554777 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:11.554657 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:38:12.326776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:12.321818 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:38:12.361109 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:12.361070 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:38:12.688659 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:12.663111 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83968442-b868-4896-84c1-415c42cfbbcc" path="/var/lib/kubelet/pods/83968442-b868-4896-84c1-415c42cfbbcc/volumes" Oct 30 01:38:12.823772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:12.822529 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-nsfcb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:38:12.823772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:12.822599 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" podUID="c1d4d67f-e18a-4152-be41-e1832bfd78ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:38:13.244622 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:13.244577 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv"] Oct 30 01:38:13.244882 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:13.244837 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" podUID="e9462ac3-f977-4dce-8c37-a89b70928484" containerName="istio-proxy" containerID="cri-o://55bf86878ed086665fe9f88163407d0bd96c516f1fefb061687ab54a114d99c3" gracePeriod=30 Oct 30 01:38:13.254691 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:13.254651 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x"] Oct 30 01:38:13.254932 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:13.254900 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" podUID="96ae6bcd-2343-4c65-b4db-f7a5106f51f2" containerName="istio-proxy" containerID="cri-o://b1102e030bb072072f9910974ad6760d1da4f18647dcf5e9a37cc49083430c55" gracePeriod=30 Oct 30 01:38:13.260530 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:13.260488 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd"] Oct 30 01:38:13.260776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:13.260727 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" podUID="46929775-3a67-4bd4-8387-52bab2ebdc0e" containerName="istio-proxy" containerID="cri-o://f10b67b9e2539aad5f526f85f46e89707c690efc2eb79e9d28c08d7bdcbd28e3" gracePeriod=30 Oct 30 01:38:13.265429 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:13.265386 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w"] Oct 30 01:38:13.265702 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:13.265639 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" podUID="6ba2dcf2-ec73-413b-8190-5e2b31cd692c" containerName="istio-proxy" containerID="cri-o://38e8aa2bb06b9712c749767dca3d4fe5b92e43db83d6cd6c636d3050a28718a0" gracePeriod=30 Oct 30 01:38:13.407034 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:13.406990 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:13.753383 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:13.753351 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:14.008783 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:14.008507 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:14.689893 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:38:14.689852 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:38:15.033952 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:15.033918 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a09d277a-stkvg"] Oct 30 01:38:15.043999 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:15.043965 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a09d277a-stkvg"] Oct 30 01:38:15.323217 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:15.323117 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:15.642978 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:15.642879 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:16.224982 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:16.224938 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:16.545452 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:16.545366 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eee5e778-212f-432e-8598-d27a840394e9" path="/var/lib/kubelet/pods/eee5e778-212f-432e-8598-d27a840394e9/volumes" Oct 30 01:38:18.030762 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.030719 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c84c4f53-rmhlb"] Oct 30 01:38:18.040574 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.040536 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c84c4f53-rmhlb"] Oct 30 01:38:18.381074 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.380034 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:18.406806 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.404417 6572 generic.go:356] "Generic (PLEG): container finished" podID="46929775-3a67-4bd4-8387-52bab2ebdc0e" containerID="f10b67b9e2539aad5f526f85f46e89707c690efc2eb79e9d28c08d7bdcbd28e3" exitCode=0 Oct 30 01:38:18.406806 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.404450 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" event={"ID":"46929775-3a67-4bd4-8387-52bab2ebdc0e","Type":"ContainerDied","Data":"f10b67b9e2539aad5f526f85f46e89707c690efc2eb79e9d28c08d7bdcbd28e3"} Oct 30 01:38:18.409004 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.408969 6572 generic.go:356] "Generic (PLEG): container finished" podID="96ae6bcd-2343-4c65-b4db-f7a5106f51f2" containerID="b1102e030bb072072f9910974ad6760d1da4f18647dcf5e9a37cc49083430c55" exitCode=0 Oct 30 01:38:18.409226 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.409199 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" event={"ID":"96ae6bcd-2343-4c65-b4db-f7a5106f51f2","Type":"ContainerDied","Data":"b1102e030bb072072f9910974ad6760d1da4f18647dcf5e9a37cc49083430c55"} Oct 30 01:38:18.411610 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.411580 6572 generic.go:356] "Generic (PLEG): container finished" podID="e9462ac3-f977-4dce-8c37-a89b70928484" containerID="55bf86878ed086665fe9f88163407d0bd96c516f1fefb061687ab54a114d99c3" exitCode=0 Oct 30 01:38:18.411983 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.411956 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" event={"ID":"e9462ac3-f977-4dce-8c37-a89b70928484","Type":"ContainerDied","Data":"55bf86878ed086665fe9f88163407d0bd96c516f1fefb061687ab54a114d99c3"} Oct 30 01:38:18.547088 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.546998 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b45c8424-ddb7-4018-9bc4-cf7986fe93b5" path="/var/lib/kubelet/pods/b45c8424-ddb7-4018-9bc4-cf7986fe93b5/volumes" Oct 30 01:38:18.589674 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.587991 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" Oct 30 01:38:18.612632 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.612600 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" Oct 30 01:38:18.673666 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.673636 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" Oct 30 01:38:18.679656 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.679622 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" Oct 30 01:38:18.699863 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.699825 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46929775-3a67-4bd4-8387-52bab2ebdc0e-kube-api-access-cmwt4\" (UniqueName: \"kubernetes.io/projected/46929775-3a67-4bd4-8387-52bab2ebdc0e-kube-api-access-cmwt4\") pod \"46929775-3a67-4bd4-8387-52bab2ebdc0e\" (UID: \"46929775-3a67-4bd4-8387-52bab2ebdc0e\") " Oct 30 01:38:18.700062 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.699881 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46929775-3a67-4bd4-8387-52bab2ebdc0e-istio-token\" (UniqueName: \"kubernetes.io/projected/46929775-3a67-4bd4-8387-52bab2ebdc0e-istio-token\") pod \"46929775-3a67-4bd4-8387-52bab2ebdc0e\" (UID: \"46929775-3a67-4bd4-8387-52bab2ebdc0e\") " Oct 30 01:38:18.700062 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.699958 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-istio-data\") pod \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\" (UID: \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\") " Oct 30 01:38:18.700062 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.700035 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46929775-3a67-4bd4-8387-52bab2ebdc0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46929775-3a67-4bd4-8387-52bab2ebdc0e-workload-socket\") pod \"46929775-3a67-4bd4-8387-52bab2ebdc0e\" (UID: \"46929775-3a67-4bd4-8387-52bab2ebdc0e\") " Oct 30 01:38:18.700268 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.700089 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-istio-envoy\") pod \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\" (UID: \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\") " Oct 30 01:38:18.700268 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.700131 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-workload-socket\") pod \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\" (UID: \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\") " Oct 30 01:38:18.700268 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.700212 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46929775-3a67-4bd4-8387-52bab2ebdc0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46929775-3a67-4bd4-8387-52bab2ebdc0e-istio-data\") pod \"46929775-3a67-4bd4-8387-52bab2ebdc0e\" (UID: \"46929775-3a67-4bd4-8387-52bab2ebdc0e\") " Oct 30 01:38:18.700454 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.700266 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-istio-podinfo\") pod \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\" (UID: \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\") " Oct 30 01:38:18.700454 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.700322 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-istio-data" pod "96ae6bcd-2343-4c65-b4db-f7a5106f51f2" (UID: "96ae6bcd-2343-4c65-b4db-f7a5106f51f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:18.700454 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.700363 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-workload-certs\") pod \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\" (UID: \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\") " Oct 30 01:38:18.700454 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.700392 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46929775-3a67-4bd4-8387-52bab2ebdc0e-workload-socket" pod "46929775-3a67-4bd4-8387-52bab2ebdc0e" (UID: "46929775-3a67-4bd4-8387-52bab2ebdc0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:18.700454 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.700439 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46929775-3a67-4bd4-8387-52bab2ebdc0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46929775-3a67-4bd4-8387-52bab2ebdc0e-istio-podinfo\") pod \"46929775-3a67-4bd4-8387-52bab2ebdc0e\" (UID: \"46929775-3a67-4bd4-8387-52bab2ebdc0e\") " Oct 30 01:38:18.700738 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.700493 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46929775-3a67-4bd4-8387-52bab2ebdc0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46929775-3a67-4bd4-8387-52bab2ebdc0e-credential-socket\") pod \"46929775-3a67-4bd4-8387-52bab2ebdc0e\" (UID: \"46929775-3a67-4bd4-8387-52bab2ebdc0e\") " Oct 30 01:38:18.700738 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.700534 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46929775-3a67-4bd4-8387-52bab2ebdc0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46929775-3a67-4bd4-8387-52bab2ebdc0e-istiod-ca-cert\") pod \"46929775-3a67-4bd4-8387-52bab2ebdc0e\" (UID: \"46929775-3a67-4bd4-8387-52bab2ebdc0e\") " Oct 30 01:38:18.700738 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.700542 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-workload-socket" pod "96ae6bcd-2343-4c65-b4db-f7a5106f51f2" (UID: "96ae6bcd-2343-4c65-b4db-f7a5106f51f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:18.700738 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.700569 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-kube-api-access-4p5ff\" (UniqueName: \"kubernetes.io/projected/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-kube-api-access-4p5ff\") pod \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\" (UID: \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\") " Oct 30 01:38:18.700738 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.700617 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46929775-3a67-4bd4-8387-52bab2ebdc0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46929775-3a67-4bd4-8387-52bab2ebdc0e-istio-envoy\") pod \"46929775-3a67-4bd4-8387-52bab2ebdc0e\" (UID: \"46929775-3a67-4bd4-8387-52bab2ebdc0e\") " Oct 30 01:38:18.700738 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.700651 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-istio-token\" (UniqueName: \"kubernetes.io/projected/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-istio-token\") pod \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\" (UID: \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\") " Oct 30 01:38:18.700738 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.700681 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-istiod-ca-cert\") pod \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\" (UID: \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\") " Oct 30 01:38:18.700738 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.700736 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-workload-certs" pod "96ae6bcd-2343-4c65-b4db-f7a5106f51f2" (UID: "96ae6bcd-2343-4c65-b4db-f7a5106f51f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:18.701172 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.700790 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-credential-socket\") pod \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\" (UID: \"96ae6bcd-2343-4c65-b4db-f7a5106f51f2\") " Oct 30 01:38:18.701172 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.700854 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46929775-3a67-4bd4-8387-52bab2ebdc0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46929775-3a67-4bd4-8387-52bab2ebdc0e-workload-certs\") pod \"46929775-3a67-4bd4-8387-52bab2ebdc0e\" (UID: \"46929775-3a67-4bd4-8387-52bab2ebdc0e\") " Oct 30 01:38:18.701172 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.700924 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46929775-3a67-4bd4-8387-52bab2ebdc0e-istio-data" pod "46929775-3a67-4bd4-8387-52bab2ebdc0e" (UID: "46929775-3a67-4bd4-8387-52bab2ebdc0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:18.702879 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.701792 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.702879 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.701822 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46929775-3a67-4bd4-8387-52bab2ebdc0e-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.702879 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.701837 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.702879 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.701852 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46929775-3a67-4bd4-8387-52bab2ebdc0e-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.702879 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.701866 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.702879 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.702039 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46929775-3a67-4bd4-8387-52bab2ebdc0e-credential-socket" pod "46929775-3a67-4bd4-8387-52bab2ebdc0e" (UID: "46929775-3a67-4bd4-8387-52bab2ebdc0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:18.702879 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.702315 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-credential-socket" pod "96ae6bcd-2343-4c65-b4db-f7a5106f51f2" (UID: "96ae6bcd-2343-4c65-b4db-f7a5106f51f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:18.702879 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.702541 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-istiod-ca-cert" pod "96ae6bcd-2343-4c65-b4db-f7a5106f51f2" (UID: "96ae6bcd-2343-4c65-b4db-f7a5106f51f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:18.702879 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.702561 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46929775-3a67-4bd4-8387-52bab2ebdc0e-workload-certs" pod "46929775-3a67-4bd4-8387-52bab2ebdc0e" (UID: "46929775-3a67-4bd4-8387-52bab2ebdc0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:18.703603 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.702885 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46929775-3a67-4bd4-8387-52bab2ebdc0e-istio-token" pod "46929775-3a67-4bd4-8387-52bab2ebdc0e" (UID: "46929775-3a67-4bd4-8387-52bab2ebdc0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:18.703603 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.702946 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-istio-envoy" pod "96ae6bcd-2343-4c65-b4db-f7a5106f51f2" (UID: "96ae6bcd-2343-4c65-b4db-f7a5106f51f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:18.703603 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.703260 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46929775-3a67-4bd4-8387-52bab2ebdc0e-istiod-ca-cert" pod "46929775-3a67-4bd4-8387-52bab2ebdc0e" (UID: "46929775-3a67-4bd4-8387-52bab2ebdc0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:18.704086 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.704057 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46929775-3a67-4bd4-8387-52bab2ebdc0e-kube-api-access-cmwt4" pod "46929775-3a67-4bd4-8387-52bab2ebdc0e" (UID: "46929775-3a67-4bd4-8387-52bab2ebdc0e"). InnerVolumeSpecName "kube-api-access-cmwt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:18.704206 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.704152 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-istio-podinfo" pod "96ae6bcd-2343-4c65-b4db-f7a5106f51f2" (UID: "96ae6bcd-2343-4c65-b4db-f7a5106f51f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:38:18.705005 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.704977 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46929775-3a67-4bd4-8387-52bab2ebdc0e-istio-envoy" pod "46929775-3a67-4bd4-8387-52bab2ebdc0e" (UID: "46929775-3a67-4bd4-8387-52bab2ebdc0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:18.705005 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.704989 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46929775-3a67-4bd4-8387-52bab2ebdc0e-istio-podinfo" pod "46929775-3a67-4bd4-8387-52bab2ebdc0e" (UID: "46929775-3a67-4bd4-8387-52bab2ebdc0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:38:18.705425 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.705395 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-kube-api-access-4p5ff" pod "96ae6bcd-2343-4c65-b4db-f7a5106f51f2" (UID: "96ae6bcd-2343-4c65-b4db-f7a5106f51f2"). InnerVolumeSpecName "kube-api-access-4p5ff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:18.706178 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.706156 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-istio-token" pod "96ae6bcd-2343-4c65-b4db-f7a5106f51f2" (UID: "96ae6bcd-2343-4c65-b4db-f7a5106f51f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:18.802699 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.802605 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-istiod-ca-cert\") pod \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\" (UID: \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\") " Oct 30 01:38:18.802699 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.802650 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9462ac3-f977-4dce-8c37-a89b70928484-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9462ac3-f977-4dce-8c37-a89b70928484-istiod-ca-cert\") pod \"e9462ac3-f977-4dce-8c37-a89b70928484\" (UID: \"e9462ac3-f977-4dce-8c37-a89b70928484\") " Oct 30 01:38:18.802699 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.802692 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9462ac3-f977-4dce-8c37-a89b70928484-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9462ac3-f977-4dce-8c37-a89b70928484-istio-envoy\") pod \"e9462ac3-f977-4dce-8c37-a89b70928484\" (UID: \"e9462ac3-f977-4dce-8c37-a89b70928484\") " Oct 30 01:38:18.803025 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.802735 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9462ac3-f977-4dce-8c37-a89b70928484-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9462ac3-f977-4dce-8c37-a89b70928484-istio-podinfo\") pod \"e9462ac3-f977-4dce-8c37-a89b70928484\" (UID: \"e9462ac3-f977-4dce-8c37-a89b70928484\") " Oct 30 01:38:18.803025 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.802819 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-istio-podinfo\") pod \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\" (UID: \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\") " Oct 30 01:38:18.803025 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.802898 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-istio-envoy\") pod \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\" (UID: \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\") " Oct 30 01:38:18.803025 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.802933 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9462ac3-f977-4dce-8c37-a89b70928484-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9462ac3-f977-4dce-8c37-a89b70928484-credential-socket\") pod \"e9462ac3-f977-4dce-8c37-a89b70928484\" (UID: \"e9462ac3-f977-4dce-8c37-a89b70928484\") " Oct 30 01:38:18.803025 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.802969 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9462ac3-f977-4dce-8c37-a89b70928484-kube-api-access-gdd9k\" (UniqueName: \"kubernetes.io/projected/e9462ac3-f977-4dce-8c37-a89b70928484-kube-api-access-gdd9k\") pod \"e9462ac3-f977-4dce-8c37-a89b70928484\" (UID: \"e9462ac3-f977-4dce-8c37-a89b70928484\") " Oct 30 01:38:18.803291 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.803042 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-workload-socket\") pod \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\" (UID: \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\") " Oct 30 01:38:18.803291 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.803094 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9462ac3-f977-4dce-8c37-a89b70928484-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9462ac3-f977-4dce-8c37-a89b70928484-workload-certs\") pod \"e9462ac3-f977-4dce-8c37-a89b70928484\" (UID: \"e9462ac3-f977-4dce-8c37-a89b70928484\") " Oct 30 01:38:18.803291 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.803143 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-workload-certs\") pod \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\" (UID: \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\") " Oct 30 01:38:18.803291 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.803212 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-istio-token\" (UniqueName: \"kubernetes.io/projected/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-istio-token\") pod \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\" (UID: \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\") " Oct 30 01:38:18.803948 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.803087 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-istiod-ca-cert" pod "6ba2dcf2-ec73-413b-8190-5e2b31cd692c" (UID: "6ba2dcf2-ec73-413b-8190-5e2b31cd692c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:18.803948 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.803112 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9462ac3-f977-4dce-8c37-a89b70928484-istiod-ca-cert" pod "e9462ac3-f977-4dce-8c37-a89b70928484" (UID: "e9462ac3-f977-4dce-8c37-a89b70928484"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:18.803948 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.803243 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9462ac3-f977-4dce-8c37-a89b70928484-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9462ac3-f977-4dce-8c37-a89b70928484-istio-data\") pod \"e9462ac3-f977-4dce-8c37-a89b70928484\" (UID: \"e9462ac3-f977-4dce-8c37-a89b70928484\") " Oct 30 01:38:18.803948 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.803505 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9462ac3-f977-4dce-8c37-a89b70928484-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9462ac3-f977-4dce-8c37-a89b70928484-workload-socket\") pod \"e9462ac3-f977-4dce-8c37-a89b70928484\" (UID: \"e9462ac3-f977-4dce-8c37-a89b70928484\") " Oct 30 01:38:18.803948 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.803557 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-credential-socket\") pod \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\" (UID: \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\") " Oct 30 01:38:18.803948 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.803630 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9462ac3-f977-4dce-8c37-a89b70928484-istio-token\" (UniqueName: \"kubernetes.io/projected/e9462ac3-f977-4dce-8c37-a89b70928484-istio-token\") pod \"e9462ac3-f977-4dce-8c37-a89b70928484\" (UID: \"e9462ac3-f977-4dce-8c37-a89b70928484\") " Oct 30 01:38:18.803948 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.803422 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-workload-socket" pod "6ba2dcf2-ec73-413b-8190-5e2b31cd692c" (UID: "6ba2dcf2-ec73-413b-8190-5e2b31cd692c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:18.803948 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.803434 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9462ac3-f977-4dce-8c37-a89b70928484-istio-data" pod "e9462ac3-f977-4dce-8c37-a89b70928484" (UID: "e9462ac3-f977-4dce-8c37-a89b70928484"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:18.803948 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.803628 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-workload-certs" pod "6ba2dcf2-ec73-413b-8190-5e2b31cd692c" (UID: "6ba2dcf2-ec73-413b-8190-5e2b31cd692c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:18.803948 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.803671 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-kube-api-access-tn5n8\" (UniqueName: \"kubernetes.io/projected/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-kube-api-access-tn5n8\") pod \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\" (UID: \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\") " Oct 30 01:38:18.803948 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.803735 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-istio-data\") pod \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\" (UID: \"6ba2dcf2-ec73-413b-8190-5e2b31cd692c\") " Oct 30 01:38:18.803948 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.803635 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9462ac3-f977-4dce-8c37-a89b70928484-credential-socket" pod "e9462ac3-f977-4dce-8c37-a89b70928484" (UID: "e9462ac3-f977-4dce-8c37-a89b70928484"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:18.804652 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.803886 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9462ac3-f977-4dce-8c37-a89b70928484-workload-certs" pod "e9462ac3-f977-4dce-8c37-a89b70928484" (UID: "e9462ac3-f977-4dce-8c37-a89b70928484"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:18.804652 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.803940 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9462ac3-f977-4dce-8c37-a89b70928484-workload-socket" pod "e9462ac3-f977-4dce-8c37-a89b70928484" (UID: "e9462ac3-f977-4dce-8c37-a89b70928484"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:18.804652 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.804248 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-credential-socket" pod "6ba2dcf2-ec73-413b-8190-5e2b31cd692c" (UID: "6ba2dcf2-ec73-413b-8190-5e2b31cd692c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:18.804652 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.804302 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-istio-data" pod "6ba2dcf2-ec73-413b-8190-5e2b31cd692c" (UID: "6ba2dcf2-ec73-413b-8190-5e2b31cd692c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:18.804900 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.804759 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.804900 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.804783 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46929775-3a67-4bd4-8387-52bab2ebdc0e-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.804900 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.804798 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.804900 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.804812 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.804900 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.804828 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9462ac3-f977-4dce-8c37-a89b70928484-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.804900 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.804841 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.804900 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.804854 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.804900 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.804868 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46929775-3a67-4bd4-8387-52bab2ebdc0e-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.804900 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.804880 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9462ac3-f977-4dce-8c37-a89b70928484-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.804900 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.804893 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9462ac3-f977-4dce-8c37-a89b70928484-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.805507 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.804906 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.805507 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.804923 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmwt4\" (UniqueName: \"kubernetes.io/projected/46929775-3a67-4bd4-8387-52bab2ebdc0e-kube-api-access-cmwt4\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.805507 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.804937 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46929775-3a67-4bd4-8387-52bab2ebdc0e-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.805507 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.804951 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.805507 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.804963 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.805507 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.804976 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9462ac3-f977-4dce-8c37-a89b70928484-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.805507 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.804990 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.805507 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.805003 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.805507 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.805015 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9462ac3-f977-4dce-8c37-a89b70928484-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.805507 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.805030 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46929775-3a67-4bd4-8387-52bab2ebdc0e-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.805507 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.805044 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46929775-3a67-4bd4-8387-52bab2ebdc0e-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.805507 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.805057 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46929775-3a67-4bd4-8387-52bab2ebdc0e-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.805507 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.805073 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p5ff\" (UniqueName: \"kubernetes.io/projected/96ae6bcd-2343-4c65-b4db-f7a5106f51f2-kube-api-access-4p5ff\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.806209 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.805707 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9462ac3-f977-4dce-8c37-a89b70928484-kube-api-access-gdd9k" pod "e9462ac3-f977-4dce-8c37-a89b70928484" (UID: "e9462ac3-f977-4dce-8c37-a89b70928484"). InnerVolumeSpecName "kube-api-access-gdd9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:18.806303 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.806266 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-istio-podinfo" pod "6ba2dcf2-ec73-413b-8190-5e2b31cd692c" (UID: "6ba2dcf2-ec73-413b-8190-5e2b31cd692c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:38:18.806370 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.806302 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9462ac3-f977-4dce-8c37-a89b70928484-istio-envoy" pod "e9462ac3-f977-4dce-8c37-a89b70928484" (UID: "e9462ac3-f977-4dce-8c37-a89b70928484"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:18.806644 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.806619 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9462ac3-f977-4dce-8c37-a89b70928484-istio-podinfo" pod "e9462ac3-f977-4dce-8c37-a89b70928484" (UID: "e9462ac3-f977-4dce-8c37-a89b70928484"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:38:18.806865 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.806840 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-kube-api-access-tn5n8" pod "6ba2dcf2-ec73-413b-8190-5e2b31cd692c" (UID: "6ba2dcf2-ec73-413b-8190-5e2b31cd692c"). InnerVolumeSpecName "kube-api-access-tn5n8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:18.806961 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.806876 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-istio-envoy" pod "6ba2dcf2-ec73-413b-8190-5e2b31cd692c" (UID: "6ba2dcf2-ec73-413b-8190-5e2b31cd692c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:18.807054 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.807030 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-istio-token" pod "6ba2dcf2-ec73-413b-8190-5e2b31cd692c" (UID: "6ba2dcf2-ec73-413b-8190-5e2b31cd692c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:18.807677 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.807659 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9462ac3-f977-4dce-8c37-a89b70928484-istio-token" pod "e9462ac3-f977-4dce-8c37-a89b70928484" (UID: "e9462ac3-f977-4dce-8c37-a89b70928484"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:18.906983 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.906942 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9462ac3-f977-4dce-8c37-a89b70928484-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.906983 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.906976 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.906983 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.906987 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.907229 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.906997 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdd9k\" (UniqueName: \"kubernetes.io/projected/e9462ac3-f977-4dce-8c37-a89b70928484-kube-api-access-gdd9k\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.907229 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.907006 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.907229 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.907018 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9462ac3-f977-4dce-8c37-a89b70928484-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.907229 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.907031 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn5n8\" (UniqueName: \"kubernetes.io/projected/6ba2dcf2-ec73-413b-8190-5e2b31cd692c-kube-api-access-tn5n8\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:18.907229 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:18.907039 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9462ac3-f977-4dce-8c37-a89b70928484-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:19.422064 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:19.422029 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" Oct 30 01:38:19.422064 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:19.422038 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv" event={"ID":"e9462ac3-f977-4dce-8c37-a89b70928484","Type":"ContainerDied","Data":"0b961230497d339260ad2e300c508d126061f87fb65d625bfe140d2938e30d39"} Oct 30 01:38:19.422627 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:19.422099 6572 scope.go:122] "RemoveContainer" containerID="55bf86878ed086665fe9f88163407d0bd96c516f1fefb061687ab54a114d99c3" Oct 30 01:38:19.424187 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:19.424160 6572 generic.go:356] "Generic (PLEG): container finished" podID="6ba2dcf2-ec73-413b-8190-5e2b31cd692c" containerID="38e8aa2bb06b9712c749767dca3d4fe5b92e43db83d6cd6c636d3050a28718a0" exitCode=0 Oct 30 01:38:19.424290 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:19.424241 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" Oct 30 01:38:19.424290 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:19.424245 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" event={"ID":"6ba2dcf2-ec73-413b-8190-5e2b31cd692c","Type":"ContainerDied","Data":"38e8aa2bb06b9712c749767dca3d4fe5b92e43db83d6cd6c636d3050a28718a0"} Oct 30 01:38:19.424408 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:19.424312 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w" event={"ID":"6ba2dcf2-ec73-413b-8190-5e2b31cd692c","Type":"ContainerDied","Data":"d9b30d1ab93cdb9085f76f065445ae5dd1c1733f5902050f755863d1781e5981"} Oct 30 01:38:19.426369 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:19.426029 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" Oct 30 01:38:19.426369 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:19.426039 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd" event={"ID":"46929775-3a67-4bd4-8387-52bab2ebdc0e","Type":"ContainerDied","Data":"f4cb6d6c0031de2eda37b667f39d667f5601a2d2200e029ea6baed28d95d8e92"} Oct 30 01:38:19.427788 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:19.427738 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" event={"ID":"96ae6bcd-2343-4c65-b4db-f7a5106f51f2","Type":"ContainerDied","Data":"fe39239b0984f2c2fc8032ff5de2a403fe87d2784dbe2dc07c2578ce9232cfdf"} Oct 30 01:38:19.427883 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:19.427803 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x" Oct 30 01:38:19.438580 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:19.438559 6572 scope.go:122] "RemoveContainer" containerID="38e8aa2bb06b9712c749767dca3d4fe5b92e43db83d6cd6c636d3050a28718a0" Oct 30 01:38:19.461516 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:19.461488 6572 scope.go:122] "RemoveContainer" containerID="38e8aa2bb06b9712c749767dca3d4fe5b92e43db83d6cd6c636d3050a28718a0" Oct 30 01:38:19.461910 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:38:19.461879 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38e8aa2bb06b9712c749767dca3d4fe5b92e43db83d6cd6c636d3050a28718a0\": container with ID starting with 38e8aa2bb06b9712c749767dca3d4fe5b92e43db83d6cd6c636d3050a28718a0 not found: ID does not exist" containerID="38e8aa2bb06b9712c749767dca3d4fe5b92e43db83d6cd6c636d3050a28718a0" Oct 30 01:38:19.461983 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:19.461925 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38e8aa2bb06b9712c749767dca3d4fe5b92e43db83d6cd6c636d3050a28718a0"} err="failed to get container status \"38e8aa2bb06b9712c749767dca3d4fe5b92e43db83d6cd6c636d3050a28718a0\": rpc error: code = NotFound desc = could not find container \"38e8aa2bb06b9712c749767dca3d4fe5b92e43db83d6cd6c636d3050a28718a0\": container with ID starting with 38e8aa2bb06b9712c749767dca3d4fe5b92e43db83d6cd6c636d3050a28718a0 not found: ID does not exist" Oct 30 01:38:19.461983 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:19.461951 6572 scope.go:122] "RemoveContainer" containerID="f10b67b9e2539aad5f526f85f46e89707c690efc2eb79e9d28c08d7bdcbd28e3" Oct 30 01:38:19.473507 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:19.473478 6572 scope.go:122] "RemoveContainer" containerID="b1102e030bb072072f9910974ad6760d1da4f18647dcf5e9a37cc49083430c55" Oct 30 01:38:19.502093 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:19.502051 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv"] Oct 30 01:38:19.513269 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:19.513227 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-f8bsv"] Oct 30 01:38:19.537827 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:19.537667 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w"] Oct 30 01:38:19.543006 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:19.542960 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-fdz5w"] Oct 30 01:38:19.575492 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:19.575331 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd"] Oct 30 01:38:19.579927 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:19.579896 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-whkbd"] Oct 30 01:38:19.613846 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:19.613805 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x"] Oct 30 01:38:19.620872 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:19.620836 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-xq97x"] Oct 30 01:38:20.159419 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:20.159386 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:20.544241 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:20.544210 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46929775-3a67-4bd4-8387-52bab2ebdc0e" path="/var/lib/kubelet/pods/46929775-3a67-4bd4-8387-52bab2ebdc0e/volumes" Oct 30 01:38:20.544697 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:20.544683 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ba2dcf2-ec73-413b-8190-5e2b31cd692c" path="/var/lib/kubelet/pods/6ba2dcf2-ec73-413b-8190-5e2b31cd692c/volumes" Oct 30 01:38:20.545186 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:20.545172 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96ae6bcd-2343-4c65-b4db-f7a5106f51f2" path="/var/lib/kubelet/pods/96ae6bcd-2343-4c65-b4db-f7a5106f51f2/volumes" Oct 30 01:38:20.545935 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:20.545921 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9462ac3-f977-4dce-8c37-a89b70928484" path="/var/lib/kubelet/pods/e9462ac3-f977-4dce-8c37-a89b70928484/volumes" Oct 30 01:38:21.257955 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:21.257917 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-799dc77c98-g76nh"] Oct 30 01:38:21.291105 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:21.291071 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-6fb94d5599-46prg"] Oct 30 01:38:21.492174 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:21.492139 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:21.726388 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:21.726350 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:22.198880 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:22.198843 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:23.850649 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:23.850613 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:23.926320 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:23.926272 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7668f5d547-dxb9b"] Oct 30 01:38:24.133222 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:24.133134 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:24.335369 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:24.335337 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:25.738701 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:25.738663 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:26.258917 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:26.258869 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-799dc77c98-g76nh" podUID="1beef457-ba1a-437d-aa79-a625c5032b19" containerName="payload-processing" containerID="cri-o://de09413faeed12daeae145c36d3d3a1fbcb1b282916b2dc56008cfdec6a6383d" gracePeriod=25 Oct 30 01:38:26.291718 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:26.291680 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-6fb94d5599-46prg" podUID="a8ce41ba-cb52-4d64-8b85-d420b3d96711" containerName="payload-pre-processing" containerID="cri-o://f19b5f03e03fed67f170b45a426f6761d15a8f183decef5917d198bfaa5e3fc8" gracePeriod=25 Oct 30 01:38:26.506299 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:26.506257 6572 generic.go:356] "Generic (PLEG): container finished" podID="1beef457-ba1a-437d-aa79-a625c5032b19" containerID="de09413faeed12daeae145c36d3d3a1fbcb1b282916b2dc56008cfdec6a6383d" exitCode=0 Oct 30 01:38:26.506500 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:26.506422 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-799dc77c98-g76nh" event={"ID":"1beef457-ba1a-437d-aa79-a625c5032b19","Type":"ContainerDied","Data":"de09413faeed12daeae145c36d3d3a1fbcb1b282916b2dc56008cfdec6a6383d"} Oct 30 01:38:26.508672 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:26.508640 6572 generic.go:356] "Generic (PLEG): container finished" podID="a8ce41ba-cb52-4d64-8b85-d420b3d96711" containerID="f19b5f03e03fed67f170b45a426f6761d15a8f183decef5917d198bfaa5e3fc8" exitCode=0 Oct 30 01:38:26.509009 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:26.508956 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-6fb94d5599-46prg" event={"ID":"a8ce41ba-cb52-4d64-8b85-d420b3d96711","Type":"ContainerDied","Data":"f19b5f03e03fed67f170b45a426f6761d15a8f183decef5917d198bfaa5e3fc8"} Oct 30 01:38:26.575940 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:26.575910 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-799dc77c98-g76nh" Oct 30 01:38:26.611470 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:26.611443 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-6fb94d5599-46prg" Oct 30 01:38:26.633036 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:26.632978 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1beef457-ba1a-437d-aa79-a625c5032b19-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1beef457-ba1a-437d-aa79-a625c5032b19-plugins-config-volume\") pod \"1beef457-ba1a-437d-aa79-a625c5032b19\" (UID: \"1beef457-ba1a-437d-aa79-a625c5032b19\") " Oct 30 01:38:26.633218 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:26.633169 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1beef457-ba1a-437d-aa79-a625c5032b19-kube-api-access-jzc7t\" (UniqueName: \"kubernetes.io/projected/1beef457-ba1a-437d-aa79-a625c5032b19-kube-api-access-jzc7t\") pod \"1beef457-ba1a-437d-aa79-a625c5032b19\" (UID: \"1beef457-ba1a-437d-aa79-a625c5032b19\") " Oct 30 01:38:26.633326 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:26.633305 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1beef457-ba1a-437d-aa79-a625c5032b19-plugins-config-volume" pod "1beef457-ba1a-437d-aa79-a625c5032b19" (UID: "1beef457-ba1a-437d-aa79-a625c5032b19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:26.633741 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:26.633717 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1beef457-ba1a-437d-aa79-a625c5032b19-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:26.635424 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:26.635394 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1beef457-ba1a-437d-aa79-a625c5032b19-kube-api-access-jzc7t" pod "1beef457-ba1a-437d-aa79-a625c5032b19" (UID: "1beef457-ba1a-437d-aa79-a625c5032b19"). InnerVolumeSpecName "kube-api-access-jzc7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:26.731445 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:26.731405 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg"] Oct 30 01:38:26.731693 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:26.731664 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" podUID="29807745-a828-46e6-ab0e-994057d74363" containerName="istio-proxy" containerID="cri-o://50f5f648e8fcbb3767a2ff2447a53fce4196ed835371402653a7bfff9939e1b2" gracePeriod=30 Oct 30 01:38:26.734654 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:26.734616 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8ce41ba-cb52-4d64-8b85-d420b3d96711-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8ce41ba-cb52-4d64-8b85-d420b3d96711-plugins-config-volume\") pod \"a8ce41ba-cb52-4d64-8b85-d420b3d96711\" (UID: \"a8ce41ba-cb52-4d64-8b85-d420b3d96711\") " Oct 30 01:38:26.734811 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:26.734727 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8ce41ba-cb52-4d64-8b85-d420b3d96711-kube-api-access-4bx44\" (UniqueName: \"kubernetes.io/projected/a8ce41ba-cb52-4d64-8b85-d420b3d96711-kube-api-access-4bx44\") pod \"a8ce41ba-cb52-4d64-8b85-d420b3d96711\" (UID: \"a8ce41ba-cb52-4d64-8b85-d420b3d96711\") " Oct 30 01:38:26.735104 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:26.735077 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8ce41ba-cb52-4d64-8b85-d420b3d96711-plugins-config-volume" pod "a8ce41ba-cb52-4d64-8b85-d420b3d96711" (UID: "a8ce41ba-cb52-4d64-8b85-d420b3d96711"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:26.735383 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:26.735362 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8ce41ba-cb52-4d64-8b85-d420b3d96711-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:26.735466 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:26.735388 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzc7t\" (UniqueName: \"kubernetes.io/projected/1beef457-ba1a-437d-aa79-a625c5032b19-kube-api-access-jzc7t\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:26.736830 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:26.736806 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8ce41ba-cb52-4d64-8b85-d420b3d96711-kube-api-access-4bx44" pod "a8ce41ba-cb52-4d64-8b85-d420b3d96711" (UID: "a8ce41ba-cb52-4d64-8b85-d420b3d96711"). InnerVolumeSpecName "kube-api-access-4bx44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:26.836468 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:26.836370 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bx44\" (UniqueName: \"kubernetes.io/projected/a8ce41ba-cb52-4d64-8b85-d420b3d96711-kube-api-access-4bx44\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:27.521184 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:27.521149 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-799dc77c98-g76nh" Oct 30 01:38:27.521421 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:27.521148 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-799dc77c98-g76nh" event={"ID":"1beef457-ba1a-437d-aa79-a625c5032b19","Type":"ContainerDied","Data":"7c163fd74a0ddd39421463d71b541495ef3176edb0b2457f236f0737c8e9d24e"} Oct 30 01:38:27.521421 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:27.521378 6572 scope.go:122] "RemoveContainer" containerID="de09413faeed12daeae145c36d3d3a1fbcb1b282916b2dc56008cfdec6a6383d" Oct 30 01:38:27.523675 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:27.523636 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-6fb94d5599-46prg" event={"ID":"a8ce41ba-cb52-4d64-8b85-d420b3d96711","Type":"ContainerDied","Data":"e5fab0abd094e6ada0c34b9571e2c5c627d868a09fc188f5229d7a5cca475a90"} Oct 30 01:38:27.523873 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:27.523653 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-6fb94d5599-46prg" Oct 30 01:38:27.542644 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:27.542611 6572 scope.go:122] "RemoveContainer" containerID="f19b5f03e03fed67f170b45a426f6761d15a8f183decef5917d198bfaa5e3fc8" Oct 30 01:38:27.544476 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:27.544441 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:27.589007 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:27.588968 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-799dc77c98-g76nh"] Oct 30 01:38:27.598901 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:27.598865 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-799dc77c98-g76nh"] Oct 30 01:38:27.632185 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:27.632148 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-6fb94d5599-46prg"] Oct 30 01:38:27.635555 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:27.635530 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-6fb94d5599-46prg"] Oct 30 01:38:28.546676 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:28.544907 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1beef457-ba1a-437d-aa79-a625c5032b19" path="/var/lib/kubelet/pods/1beef457-ba1a-437d-aa79-a625c5032b19/volumes" Oct 30 01:38:28.546676 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:28.545516 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8ce41ba-cb52-4d64-8b85-d420b3d96711" path="/var/lib/kubelet/pods/a8ce41ba-cb52-4d64-8b85-d420b3d96711/volumes" Oct 30 01:38:28.926874 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:28.926827 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7668f5d547-dxb9b" podUID="1a147d7d-b019-4960-9813-9cc1495d9901" containerName="payload-pre-processing" containerID="cri-o://8ed19ac9d4a09bf20c4eee39e915f2b59f7e6f1f29d09cae6a9b25ae881c46b5" gracePeriod=25 Oct 30 01:38:29.245690 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:29.245657 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7668f5d547-dxb9b" Oct 30 01:38:29.251327 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:29.251277 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:29.374393 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:29.374351 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a147d7d-b019-4960-9813-9cc1495d9901-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a147d7d-b019-4960-9813-9cc1495d9901-plugins-config-volume\") pod \"1a147d7d-b019-4960-9813-9cc1495d9901\" (UID: \"1a147d7d-b019-4960-9813-9cc1495d9901\") " Oct 30 01:38:29.374569 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:29.374408 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a147d7d-b019-4960-9813-9cc1495d9901-kube-api-access-jq2f7\" (UniqueName: \"kubernetes.io/projected/1a147d7d-b019-4960-9813-9cc1495d9901-kube-api-access-jq2f7\") pod \"1a147d7d-b019-4960-9813-9cc1495d9901\" (UID: \"1a147d7d-b019-4960-9813-9cc1495d9901\") " Oct 30 01:38:29.374838 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:29.374806 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a147d7d-b019-4960-9813-9cc1495d9901-plugins-config-volume" pod "1a147d7d-b019-4960-9813-9cc1495d9901" (UID: "1a147d7d-b019-4960-9813-9cc1495d9901"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:29.376602 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:29.376571 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a147d7d-b019-4960-9813-9cc1495d9901-kube-api-access-jq2f7" pod "1a147d7d-b019-4960-9813-9cc1495d9901" (UID: "1a147d7d-b019-4960-9813-9cc1495d9901"). InnerVolumeSpecName "kube-api-access-jq2f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:29.475741 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:29.475638 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a147d7d-b019-4960-9813-9cc1495d9901-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:29.475741 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:29.475682 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq2f7\" (UniqueName: \"kubernetes.io/projected/1a147d7d-b019-4960-9813-9cc1495d9901-kube-api-access-jq2f7\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:29.544939 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:29.544897 6572 generic.go:356] "Generic (PLEG): container finished" podID="1a147d7d-b019-4960-9813-9cc1495d9901" containerID="8ed19ac9d4a09bf20c4eee39e915f2b59f7e6f1f29d09cae6a9b25ae881c46b5" exitCode=0 Oct 30 01:38:29.545154 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:29.544946 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7668f5d547-dxb9b" Oct 30 01:38:29.545154 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:29.544984 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7668f5d547-dxb9b" event={"ID":"1a147d7d-b019-4960-9813-9cc1495d9901","Type":"ContainerDied","Data":"8ed19ac9d4a09bf20c4eee39e915f2b59f7e6f1f29d09cae6a9b25ae881c46b5"} Oct 30 01:38:29.545154 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:29.545045 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7668f5d547-dxb9b" event={"ID":"1a147d7d-b019-4960-9813-9cc1495d9901","Type":"ContainerDied","Data":"df1f0932abb0700acb22269d32353a2e840325d41a03e6a219f98fb3b30b4069"} Oct 30 01:38:29.545154 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:29.545087 6572 scope.go:122] "RemoveContainer" containerID="8ed19ac9d4a09bf20c4eee39e915f2b59f7e6f1f29d09cae6a9b25ae881c46b5" Oct 30 01:38:29.557419 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:29.557394 6572 scope.go:122] "RemoveContainer" containerID="8ed19ac9d4a09bf20c4eee39e915f2b59f7e6f1f29d09cae6a9b25ae881c46b5" Oct 30 01:38:29.557826 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:38:29.557799 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ed19ac9d4a09bf20c4eee39e915f2b59f7e6f1f29d09cae6a9b25ae881c46b5\": container with ID starting with 8ed19ac9d4a09bf20c4eee39e915f2b59f7e6f1f29d09cae6a9b25ae881c46b5 not found: ID does not exist" containerID="8ed19ac9d4a09bf20c4eee39e915f2b59f7e6f1f29d09cae6a9b25ae881c46b5" Oct 30 01:38:29.557882 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:29.557835 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ed19ac9d4a09bf20c4eee39e915f2b59f7e6f1f29d09cae6a9b25ae881c46b5"} err="failed to get container status \"8ed19ac9d4a09bf20c4eee39e915f2b59f7e6f1f29d09cae6a9b25ae881c46b5\": rpc error: code = NotFound desc = could not find container \"8ed19ac9d4a09bf20c4eee39e915f2b59f7e6f1f29d09cae6a9b25ae881c46b5\": container with ID starting with 8ed19ac9d4a09bf20c4eee39e915f2b59f7e6f1f29d09cae6a9b25ae881c46b5 not found: ID does not exist" Oct 30 01:38:29.603228 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:29.603180 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7668f5d547-dxb9b"] Oct 30 01:38:29.611473 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:29.611441 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7668f5d547-dxb9b"] Oct 30 01:38:30.543313 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:30.543271 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a147d7d-b019-4960-9813-9cc1495d9901" path="/var/lib/kubelet/pods/1a147d7d-b019-4960-9813-9cc1495d9901/volumes" Oct 30 01:38:31.047870 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:31.047830 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:32.016687 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.016614 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" Oct 30 01:38:32.110180 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.110146 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29807745-a828-46e6-ab0e-994057d74363-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29807745-a828-46e6-ab0e-994057d74363-workload-socket\") pod \"29807745-a828-46e6-ab0e-994057d74363\" (UID: \"29807745-a828-46e6-ab0e-994057d74363\") " Oct 30 01:38:32.110674 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.110229 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29807745-a828-46e6-ab0e-994057d74363-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29807745-a828-46e6-ab0e-994057d74363-istio-envoy\") pod \"29807745-a828-46e6-ab0e-994057d74363\" (UID: \"29807745-a828-46e6-ab0e-994057d74363\") " Oct 30 01:38:32.110674 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.110346 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29807745-a828-46e6-ab0e-994057d74363-istio-token\" (UniqueName: \"kubernetes.io/projected/29807745-a828-46e6-ab0e-994057d74363-istio-token\") pod \"29807745-a828-46e6-ab0e-994057d74363\" (UID: \"29807745-a828-46e6-ab0e-994057d74363\") " Oct 30 01:38:32.110674 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.110393 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29807745-a828-46e6-ab0e-994057d74363-kube-api-access-lgn94\" (UniqueName: \"kubernetes.io/projected/29807745-a828-46e6-ab0e-994057d74363-kube-api-access-lgn94\") pod \"29807745-a828-46e6-ab0e-994057d74363\" (UID: \"29807745-a828-46e6-ab0e-994057d74363\") " Oct 30 01:38:32.110674 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.110442 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29807745-a828-46e6-ab0e-994057d74363-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29807745-a828-46e6-ab0e-994057d74363-istio-data\") pod \"29807745-a828-46e6-ab0e-994057d74363\" (UID: \"29807745-a828-46e6-ab0e-994057d74363\") " Oct 30 01:38:32.110674 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.110469 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29807745-a828-46e6-ab0e-994057d74363-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29807745-a828-46e6-ab0e-994057d74363-istiod-ca-cert\") pod \"29807745-a828-46e6-ab0e-994057d74363\" (UID: \"29807745-a828-46e6-ab0e-994057d74363\") " Oct 30 01:38:32.110674 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.110504 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29807745-a828-46e6-ab0e-994057d74363-workload-socket" pod "29807745-a828-46e6-ab0e-994057d74363" (UID: "29807745-a828-46e6-ab0e-994057d74363"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:32.110674 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.110518 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29807745-a828-46e6-ab0e-994057d74363-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29807745-a828-46e6-ab0e-994057d74363-istio-podinfo\") pod \"29807745-a828-46e6-ab0e-994057d74363\" (UID: \"29807745-a828-46e6-ab0e-994057d74363\") " Oct 30 01:38:32.110674 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.110626 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29807745-a828-46e6-ab0e-994057d74363-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29807745-a828-46e6-ab0e-994057d74363-workload-certs\") pod \"29807745-a828-46e6-ab0e-994057d74363\" (UID: \"29807745-a828-46e6-ab0e-994057d74363\") " Oct 30 01:38:32.111219 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.110689 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29807745-a828-46e6-ab0e-994057d74363-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29807745-a828-46e6-ab0e-994057d74363-credential-socket\") pod \"29807745-a828-46e6-ab0e-994057d74363\" (UID: \"29807745-a828-46e6-ab0e-994057d74363\") " Oct 30 01:38:32.111219 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.110828 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29807745-a828-46e6-ab0e-994057d74363-istio-data" pod "29807745-a828-46e6-ab0e-994057d74363" (UID: "29807745-a828-46e6-ab0e-994057d74363"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:32.111219 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.110972 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29807745-a828-46e6-ab0e-994057d74363-workload-certs" pod "29807745-a828-46e6-ab0e-994057d74363" (UID: "29807745-a828-46e6-ab0e-994057d74363"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:32.111219 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.111186 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29807745-a828-46e6-ab0e-994057d74363-istiod-ca-cert" pod "29807745-a828-46e6-ab0e-994057d74363" (UID: "29807745-a828-46e6-ab0e-994057d74363"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:32.111412 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.111217 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29807745-a828-46e6-ab0e-994057d74363-credential-socket" pod "29807745-a828-46e6-ab0e-994057d74363" (UID: "29807745-a828-46e6-ab0e-994057d74363"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:32.111529 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.111468 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29807745-a828-46e6-ab0e-994057d74363-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:32.111529 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.111488 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29807745-a828-46e6-ab0e-994057d74363-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:32.111529 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.111502 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29807745-a828-46e6-ab0e-994057d74363-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:32.111529 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.111515 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29807745-a828-46e6-ab0e-994057d74363-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:32.111659 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.111530 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29807745-a828-46e6-ab0e-994057d74363-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:32.113815 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.113786 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29807745-a828-46e6-ab0e-994057d74363-istio-envoy" pod "29807745-a828-46e6-ab0e-994057d74363" (UID: "29807745-a828-46e6-ab0e-994057d74363"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:32.114700 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.114671 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29807745-a828-46e6-ab0e-994057d74363-istio-podinfo" pod "29807745-a828-46e6-ab0e-994057d74363" (UID: "29807745-a828-46e6-ab0e-994057d74363"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:38:32.115002 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.114984 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29807745-a828-46e6-ab0e-994057d74363-kube-api-access-lgn94" pod "29807745-a828-46e6-ab0e-994057d74363" (UID: "29807745-a828-46e6-ab0e-994057d74363"). InnerVolumeSpecName "kube-api-access-lgn94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:32.115113 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.115100 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29807745-a828-46e6-ab0e-994057d74363-istio-token" pod "29807745-a828-46e6-ab0e-994057d74363" (UID: "29807745-a828-46e6-ab0e-994057d74363"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:32.213422 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.213373 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29807745-a828-46e6-ab0e-994057d74363-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:32.213422 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.213415 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29807745-a828-46e6-ab0e-994057d74363-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:32.213422 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.213430 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgn94\" (UniqueName: \"kubernetes.io/projected/29807745-a828-46e6-ab0e-994057d74363-kube-api-access-lgn94\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:32.213669 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.213443 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29807745-a828-46e6-ab0e-994057d74363-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:32.573381 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.573342 6572 generic.go:356] "Generic (PLEG): container finished" podID="29807745-a828-46e6-ab0e-994057d74363" containerID="50f5f648e8fcbb3767a2ff2447a53fce4196ed835371402653a7bfff9939e1b2" exitCode=0 Oct 30 01:38:32.573537 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.573393 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" Oct 30 01:38:32.573537 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.573427 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" event={"ID":"29807745-a828-46e6-ab0e-994057d74363","Type":"ContainerDied","Data":"50f5f648e8fcbb3767a2ff2447a53fce4196ed835371402653a7bfff9939e1b2"} Oct 30 01:38:32.573537 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.573485 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg" event={"ID":"29807745-a828-46e6-ab0e-994057d74363","Type":"ContainerDied","Data":"4a3d97f3bed1d8b173040a8268525a61cd41be971a174d7e06f2155d4a17e589"} Oct 30 01:38:32.573537 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.573515 6572 scope.go:122] "RemoveContainer" containerID="50f5f648e8fcbb3767a2ff2447a53fce4196ed835371402653a7bfff9939e1b2" Oct 30 01:38:32.585426 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.585401 6572 scope.go:122] "RemoveContainer" containerID="50f5f648e8fcbb3767a2ff2447a53fce4196ed835371402653a7bfff9939e1b2" Oct 30 01:38:32.585808 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:38:32.585739 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50f5f648e8fcbb3767a2ff2447a53fce4196ed835371402653a7bfff9939e1b2\": container with ID starting with 50f5f648e8fcbb3767a2ff2447a53fce4196ed835371402653a7bfff9939e1b2 not found: ID does not exist" containerID="50f5f648e8fcbb3767a2ff2447a53fce4196ed835371402653a7bfff9939e1b2" Oct 30 01:38:32.585869 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.585821 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50f5f648e8fcbb3767a2ff2447a53fce4196ed835371402653a7bfff9939e1b2"} err="failed to get container status \"50f5f648e8fcbb3767a2ff2447a53fce4196ed835371402653a7bfff9939e1b2\": rpc error: code = NotFound desc = could not find container \"50f5f648e8fcbb3767a2ff2447a53fce4196ed835371402653a7bfff9939e1b2\": container with ID starting with 50f5f648e8fcbb3767a2ff2447a53fce4196ed835371402653a7bfff9939e1b2 not found: ID does not exist" Oct 30 01:38:32.625925 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.625887 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg"] Oct 30 01:38:32.636726 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.636686 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-b84rg"] Oct 30 01:38:32.903091 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:32.902980 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:33.105034 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:33.105003 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:33.297583 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:33.297544 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-56668cb6-85kf8"] Oct 30 01:38:33.658438 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:33.658329 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:33.912191 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:33.912156 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:34.546402 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:34.546355 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29807745-a828-46e6-ab0e-994057d74363" path="/var/lib/kubelet/pods/29807745-a828-46e6-ab0e-994057d74363/volumes" Oct 30 01:38:34.867001 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:34.866904 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:35.593590 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:35.591838 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:35.721776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:35.711122 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f09934-htnf2"] Oct 30 01:38:35.725775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:35.722653 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f09934-htnf2"] Oct 30 01:38:35.812771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:35.811131 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:36.047771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:36.047065 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:36.388129 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:36.388040 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:36.546307 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:36.546265 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a618ddde-ca90-4180-a97e-8321e808f93a" path="/var/lib/kubelet/pods/a618ddde-ca90-4180-a97e-8321e808f93a/volumes" Oct 30 01:38:37.661775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:37.654579 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:37.858882 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:37.858849 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:38.079313 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.079233 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-cbfd98f6-9nnfl"] Oct 30 01:38:38.079876 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.079851 6572 cpu_manager.go:414] "containerMap: removing container" podUID="a8ce41ba-cb52-4d64-8b85-d420b3d96711" containerName="payload-pre-processing" Oct 30 01:38:38.079972 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.079875 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8ce41ba-cb52-4d64-8b85-d420b3d96711" containerName="payload-pre-processing" Oct 30 01:38:38.079972 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.079894 6572 cpu_manager.go:414] "containerMap: removing container" podUID="1beef457-ba1a-437d-aa79-a625c5032b19" containerName="payload-processing" Oct 30 01:38:38.079972 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.079901 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1beef457-ba1a-437d-aa79-a625c5032b19" containerName="payload-processing" Oct 30 01:38:38.079972 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.079943 6572 cpu_manager.go:414] "containerMap: removing container" podUID="29807745-a828-46e6-ab0e-994057d74363" containerName="istio-proxy" Oct 30 01:38:38.079972 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.079950 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29807745-a828-46e6-ab0e-994057d74363" containerName="istio-proxy" Oct 30 01:38:38.080137 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.079979 6572 cpu_manager.go:414] "containerMap: removing container" podUID="6ba2dcf2-ec73-413b-8190-5e2b31cd692c" containerName="istio-proxy" Oct 30 01:38:38.080137 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.079986 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ba2dcf2-ec73-413b-8190-5e2b31cd692c" containerName="istio-proxy" Oct 30 01:38:38.080137 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.080009 6572 cpu_manager.go:414] "containerMap: removing container" podUID="46929775-3a67-4bd4-8387-52bab2ebdc0e" containerName="istio-proxy" Oct 30 01:38:38.080137 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.080017 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46929775-3a67-4bd4-8387-52bab2ebdc0e" containerName="istio-proxy" Oct 30 01:38:38.080137 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.080029 6572 cpu_manager.go:414] "containerMap: removing container" podUID="e9462ac3-f977-4dce-8c37-a89b70928484" containerName="istio-proxy" Oct 30 01:38:38.080137 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.080036 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9462ac3-f977-4dce-8c37-a89b70928484" containerName="istio-proxy" Oct 30 01:38:38.080137 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.080052 6572 cpu_manager.go:414] "containerMap: removing container" podUID="1a147d7d-b019-4960-9813-9cc1495d9901" containerName="payload-pre-processing" Oct 30 01:38:38.080137 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.080059 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a147d7d-b019-4960-9813-9cc1495d9901" containerName="payload-pre-processing" Oct 30 01:38:38.080137 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.080070 6572 cpu_manager.go:414] "containerMap: removing container" podUID="96ae6bcd-2343-4c65-b4db-f7a5106f51f2" containerName="istio-proxy" Oct 30 01:38:38.080137 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.080076 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96ae6bcd-2343-4c65-b4db-f7a5106f51f2" containerName="istio-proxy" Oct 30 01:38:38.080569 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.080210 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ba2dcf2-ec73-413b-8190-5e2b31cd692c" containerName="istio-proxy" Oct 30 01:38:38.080569 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.080223 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="46929775-3a67-4bd4-8387-52bab2ebdc0e" containerName="istio-proxy" Oct 30 01:38:38.080569 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.080231 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9462ac3-f977-4dce-8c37-a89b70928484" containerName="istio-proxy" Oct 30 01:38:38.080569 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.080237 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a147d7d-b019-4960-9813-9cc1495d9901" containerName="payload-pre-processing" Oct 30 01:38:38.080569 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.080243 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="96ae6bcd-2343-4c65-b4db-f7a5106f51f2" containerName="istio-proxy" Oct 30 01:38:38.080569 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.080255 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8ce41ba-cb52-4d64-8b85-d420b3d96711" containerName="payload-pre-processing" Oct 30 01:38:38.080569 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.080264 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="1beef457-ba1a-437d-aa79-a625c5032b19" containerName="payload-processing" Oct 30 01:38:38.080569 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.080274 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="29807745-a828-46e6-ab0e-994057d74363" containerName="istio-proxy" Oct 30 01:38:38.085412 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.085388 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-cbfd98f6-9nnfl" Oct 30 01:38:38.112932 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.112895 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d9ede0\"" Oct 30 01:38:38.113118 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.112999 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d9ede0-dockercfg-cjhz9\"" Oct 30 01:38:38.123254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.123212 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-cbfd98f6-9nnfl"] Oct 30 01:38:38.168434 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.168388 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85fef8a6-8af8-46e7-b994-475da53e403b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d9ede0-cbfd98f6-9nnfl\" (UID: \"85fef8a6-8af8-46e7-b994-475da53e403b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-cbfd98f6-9nnfl" Oct 30 01:38:38.168668 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.168480 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dscrr\" (UniqueName: \"kubernetes.io/projected/85fef8a6-8af8-46e7-b994-475da53e403b-kube-api-access-dscrr\") pod \"payload-processing-e2e-ipp-a-d9ede0-cbfd98f6-9nnfl\" (UID: \"85fef8a6-8af8-46e7-b994-475da53e403b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-cbfd98f6-9nnfl" Oct 30 01:38:38.257271 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.257227 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-96888c8c9-wzmnm"] Oct 30 01:38:38.268172 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.268130 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-96888c8c9-wzmnm" Oct 30 01:38:38.270251 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.270203 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85fef8a6-8af8-46e7-b994-475da53e403b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d9ede0-cbfd98f6-9nnfl\" (UID: \"85fef8a6-8af8-46e7-b994-475da53e403b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-cbfd98f6-9nnfl" Oct 30 01:38:38.270505 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.270297 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dscrr\" (UniqueName: \"kubernetes.io/projected/85fef8a6-8af8-46e7-b994-475da53e403b-kube-api-access-dscrr\") pod \"payload-processing-e2e-ipp-a-d9ede0-cbfd98f6-9nnfl\" (UID: \"85fef8a6-8af8-46e7-b994-475da53e403b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-cbfd98f6-9nnfl" Oct 30 01:38:38.271121 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.271094 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85fef8a6-8af8-46e7-b994-475da53e403b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d9ede0-cbfd98f6-9nnfl\" (UID: \"85fef8a6-8af8-46e7-b994-475da53e403b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-cbfd98f6-9nnfl" Oct 30 01:38:38.280029 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.278015 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-96888c8c9-wzmnm"] Oct 30 01:38:38.291038 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.291011 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dscrr\" (UniqueName: \"kubernetes.io/projected/85fef8a6-8af8-46e7-b994-475da53e403b-kube-api-access-dscrr\") pod \"payload-processing-e2e-ipp-a-d9ede0-cbfd98f6-9nnfl\" (UID: \"85fef8a6-8af8-46e7-b994-475da53e403b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-cbfd98f6-9nnfl" Oct 30 01:38:38.298763 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.298715 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-56668cb6-85kf8" podUID="f485bb6f-8dcb-40de-9661-d12993b82934" containerName="payload-processing" containerID="cri-o://bebdfbcedcb36bc288516cadb785f552f906d6a228c614c619920db05025ee1c" gracePeriod=25 Oct 30 01:38:38.372086 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.371985 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n685q\" (UniqueName: \"kubernetes.io/projected/5349aa56-fc17-4286-9390-3a68ff65542f-kube-api-access-n685q\") pod \"payload-pre-processing-e2e-ipp-a-d9ede0-96888c8c9-wzmnm\" (UID: \"5349aa56-fc17-4286-9390-3a68ff65542f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-96888c8c9-wzmnm" Oct 30 01:38:38.372253 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.372188 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5349aa56-fc17-4286-9390-3a68ff65542f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d9ede0-96888c8c9-wzmnm\" (UID: \"5349aa56-fc17-4286-9390-3a68ff65542f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-96888c8c9-wzmnm" Oct 30 01:38:38.401088 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.401056 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-cbfd98f6-9nnfl" Oct 30 01:38:38.473934 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.473879 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5349aa56-fc17-4286-9390-3a68ff65542f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d9ede0-96888c8c9-wzmnm\" (UID: \"5349aa56-fc17-4286-9390-3a68ff65542f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-96888c8c9-wzmnm" Oct 30 01:38:38.474100 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.474087 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n685q\" (UniqueName: \"kubernetes.io/projected/5349aa56-fc17-4286-9390-3a68ff65542f-kube-api-access-n685q\") pod \"payload-pre-processing-e2e-ipp-a-d9ede0-96888c8c9-wzmnm\" (UID: \"5349aa56-fc17-4286-9390-3a68ff65542f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-96888c8c9-wzmnm" Oct 30 01:38:38.475316 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.475285 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5349aa56-fc17-4286-9390-3a68ff65542f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d9ede0-96888c8c9-wzmnm\" (UID: \"5349aa56-fc17-4286-9390-3a68ff65542f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-96888c8c9-wzmnm" Oct 30 01:38:38.496514 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.496479 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n685q\" (UniqueName: \"kubernetes.io/projected/5349aa56-fc17-4286-9390-3a68ff65542f-kube-api-access-n685q\") pod \"payload-pre-processing-e2e-ipp-a-d9ede0-96888c8c9-wzmnm\" (UID: \"5349aa56-fc17-4286-9390-3a68ff65542f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-96888c8c9-wzmnm" Oct 30 01:38:38.592668 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.592624 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-96888c8c9-wzmnm" Oct 30 01:38:38.620234 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.620204 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-56668cb6-85kf8" Oct 30 01:38:38.677593 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.676666 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f485bb6f-8dcb-40de-9661-d12993b82934-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f485bb6f-8dcb-40de-9661-d12993b82934-plugins-config-volume\") pod \"f485bb6f-8dcb-40de-9661-d12993b82934\" (UID: \"f485bb6f-8dcb-40de-9661-d12993b82934\") " Oct 30 01:38:38.677593 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.676881 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f485bb6f-8dcb-40de-9661-d12993b82934-kube-api-access-dk7dt\" (UniqueName: \"kubernetes.io/projected/f485bb6f-8dcb-40de-9661-d12993b82934-kube-api-access-dk7dt\") pod \"f485bb6f-8dcb-40de-9661-d12993b82934\" (UID: \"f485bb6f-8dcb-40de-9661-d12993b82934\") " Oct 30 01:38:38.677593 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.677380 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f485bb6f-8dcb-40de-9661-d12993b82934-plugins-config-volume" pod "f485bb6f-8dcb-40de-9661-d12993b82934" (UID: "f485bb6f-8dcb-40de-9661-d12993b82934"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:38.677593 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.677544 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f485bb6f-8dcb-40de-9661-d12993b82934-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:38.685691 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.685339 6572 generic.go:356] "Generic (PLEG): container finished" podID="f485bb6f-8dcb-40de-9661-d12993b82934" containerID="bebdfbcedcb36bc288516cadb785f552f906d6a228c614c619920db05025ee1c" exitCode=0 Oct 30 01:38:38.685691 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.685390 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-56668cb6-85kf8" Oct 30 01:38:38.685691 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.685381 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-56668cb6-85kf8" event={"ID":"f485bb6f-8dcb-40de-9661-d12993b82934","Type":"ContainerDied","Data":"bebdfbcedcb36bc288516cadb785f552f906d6a228c614c619920db05025ee1c"} Oct 30 01:38:38.685691 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.685513 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-56668cb6-85kf8" event={"ID":"f485bb6f-8dcb-40de-9661-d12993b82934","Type":"ContainerDied","Data":"414d82200f5e0080c410769749bfe992ffd64c99d0dd38990c978ceb6c501db1"} Oct 30 01:38:38.685691 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.685542 6572 scope.go:122] "RemoveContainer" containerID="bebdfbcedcb36bc288516cadb785f552f906d6a228c614c619920db05025ee1c" Oct 30 01:38:38.686104 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.685978 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f485bb6f-8dcb-40de-9661-d12993b82934-kube-api-access-dk7dt" pod "f485bb6f-8dcb-40de-9661-d12993b82934" (UID: "f485bb6f-8dcb-40de-9661-d12993b82934"). InnerVolumeSpecName "kube-api-access-dk7dt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:38.697553 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.697529 6572 scope.go:122] "RemoveContainer" containerID="bebdfbcedcb36bc288516cadb785f552f906d6a228c614c619920db05025ee1c" Oct 30 01:38:38.697941 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:38:38.697916 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bebdfbcedcb36bc288516cadb785f552f906d6a228c614c619920db05025ee1c\": container with ID starting with bebdfbcedcb36bc288516cadb785f552f906d6a228c614c619920db05025ee1c not found: ID does not exist" containerID="bebdfbcedcb36bc288516cadb785f552f906d6a228c614c619920db05025ee1c" Oct 30 01:38:38.698043 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.697950 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bebdfbcedcb36bc288516cadb785f552f906d6a228c614c619920db05025ee1c"} err="failed to get container status \"bebdfbcedcb36bc288516cadb785f552f906d6a228c614c619920db05025ee1c\": rpc error: code = NotFound desc = could not find container \"bebdfbcedcb36bc288516cadb785f552f906d6a228c614c619920db05025ee1c\": container with ID starting with bebdfbcedcb36bc288516cadb785f552f906d6a228c614c619920db05025ee1c not found: ID does not exist" Oct 30 01:38:38.724592 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.723878 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-cbfd98f6-9nnfl"] Oct 30 01:38:38.725063 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:38:38.725036 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85fef8a6_8af8_46e7_b994_475da53e403b.slice/crio-c50b39d2338178382b0ebfc465f5627b4b1e1d0859d66441a86a6032daa75e5a WatchSource:0}: Error finding container c50b39d2338178382b0ebfc465f5627b4b1e1d0859d66441a86a6032daa75e5a: Status 404 returned error can't find the container with id c50b39d2338178382b0ebfc465f5627b4b1e1d0859d66441a86a6032daa75e5a Oct 30 01:38:38.778659 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.778617 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk7dt\" (UniqueName: \"kubernetes.io/projected/f485bb6f-8dcb-40de-9661-d12993b82934-kube-api-access-dk7dt\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:38.909603 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:38.909568 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-96888c8c9-wzmnm"] Oct 30 01:38:38.910116 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:38:38.910093 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5349aa56_fc17_4286_9390_3a68ff65542f.slice/crio-be2d51149dddce7fcd5a716b455e717849de2a5138e2dac4a5a896eeda75eb2f WatchSource:0}: Error finding container be2d51149dddce7fcd5a716b455e717849de2a5138e2dac4a5a896eeda75eb2f: Status 404 returned error can't find the container with id be2d51149dddce7fcd5a716b455e717849de2a5138e2dac4a5a896eeda75eb2f Oct 30 01:38:39.030245 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:39.030206 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-56668cb6-85kf8"] Oct 30 01:38:39.033831 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:39.033788 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-56668cb6-85kf8"] Oct 30 01:38:39.212929 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:39.212880 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:39.704934 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:39.704888 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-cbfd98f6-9nnfl" event={"ID":"85fef8a6-8af8-46e7-b994-475da53e403b","Type":"ContainerStarted","Data":"c50b39d2338178382b0ebfc465f5627b4b1e1d0859d66441a86a6032daa75e5a"} Oct 30 01:38:39.707167 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:39.707130 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-96888c8c9-wzmnm" event={"ID":"5349aa56-fc17-4286-9390-3a68ff65542f","Type":"ContainerStarted","Data":"be2d51149dddce7fcd5a716b455e717849de2a5138e2dac4a5a896eeda75eb2f"} Oct 30 01:38:40.547522 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:40.547486 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f485bb6f-8dcb-40de-9661-d12993b82934" path="/var/lib/kubelet/pods/f485bb6f-8dcb-40de-9661-d12993b82934/volumes" Oct 30 01:38:41.167766 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:41.167708 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:41.735268 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:41.735230 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-cbfd98f6-9nnfl" event={"ID":"85fef8a6-8af8-46e7-b994-475da53e403b","Type":"ContainerStarted","Data":"29471aa1d401da2551b86bb256ba879ba7c43be1fd062a5c986a7b479a5edc5b"} Oct 30 01:38:41.735484 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:41.735327 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-cbfd98f6-9nnfl" Oct 30 01:38:41.737123 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:41.737077 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-96888c8c9-wzmnm" event={"ID":"5349aa56-fc17-4286-9390-3a68ff65542f","Type":"ContainerStarted","Data":"4688126a5999c457b849494fb010714e75976499dcf538eb0dfc55f6e9633d0d"} Oct 30 01:38:41.737329 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:41.737141 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-96888c8c9-wzmnm" Oct 30 01:38:41.774556 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:41.774497 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-cbfd98f6-9nnfl" podStartSLOduration=1.846025917 podStartE2EDuration="3.774482324s" totalImagesPullingTime="1.928456407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:38:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:38:41.773906747 +0000 UTC m=+1247.350332161" watchObservedRunningTime="2026-10-30 01:38:41.774482324 +0000 UTC m=+1247.350907765" Oct 30 01:38:41.814905 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:41.814839 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-96888c8c9-wzmnm" podStartSLOduration=2.057784622 podStartE2EDuration="3.814174817s" totalImagesPullingTime="1.756390195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:38:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:38:41.813421335 +0000 UTC m=+1247.389846737" watchObservedRunningTime="2026-10-30 01:38:41.814174817 +0000 UTC m=+1247.390600221" Oct 30 01:38:42.481835 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:42.481797 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:42.679967 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:42.679930 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:42.815829 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:42.814810 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7dc7f7bb59-mc2jj"] Oct 30 01:38:42.815829 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:42.815776 6572 cpu_manager.go:414] "containerMap: removing container" podUID="f485bb6f-8dcb-40de-9661-d12993b82934" containerName="payload-processing" Oct 30 01:38:42.815829 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:42.815792 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f485bb6f-8dcb-40de-9661-d12993b82934" containerName="payload-processing" Oct 30 01:38:42.816111 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:42.815986 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="f485bb6f-8dcb-40de-9661-d12993b82934" containerName="payload-processing" Oct 30 01:38:42.822632 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:42.821020 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7dc7f7bb59-mc2jj" Oct 30 01:38:42.879802 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:42.879678 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-19a5f1\"" Oct 30 01:38:42.884608 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:42.884517 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7dc7f7bb59-mc2jj"] Oct 30 01:38:42.889566 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:42.889529 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-19a5f1-dockercfg-lffc4\"" Oct 30 01:38:42.954820 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:42.954776 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-7dd7645496-mmljt"] Oct 30 01:38:42.965809 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:42.962191 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-7dd7645496-mmljt" Oct 30 01:38:42.982961 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:42.981927 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42fcg\" (UniqueName: \"kubernetes.io/projected/00aac51a-0f35-4f98-95cc-8ee9a919a3fe-kube-api-access-42fcg\") pod \"payload-pre-processing-e2e-ipp-a-19a5f1-7dc7f7bb59-mc2jj\" (UID: \"00aac51a-0f35-4f98-95cc-8ee9a919a3fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7dc7f7bb59-mc2jj" Oct 30 01:38:42.982961 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:42.982049 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00aac51a-0f35-4f98-95cc-8ee9a919a3fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-19a5f1-7dc7f7bb59-mc2jj\" (UID: \"00aac51a-0f35-4f98-95cc-8ee9a919a3fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7dc7f7bb59-mc2jj" Oct 30 01:38:42.986853 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:42.986817 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-7dd7645496-mmljt"] Oct 30 01:38:43.036612 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.036565 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:43.114838 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.098854 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42fcg\" (UniqueName: \"kubernetes.io/projected/00aac51a-0f35-4f98-95cc-8ee9a919a3fe-kube-api-access-42fcg\") pod \"payload-pre-processing-e2e-ipp-a-19a5f1-7dc7f7bb59-mc2jj\" (UID: \"00aac51a-0f35-4f98-95cc-8ee9a919a3fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7dc7f7bb59-mc2jj" Oct 30 01:38:43.114838 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.099000 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qngpp\" (UniqueName: \"kubernetes.io/projected/773200f9-3e31-41f4-817f-d6fb5b97e12f-kube-api-access-qngpp\") pod \"payload-processing-e2e-ipp-a-19a5f1-7dd7645496-mmljt\" (UID: \"773200f9-3e31-41f4-817f-d6fb5b97e12f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-7dd7645496-mmljt" Oct 30 01:38:43.114838 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.099050 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00aac51a-0f35-4f98-95cc-8ee9a919a3fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-19a5f1-7dc7f7bb59-mc2jj\" (UID: \"00aac51a-0f35-4f98-95cc-8ee9a919a3fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7dc7f7bb59-mc2jj" Oct 30 01:38:43.114838 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.099117 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/773200f9-3e31-41f4-817f-d6fb5b97e12f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-19a5f1-7dd7645496-mmljt\" (UID: \"773200f9-3e31-41f4-817f-d6fb5b97e12f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-7dd7645496-mmljt" Oct 30 01:38:43.114838 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.100619 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00aac51a-0f35-4f98-95cc-8ee9a919a3fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-19a5f1-7dc7f7bb59-mc2jj\" (UID: \"00aac51a-0f35-4f98-95cc-8ee9a919a3fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7dc7f7bb59-mc2jj" Oct 30 01:38:43.142071 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.142040 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42fcg\" (UniqueName: \"kubernetes.io/projected/00aac51a-0f35-4f98-95cc-8ee9a919a3fe-kube-api-access-42fcg\") pod \"payload-pre-processing-e2e-ipp-a-19a5f1-7dc7f7bb59-mc2jj\" (UID: \"00aac51a-0f35-4f98-95cc-8ee9a919a3fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7dc7f7bb59-mc2jj" Oct 30 01:38:43.193011 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.192966 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7dc7f7bb59-mc2jj" Oct 30 01:38:43.201087 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.201052 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qngpp\" (UniqueName: \"kubernetes.io/projected/773200f9-3e31-41f4-817f-d6fb5b97e12f-kube-api-access-qngpp\") pod \"payload-processing-e2e-ipp-a-19a5f1-7dd7645496-mmljt\" (UID: \"773200f9-3e31-41f4-817f-d6fb5b97e12f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-7dd7645496-mmljt" Oct 30 01:38:43.201233 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.201118 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/773200f9-3e31-41f4-817f-d6fb5b97e12f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-19a5f1-7dd7645496-mmljt\" (UID: \"773200f9-3e31-41f4-817f-d6fb5b97e12f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-7dd7645496-mmljt" Oct 30 01:38:43.201758 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.201723 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/773200f9-3e31-41f4-817f-d6fb5b97e12f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-19a5f1-7dd7645496-mmljt\" (UID: \"773200f9-3e31-41f4-817f-d6fb5b97e12f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-7dd7645496-mmljt" Oct 30 01:38:43.224269 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.224238 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qngpp\" (UniqueName: \"kubernetes.io/projected/773200f9-3e31-41f4-817f-d6fb5b97e12f-kube-api-access-qngpp\") pod \"payload-processing-e2e-ipp-a-19a5f1-7dd7645496-mmljt\" (UID: \"773200f9-3e31-41f4-817f-d6fb5b97e12f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-7dd7645496-mmljt" Oct 30 01:38:43.286927 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.286625 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-7dd7645496-mmljt" Oct 30 01:38:43.502778 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.502734 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7dc7f7bb59-mc2jj"] Oct 30 01:38:43.504153 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:38:43.504126 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00aac51a_0f35_4f98_95cc_8ee9a919a3fe.slice/crio-8c68ebd4a872358bb9dc738b79c2946887536728a5fc81e667bf1ff8a3dfd66f WatchSource:0}: Error finding container 8c68ebd4a872358bb9dc738b79c2946887536728a5fc81e667bf1ff8a3dfd66f: Status 404 returned error can't find the container with id 8c68ebd4a872358bb9dc738b79c2946887536728a5fc81e667bf1ff8a3dfd66f Oct 30 01:38:43.604449 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:38:43.604417 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod773200f9_3e31_41f4_817f_d6fb5b97e12f.slice/crio-c374accc81a7cc255a82f2a49f5c460f34b25059ca989f59bcf0d33a7eda7ef2 WatchSource:0}: Error finding container c374accc81a7cc255a82f2a49f5c460f34b25059ca989f59bcf0d33a7eda7ef2: Status 404 returned error can't find the container with id c374accc81a7cc255a82f2a49f5c460f34b25059ca989f59bcf0d33a7eda7ef2 Oct 30 01:38:43.605267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.605245 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-7dd7645496-mmljt"] Oct 30 01:38:43.705851 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.705814 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18560b-62sdq"] Oct 30 01:38:43.710312 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.710283 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18560b-62sdq" Oct 30 01:38:43.727054 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.727000 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:38:43.727054 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.727005 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:38:43.729822 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.729731 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18560b-62sdq"] Oct 30 01:38:43.734636 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.734612 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:38:43.736136 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.736098 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:38:43.782223 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.782122 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7dc7f7bb59-mc2jj" event={"ID":"00aac51a-0f35-4f98-95cc-8ee9a919a3fe","Type":"ContainerStarted","Data":"f5117d5f02a2b64be2b08783dd9d90886692168c5b21d5077057c2fb8c74d3b9"} Oct 30 01:38:43.782223 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.782203 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7dc7f7bb59-mc2jj" event={"ID":"00aac51a-0f35-4f98-95cc-8ee9a919a3fe","Type":"ContainerStarted","Data":"8c68ebd4a872358bb9dc738b79c2946887536728a5fc81e667bf1ff8a3dfd66f"} Oct 30 01:38:43.782457 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.782270 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7dc7f7bb59-mc2jj" Oct 30 01:38:43.784350 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.784318 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-7dd7645496-mmljt" event={"ID":"773200f9-3e31-41f4-817f-d6fb5b97e12f","Type":"ContainerStarted","Data":"36c139386af68504ca1aa9f945c2dda21376f92deb16e15fb4cdd5d7fd090b8a"} Oct 30 01:38:43.784500 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.784381 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-7dd7645496-mmljt" event={"ID":"773200f9-3e31-41f4-817f-d6fb5b97e12f","Type":"ContainerStarted","Data":"c374accc81a7cc255a82f2a49f5c460f34b25059ca989f59bcf0d33a7eda7ef2"} Oct 30 01:38:43.784624 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.784602 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-7dd7645496-mmljt" Oct 30 01:38:43.812617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.812585 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkr7r\" (UniqueName: \"kubernetes.io/projected/049c1151-7da9-4331-97c4-7cdc7f6e4e6b-kube-api-access-jkr7r\") pod \"maas-api-revoke-keys-e2e-api-b-18560b-62sdq\" (UID: \"049c1151-7da9-4331-97c4-7cdc7f6e4e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18560b-62sdq" Oct 30 01:38:43.812830 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.812623 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/049c1151-7da9-4331-97c4-7cdc7f6e4e6b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-18560b-62sdq\" (UID: \"049c1151-7da9-4331-97c4-7cdc7f6e4e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18560b-62sdq" Oct 30 01:38:43.812830 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.812807 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/049c1151-7da9-4331-97c4-7cdc7f6e4e6b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-18560b-62sdq\" (UID: \"049c1151-7da9-4331-97c4-7cdc7f6e4e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18560b-62sdq" Oct 30 01:38:43.833297 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.833226 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7dc7f7bb59-mc2jj" podStartSLOduration=1.833202623 podStartE2EDuration="1.833202623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:38:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:38:43.826057848 +0000 UTC m=+1249.402483250" watchObservedRunningTime="2026-10-30 01:38:43.833202623 +0000 UTC m=+1249.409628030" Oct 30 01:38:43.874394 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.874302 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-7dd7645496-mmljt" podStartSLOduration=1.874276464 podStartE2EDuration="1.874276464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:38:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:38:43.869661978 +0000 UTC m=+1249.446087375" watchObservedRunningTime="2026-10-30 01:38:43.874276464 +0000 UTC m=+1249.450701865" Oct 30 01:38:43.914320 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.914284 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkr7r\" (UniqueName: \"kubernetes.io/projected/049c1151-7da9-4331-97c4-7cdc7f6e4e6b-kube-api-access-jkr7r\") pod \"maas-api-revoke-keys-e2e-api-b-18560b-62sdq\" (UID: \"049c1151-7da9-4331-97c4-7cdc7f6e4e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18560b-62sdq" Oct 30 01:38:43.914494 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.914349 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/049c1151-7da9-4331-97c4-7cdc7f6e4e6b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-18560b-62sdq\" (UID: \"049c1151-7da9-4331-97c4-7cdc7f6e4e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18560b-62sdq" Oct 30 01:38:43.914494 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.914454 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/049c1151-7da9-4331-97c4-7cdc7f6e4e6b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-18560b-62sdq\" (UID: \"049c1151-7da9-4331-97c4-7cdc7f6e4e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18560b-62sdq" Oct 30 01:38:43.915255 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.915234 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/049c1151-7da9-4331-97c4-7cdc7f6e4e6b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-18560b-62sdq\" (UID: \"049c1151-7da9-4331-97c4-7cdc7f6e4e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18560b-62sdq" Oct 30 01:38:43.917032 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.917013 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/049c1151-7da9-4331-97c4-7cdc7f6e4e6b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-18560b-62sdq\" (UID: \"049c1151-7da9-4331-97c4-7cdc7f6e4e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18560b-62sdq" Oct 30 01:38:43.933215 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:43.933184 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkr7r\" (UniqueName: \"kubernetes.io/projected/049c1151-7da9-4331-97c4-7cdc7f6e4e6b-kube-api-access-jkr7r\") pod \"maas-api-revoke-keys-e2e-api-b-18560b-62sdq\" (UID: \"049c1151-7da9-4331-97c4-7cdc7f6e4e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18560b-62sdq" Oct 30 01:38:44.010566 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:44.010528 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:44.026590 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:44.026554 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18560b-62sdq" Oct 30 01:38:44.345162 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:44.345134 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18560b-62sdq"] Oct 30 01:38:44.346431 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:38:44.346405 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod049c1151_7da9_4331_97c4_7cdc7f6e4e6b.slice/crio-66664bb17025554c08ad1bcdaccf11c7702404b4e9a14e471d88d92e4ed861b6 WatchSource:0}: Error finding container 66664bb17025554c08ad1bcdaccf11c7702404b4e9a14e471d88d92e4ed861b6: Status 404 returned error can't find the container with id 66664bb17025554c08ad1bcdaccf11c7702404b4e9a14e471d88d92e4ed861b6 Oct 30 01:38:44.688770 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:38:44.688705 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:38:44.794994 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:44.794953 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18560b-62sdq" event={"ID":"049c1151-7da9-4331-97c4-7cdc7f6e4e6b","Type":"ContainerStarted","Data":"f86982ce81697156fde84f84b59973662e196d6bf72eea730fcd43f3ea0765b1"} Oct 30 01:38:44.795169 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:44.795015 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18560b-62sdq" event={"ID":"049c1151-7da9-4331-97c4-7cdc7f6e4e6b","Type":"ContainerStarted","Data":"66664bb17025554c08ad1bcdaccf11c7702404b4e9a14e471d88d92e4ed861b6"} Oct 30 01:38:44.839912 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:44.839851 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18560b-62sdq" podStartSLOduration=1.839836922 podStartE2EDuration="1.839836922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:38:44.836849827 +0000 UTC m=+1250.413275230" watchObservedRunningTime="2026-10-30 01:38:44.839836922 +0000 UTC m=+1250.416262324" Oct 30 01:38:45.805142 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:45.805107 6572 generic.go:356] "Generic (PLEG): container finished" podID="049c1151-7da9-4331-97c4-7cdc7f6e4e6b" containerID="f86982ce81697156fde84f84b59973662e196d6bf72eea730fcd43f3ea0765b1" exitCode=0 Oct 30 01:38:45.805597 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:45.805192 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18560b-62sdq" event={"ID":"049c1151-7da9-4331-97c4-7cdc7f6e4e6b","Type":"ContainerDied","Data":"f86982ce81697156fde84f84b59973662e196d6bf72eea730fcd43f3ea0765b1"} Oct 30 01:38:45.932102 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:45.932065 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:46.983099 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:46.983068 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18560b-62sdq" Oct 30 01:38:47.066899 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:47.066856 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/049c1151-7da9-4331-97c4-7cdc7f6e4e6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/049c1151-7da9-4331-97c4-7cdc7f6e4e6b-maas-api-xks-ca\") pod \"049c1151-7da9-4331-97c4-7cdc7f6e4e6b\" (UID: \"049c1151-7da9-4331-97c4-7cdc7f6e4e6b\") " Oct 30 01:38:47.067074 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:47.066923 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/049c1151-7da9-4331-97c4-7cdc7f6e4e6b-kube-api-access-jkr7r\" (UniqueName: \"kubernetes.io/projected/049c1151-7da9-4331-97c4-7cdc7f6e4e6b-kube-api-access-jkr7r\") pod \"049c1151-7da9-4331-97c4-7cdc7f6e4e6b\" (UID: \"049c1151-7da9-4331-97c4-7cdc7f6e4e6b\") " Oct 30 01:38:47.067074 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:47.067022 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/049c1151-7da9-4331-97c4-7cdc7f6e4e6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/049c1151-7da9-4331-97c4-7cdc7f6e4e6b-maas-api-openshift-ca\") pod \"049c1151-7da9-4331-97c4-7cdc7f6e4e6b\" (UID: \"049c1151-7da9-4331-97c4-7cdc7f6e4e6b\") " Oct 30 01:38:47.067419 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:47.067395 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/049c1151-7da9-4331-97c4-7cdc7f6e4e6b-maas-api-openshift-ca" pod "049c1151-7da9-4331-97c4-7cdc7f6e4e6b" (UID: "049c1151-7da9-4331-97c4-7cdc7f6e4e6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:47.067630 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:47.067606 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/049c1151-7da9-4331-97c4-7cdc7f6e4e6b-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:47.069230 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:47.069178 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/049c1151-7da9-4331-97c4-7cdc7f6e4e6b-kube-api-access-jkr7r" pod "049c1151-7da9-4331-97c4-7cdc7f6e4e6b" (UID: "049c1151-7da9-4331-97c4-7cdc7f6e4e6b"). InnerVolumeSpecName "kube-api-access-jkr7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:47.069673 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:47.069659 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/049c1151-7da9-4331-97c4-7cdc7f6e4e6b-maas-api-xks-ca" pod "049c1151-7da9-4331-97c4-7cdc7f6e4e6b" (UID: "049c1151-7da9-4331-97c4-7cdc7f6e4e6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:38:47.168861 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:47.168822 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/049c1151-7da9-4331-97c4-7cdc7f6e4e6b-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:47.168861 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:47.168855 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkr7r\" (UniqueName: \"kubernetes.io/projected/049c1151-7da9-4331-97c4-7cdc7f6e4e6b-kube-api-access-jkr7r\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:47.830938 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:47.830898 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18560b-62sdq" event={"ID":"049c1151-7da9-4331-97c4-7cdc7f6e4e6b","Type":"ContainerDied","Data":"66664bb17025554c08ad1bcdaccf11c7702404b4e9a14e471d88d92e4ed861b6"} Oct 30 01:38:47.830938 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:47.830935 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18560b-62sdq" Oct 30 01:38:47.831174 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:47.830982 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66664bb17025554c08ad1bcdaccf11c7702404b4e9a14e471d88d92e4ed861b6" Oct 30 01:38:47.874003 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:47.873969 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:49.667530 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:49.667488 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:51.506309 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:51.506270 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:52.771814 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:52.771780 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-96888c8c9-wzmnm" Oct 30 01:38:52.772293 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:52.772145 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-cbfd98f6-9nnfl" Oct 30 01:38:53.039196 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:53.039109 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d361c1d-dxsn9"] Oct 30 01:38:53.044509 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:53.044471 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d361c1d-dxsn9"] Oct 30 01:38:53.412831 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:53.411724 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:54.235783 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:54.235733 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-18560b-ffb64bd8c-f4c2s"] Oct 30 01:38:54.542929 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:54.542821 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:54.552656 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:54.552623 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c938ec61-ba37-43cc-84f1-00f917a24b6d" path="/var/lib/kubelet/pods/c938ec61-ba37-43cc-84f1-00f917a24b6d/volumes" Oct 30 01:38:54.785967 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:54.785933 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:54.797206 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:54.797120 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7dc7f7bb59-mc2jj" Oct 30 01:38:54.797528 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:54.797513 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-7dd7645496-mmljt" Oct 30 01:38:55.032331 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:55.032293 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:55.255382 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:55.255342 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:55.272657 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:55.272627 6572 scope.go:122] "RemoveContainer" containerID="8870fc64531782ca4670c69cf431145c70a36f90291adcc76f83e58f96ebc845" Oct 30 01:38:55.370257 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:55.369690 6572 scope.go:122] "RemoveContainer" containerID="64392540aaa61c122acfc1af0353c2b6543ebca152a4c2c415cb362f17196755" Oct 30 01:38:55.391965 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:55.391930 6572 scope.go:122] "RemoveContainer" containerID="04eb97acfe1994c72454bfaf1078733f44e22f8338a8c0786ca58417d21fce3b" Oct 30 01:38:55.438301 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:55.437282 6572 scope.go:122] "RemoveContainer" containerID="c6eed83069a67c1a85a6d56020eed776746799faa575b9e60521d9daef96b907" Oct 30 01:38:55.456003 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:55.455863 6572 scope.go:122] "RemoveContainer" containerID="6ef1a1ed0f8fdd112a16251180621502c0898fbe6128b2e8a293c28980412a3e" Oct 30 01:38:55.471035 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:55.470990 6572 scope.go:122] "RemoveContainer" containerID="ba961df5c71ab81c34f613c3a5acbc8795582ae6810febbf1f041c7d116a7a41" Oct 30 01:38:55.482075 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:55.482049 6572 scope.go:122] "RemoveContainer" containerID="43bb85bee88ebe879c9dd4c59f19f3bb95c59589b883f2dca98659b8d755e043" Oct 30 01:38:55.497125 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:55.497096 6572 scope.go:122] "RemoveContainer" containerID="b04f8ecdadc5fa7b8a73f3e5f9844dd43ef45d64be8b0aa859857c81c412b07c" Oct 30 01:38:55.532796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:55.532737 6572 scope.go:122] "RemoveContainer" containerID="7aac4fc1580136722c073a0d4c60f2735c33cc18b4807bd5ce7ae0126af15da4" Oct 30 01:38:55.544027 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:55.544005 6572 scope.go:122] "RemoveContainer" containerID="d3cfd15de9ecf912b6d1940236b94e2bbd5ca5a94db178e9d2a004a13a11334b" Oct 30 01:38:55.555188 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:55.555159 6572 scope.go:122] "RemoveContainer" containerID="d9d67de54334c98147ea6074fe07340d2ce5cfea6df7c26a6fdc9932de4e6c3c" Oct 30 01:38:55.565314 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:55.565292 6572 scope.go:122] "RemoveContainer" containerID="44dedf7efa8d2e3fbeba9f5d1da22e7b919192af8691317efdad783a8e4ac85f" Oct 30 01:38:55.575620 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:55.575600 6572 scope.go:122] "RemoveContainer" containerID="92e064dc3aedb1117e3d505de23d71ffe4c5c802af0d57b2e883831dddcdf68c" Oct 30 01:38:56.155776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:56.155364 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:56.391563 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:56.391530 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:56.650175 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:56.648337 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:56.883654 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:56.883618 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:58.141947 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:58.141907 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:58.360963 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:58.360911 6572 patch_prober.go:29] interesting pod/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:38:58.361148 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:58.360980 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" podUID="eced80d4-0bf9-4f4e-a25e-0448b3a2f70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:38:58.495194 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:58.495150 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:58.967856 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:58.967820 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:59.254825 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:59.242833 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18560b-ffb64bd8c-f4c2s" podUID="24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad" containerName="payload-pre-processing" containerID="cri-o://72820a42e18e739e86733d7f830c029e5f970fad23d2fd846c9c7d90bde50d76" gracePeriod=25 Oct 30 01:38:59.299657 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:59.299622 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:59.562428 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:59.562400 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18560b-ffb64bd8c-f4c2s" Oct 30 01:38:59.746606 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:59.746508 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad-kube-api-access-6fsgs\" (UniqueName: \"kubernetes.io/projected/24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad-kube-api-access-6fsgs\") pod \"24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad\" (UID: \"24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad\") " Oct 30 01:38:59.747412 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:59.747386 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad-plugins-config-volume\") pod \"24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad\" (UID: \"24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad\") " Oct 30 01:38:59.749068 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:59.747967 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad-plugins-config-volume" pod "24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad" (UID: "24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:59.772132 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:59.772075 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad-kube-api-access-6fsgs" pod "24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad" (UID: "24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad"). InnerVolumeSpecName "kube-api-access-6fsgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:59.850079 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:59.849996 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fsgs\" (UniqueName: \"kubernetes.io/projected/24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad-kube-api-access-6fsgs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:38:59.850079 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:38:59.850036 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:00.033491 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:00.033457 6572 generic.go:356] "Generic (PLEG): container finished" podID="24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad" containerID="72820a42e18e739e86733d7f830c029e5f970fad23d2fd846c9c7d90bde50d76" exitCode=0 Oct 30 01:39:00.033686 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:00.033524 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18560b-ffb64bd8c-f4c2s" event={"ID":"24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad","Type":"ContainerDied","Data":"72820a42e18e739e86733d7f830c029e5f970fad23d2fd846c9c7d90bde50d76"} Oct 30 01:39:00.033686 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:00.033541 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18560b-ffb64bd8c-f4c2s" Oct 30 01:39:00.033686 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:00.033639 6572 scope.go:122] "RemoveContainer" containerID="72820a42e18e739e86733d7f830c029e5f970fad23d2fd846c9c7d90bde50d76" Oct 30 01:39:00.033901 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:00.033626 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18560b-ffb64bd8c-f4c2s" event={"ID":"24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad","Type":"ContainerDied","Data":"ffcd02f156ba0284d95ec2a9081ee8b3f4b779515738c42c01ee0d1de22e5d1c"} Oct 30 01:39:00.051535 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:00.051511 6572 scope.go:122] "RemoveContainer" containerID="72820a42e18e739e86733d7f830c029e5f970fad23d2fd846c9c7d90bde50d76" Oct 30 01:39:00.051974 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:39:00.051941 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72820a42e18e739e86733d7f830c029e5f970fad23d2fd846c9c7d90bde50d76\": container with ID starting with 72820a42e18e739e86733d7f830c029e5f970fad23d2fd846c9c7d90bde50d76 not found: ID does not exist" containerID="72820a42e18e739e86733d7f830c029e5f970fad23d2fd846c9c7d90bde50d76" Oct 30 01:39:00.052076 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:00.051987 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72820a42e18e739e86733d7f830c029e5f970fad23d2fd846c9c7d90bde50d76"} err="failed to get container status \"72820a42e18e739e86733d7f830c029e5f970fad23d2fd846c9c7d90bde50d76\": rpc error: code = NotFound desc = could not find container \"72820a42e18e739e86733d7f830c029e5f970fad23d2fd846c9c7d90bde50d76\": container with ID starting with 72820a42e18e739e86733d7f830c029e5f970fad23d2fd846c9c7d90bde50d76 not found: ID does not exist" Oct 30 01:39:00.087231 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:00.087191 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-18560b-ffb64bd8c-f4c2s"] Oct 30 01:39:00.094929 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:00.094894 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-18560b-ffb64bd8c-f4c2s"] Oct 30 01:39:00.240048 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:00.240009 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:00.453097 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:00.453059 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:00.543995 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:00.543915 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad" path="/var/lib/kubelet/pods/24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad/volumes" Oct 30 01:39:01.043411 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:01.043323 6572 generic.go:356] "Generic (PLEG): container finished" podID="eced80d4-0bf9-4f4e-a25e-0448b3a2f70b" containerID="6e137d476a869f00b916844a51918cba2c9b8f33aaccdbcba7b970bea4ff74c3" exitCode=0 Oct 30 01:39:01.043411 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:01.043396 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" event={"ID":"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b","Type":"ContainerDied","Data":"6e137d476a869f00b916844a51918cba2c9b8f33aaccdbcba7b970bea4ff74c3"} Oct 30 01:39:01.043928 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:01.043908 6572 scope.go:122] "RemoveContainer" containerID="6e137d476a869f00b916844a51918cba2c9b8f33aaccdbcba7b970bea4ff74c3" Oct 30 01:39:01.045624 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:01.045595 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:39:01.045692 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:01.045679 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:39:01.045740 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:01.045714 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:39:02.055741 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:02.055705 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" event={"ID":"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b","Type":"ContainerStarted","Data":"646812fd6e7a54a86ed8e647687f2b7f90232210e08da4ce53a5cd1afb084e21"} Oct 30 01:39:02.162679 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:02.162644 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:02.202818 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:02.202781 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-6b6795975-2hdzq"] Oct 30 01:39:02.203438 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:02.203423 6572 cpu_manager.go:414] "containerMap: removing container" podUID="24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad" containerName="payload-pre-processing" Oct 30 01:39:02.203485 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:02.203440 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad" containerName="payload-pre-processing" Oct 30 01:39:02.203485 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:02.203461 6572 cpu_manager.go:414] "containerMap: removing container" podUID="049c1151-7da9-4331-97c4-7cdc7f6e4e6b" containerName="revoke-keys" Oct 30 01:39:02.203485 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:02.203466 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="049c1151-7da9-4331-97c4-7cdc7f6e4e6b" containerName="revoke-keys" Oct 30 01:39:02.203593 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:02.203552 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="24a9c2bd-b4e6-4f34-ae3e-bdc63569b8ad" containerName="payload-pre-processing" Oct 30 01:39:02.203593 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:02.203563 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="049c1151-7da9-4331-97c4-7cdc7f6e4e6b" containerName="revoke-keys" Oct 30 01:39:02.207459 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:02.207428 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-6b6795975-2hdzq" Oct 30 01:39:02.241373 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:02.241343 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-fc51e9\"" Oct 30 01:39:02.242284 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:02.242244 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-fc51e9-dockercfg-8vpxx\"" Oct 30 01:39:02.245386 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:02.245359 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-6b6795975-2hdzq"] Oct 30 01:39:02.400883 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:02.379809 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-bf96646b8-n6q7h"] Oct 30 01:39:02.402256 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:02.401965 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-bf96646b8-n6q7h" Oct 30 01:39:02.466133 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:02.466054 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:02.479208 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:02.478711 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-bf96646b8-n6q7h"] Oct 30 01:39:02.489073 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:02.489033 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxcv9\" (UniqueName: \"kubernetes.io/projected/3c3467ed-9f98-4311-9943-f97b7c9f2ed6-kube-api-access-vxcv9\") pod \"payload-pre-processing-e2e-ipp-b-fc51e9-6b6795975-2hdzq\" (UID: \"3c3467ed-9f98-4311-9943-f97b7c9f2ed6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-6b6795975-2hdzq" Oct 30 01:39:02.489392 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:02.489360 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c3467ed-9f98-4311-9943-f97b7c9f2ed6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fc51e9-6b6795975-2hdzq\" (UID: \"3c3467ed-9f98-4311-9943-f97b7c9f2ed6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-6b6795975-2hdzq" Oct 30 01:39:02.533238 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:02.533205 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:39:02.533606 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:02.533584 6572 patch_prober.go:29] interesting pod/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 01:39:02.533839 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:02.533733 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" podUID="eced80d4-0bf9-4f4e-a25e-0448b3a2f70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 01:39:02.591185 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:02.591150 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56b43cf-29cd-4f22-b8e1-48feaaea009b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-fc51e9-bf96646b8-n6q7h\" (UID: \"c56b43cf-29cd-4f22-b8e1-48feaaea009b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-bf96646b8-n6q7h" Oct 30 01:39:02.591387 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:02.591227 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxcv9\" (UniqueName: \"kubernetes.io/projected/3c3467ed-9f98-4311-9943-f97b7c9f2ed6-kube-api-access-vxcv9\") pod \"payload-pre-processing-e2e-ipp-b-fc51e9-6b6795975-2hdzq\" (UID: \"3c3467ed-9f98-4311-9943-f97b7c9f2ed6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-6b6795975-2hdzq" Oct 30 01:39:02.591387 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:02.591314 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzgnv\" (UniqueName: \"kubernetes.io/projected/c56b43cf-29cd-4f22-b8e1-48feaaea009b-kube-api-access-fzgnv\") pod \"payload-processing-e2e-ipp-b-fc51e9-bf96646b8-n6q7h\" (UID: \"c56b43cf-29cd-4f22-b8e1-48feaaea009b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-bf96646b8-n6q7h" Oct 30 01:39:02.591387 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:02.591357 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c3467ed-9f98-4311-9943-f97b7c9f2ed6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fc51e9-6b6795975-2hdzq\" (UID: \"3c3467ed-9f98-4311-9943-f97b7c9f2ed6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-6b6795975-2hdzq" Oct 30 01:39:02.592159 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:02.592139 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c3467ed-9f98-4311-9943-f97b7c9f2ed6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fc51e9-6b6795975-2hdzq\" (UID: \"3c3467ed-9f98-4311-9943-f97b7c9f2ed6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-6b6795975-2hdzq" Oct 30 01:39:02.634815 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:02.634766 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxcv9\" (UniqueName: \"kubernetes.io/projected/3c3467ed-9f98-4311-9943-f97b7c9f2ed6-kube-api-access-vxcv9\") pod \"payload-pre-processing-e2e-ipp-b-fc51e9-6b6795975-2hdzq\" (UID: \"3c3467ed-9f98-4311-9943-f97b7c9f2ed6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-6b6795975-2hdzq" Oct 30 01:39:02.692706 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:02.692665 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56b43cf-29cd-4f22-b8e1-48feaaea009b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-fc51e9-bf96646b8-n6q7h\" (UID: \"c56b43cf-29cd-4f22-b8e1-48feaaea009b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-bf96646b8-n6q7h" Oct 30 01:39:02.692967 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:02.692807 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzgnv\" (UniqueName: \"kubernetes.io/projected/c56b43cf-29cd-4f22-b8e1-48feaaea009b-kube-api-access-fzgnv\") pod \"payload-processing-e2e-ipp-b-fc51e9-bf96646b8-n6q7h\" (UID: \"c56b43cf-29cd-4f22-b8e1-48feaaea009b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-bf96646b8-n6q7h" Oct 30 01:39:02.693489 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:02.693465 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56b43cf-29cd-4f22-b8e1-48feaaea009b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-fc51e9-bf96646b8-n6q7h\" (UID: \"c56b43cf-29cd-4f22-b8e1-48feaaea009b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-bf96646b8-n6q7h" Oct 30 01:39:02.731958 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:02.731918 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzgnv\" (UniqueName: \"kubernetes.io/projected/c56b43cf-29cd-4f22-b8e1-48feaaea009b-kube-api-access-fzgnv\") pod \"payload-processing-e2e-ipp-b-fc51e9-bf96646b8-n6q7h\" (UID: \"c56b43cf-29cd-4f22-b8e1-48feaaea009b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-bf96646b8-n6q7h" Oct 30 01:39:02.767017 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:02.766981 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-bf96646b8-n6q7h" Oct 30 01:39:02.839583 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:02.839519 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-6b6795975-2hdzq" Oct 30 01:39:03.076185 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:03.076124 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:03.092769 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:03.088799 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s"] Oct 30 01:39:03.095884 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:03.094166 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" podUID="35fa0269-eb44-4cf3-91df-50cf13230d1e" containerName="istio-proxy" containerID="cri-o://03137dcd2352e94cc2a793cada615a905f3d7e1096fc10fe889c04e670280f5e" gracePeriod=30 Oct 30 01:39:03.163210 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:03.163182 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx"] Oct 30 01:39:03.165144 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:03.165093 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" podUID="11caba41-1413-482d-8c53-03da9c396b26" containerName="istio-proxy" containerID="cri-o://aa426fc35ad5dda05e6838068e05d634ca4a6e7f4169f24f322e17e46a31581c" gracePeriod=30 Oct 30 01:39:03.179013 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:03.178637 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm"] Oct 30 01:39:03.179013 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:03.178925 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" podUID="eb45f4d6-fea9-4316-97c7-0fa7911073cc" containerName="istio-proxy" containerID="cri-o://26ee68b38be5a726933e3a4b35f1bdfd18674eb69cc9b9f71fb39aa566f38bd5" gracePeriod=30 Oct 30 01:39:03.183860 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:03.183827 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg"] Oct 30 01:39:03.184123 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:03.184099 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" podUID="7ca3df60-4274-4031-8eef-04b1416d22b8" containerName="istio-proxy" containerID="cri-o://2ef6caa5f52cf812ae4ae9c15500c040ad3d27bb51f303739dbb9470555f3d46" gracePeriod=30 Oct 30 01:39:03.189229 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:03.189199 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-bf96646b8-n6q7h"] Oct 30 01:39:03.195395 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:39:03.195370 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3467ed_9f98_4311_9943_f97b7c9f2ed6.slice/crio-71fdf1e7f04380d0907ec90f1d0879390b94b3da2ea5538641808bca61bbe4ba WatchSource:0}: Error finding container 71fdf1e7f04380d0907ec90f1d0879390b94b3da2ea5538641808bca61bbe4ba: Status 404 returned error can't find the container with id 71fdf1e7f04380d0907ec90f1d0879390b94b3da2ea5538641808bca61bbe4ba Oct 30 01:39:03.205149 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:03.205117 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-6b6795975-2hdzq"] Oct 30 01:39:03.532395 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:03.532230 6572 patch_prober.go:29] interesting pod/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 01:39:03.532395 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:03.532293 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" podUID="eced80d4-0bf9-4f4e-a25e-0448b3a2f70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 01:39:03.562590 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:03.562538 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:03.791171 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:03.791049 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:04.105049 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:04.104157 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-6b6795975-2hdzq" event={"ID":"3c3467ed-9f98-4311-9943-f97b7c9f2ed6","Type":"ContainerStarted","Data":"42dadc6e12777500b208c90fc732164aca4b53b21d9623fed93ae2274b9adde8"} Oct 30 01:39:04.105049 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:04.104255 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-6b6795975-2hdzq" event={"ID":"3c3467ed-9f98-4311-9943-f97b7c9f2ed6","Type":"ContainerStarted","Data":"71fdf1e7f04380d0907ec90f1d0879390b94b3da2ea5538641808bca61bbe4ba"} Oct 30 01:39:04.105049 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:04.104396 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-6b6795975-2hdzq" Oct 30 01:39:04.140368 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:04.140319 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-bf96646b8-n6q7h" event={"ID":"c56b43cf-29cd-4f22-b8e1-48feaaea009b","Type":"ContainerStarted","Data":"a6474b848c0df9bb38ee81935affb80b68cfce0f790433b3060afad030628fc7"} Oct 30 01:39:04.140563 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:04.140407 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-bf96646b8-n6q7h" event={"ID":"c56b43cf-29cd-4f22-b8e1-48feaaea009b","Type":"ContainerStarted","Data":"1bda6a9c862a1dd326e3500db14e528903cf261fb3a79587b2d660559f139255"} Oct 30 01:39:04.140563 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:04.140529 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-bf96646b8-n6q7h" Oct 30 01:39:04.163971 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:04.163903 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-6b6795975-2hdzq" podStartSLOduration=2.163879268 podStartE2EDuration="2.163879268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:39:04.159203045 +0000 UTC m=+1269.735628450" watchObservedRunningTime="2026-10-30 01:39:04.163879268 +0000 UTC m=+1269.740304670" Oct 30 01:39:04.199897 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:04.199832 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-bf96646b8-n6q7h" podStartSLOduration=2.199814399 podStartE2EDuration="2.199814399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:39:04.198899571 +0000 UTC m=+1269.775324974" watchObservedRunningTime="2026-10-30 01:39:04.199814399 +0000 UTC m=+1269.776239801" Oct 30 01:39:04.243450 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:04.243416 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:04.531877 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:04.531507 6572 patch_prober.go:29] interesting pod/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 01:39:04.531877 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:04.531560 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" podUID="eced80d4-0bf9-4f4e-a25e-0448b3a2f70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 01:39:04.543816 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:04.543784 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:05.040956 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:05.040915 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9121606-wg84d"] Oct 30 01:39:05.046680 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:05.046647 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9121606-wg84d"] Oct 30 01:39:05.434425 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:05.434391 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:05.532182 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:05.531600 6572 patch_prober.go:29] interesting pod/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 01:39:05.532182 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:05.531663 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" podUID="eced80d4-0bf9-4f4e-a25e-0448b3a2f70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 01:39:05.535376 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:05.535345 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5c97978cff-n6khx"] Oct 30 01:39:05.540543 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:05.540505 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5c97978cff-n6khx" Oct 30 01:39:05.565276 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:05.565109 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d8eb06-dockercfg-bsdgp\"" Oct 30 01:39:05.566114 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:05.566035 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d8eb06\"" Oct 30 01:39:05.568450 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:05.567970 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spc4m\" (UniqueName: \"kubernetes.io/projected/3c145c57-8343-42ed-9a5f-5c333ad1f598-kube-api-access-spc4m\") pod \"payload-pre-processing-e2e-ipp-b-d8eb06-5c97978cff-n6khx\" (UID: \"3c145c57-8343-42ed-9a5f-5c333ad1f598\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5c97978cff-n6khx" Oct 30 01:39:05.568450 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:05.568316 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c145c57-8343-42ed-9a5f-5c333ad1f598-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d8eb06-5c97978cff-n6khx\" (UID: \"3c145c57-8343-42ed-9a5f-5c333ad1f598\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5c97978cff-n6khx" Oct 30 01:39:05.571126 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:05.571101 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5c97978cff-n6khx"] Oct 30 01:39:05.691875 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:05.669622 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spc4m\" (UniqueName: \"kubernetes.io/projected/3c145c57-8343-42ed-9a5f-5c333ad1f598-kube-api-access-spc4m\") pod \"payload-pre-processing-e2e-ipp-b-d8eb06-5c97978cff-n6khx\" (UID: \"3c145c57-8343-42ed-9a5f-5c333ad1f598\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5c97978cff-n6khx" Oct 30 01:39:05.691875 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:05.669768 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c145c57-8343-42ed-9a5f-5c333ad1f598-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d8eb06-5c97978cff-n6khx\" (UID: \"3c145c57-8343-42ed-9a5f-5c333ad1f598\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5c97978cff-n6khx" Oct 30 01:39:05.691875 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:05.670942 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c145c57-8343-42ed-9a5f-5c333ad1f598-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d8eb06-5c97978cff-n6khx\" (UID: \"3c145c57-8343-42ed-9a5f-5c333ad1f598\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5c97978cff-n6khx" Oct 30 01:39:05.770782 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:05.770552 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spc4m\" (UniqueName: \"kubernetes.io/projected/3c145c57-8343-42ed-9a5f-5c333ad1f598-kube-api-access-spc4m\") pod \"payload-pre-processing-e2e-ipp-b-d8eb06-5c97978cff-n6khx\" (UID: \"3c145c57-8343-42ed-9a5f-5c333ad1f598\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5c97978cff-n6khx" Oct 30 01:39:05.783788 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:05.778062 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:05.858972 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:05.856842 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-587d749d6d-nd7r5"] Oct 30 01:39:05.858972 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:05.857991 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5c97978cff-n6khx" Oct 30 01:39:05.862315 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:05.862296 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-587d749d6d-nd7r5" Oct 30 01:39:05.863797 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:05.863768 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-587d749d6d-nd7r5"] Oct 30 01:39:05.880664 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:05.880631 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dac431a9-b17e-4b64-bad9-98cccf1e244b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d8eb06-587d749d6d-nd7r5\" (UID: \"dac431a9-b17e-4b64-bad9-98cccf1e244b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-587d749d6d-nd7r5" Oct 30 01:39:05.881183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:05.881141 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvzbd\" (UniqueName: \"kubernetes.io/projected/dac431a9-b17e-4b64-bad9-98cccf1e244b-kube-api-access-tvzbd\") pod \"payload-processing-e2e-ipp-b-d8eb06-587d749d6d-nd7r5\" (UID: \"dac431a9-b17e-4b64-bad9-98cccf1e244b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-587d749d6d-nd7r5" Oct 30 01:39:05.982428 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:05.982186 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvzbd\" (UniqueName: \"kubernetes.io/projected/dac431a9-b17e-4b64-bad9-98cccf1e244b-kube-api-access-tvzbd\") pod \"payload-processing-e2e-ipp-b-d8eb06-587d749d6d-nd7r5\" (UID: \"dac431a9-b17e-4b64-bad9-98cccf1e244b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-587d749d6d-nd7r5" Oct 30 01:39:05.982428 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:05.982264 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dac431a9-b17e-4b64-bad9-98cccf1e244b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d8eb06-587d749d6d-nd7r5\" (UID: \"dac431a9-b17e-4b64-bad9-98cccf1e244b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-587d749d6d-nd7r5" Oct 30 01:39:05.985107 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:05.983119 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dac431a9-b17e-4b64-bad9-98cccf1e244b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d8eb06-587d749d6d-nd7r5\" (UID: \"dac431a9-b17e-4b64-bad9-98cccf1e244b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-587d749d6d-nd7r5" Oct 30 01:39:06.003350 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:06.003273 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvzbd\" (UniqueName: \"kubernetes.io/projected/dac431a9-b17e-4b64-bad9-98cccf1e244b-kube-api-access-tvzbd\") pod \"payload-processing-e2e-ipp-b-d8eb06-587d749d6d-nd7r5\" (UID: \"dac431a9-b17e-4b64-bad9-98cccf1e244b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-587d749d6d-nd7r5" Oct 30 01:39:06.192120 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:06.192071 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-587d749d6d-nd7r5" Oct 30 01:39:06.235417 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:06.235260 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5c97978cff-n6khx"] Oct 30 01:39:06.238906 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:39:06.238871 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c145c57_8343_42ed_9a5f_5c333ad1f598.slice/crio-bf049505231d6196a7eddb97b093bc17fdf9c710e62d1f0e9dbc1fd37ffd8345 WatchSource:0}: Error finding container bf049505231d6196a7eddb97b093bc17fdf9c710e62d1f0e9dbc1fd37ffd8345: Status 404 returned error can't find the container with id bf049505231d6196a7eddb97b093bc17fdf9c710e62d1f0e9dbc1fd37ffd8345 Oct 30 01:39:06.537193 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:06.536250 6572 patch_prober.go:29] interesting pod/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 01:39:06.537193 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:06.536309 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" podUID="eced80d4-0bf9-4f4e-a25e-0448b3a2f70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 01:39:06.540900 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:39:06.540865 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac431a9_b17e_4b64_bad9_98cccf1e244b.slice/crio-2fd12fbed8c003e8b361e0bbc62439680e664915ced93f4a8a8b9eb3240fc293 WatchSource:0}: Error finding container 2fd12fbed8c003e8b361e0bbc62439680e664915ced93f4a8a8b9eb3240fc293: Status 404 returned error can't find the container with id 2fd12fbed8c003e8b361e0bbc62439680e664915ced93f4a8a8b9eb3240fc293 Oct 30 01:39:06.547673 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:06.547640 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9e2aef8-2ffc-4b17-8c7f-25265e2056e7" path="/var/lib/kubelet/pods/f9e2aef8-2ffc-4b17-8c7f-25265e2056e7/volumes" Oct 30 01:39:06.548063 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:06.548044 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-587d749d6d-nd7r5"] Oct 30 01:39:07.169264 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:07.169219 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-587d749d6d-nd7r5" event={"ID":"dac431a9-b17e-4b64-bad9-98cccf1e244b","Type":"ContainerStarted","Data":"bff63ad3fe909c7d59a6a86ae6a2deb8754e6d15537cf4090350ae9924ccc680"} Oct 30 01:39:07.169480 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:07.169277 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-587d749d6d-nd7r5" event={"ID":"dac431a9-b17e-4b64-bad9-98cccf1e244b","Type":"ContainerStarted","Data":"2fd12fbed8c003e8b361e0bbc62439680e664915ced93f4a8a8b9eb3240fc293"} Oct 30 01:39:07.169480 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:07.169453 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-587d749d6d-nd7r5" Oct 30 01:39:07.171099 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:07.171066 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5c97978cff-n6khx" event={"ID":"3c145c57-8343-42ed-9a5f-5c333ad1f598","Type":"ContainerStarted","Data":"2b38acca3f81dafbb2652e9b0e333762bee1e380b08d862c96410b877c655d17"} Oct 30 01:39:07.171238 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:07.171114 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5c97978cff-n6khx" event={"ID":"3c145c57-8343-42ed-9a5f-5c333ad1f598","Type":"ContainerStarted","Data":"bf049505231d6196a7eddb97b093bc17fdf9c710e62d1f0e9dbc1fd37ffd8345"} Oct 30 01:39:07.171238 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:07.171197 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5c97978cff-n6khx" Oct 30 01:39:07.230814 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:07.230024 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-587d749d6d-nd7r5" podStartSLOduration=2.22999982 podStartE2EDuration="2.22999982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:39:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:39:07.2102689 +0000 UTC m=+1272.786694307" watchObservedRunningTime="2026-10-30 01:39:07.22999982 +0000 UTC m=+1272.806425223" Oct 30 01:39:07.260636 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:07.260563 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5c97978cff-n6khx" podStartSLOduration=2.260547498 podStartE2EDuration="2.260547498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:39:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:39:07.25792347 +0000 UTC m=+1272.834348871" watchObservedRunningTime="2026-10-30 01:39:07.260547498 +0000 UTC m=+1272.836972900" Oct 30 01:39:07.531735 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:07.531333 6572 patch_prober.go:29] interesting pod/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 01:39:07.531735 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:07.531385 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" podUID="eced80d4-0bf9-4f4e-a25e-0448b3a2f70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 01:39:07.941439 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:07.941402 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:08.491991 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.490873 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" Oct 30 01:39:08.517946 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.517863 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11caba41-1413-482d-8c53-03da9c396b26-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11caba41-1413-482d-8c53-03da9c396b26-credential-socket\") pod \"11caba41-1413-482d-8c53-03da9c396b26\" (UID: \"11caba41-1413-482d-8c53-03da9c396b26\") " Oct 30 01:39:08.517946 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.517930 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11caba41-1413-482d-8c53-03da9c396b26-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11caba41-1413-482d-8c53-03da9c396b26-istio-data\") pod \"11caba41-1413-482d-8c53-03da9c396b26\" (UID: \"11caba41-1413-482d-8c53-03da9c396b26\") " Oct 30 01:39:08.518130 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.517984 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11caba41-1413-482d-8c53-03da9c396b26-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11caba41-1413-482d-8c53-03da9c396b26-istiod-ca-cert\") pod \"11caba41-1413-482d-8c53-03da9c396b26\" (UID: \"11caba41-1413-482d-8c53-03da9c396b26\") " Oct 30 01:39:08.518130 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.518123 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11caba41-1413-482d-8c53-03da9c396b26-istio-token\" (UniqueName: \"kubernetes.io/projected/11caba41-1413-482d-8c53-03da9c396b26-istio-token\") pod \"11caba41-1413-482d-8c53-03da9c396b26\" (UID: \"11caba41-1413-482d-8c53-03da9c396b26\") " Oct 30 01:39:08.519817 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.518201 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11caba41-1413-482d-8c53-03da9c396b26-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11caba41-1413-482d-8c53-03da9c396b26-istio-envoy\") pod \"11caba41-1413-482d-8c53-03da9c396b26\" (UID: \"11caba41-1413-482d-8c53-03da9c396b26\") " Oct 30 01:39:08.519817 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.518313 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11caba41-1413-482d-8c53-03da9c396b26-istio-data" pod "11caba41-1413-482d-8c53-03da9c396b26" (UID: "11caba41-1413-482d-8c53-03da9c396b26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:08.519817 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.518325 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11caba41-1413-482d-8c53-03da9c396b26-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11caba41-1413-482d-8c53-03da9c396b26-istio-podinfo\") pod \"11caba41-1413-482d-8c53-03da9c396b26\" (UID: \"11caba41-1413-482d-8c53-03da9c396b26\") " Oct 30 01:39:08.519817 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.518366 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11caba41-1413-482d-8c53-03da9c396b26-kube-api-access-m56nm\" (UniqueName: \"kubernetes.io/projected/11caba41-1413-482d-8c53-03da9c396b26-kube-api-access-m56nm\") pod \"11caba41-1413-482d-8c53-03da9c396b26\" (UID: \"11caba41-1413-482d-8c53-03da9c396b26\") " Oct 30 01:39:08.519817 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.518431 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11caba41-1413-482d-8c53-03da9c396b26-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11caba41-1413-482d-8c53-03da9c396b26-workload-certs\") pod \"11caba41-1413-482d-8c53-03da9c396b26\" (UID: \"11caba41-1413-482d-8c53-03da9c396b26\") " Oct 30 01:39:08.519817 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.518482 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11caba41-1413-482d-8c53-03da9c396b26-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11caba41-1413-482d-8c53-03da9c396b26-workload-socket\") pod \"11caba41-1413-482d-8c53-03da9c396b26\" (UID: \"11caba41-1413-482d-8c53-03da9c396b26\") " Oct 30 01:39:08.519817 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.519005 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11caba41-1413-482d-8c53-03da9c396b26-credential-socket" pod "11caba41-1413-482d-8c53-03da9c396b26" (UID: "11caba41-1413-482d-8c53-03da9c396b26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:08.519817 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.519243 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11caba41-1413-482d-8c53-03da9c396b26-workload-socket" pod "11caba41-1413-482d-8c53-03da9c396b26" (UID: "11caba41-1413-482d-8c53-03da9c396b26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:08.519817 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.519247 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11caba41-1413-482d-8c53-03da9c396b26-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:08.519817 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.519287 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11caba41-1413-482d-8c53-03da9c396b26-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:08.519817 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.519334 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11caba41-1413-482d-8c53-03da9c396b26-istiod-ca-cert" pod "11caba41-1413-482d-8c53-03da9c396b26" (UID: "11caba41-1413-482d-8c53-03da9c396b26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:39:08.519817 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.519621 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11caba41-1413-482d-8c53-03da9c396b26-workload-certs" pod "11caba41-1413-482d-8c53-03da9c396b26" (UID: "11caba41-1413-482d-8c53-03da9c396b26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:08.523384 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.523072 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11caba41-1413-482d-8c53-03da9c396b26-istio-envoy" pod "11caba41-1413-482d-8c53-03da9c396b26" (UID: "11caba41-1413-482d-8c53-03da9c396b26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:08.523384 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.523291 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11caba41-1413-482d-8c53-03da9c396b26-istio-token" pod "11caba41-1413-482d-8c53-03da9c396b26" (UID: "11caba41-1413-482d-8c53-03da9c396b26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:39:08.524894 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.524870 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11caba41-1413-482d-8c53-03da9c396b26-kube-api-access-m56nm" pod "11caba41-1413-482d-8c53-03da9c396b26" (UID: "11caba41-1413-482d-8c53-03da9c396b26"). InnerVolumeSpecName "kube-api-access-m56nm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:39:08.525332 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.525310 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11caba41-1413-482d-8c53-03da9c396b26-istio-podinfo" pod "11caba41-1413-482d-8c53-03da9c396b26" (UID: "11caba41-1413-482d-8c53-03da9c396b26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:39:08.534223 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.534158 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" Oct 30 01:39:08.538456 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.538422 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:39:08.557800 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.557339 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:39:08.558555 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.558174 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:39:08.559283 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.559232 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:39:08.582370 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.582335 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" Oct 30 01:39:08.620711 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.620669 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb45f4d6-fea9-4316-97c7-0fa7911073cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb45f4d6-fea9-4316-97c7-0fa7911073cc-credential-socket\") pod \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\" (UID: \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\") " Oct 30 01:39:08.620908 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.620791 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb45f4d6-fea9-4316-97c7-0fa7911073cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb45f4d6-fea9-4316-97c7-0fa7911073cc-istio-envoy\") pod \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\" (UID: \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\") " Oct 30 01:39:08.620908 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.620829 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb45f4d6-fea9-4316-97c7-0fa7911073cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb45f4d6-fea9-4316-97c7-0fa7911073cc-istio-podinfo\") pod \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\" (UID: \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\") " Oct 30 01:39:08.620908 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.620867 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35fa0269-eb44-4cf3-91df-50cf13230d1e-kube-api-access-85f65\" (UniqueName: \"kubernetes.io/projected/35fa0269-eb44-4cf3-91df-50cf13230d1e-kube-api-access-85f65\") pod \"35fa0269-eb44-4cf3-91df-50cf13230d1e\" (UID: \"35fa0269-eb44-4cf3-91df-50cf13230d1e\") " Oct 30 01:39:08.620908 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.620901 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35fa0269-eb44-4cf3-91df-50cf13230d1e-istio-token\" (UniqueName: \"kubernetes.io/projected/35fa0269-eb44-4cf3-91df-50cf13230d1e-istio-token\") pod \"35fa0269-eb44-4cf3-91df-50cf13230d1e\" (UID: \"35fa0269-eb44-4cf3-91df-50cf13230d1e\") " Oct 30 01:39:08.621076 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.620930 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ca3df60-4274-4031-8eef-04b1416d22b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca3df60-4274-4031-8eef-04b1416d22b8-istio-podinfo\") pod \"7ca3df60-4274-4031-8eef-04b1416d22b8\" (UID: \"7ca3df60-4274-4031-8eef-04b1416d22b8\") " Oct 30 01:39:08.621076 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.621012 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb45f4d6-fea9-4316-97c7-0fa7911073cc-credential-socket" pod "eb45f4d6-fea9-4316-97c7-0fa7911073cc" (UID: "eb45f4d6-fea9-4316-97c7-0fa7911073cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:08.621076 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.620965 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca3df60-4274-4031-8eef-04b1416d22b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca3df60-4274-4031-8eef-04b1416d22b8-istio-data\") pod \"7ca3df60-4274-4031-8eef-04b1416d22b8\" (UID: \"7ca3df60-4274-4031-8eef-04b1416d22b8\") " Oct 30 01:39:08.621345 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.621307 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca3df60-4274-4031-8eef-04b1416d22b8-istio-data" pod "7ca3df60-4274-4031-8eef-04b1416d22b8" (UID: "7ca3df60-4274-4031-8eef-04b1416d22b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:08.622029 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.622004 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ca3df60-4274-4031-8eef-04b1416d22b8-kube-api-access-swv8h\" (UniqueName: \"kubernetes.io/projected/7ca3df60-4274-4031-8eef-04b1416d22b8-kube-api-access-swv8h\") pod \"7ca3df60-4274-4031-8eef-04b1416d22b8\" (UID: \"7ca3df60-4274-4031-8eef-04b1416d22b8\") " Oct 30 01:39:08.622420 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.622398 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35fa0269-eb44-4cf3-91df-50cf13230d1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35fa0269-eb44-4cf3-91df-50cf13230d1e-istio-podinfo\") pod \"35fa0269-eb44-4cf3-91df-50cf13230d1e\" (UID: \"35fa0269-eb44-4cf3-91df-50cf13230d1e\") " Oct 30 01:39:08.622849 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.622827 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca3df60-4274-4031-8eef-04b1416d22b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca3df60-4274-4031-8eef-04b1416d22b8-istio-envoy\") pod \"7ca3df60-4274-4031-8eef-04b1416d22b8\" (UID: \"7ca3df60-4274-4031-8eef-04b1416d22b8\") " Oct 30 01:39:08.622954 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.622885 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35fa0269-eb44-4cf3-91df-50cf13230d1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fa0269-eb44-4cf3-91df-50cf13230d1e-workload-socket\") pod \"35fa0269-eb44-4cf3-91df-50cf13230d1e\" (UID: \"35fa0269-eb44-4cf3-91df-50cf13230d1e\") " Oct 30 01:39:08.622954 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.622924 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ca3df60-4274-4031-8eef-04b1416d22b8-istio-token\" (UniqueName: \"kubernetes.io/projected/7ca3df60-4274-4031-8eef-04b1416d22b8-istio-token\") pod \"7ca3df60-4274-4031-8eef-04b1416d22b8\" (UID: \"7ca3df60-4274-4031-8eef-04b1416d22b8\") " Oct 30 01:39:08.623091 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.622985 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb45f4d6-fea9-4316-97c7-0fa7911073cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb45f4d6-fea9-4316-97c7-0fa7911073cc-istiod-ca-cert\") pod \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\" (UID: \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\") " Oct 30 01:39:08.623091 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.623059 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb45f4d6-fea9-4316-97c7-0fa7911073cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb45f4d6-fea9-4316-97c7-0fa7911073cc-workload-socket\") pod \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\" (UID: \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\") " Oct 30 01:39:08.623202 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.623095 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35fa0269-eb44-4cf3-91df-50cf13230d1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35fa0269-eb44-4cf3-91df-50cf13230d1e-istio-envoy\") pod \"35fa0269-eb44-4cf3-91df-50cf13230d1e\" (UID: \"35fa0269-eb44-4cf3-91df-50cf13230d1e\") " Oct 30 01:39:08.623202 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.623150 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb45f4d6-fea9-4316-97c7-0fa7911073cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb45f4d6-fea9-4316-97c7-0fa7911073cc-workload-certs\") pod \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\" (UID: \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\") " Oct 30 01:39:08.623313 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.623242 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb45f4d6-fea9-4316-97c7-0fa7911073cc-istio-token\" (UniqueName: \"kubernetes.io/projected/eb45f4d6-fea9-4316-97c7-0fa7911073cc-istio-token\") pod \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\" (UID: \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\") " Oct 30 01:39:08.623375 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.623342 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca3df60-4274-4031-8eef-04b1416d22b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca3df60-4274-4031-8eef-04b1416d22b8-workload-socket\") pod \"7ca3df60-4274-4031-8eef-04b1416d22b8\" (UID: \"7ca3df60-4274-4031-8eef-04b1416d22b8\") " Oct 30 01:39:08.623440 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.623400 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb45f4d6-fea9-4316-97c7-0fa7911073cc-kube-api-access-49dlf\" (UniqueName: \"kubernetes.io/projected/eb45f4d6-fea9-4316-97c7-0fa7911073cc-kube-api-access-49dlf\") pod \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\" (UID: \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\") " Oct 30 01:39:08.623505 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.623457 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35fa0269-eb44-4cf3-91df-50cf13230d1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35fa0269-eb44-4cf3-91df-50cf13230d1e-istio-data\") pod \"35fa0269-eb44-4cf3-91df-50cf13230d1e\" (UID: \"35fa0269-eb44-4cf3-91df-50cf13230d1e\") " Oct 30 01:39:08.623601 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.623505 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ca3df60-4274-4031-8eef-04b1416d22b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca3df60-4274-4031-8eef-04b1416d22b8-istiod-ca-cert\") pod \"7ca3df60-4274-4031-8eef-04b1416d22b8\" (UID: \"7ca3df60-4274-4031-8eef-04b1416d22b8\") " Oct 30 01:39:08.623601 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.623576 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca3df60-4274-4031-8eef-04b1416d22b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca3df60-4274-4031-8eef-04b1416d22b8-credential-socket\") pod \"7ca3df60-4274-4031-8eef-04b1416d22b8\" (UID: \"7ca3df60-4274-4031-8eef-04b1416d22b8\") " Oct 30 01:39:08.623710 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.623699 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb45f4d6-fea9-4316-97c7-0fa7911073cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb45f4d6-fea9-4316-97c7-0fa7911073cc-istio-data\") pod \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\" (UID: \"eb45f4d6-fea9-4316-97c7-0fa7911073cc\") " Oct 30 01:39:08.623976 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.623949 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35fa0269-eb44-4cf3-91df-50cf13230d1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35fa0269-eb44-4cf3-91df-50cf13230d1e-istiod-ca-cert\") pod \"35fa0269-eb44-4cf3-91df-50cf13230d1e\" (UID: \"35fa0269-eb44-4cf3-91df-50cf13230d1e\") " Oct 30 01:39:08.624088 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.623990 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca3df60-4274-4031-8eef-04b1416d22b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca3df60-4274-4031-8eef-04b1416d22b8-workload-certs\") pod \"7ca3df60-4274-4031-8eef-04b1416d22b8\" (UID: \"7ca3df60-4274-4031-8eef-04b1416d22b8\") " Oct 30 01:39:08.624088 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.624044 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35fa0269-eb44-4cf3-91df-50cf13230d1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35fa0269-eb44-4cf3-91df-50cf13230d1e-workload-certs\") pod \"35fa0269-eb44-4cf3-91df-50cf13230d1e\" (UID: \"35fa0269-eb44-4cf3-91df-50cf13230d1e\") " Oct 30 01:39:08.624232 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.624109 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35fa0269-eb44-4cf3-91df-50cf13230d1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fa0269-eb44-4cf3-91df-50cf13230d1e-credential-socket\") pod \"35fa0269-eb44-4cf3-91df-50cf13230d1e\" (UID: \"35fa0269-eb44-4cf3-91df-50cf13230d1e\") " Oct 30 01:39:08.624396 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.624368 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca3df60-4274-4031-8eef-04b1416d22b8-workload-socket" pod "7ca3df60-4274-4031-8eef-04b1416d22b8" (UID: "7ca3df60-4274-4031-8eef-04b1416d22b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:08.624787 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.624720 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35fa0269-eb44-4cf3-91df-50cf13230d1e-workload-certs" pod "35fa0269-eb44-4cf3-91df-50cf13230d1e" (UID: "35fa0269-eb44-4cf3-91df-50cf13230d1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:08.624787 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.624727 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb45f4d6-fea9-4316-97c7-0fa7911073cc-istio-data" pod "eb45f4d6-fea9-4316-97c7-0fa7911073cc" (UID: "eb45f4d6-fea9-4316-97c7-0fa7911073cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:08.624948 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.624738 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb45f4d6-fea9-4316-97c7-0fa7911073cc-workload-socket" pod "eb45f4d6-fea9-4316-97c7-0fa7911073cc" (UID: "eb45f4d6-fea9-4316-97c7-0fa7911073cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:08.625034 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.625007 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35fa0269-eb44-4cf3-91df-50cf13230d1e-workload-socket" pod "35fa0269-eb44-4cf3-91df-50cf13230d1e" (UID: "35fa0269-eb44-4cf3-91df-50cf13230d1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:08.625034 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.625024 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35fa0269-eb44-4cf3-91df-50cf13230d1e-istio-data" pod "35fa0269-eb44-4cf3-91df-50cf13230d1e" (UID: "35fa0269-eb44-4cf3-91df-50cf13230d1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:08.625292 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.625130 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca3df60-4274-4031-8eef-04b1416d22b8-kube-api-access-swv8h" pod "7ca3df60-4274-4031-8eef-04b1416d22b8" (UID: "7ca3df60-4274-4031-8eef-04b1416d22b8"). InnerVolumeSpecName "kube-api-access-swv8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:39:08.625292 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.625253 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35fa0269-eb44-4cf3-91df-50cf13230d1e-kube-api-access-85f65" pod "35fa0269-eb44-4cf3-91df-50cf13230d1e" (UID: "35fa0269-eb44-4cf3-91df-50cf13230d1e"). InnerVolumeSpecName "kube-api-access-85f65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:39:08.625292 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.625265 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca3df60-4274-4031-8eef-04b1416d22b8-workload-certs" pod "7ca3df60-4274-4031-8eef-04b1416d22b8" (UID: "7ca3df60-4274-4031-8eef-04b1416d22b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:08.626025 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.625432 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb45f4d6-fea9-4316-97c7-0fa7911073cc-workload-certs" pod "eb45f4d6-fea9-4316-97c7-0fa7911073cc" (UID: "eb45f4d6-fea9-4316-97c7-0fa7911073cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:08.626025 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.625474 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35fa0269-eb44-4cf3-91df-50cf13230d1e-credential-socket" pod "35fa0269-eb44-4cf3-91df-50cf13230d1e" (UID: "35fa0269-eb44-4cf3-91df-50cf13230d1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:08.626025 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.625516 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca3df60-4274-4031-8eef-04b1416d22b8-credential-socket" pod "7ca3df60-4274-4031-8eef-04b1416d22b8" (UID: "7ca3df60-4274-4031-8eef-04b1416d22b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:08.626025 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.625864 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35fa0269-eb44-4cf3-91df-50cf13230d1e-istiod-ca-cert" pod "35fa0269-eb44-4cf3-91df-50cf13230d1e" (UID: "35fa0269-eb44-4cf3-91df-50cf13230d1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:39:08.626869 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.626443 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11caba41-1413-482d-8c53-03da9c396b26-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:08.626869 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.626466 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fa0269-eb44-4cf3-91df-50cf13230d1e-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:08.626869 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.626483 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb45f4d6-fea9-4316-97c7-0fa7911073cc-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:08.626869 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.626699 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb45f4d6-fea9-4316-97c7-0fa7911073cc-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:08.626869 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.626716 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11caba41-1413-482d-8c53-03da9c396b26-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:08.626869 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.626732 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m56nm\" (UniqueName: \"kubernetes.io/projected/11caba41-1413-482d-8c53-03da9c396b26-kube-api-access-m56nm\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:08.626869 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.626786 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11caba41-1413-482d-8c53-03da9c396b26-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:08.626869 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.626801 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca3df60-4274-4031-8eef-04b1416d22b8-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:08.626869 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.626816 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11caba41-1413-482d-8c53-03da9c396b26-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:08.626869 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.626849 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35fa0269-eb44-4cf3-91df-50cf13230d1e-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:08.626869 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.626866 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca3df60-4274-4031-8eef-04b1416d22b8-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:08.628987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.626881 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb45f4d6-fea9-4316-97c7-0fa7911073cc-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:08.628987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.626896 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35fa0269-eb44-4cf3-91df-50cf13230d1e-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:08.628987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.626937 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca3df60-4274-4031-8eef-04b1416d22b8-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:08.628987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.626952 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35fa0269-eb44-4cf3-91df-50cf13230d1e-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:08.628987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.626968 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11caba41-1413-482d-8c53-03da9c396b26-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:08.628987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.627007 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fa0269-eb44-4cf3-91df-50cf13230d1e-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:08.628987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.627023 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb45f4d6-fea9-4316-97c7-0fa7911073cc-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:08.628987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.627038 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11caba41-1413-482d-8c53-03da9c396b26-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:08.628987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.627053 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85f65\" (UniqueName: \"kubernetes.io/projected/35fa0269-eb44-4cf3-91df-50cf13230d1e-kube-api-access-85f65\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:08.628987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.627071 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca3df60-4274-4031-8eef-04b1416d22b8-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:08.628987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.627088 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swv8h\" (UniqueName: \"kubernetes.io/projected/7ca3df60-4274-4031-8eef-04b1416d22b8-kube-api-access-swv8h\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:08.628987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.627341 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb45f4d6-fea9-4316-97c7-0fa7911073cc-istiod-ca-cert" pod "eb45f4d6-fea9-4316-97c7-0fa7911073cc" (UID: "eb45f4d6-fea9-4316-97c7-0fa7911073cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:39:08.628987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.627446 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca3df60-4274-4031-8eef-04b1416d22b8-istio-envoy" pod "7ca3df60-4274-4031-8eef-04b1416d22b8" (UID: "7ca3df60-4274-4031-8eef-04b1416d22b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:08.628987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.627521 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ca3df60-4274-4031-8eef-04b1416d22b8-istiod-ca-cert" pod "7ca3df60-4274-4031-8eef-04b1416d22b8" (UID: "7ca3df60-4274-4031-8eef-04b1416d22b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:39:08.628987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.627919 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35fa0269-eb44-4cf3-91df-50cf13230d1e-istio-token" pod "35fa0269-eb44-4cf3-91df-50cf13230d1e" (UID: "35fa0269-eb44-4cf3-91df-50cf13230d1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:39:08.629910 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.629626 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb45f4d6-fea9-4316-97c7-0fa7911073cc-istio-token" pod "eb45f4d6-fea9-4316-97c7-0fa7911073cc" (UID: "eb45f4d6-fea9-4316-97c7-0fa7911073cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:39:08.630120 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.630095 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35fa0269-eb44-4cf3-91df-50cf13230d1e-istio-envoy" pod "35fa0269-eb44-4cf3-91df-50cf13230d1e" (UID: "35fa0269-eb44-4cf3-91df-50cf13230d1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:08.630276 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.630176 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb45f4d6-fea9-4316-97c7-0fa7911073cc-istio-podinfo" pod "eb45f4d6-fea9-4316-97c7-0fa7911073cc" (UID: "eb45f4d6-fea9-4316-97c7-0fa7911073cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:39:08.630361 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.630302 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb45f4d6-fea9-4316-97c7-0fa7911073cc-kube-api-access-49dlf" pod "eb45f4d6-fea9-4316-97c7-0fa7911073cc" (UID: "eb45f4d6-fea9-4316-97c7-0fa7911073cc"). InnerVolumeSpecName "kube-api-access-49dlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:39:08.630454 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.630425 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb45f4d6-fea9-4316-97c7-0fa7911073cc-istio-envoy" pod "eb45f4d6-fea9-4316-97c7-0fa7911073cc" (UID: "eb45f4d6-fea9-4316-97c7-0fa7911073cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:08.630538 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.630459 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ca3df60-4274-4031-8eef-04b1416d22b8-istio-podinfo" pod "7ca3df60-4274-4031-8eef-04b1416d22b8" (UID: "7ca3df60-4274-4031-8eef-04b1416d22b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:39:08.630609 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.630574 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35fa0269-eb44-4cf3-91df-50cf13230d1e-istio-podinfo" pod "35fa0269-eb44-4cf3-91df-50cf13230d1e" (UID: "35fa0269-eb44-4cf3-91df-50cf13230d1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:39:08.632215 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.632183 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca3df60-4274-4031-8eef-04b1416d22b8-istio-token" pod "7ca3df60-4274-4031-8eef-04b1416d22b8" (UID: "7ca3df60-4274-4031-8eef-04b1416d22b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:39:08.728743 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.728633 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca3df60-4274-4031-8eef-04b1416d22b8-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:08.728743 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.728667 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ca3df60-4274-4031-8eef-04b1416d22b8-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:08.728743 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.728676 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb45f4d6-fea9-4316-97c7-0fa7911073cc-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:08.728743 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.728687 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35fa0269-eb44-4cf3-91df-50cf13230d1e-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:08.728743 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.728695 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb45f4d6-fea9-4316-97c7-0fa7911073cc-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:08.728743 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.728704 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49dlf\" (UniqueName: \"kubernetes.io/projected/eb45f4d6-fea9-4316-97c7-0fa7911073cc-kube-api-access-49dlf\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:08.728743 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.728718 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca3df60-4274-4031-8eef-04b1416d22b8-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:08.728743 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.728730 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb45f4d6-fea9-4316-97c7-0fa7911073cc-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:08.728743 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.728739 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb45f4d6-fea9-4316-97c7-0fa7911073cc-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:08.728743 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.728760 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35fa0269-eb44-4cf3-91df-50cf13230d1e-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:08.728743 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.728768 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca3df60-4274-4031-8eef-04b1416d22b8-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:08.728743 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:08.728776 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35fa0269-eb44-4cf3-91df-50cf13230d1e-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:09.196682 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.196646 6572 generic.go:356] "Generic (PLEG): container finished" podID="eb45f4d6-fea9-4316-97c7-0fa7911073cc" containerID="26ee68b38be5a726933e3a4b35f1bdfd18674eb69cc9b9f71fb39aa566f38bd5" exitCode=0 Oct 30 01:39:09.197155 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.196733 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" Oct 30 01:39:09.197155 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.196769 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" event={"ID":"eb45f4d6-fea9-4316-97c7-0fa7911073cc","Type":"ContainerDied","Data":"26ee68b38be5a726933e3a4b35f1bdfd18674eb69cc9b9f71fb39aa566f38bd5"} Oct 30 01:39:09.197155 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.196860 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm" event={"ID":"eb45f4d6-fea9-4316-97c7-0fa7911073cc","Type":"ContainerDied","Data":"d9c8ab09ce7e0b547369f5ec92eccca00ac04949985a372d63646696e654dd1b"} Oct 30 01:39:09.197155 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.196893 6572 scope.go:122] "RemoveContainer" containerID="26ee68b38be5a726933e3a4b35f1bdfd18674eb69cc9b9f71fb39aa566f38bd5" Oct 30 01:39:09.198722 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.198688 6572 generic.go:356] "Generic (PLEG): container finished" podID="35fa0269-eb44-4cf3-91df-50cf13230d1e" containerID="03137dcd2352e94cc2a793cada615a905f3d7e1096fc10fe889c04e670280f5e" exitCode=0 Oct 30 01:39:09.198859 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.198786 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" Oct 30 01:39:09.198947 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.198875 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" event={"ID":"35fa0269-eb44-4cf3-91df-50cf13230d1e","Type":"ContainerDied","Data":"03137dcd2352e94cc2a793cada615a905f3d7e1096fc10fe889c04e670280f5e"} Oct 30 01:39:09.199008 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.198985 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s" event={"ID":"35fa0269-eb44-4cf3-91df-50cf13230d1e","Type":"ContainerDied","Data":"2e24cf56fc89b78b626c95845be9966098f80d289f035262f717caff0910fac3"} Oct 30 01:39:09.202080 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.202057 6572 generic.go:356] "Generic (PLEG): container finished" podID="7ca3df60-4274-4031-8eef-04b1416d22b8" containerID="2ef6caa5f52cf812ae4ae9c15500c040ad3d27bb51f303739dbb9470555f3d46" exitCode=0 Oct 30 01:39:09.202217 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.202087 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" event={"ID":"7ca3df60-4274-4031-8eef-04b1416d22b8","Type":"ContainerDied","Data":"2ef6caa5f52cf812ae4ae9c15500c040ad3d27bb51f303739dbb9470555f3d46"} Oct 30 01:39:09.202217 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.202134 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" Oct 30 01:39:09.202217 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.202142 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg" event={"ID":"7ca3df60-4274-4031-8eef-04b1416d22b8","Type":"ContainerDied","Data":"139f2dc64dd3756744dcc4c9e94d2dd6737481db6f7073d2c3e2be6dc1e23344"} Oct 30 01:39:09.203894 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.203874 6572 generic.go:356] "Generic (PLEG): container finished" podID="11caba41-1413-482d-8c53-03da9c396b26" containerID="aa426fc35ad5dda05e6838068e05d634ca4a6e7f4169f24f322e17e46a31581c" exitCode=0 Oct 30 01:39:09.204022 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.203900 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" event={"ID":"11caba41-1413-482d-8c53-03da9c396b26","Type":"ContainerDied","Data":"aa426fc35ad5dda05e6838068e05d634ca4a6e7f4169f24f322e17e46a31581c"} Oct 30 01:39:09.204022 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.203947 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" event={"ID":"11caba41-1413-482d-8c53-03da9c396b26","Type":"ContainerDied","Data":"8167e3193f2f17090e559215a91ac4b3b9d90f2933af92945d2ec5af47cf30e2"} Oct 30 01:39:09.204022 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.203947 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx" Oct 30 01:39:09.222596 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.222578 6572 scope.go:122] "RemoveContainer" containerID="44f47ba8ff23967e0342eae0a84955d77338ce62231d4e898a68f1dd83c74a9a" Oct 30 01:39:09.234373 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.234353 6572 scope.go:122] "RemoveContainer" containerID="26ee68b38be5a726933e3a4b35f1bdfd18674eb69cc9b9f71fb39aa566f38bd5" Oct 30 01:39:09.234700 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:39:09.234672 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26ee68b38be5a726933e3a4b35f1bdfd18674eb69cc9b9f71fb39aa566f38bd5\": container with ID starting with 26ee68b38be5a726933e3a4b35f1bdfd18674eb69cc9b9f71fb39aa566f38bd5 not found: ID does not exist" containerID="26ee68b38be5a726933e3a4b35f1bdfd18674eb69cc9b9f71fb39aa566f38bd5" Oct 30 01:39:09.234788 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.234704 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26ee68b38be5a726933e3a4b35f1bdfd18674eb69cc9b9f71fb39aa566f38bd5"} err="failed to get container status \"26ee68b38be5a726933e3a4b35f1bdfd18674eb69cc9b9f71fb39aa566f38bd5\": rpc error: code = NotFound desc = could not find container \"26ee68b38be5a726933e3a4b35f1bdfd18674eb69cc9b9f71fb39aa566f38bd5\": container with ID starting with 26ee68b38be5a726933e3a4b35f1bdfd18674eb69cc9b9f71fb39aa566f38bd5 not found: ID does not exist" Oct 30 01:39:09.234788 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.234722 6572 scope.go:122] "RemoveContainer" containerID="44f47ba8ff23967e0342eae0a84955d77338ce62231d4e898a68f1dd83c74a9a" Oct 30 01:39:09.235045 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:39:09.235021 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44f47ba8ff23967e0342eae0a84955d77338ce62231d4e898a68f1dd83c74a9a\": container with ID starting with 44f47ba8ff23967e0342eae0a84955d77338ce62231d4e898a68f1dd83c74a9a not found: ID does not exist" containerID="44f47ba8ff23967e0342eae0a84955d77338ce62231d4e898a68f1dd83c74a9a" Oct 30 01:39:09.235103 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.235051 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44f47ba8ff23967e0342eae0a84955d77338ce62231d4e898a68f1dd83c74a9a"} err="failed to get container status \"44f47ba8ff23967e0342eae0a84955d77338ce62231d4e898a68f1dd83c74a9a\": rpc error: code = NotFound desc = could not find container \"44f47ba8ff23967e0342eae0a84955d77338ce62231d4e898a68f1dd83c74a9a\": container with ID starting with 44f47ba8ff23967e0342eae0a84955d77338ce62231d4e898a68f1dd83c74a9a not found: ID does not exist" Oct 30 01:39:09.235103 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.235066 6572 scope.go:122] "RemoveContainer" containerID="03137dcd2352e94cc2a793cada615a905f3d7e1096fc10fe889c04e670280f5e" Oct 30 01:39:09.245271 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.245253 6572 scope.go:122] "RemoveContainer" containerID="03137dcd2352e94cc2a793cada615a905f3d7e1096fc10fe889c04e670280f5e" Oct 30 01:39:09.245566 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:39:09.245543 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03137dcd2352e94cc2a793cada615a905f3d7e1096fc10fe889c04e670280f5e\": container with ID starting with 03137dcd2352e94cc2a793cada615a905f3d7e1096fc10fe889c04e670280f5e not found: ID does not exist" containerID="03137dcd2352e94cc2a793cada615a905f3d7e1096fc10fe889c04e670280f5e" Oct 30 01:39:09.245614 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.245573 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03137dcd2352e94cc2a793cada615a905f3d7e1096fc10fe889c04e670280f5e"} err="failed to get container status \"03137dcd2352e94cc2a793cada615a905f3d7e1096fc10fe889c04e670280f5e\": rpc error: code = NotFound desc = could not find container \"03137dcd2352e94cc2a793cada615a905f3d7e1096fc10fe889c04e670280f5e\": container with ID starting with 03137dcd2352e94cc2a793cada615a905f3d7e1096fc10fe889c04e670280f5e not found: ID does not exist" Oct 30 01:39:09.245614 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.245588 6572 scope.go:122] "RemoveContainer" containerID="2ef6caa5f52cf812ae4ae9c15500c040ad3d27bb51f303739dbb9470555f3d46" Oct 30 01:39:09.256024 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.256001 6572 scope.go:122] "RemoveContainer" containerID="2ef6caa5f52cf812ae4ae9c15500c040ad3d27bb51f303739dbb9470555f3d46" Oct 30 01:39:09.256457 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:39:09.256434 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ef6caa5f52cf812ae4ae9c15500c040ad3d27bb51f303739dbb9470555f3d46\": container with ID starting with 2ef6caa5f52cf812ae4ae9c15500c040ad3d27bb51f303739dbb9470555f3d46 not found: ID does not exist" containerID="2ef6caa5f52cf812ae4ae9c15500c040ad3d27bb51f303739dbb9470555f3d46" Oct 30 01:39:09.256544 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.256462 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ef6caa5f52cf812ae4ae9c15500c040ad3d27bb51f303739dbb9470555f3d46"} err="failed to get container status \"2ef6caa5f52cf812ae4ae9c15500c040ad3d27bb51f303739dbb9470555f3d46\": rpc error: code = NotFound desc = could not find container \"2ef6caa5f52cf812ae4ae9c15500c040ad3d27bb51f303739dbb9470555f3d46\": container with ID starting with 2ef6caa5f52cf812ae4ae9c15500c040ad3d27bb51f303739dbb9470555f3d46 not found: ID does not exist" Oct 30 01:39:09.256544 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.256493 6572 scope.go:122] "RemoveContainer" containerID="aa426fc35ad5dda05e6838068e05d634ca4a6e7f4169f24f322e17e46a31581c" Oct 30 01:39:09.266843 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.266806 6572 scope.go:122] "RemoveContainer" containerID="aa426fc35ad5dda05e6838068e05d634ca4a6e7f4169f24f322e17e46a31581c" Oct 30 01:39:09.267143 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:39:09.267119 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa426fc35ad5dda05e6838068e05d634ca4a6e7f4169f24f322e17e46a31581c\": container with ID starting with aa426fc35ad5dda05e6838068e05d634ca4a6e7f4169f24f322e17e46a31581c not found: ID does not exist" containerID="aa426fc35ad5dda05e6838068e05d634ca4a6e7f4169f24f322e17e46a31581c" Oct 30 01:39:09.267193 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.267151 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa426fc35ad5dda05e6838068e05d634ca4a6e7f4169f24f322e17e46a31581c"} err="failed to get container status \"aa426fc35ad5dda05e6838068e05d634ca4a6e7f4169f24f322e17e46a31581c\": rpc error: code = NotFound desc = could not find container \"aa426fc35ad5dda05e6838068e05d634ca4a6e7f4169f24f322e17e46a31581c\": container with ID starting with aa426fc35ad5dda05e6838068e05d634ca4a6e7f4169f24f322e17e46a31581c not found: ID does not exist" Oct 30 01:39:09.294030 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.293991 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg"] Oct 30 01:39:09.306163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.306128 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qsjsg"] Oct 30 01:39:09.341970 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.341937 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm"] Oct 30 01:39:09.349173 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.349138 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-qksxm"] Oct 30 01:39:09.383477 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.383443 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx"] Oct 30 01:39:09.394996 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.394680 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-phmlx"] Oct 30 01:39:09.422888 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.422855 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s"] Oct 30 01:39:09.432228 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.432195 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fwb5s"] Oct 30 01:39:09.661582 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:09.661549 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:10.544199 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:10.544163 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11caba41-1413-482d-8c53-03da9c396b26" path="/var/lib/kubelet/pods/11caba41-1413-482d-8c53-03da9c396b26/volumes" Oct 30 01:39:10.544662 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:10.544647 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35fa0269-eb44-4cf3-91df-50cf13230d1e" path="/var/lib/kubelet/pods/35fa0269-eb44-4cf3-91df-50cf13230d1e/volumes" Oct 30 01:39:10.545201 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:10.545181 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ca3df60-4274-4031-8eef-04b1416d22b8" path="/var/lib/kubelet/pods/7ca3df60-4274-4031-8eef-04b1416d22b8/volumes" Oct 30 01:39:10.545983 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:10.545960 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb45f4d6-fea9-4316-97c7-0fa7911073cc" path="/var/lib/kubelet/pods/eb45f4d6-fea9-4316-97c7-0fa7911073cc/volumes" Oct 30 01:39:11.397943 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:11.397913 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:13.167477 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:13.167435 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:14.687480 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:39:14.687434 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:39:14.972384 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:14.972333 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:15.154485 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:15.154456 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-6b6795975-2hdzq" Oct 30 01:39:15.154656 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:15.154544 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-bf96646b8-n6q7h" Oct 30 01:39:15.817976 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:15.817936 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:16.172674 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.172636 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv"] Oct 30 01:39:16.186425 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.186243 6572 cpu_manager.go:414] "containerMap: removing container" podUID="11caba41-1413-482d-8c53-03da9c396b26" containerName="istio-proxy" Oct 30 01:39:16.186425 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.186275 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11caba41-1413-482d-8c53-03da9c396b26" containerName="istio-proxy" Oct 30 01:39:16.186425 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.186295 6572 cpu_manager.go:414] "containerMap: removing container" podUID="eb45f4d6-fea9-4316-97c7-0fa7911073cc" containerName="istio-proxy" Oct 30 01:39:16.186425 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.186302 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb45f4d6-fea9-4316-97c7-0fa7911073cc" containerName="istio-proxy" Oct 30 01:39:16.186425 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.186315 6572 cpu_manager.go:414] "containerMap: removing container" podUID="35fa0269-eb44-4cf3-91df-50cf13230d1e" containerName="istio-proxy" Oct 30 01:39:16.186425 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.186322 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35fa0269-eb44-4cf3-91df-50cf13230d1e" containerName="istio-proxy" Oct 30 01:39:16.186425 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.186335 6572 cpu_manager.go:414] "containerMap: removing container" podUID="7ca3df60-4274-4031-8eef-04b1416d22b8" containerName="istio-proxy" Oct 30 01:39:16.186425 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.186342 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ca3df60-4274-4031-8eef-04b1416d22b8" containerName="istio-proxy" Oct 30 01:39:16.189343 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.187156 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb45f4d6-fea9-4316-97c7-0fa7911073cc" containerName="istio-proxy" Oct 30 01:39:16.189343 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.187196 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="11caba41-1413-482d-8c53-03da9c396b26" containerName="istio-proxy" Oct 30 01:39:16.189343 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.187210 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="35fa0269-eb44-4cf3-91df-50cf13230d1e" containerName="istio-proxy" Oct 30 01:39:16.189343 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.187221 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ca3df60-4274-4031-8eef-04b1416d22b8" containerName="istio-proxy" Oct 30 01:39:16.191678 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.191651 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:39:16.210189 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.209093 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv"] Oct 30 01:39:16.210262 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.210188 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-0db534-openshift-default-dockercfg-c27wt\"" Oct 30 01:39:16.324967 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.324914 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwcgh\" (UniqueName: \"kubernetes.io/projected/bcef90f8-7163-4fcc-94df-f10f46f8aa56-kube-api-access-vwcgh\") pod \"e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv\" (UID: \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\") " pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:39:16.325281 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.325034 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcef90f8-7163-4fcc-94df-f10f46f8aa56-istio-token\") pod \"e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv\" (UID: \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\") " pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:39:16.325281 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.325113 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcef90f8-7163-4fcc-94df-f10f46f8aa56-istio-podinfo\") pod \"e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv\" (UID: \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\") " pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:39:16.325281 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.325184 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcef90f8-7163-4fcc-94df-f10f46f8aa56-credential-socket\") pod \"e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv\" (UID: \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\") " pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:39:16.325281 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.325237 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcef90f8-7163-4fcc-94df-f10f46f8aa56-workload-socket\") pod \"e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv\" (UID: \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\") " pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:39:16.325281 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.325262 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcef90f8-7163-4fcc-94df-f10f46f8aa56-istio-envoy\") pod \"e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv\" (UID: \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\") " pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:39:16.325574 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.325333 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcef90f8-7163-4fcc-94df-f10f46f8aa56-istio-data\") pod \"e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv\" (UID: \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\") " pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:39:16.325574 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.325351 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcef90f8-7163-4fcc-94df-f10f46f8aa56-workload-certs\") pod \"e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv\" (UID: \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\") " pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:39:16.325574 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.325400 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcef90f8-7163-4fcc-94df-f10f46f8aa56-istiod-ca-cert\") pod \"e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv\" (UID: \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\") " pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:39:16.349285 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.349242 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:16.427082 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.426977 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcef90f8-7163-4fcc-94df-f10f46f8aa56-credential-socket\") pod \"e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv\" (UID: \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\") " pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:39:16.427082 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.427039 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcef90f8-7163-4fcc-94df-f10f46f8aa56-workload-socket\") pod \"e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv\" (UID: \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\") " pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:39:16.427082 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.427065 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcef90f8-7163-4fcc-94df-f10f46f8aa56-istio-envoy\") pod \"e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv\" (UID: \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\") " pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:39:16.427390 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.427216 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcef90f8-7163-4fcc-94df-f10f46f8aa56-istio-data\") pod \"e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv\" (UID: \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\") " pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:39:16.427390 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.427241 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcef90f8-7163-4fcc-94df-f10f46f8aa56-workload-certs\") pod \"e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv\" (UID: \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\") " pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:39:16.427390 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.427319 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcef90f8-7163-4fcc-94df-f10f46f8aa56-istiod-ca-cert\") pod \"e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv\" (UID: \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\") " pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:39:16.427390 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.427389 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwcgh\" (UniqueName: \"kubernetes.io/projected/bcef90f8-7163-4fcc-94df-f10f46f8aa56-kube-api-access-vwcgh\") pod \"e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv\" (UID: \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\") " pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:39:16.427584 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.427451 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcef90f8-7163-4fcc-94df-f10f46f8aa56-istio-token\") pod \"e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv\" (UID: \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\") " pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:39:16.427584 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.427449 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcef90f8-7163-4fcc-94df-f10f46f8aa56-credential-socket\") pod \"e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv\" (UID: \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\") " pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:39:16.427584 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.427489 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcef90f8-7163-4fcc-94df-f10f46f8aa56-workload-socket\") pod \"e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv\" (UID: \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\") " pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:39:16.427584 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.427549 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcef90f8-7163-4fcc-94df-f10f46f8aa56-istio-podinfo\") pod \"e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv\" (UID: \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\") " pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:39:16.427903 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.427876 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcef90f8-7163-4fcc-94df-f10f46f8aa56-workload-certs\") pod \"e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv\" (UID: \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\") " pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:39:16.428558 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.428530 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcef90f8-7163-4fcc-94df-f10f46f8aa56-istiod-ca-cert\") pod \"e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv\" (UID: \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\") " pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:39:16.428691 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.428680 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcef90f8-7163-4fcc-94df-f10f46f8aa56-istio-data\") pod \"e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv\" (UID: \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\") " pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:39:16.429995 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.429971 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcef90f8-7163-4fcc-94df-f10f46f8aa56-istio-envoy\") pod \"e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv\" (UID: \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\") " pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:39:16.430307 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.430288 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcef90f8-7163-4fcc-94df-f10f46f8aa56-istio-podinfo\") pod \"e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv\" (UID: \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\") " pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:39:16.448824 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.448790 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwcgh\" (UniqueName: \"kubernetes.io/projected/bcef90f8-7163-4fcc-94df-f10f46f8aa56-kube-api-access-vwcgh\") pod \"e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv\" (UID: \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\") " pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:39:16.448974 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.448856 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcef90f8-7163-4fcc-94df-f10f46f8aa56-istio-token\") pod \"e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv\" (UID: \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\") " pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:39:16.515951 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.515907 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:39:16.547392 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.547336 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:16.843928 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.843899 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv"] Oct 30 01:39:16.844643 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:39:16.844613 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcef90f8_7163_4fcc_94df_f10f46f8aa56.slice/crio-71200f73d4c4ab9bcaef2831339755e3786a2a9c4083b7be08487c6e4efdd6b5 WatchSource:0}: Error finding container 71200f73d4c4ab9bcaef2831339755e3786a2a9c4083b7be08487c6e4efdd6b5: Status 404 returned error can't find the container with id 71200f73d4c4ab9bcaef2831339755e3786a2a9c4083b7be08487c6e4efdd6b5 Oct 30 01:39:16.847224 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.847190 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:39:16.847334 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.847267 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:39:16.847334 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:16.847312 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:39:17.278383 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:17.278341 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" event={"ID":"bcef90f8-7163-4fcc-94df-f10f46f8aa56","Type":"ContainerStarted","Data":"8deeafbbc128ac76469ef1bb3adcabb7ddf26e84a96ef395a10f33c865e21d20"} Oct 30 01:39:17.278552 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:17.278411 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" event={"ID":"bcef90f8-7163-4fcc-94df-f10f46f8aa56","Type":"ContainerStarted","Data":"71200f73d4c4ab9bcaef2831339755e3786a2a9c4083b7be08487c6e4efdd6b5"} Oct 30 01:39:17.336622 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:17.336566 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" podStartSLOduration=1.336550592 podStartE2EDuration="1.336550592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:39:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:39:17.335966668 +0000 UTC m=+1282.912392106" watchObservedRunningTime="2026-10-30 01:39:17.336550592 +0000 UTC m=+1282.912975993" Oct 30 01:39:17.488818 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:17.488773 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:17.517014 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:17.516982 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:39:17.523925 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:17.523896 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:39:18.188465 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:18.188436 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-587d749d6d-nd7r5" Oct 30 01:39:18.188899 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:18.188513 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5c97978cff-n6khx" Oct 30 01:39:18.288255 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:18.288226 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:39:18.289391 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:18.289363 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:39:19.278966 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:19.278926 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:19.820725 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:19.820669 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:20.041227 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:20.041186 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:20.737769 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:20.732353 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:20.923469 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:20.923437 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:21.044887 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:21.042011 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-85bd8d7b8c-bhtcj"] Oct 30 01:39:21.044887 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:21.043226 6572 cpu_manager.go:414] "containerMap: removing container" podUID="eb45f4d6-fea9-4316-97c7-0fa7911073cc" containerName="istio-proxy" Oct 30 01:39:21.044887 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:21.043247 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb45f4d6-fea9-4316-97c7-0fa7911073cc" containerName="istio-proxy" Oct 30 01:39:21.044887 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:21.043445 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb45f4d6-fea9-4316-97c7-0fa7911073cc" containerName="istio-proxy" Oct 30 01:39:21.053253 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:21.053224 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-85bd8d7b8c-bhtcj" Oct 30 01:39:21.066683 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:21.066027 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-85bd8d7b8c-bhtcj"] Oct 30 01:39:21.081705 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:21.081573 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6b3dbd-dockercfg-wf8q6\"" Oct 30 01:39:21.081705 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:21.081664 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6b3dbd\"" Oct 30 01:39:21.194015 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:21.193979 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6fbdd58c6d-5xdjd"] Oct 30 01:39:21.197545 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:21.197512 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75596710-fb32-477b-93e7-37f13e5b0099-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6b3dbd-85bd8d7b8c-bhtcj\" (UID: \"75596710-fb32-477b-93e7-37f13e5b0099\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-85bd8d7b8c-bhtcj" Oct 30 01:39:21.197776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:21.197698 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xrc8\" (UniqueName: \"kubernetes.io/projected/75596710-fb32-477b-93e7-37f13e5b0099-kube-api-access-2xrc8\") pod \"payload-pre-processing-e2e-ipp-a-6b3dbd-85bd8d7b8c-bhtcj\" (UID: \"75596710-fb32-477b-93e7-37f13e5b0099\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-85bd8d7b8c-bhtcj" Oct 30 01:39:21.198564 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:21.198544 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6fbdd58c6d-5xdjd" Oct 30 01:39:21.220939 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:21.220904 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6fbdd58c6d-5xdjd"] Oct 30 01:39:21.298767 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:21.298662 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d90b45f-4155-4aee-a02e-b7a69295a8a7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6b3dbd-6fbdd58c6d-5xdjd\" (UID: \"7d90b45f-4155-4aee-a02e-b7a69295a8a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6fbdd58c6d-5xdjd" Oct 30 01:39:21.298767 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:21.298717 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xrc8\" (UniqueName: \"kubernetes.io/projected/75596710-fb32-477b-93e7-37f13e5b0099-kube-api-access-2xrc8\") pod \"payload-pre-processing-e2e-ipp-a-6b3dbd-85bd8d7b8c-bhtcj\" (UID: \"75596710-fb32-477b-93e7-37f13e5b0099\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-85bd8d7b8c-bhtcj" Oct 30 01:39:21.298989 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:21.298918 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75596710-fb32-477b-93e7-37f13e5b0099-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6b3dbd-85bd8d7b8c-bhtcj\" (UID: \"75596710-fb32-477b-93e7-37f13e5b0099\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-85bd8d7b8c-bhtcj" Oct 30 01:39:21.299085 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:21.299038 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkjtz\" (UniqueName: \"kubernetes.io/projected/7d90b45f-4155-4aee-a02e-b7a69295a8a7-kube-api-access-dkjtz\") pod \"payload-processing-e2e-ipp-a-6b3dbd-6fbdd58c6d-5xdjd\" (UID: \"7d90b45f-4155-4aee-a02e-b7a69295a8a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6fbdd58c6d-5xdjd" Oct 30 01:39:21.299580 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:21.299558 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75596710-fb32-477b-93e7-37f13e5b0099-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6b3dbd-85bd8d7b8c-bhtcj\" (UID: \"75596710-fb32-477b-93e7-37f13e5b0099\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-85bd8d7b8c-bhtcj" Oct 30 01:39:21.318946 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:21.318916 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xrc8\" (UniqueName: \"kubernetes.io/projected/75596710-fb32-477b-93e7-37f13e5b0099-kube-api-access-2xrc8\") pod \"payload-pre-processing-e2e-ipp-a-6b3dbd-85bd8d7b8c-bhtcj\" (UID: \"75596710-fb32-477b-93e7-37f13e5b0099\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-85bd8d7b8c-bhtcj" Oct 30 01:39:21.400289 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:21.399906 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-85bd8d7b8c-bhtcj" Oct 30 01:39:21.400289 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:21.400147 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkjtz\" (UniqueName: \"kubernetes.io/projected/7d90b45f-4155-4aee-a02e-b7a69295a8a7-kube-api-access-dkjtz\") pod \"payload-processing-e2e-ipp-a-6b3dbd-6fbdd58c6d-5xdjd\" (UID: \"7d90b45f-4155-4aee-a02e-b7a69295a8a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6fbdd58c6d-5xdjd" Oct 30 01:39:21.400289 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:21.400265 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d90b45f-4155-4aee-a02e-b7a69295a8a7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6b3dbd-6fbdd58c6d-5xdjd\" (UID: \"7d90b45f-4155-4aee-a02e-b7a69295a8a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6fbdd58c6d-5xdjd" Oct 30 01:39:21.401236 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:21.401213 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d90b45f-4155-4aee-a02e-b7a69295a8a7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6b3dbd-6fbdd58c6d-5xdjd\" (UID: \"7d90b45f-4155-4aee-a02e-b7a69295a8a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6fbdd58c6d-5xdjd" Oct 30 01:39:21.426495 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:21.426458 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkjtz\" (UniqueName: \"kubernetes.io/projected/7d90b45f-4155-4aee-a02e-b7a69295a8a7-kube-api-access-dkjtz\") pod \"payload-processing-e2e-ipp-a-6b3dbd-6fbdd58c6d-5xdjd\" (UID: \"7d90b45f-4155-4aee-a02e-b7a69295a8a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6fbdd58c6d-5xdjd" Oct 30 01:39:21.526258 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:21.525908 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6fbdd58c6d-5xdjd" Oct 30 01:39:21.711867 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:21.711829 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-85bd8d7b8c-bhtcj"] Oct 30 01:39:21.713978 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:39:21.713941 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75596710_fb32_477b_93e7_37f13e5b0099.slice/crio-7e3714fb74c66f82498464122ae4a8a5556d0f3f6d6774414f5184a177200404 WatchSource:0}: Error finding container 7e3714fb74c66f82498464122ae4a8a5556d0f3f6d6774414f5184a177200404: Status 404 returned error can't find the container with id 7e3714fb74c66f82498464122ae4a8a5556d0f3f6d6774414f5184a177200404 Oct 30 01:39:21.731267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:21.731234 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:21.757911 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:21.757863 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0db534-5d88d59d4b-z4jpk"] Oct 30 01:39:21.764461 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:21.764430 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db534-5d88d59d4b-z4jpk" Oct 30 01:39:21.779922 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:21.779885 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-0db534\"" Oct 30 01:39:21.782165 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:21.782139 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0db534-5d88d59d4b-z4jpk"] Oct 30 01:39:21.883851 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:21.883046 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6fbdd58c6d-5xdjd"] Oct 30 01:39:21.914962 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:21.914925 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0db534-5d88d59d4b-z4jpk\" (UID: \"a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0db534-5d88d59d4b-z4jpk" Oct 30 01:39:21.914962 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:21.914966 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-966l9\" (UniqueName: \"kubernetes.io/projected/a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196-kube-api-access-966l9\") pod \"payload-processing-e2e-ipp-b-0db534-5d88d59d4b-z4jpk\" (UID: \"a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0db534-5d88d59d4b-z4jpk" Oct 30 01:39:21.951281 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:21.951244 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:22.016586 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:22.016535 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0db534-5d88d59d4b-z4jpk\" (UID: \"a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0db534-5d88d59d4b-z4jpk" Oct 30 01:39:22.016586 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:22.016587 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-966l9\" (UniqueName: \"kubernetes.io/projected/a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196-kube-api-access-966l9\") pod \"payload-processing-e2e-ipp-b-0db534-5d88d59d4b-z4jpk\" (UID: \"a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0db534-5d88d59d4b-z4jpk" Oct 30 01:39:22.018309 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:22.018276 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0db534-5d88d59d4b-z4jpk\" (UID: \"a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0db534-5d88d59d4b-z4jpk" Oct 30 01:39:22.036641 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:22.036605 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-966l9\" (UniqueName: \"kubernetes.io/projected/a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196-kube-api-access-966l9\") pod \"payload-processing-e2e-ipp-b-0db534-5d88d59d4b-z4jpk\" (UID: \"a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0db534-5d88d59d4b-z4jpk" Oct 30 01:39:22.096505 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:22.096215 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db534-5d88d59d4b-z4jpk" Oct 30 01:39:22.140785 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:22.140689 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:22.337806 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:22.333826 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6fbdd58c6d-5xdjd" event={"ID":"7d90b45f-4155-4aee-a02e-b7a69295a8a7","Type":"ContainerStarted","Data":"8fc13abda953cd92747a44b22ea2cee04a467286c4ca36f7fc25f79babe51707"} Oct 30 01:39:22.337806 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:22.333931 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6fbdd58c6d-5xdjd" event={"ID":"7d90b45f-4155-4aee-a02e-b7a69295a8a7","Type":"ContainerStarted","Data":"1b540790c3f26379176ae718d988c8df29180ed0003815a6c96724125882ffcf"} Oct 30 01:39:22.337806 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:22.334421 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6fbdd58c6d-5xdjd" Oct 30 01:39:22.337806 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:22.336936 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-85bd8d7b8c-bhtcj" event={"ID":"75596710-fb32-477b-93e7-37f13e5b0099","Type":"ContainerStarted","Data":"4720f94b9237b7ebbe152d5e48f77cf1bf32264f4c3418c6c9933b1e2caa010a"} Oct 30 01:39:22.337806 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:22.336992 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-85bd8d7b8c-bhtcj" event={"ID":"75596710-fb32-477b-93e7-37f13e5b0099","Type":"ContainerStarted","Data":"7e3714fb74c66f82498464122ae4a8a5556d0f3f6d6774414f5184a177200404"} Oct 30 01:39:22.337806 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:22.337630 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-85bd8d7b8c-bhtcj" Oct 30 01:39:22.379215 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:22.379157 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6fbdd58c6d-5xdjd" podStartSLOduration=1.379142297 podStartE2EDuration="1.379142297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:39:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:39:22.374270614 +0000 UTC m=+1287.950696018" watchObservedRunningTime="2026-10-30 01:39:22.379142297 +0000 UTC m=+1287.955567699" Oct 30 01:39:22.425194 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:22.416156 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-85bd8d7b8c-bhtcj" podStartSLOduration=1.41613629 podStartE2EDuration="1.41613629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:39:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:39:22.412072297 +0000 UTC m=+1287.988497699" watchObservedRunningTime="2026-10-30 01:39:22.41613629 +0000 UTC m=+1287.992561700" Oct 30 01:39:22.444380 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:22.442554 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0db534-5d88d59d4b-z4jpk"] Oct 30 01:39:22.446905 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:39:22.446880 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda17631e3_0dc4_4ce0_bfc1_6f6c6b02f196.slice/crio-a80c304efd4fa10879b9dce40469f66d67b0ac5f74fb398351c51671ccdcf191 WatchSource:0}: Error finding container a80c304efd4fa10879b9dce40469f66d67b0ac5f74fb398351c51671ccdcf191: Status 404 returned error can't find the container with id a80c304efd4fa10879b9dce40469f66d67b0ac5f74fb398351c51671ccdcf191 Oct 30 01:39:23.346396 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:23.346354 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db534-5d88d59d4b-z4jpk" event={"ID":"a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196","Type":"ContainerStarted","Data":"7351daed5dc085d4868fd0d0f35ae9d6cbf5d3da08ac1846df9d3601918811ce"} Oct 30 01:39:23.347051 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:23.346423 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db534-5d88d59d4b-z4jpk" event={"ID":"a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196","Type":"ContainerStarted","Data":"a80c304efd4fa10879b9dce40469f66d67b0ac5f74fb398351c51671ccdcf191"} Oct 30 01:39:23.347051 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:23.346563 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db534-5d88d59d4b-z4jpk" Oct 30 01:39:23.385200 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:23.384916 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db534-5d88d59d4b-z4jpk" podStartSLOduration=2.384901615 podStartE2EDuration="2.384901615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:39:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:39:23.383958881 +0000 UTC m=+1288.960384282" watchObservedRunningTime="2026-10-30 01:39:23.384901615 +0000 UTC m=+1288.961327016" Oct 30 01:39:24.008981 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:24.008942 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:26.059038 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:26.059006 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:26.286759 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:26.286716 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:27.893232 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:27.893183 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:29.832217 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:29.832167 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:32.033394 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:32.033365 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:33.349624 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:33.349595 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6fbdd58c6d-5xdjd" Oct 30 01:39:33.733899 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:33.733858 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:34.357101 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:34.357057 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-85bd8d7b8c-bhtcj" Oct 30 01:39:34.358586 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:34.358563 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db534-5d88d59d4b-z4jpk" Oct 30 01:39:35.451035 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:35.451001 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:36.325719 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:36.325688 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:36.630021 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:36.629921 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859"] Oct 30 01:39:36.636123 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:36.636095 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:39:36.654920 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:36.654886 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859"] Oct 30 01:39:36.660403 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:36.660362 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-f01cbc-openshift-default-dockercfg-b7tzw\"" Oct 30 01:39:36.736619 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:36.736573 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-workload-certs\") pod \"e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859\" (UID: \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\") " pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:39:36.736868 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:36.736628 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-istiod-ca-cert\") pod \"e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859\" (UID: \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\") " pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:39:36.736868 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:36.736788 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-credential-socket\") pod \"e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859\" (UID: \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\") " pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:39:36.736868 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:36.736856 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmzg2\" (UniqueName: \"kubernetes.io/projected/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-kube-api-access-zmzg2\") pod \"e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859\" (UID: \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\") " pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:39:36.737072 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:36.736965 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-istio-data\") pod \"e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859\" (UID: \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\") " pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:39:36.737072 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:36.737021 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-istio-envoy\") pod \"e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859\" (UID: \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\") " pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:39:36.737072 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:36.737057 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-istio-token\") pod \"e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859\" (UID: \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\") " pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:39:36.737224 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:36.737093 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-istio-podinfo\") pod \"e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859\" (UID: \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\") " pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:39:36.737224 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:36.737191 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-workload-socket\") pod \"e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859\" (UID: \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\") " pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:39:36.782772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:36.774419 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:36.838427 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:36.838391 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-workload-certs\") pod \"e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859\" (UID: \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\") " pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:39:36.838427 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:36.838431 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-istiod-ca-cert\") pod \"e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859\" (UID: \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\") " pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:39:36.838710 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:36.838467 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-credential-socket\") pod \"e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859\" (UID: \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\") " pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:39:36.838710 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:36.838516 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmzg2\" (UniqueName: \"kubernetes.io/projected/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-kube-api-access-zmzg2\") pod \"e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859\" (UID: \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\") " pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:39:36.838710 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:36.838610 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-istio-data\") pod \"e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859\" (UID: \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\") " pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:39:36.838710 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:36.838660 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-istio-envoy\") pod \"e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859\" (UID: \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\") " pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:39:36.838710 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:36.838693 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-istio-token\") pod \"e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859\" (UID: \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\") " pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:39:36.838918 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:36.838720 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-istio-podinfo\") pod \"e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859\" (UID: \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\") " pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:39:36.838918 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:36.838816 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-workload-socket\") pod \"e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859\" (UID: \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\") " pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:39:36.839287 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:36.839261 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-credential-socket\") pod \"e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859\" (UID: \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\") " pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:39:36.839603 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:36.839578 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-workload-socket\") pod \"e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859\" (UID: \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\") " pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:39:36.839716 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:36.839682 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-istio-data\") pod \"e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859\" (UID: \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\") " pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:39:36.839904 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:36.839879 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-istiod-ca-cert\") pod \"e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859\" (UID: \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\") " pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:39:36.840138 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:36.840116 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-workload-certs\") pod \"e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859\" (UID: \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\") " pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:39:36.842172 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:36.842152 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-istio-envoy\") pod \"e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859\" (UID: \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\") " pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:39:36.842571 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:36.842548 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-istio-podinfo\") pod \"e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859\" (UID: \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\") " pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:39:36.858901 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:36.858868 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-istio-token\") pod \"e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859\" (UID: \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\") " pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:39:36.859214 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:36.859193 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmzg2\" (UniqueName: \"kubernetes.io/projected/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-kube-api-access-zmzg2\") pod \"e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859\" (UID: \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\") " pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:39:36.952555 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:36.952524 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:39:37.268602 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:37.268575 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859"] Oct 30 01:39:37.269920 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:39:37.269898 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd2951a_980f_4a1a_bb49_20e1d9fce44d.slice/crio-34035b8b52676aaaff038657218eecb4b2883da100059f8bfd2bc5a6e60f127d WatchSource:0}: Error finding container 34035b8b52676aaaff038657218eecb4b2883da100059f8bfd2bc5a6e60f127d: Status 404 returned error can't find the container with id 34035b8b52676aaaff038657218eecb4b2883da100059f8bfd2bc5a6e60f127d Oct 30 01:39:37.272628 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:37.272593 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:39:37.272729 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:37.272656 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:39:37.272729 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:37.272686 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:39:37.349796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:37.347332 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:37.498090 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:37.498056 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" event={"ID":"bfd2951a-980f-4a1a-bb49-20e1d9fce44d","Type":"ContainerStarted","Data":"cabbabac34140d1126a01d4d0a722cd69a31736a775ed126cbfff1e32c6c74d2"} Oct 30 01:39:37.498257 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:37.498112 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" event={"ID":"bfd2951a-980f-4a1a-bb49-20e1d9fce44d","Type":"ContainerStarted","Data":"34035b8b52676aaaff038657218eecb4b2883da100059f8bfd2bc5a6e60f127d"} Oct 30 01:39:37.540876 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:37.540773 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:37.548437 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:37.548382 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" podStartSLOduration=1.548366602 podStartE2EDuration="1.548366602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:39:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:39:37.545038122 +0000 UTC m=+1303.121463524" watchObservedRunningTime="2026-10-30 01:39:37.548366602 +0000 UTC m=+1303.124791998" Oct 30 01:39:37.952677 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:37.952639 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:39:38.958760 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:38.958710 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:39:39.094857 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:39.094820 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:39.516759 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:39.516712 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:39:39.517731 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:39.517712 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:39:39.997223 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:39.997192 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:40.185313 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:40.185268 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:41.409190 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:41.409154 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:41.701481 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:41.701442 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:41.892794 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:41.892507 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:42.912665 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:42.912629 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:43.463264 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:43.463229 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:43.638138 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:43.638100 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18560b-62sdq"] Oct 30 01:39:43.642980 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:43.642949 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18560b-62sdq"] Oct 30 01:39:43.673413 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:43.673380 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:43.867600 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:43.867516 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:44.188309 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:44.188254 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:44.544734 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:44.544649 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="049c1151-7da9-4331-97c4-7cdc7f6e4e6b" path="/var/lib/kubelet/pods/049c1151-7da9-4331-97c4-7cdc7f6e4e6b/volumes" Oct 30 01:39:44.703907 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:39:44.703863 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:39:45.740007 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:45.739961 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:46.032586 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:46.032492 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04cef553-42l96"] Oct 30 01:39:46.040147 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:46.040116 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04cef553-42l96"] Oct 30 01:39:46.543860 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:46.543827 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79828b67-08a2-4d32-9dc7-5ecf367791a9" path="/var/lib/kubelet/pods/79828b67-08a2-4d32-9dc7-5ecf367791a9/volumes" Oct 30 01:39:48.096981 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:48.096946 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:50.124341 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:50.124284 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:50.350958 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:50.350923 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:50.730443 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:50.730408 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:51.237315 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:51.237282 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-bf96646b8-n6q7h"] Oct 30 01:39:51.269284 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:51.269243 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-6b6795975-2hdzq"] Oct 30 01:39:51.477902 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:51.477865 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:51.681148 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:51.681104 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:52.030661 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:52.030571 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebdd4dab-hnc55"] Oct 30 01:39:52.041800 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:52.041766 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebdd4dab-hnc55"] Oct 30 01:39:52.543759 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:52.543723 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d14081a-d8ef-49af-8792-26f6956b6141" path="/var/lib/kubelet/pods/4d14081a-d8ef-49af-8792-26f6956b6141/volumes" Oct 30 01:39:53.229948 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:53.229910 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:54.275186 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:54.275151 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1050396f-r58zb"] Oct 30 01:39:54.280892 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:54.280876 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1050396f-r58zb" Oct 30 01:39:54.298987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:54.298961 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1050396f-r58zb"] Oct 30 01:39:54.306182 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:54.306135 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:39:54.306182 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:54.306137 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:39:54.307334 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:54.307312 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:39:54.307468 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:54.307420 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:39:54.425894 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:54.425620 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnmqs\" (UniqueName: \"kubernetes.io/projected/46ee12a0-1673-4ba6-b16d-2be18884f588-kube-api-access-lnmqs\") pod \"maas-api-revoke-sub-1050396f-r58zb\" (UID: \"46ee12a0-1673-4ba6-b16d-2be18884f588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1050396f-r58zb" Oct 30 01:39:54.425894 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:54.425806 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46ee12a0-1673-4ba6-b16d-2be18884f588-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1050396f-r58zb\" (UID: \"46ee12a0-1673-4ba6-b16d-2be18884f588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1050396f-r58zb" Oct 30 01:39:54.425894 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:54.425859 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46ee12a0-1673-4ba6-b16d-2be18884f588-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1050396f-r58zb\" (UID: \"46ee12a0-1673-4ba6-b16d-2be18884f588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1050396f-r58zb" Oct 30 01:39:54.527690 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:54.527602 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnmqs\" (UniqueName: \"kubernetes.io/projected/46ee12a0-1673-4ba6-b16d-2be18884f588-kube-api-access-lnmqs\") pod \"maas-api-revoke-sub-1050396f-r58zb\" (UID: \"46ee12a0-1673-4ba6-b16d-2be18884f588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1050396f-r58zb" Oct 30 01:39:54.527881 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:54.527723 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46ee12a0-1673-4ba6-b16d-2be18884f588-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1050396f-r58zb\" (UID: \"46ee12a0-1673-4ba6-b16d-2be18884f588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1050396f-r58zb" Oct 30 01:39:54.527881 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:54.527804 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46ee12a0-1673-4ba6-b16d-2be18884f588-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1050396f-r58zb\" (UID: \"46ee12a0-1673-4ba6-b16d-2be18884f588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1050396f-r58zb" Oct 30 01:39:54.540560 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:54.540534 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:39:54.540693 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:54.540599 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:39:54.548368 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:54.548341 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46ee12a0-1673-4ba6-b16d-2be18884f588-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1050396f-r58zb\" (UID: \"46ee12a0-1673-4ba6-b16d-2be18884f588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1050396f-r58zb" Oct 30 01:39:54.550045 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:54.550025 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46ee12a0-1673-4ba6-b16d-2be18884f588-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1050396f-r58zb\" (UID: \"46ee12a0-1673-4ba6-b16d-2be18884f588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1050396f-r58zb" Oct 30 01:39:54.556845 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:54.556824 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:39:54.566185 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:54.566164 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnmqs\" (UniqueName: \"kubernetes.io/projected/46ee12a0-1673-4ba6-b16d-2be18884f588-kube-api-access-lnmqs\") pod \"maas-api-revoke-sub-1050396f-r58zb\" (UID: \"46ee12a0-1673-4ba6-b16d-2be18884f588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1050396f-r58zb" Oct 30 01:39:54.608923 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:54.608899 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:39:54.616613 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:54.616592 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1050396f-r58zb" Oct 30 01:39:54.937010 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:54.936966 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1050396f-r58zb"] Oct 30 01:39:54.937221 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:39:54.937194 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46ee12a0_1673_4ba6_b16d_2be18884f588.slice/crio-def02e1eea0deaf4d09eada03d4c133ac34952fc3497c66f926e90f74fb8df40 WatchSource:0}: Error finding container def02e1eea0deaf4d09eada03d4c133ac34952fc3497c66f926e90f74fb8df40: Status 404 returned error can't find the container with id def02e1eea0deaf4d09eada03d4c133ac34952fc3497c66f926e90f74fb8df40 Oct 30 01:39:55.571535 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:55.571450 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:55.665260 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:55.665228 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1050396f-r58zb" event={"ID":"46ee12a0-1673-4ba6-b16d-2be18884f588","Type":"ContainerStarted","Data":"e860ab11401c78a99f213c39f97a5f308e3db9759277e120bc1d6e01cabf44df"} Oct 30 01:39:55.665441 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:55.665280 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1050396f-r58zb" event={"ID":"46ee12a0-1673-4ba6-b16d-2be18884f588","Type":"ContainerStarted","Data":"def02e1eea0deaf4d09eada03d4c133ac34952fc3497c66f926e90f74fb8df40"} Oct 30 01:39:55.704125 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:55.704068 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1050396f-r58zb" podStartSLOduration=1.70405425 podStartE2EDuration="1.70405425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:39:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:39:55.701763283 +0000 UTC m=+1321.278188682" watchObservedRunningTime="2026-10-30 01:39:55.70405425 +0000 UTC m=+1321.280479649" Oct 30 01:39:55.791174 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:55.791141 6572 scope.go:122] "RemoveContainer" containerID="25817e4f480731621ce956631f9f3fac978346fd715c7bf062ec3d77c6f9add1" Oct 30 01:39:55.802012 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:55.801993 6572 scope.go:122] "RemoveContainer" containerID="750e99744759bf76a092ffe03f50598aee979e60b9e4794ec358881924c3f4fb" Oct 30 01:39:55.812184 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:55.812157 6572 scope.go:122] "RemoveContainer" containerID="492b5814d0de317ef16afc0ef49b9ee188677c7897e6daad482b32818ac22594" Oct 30 01:39:55.821723 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:55.821685 6572 scope.go:122] "RemoveContainer" containerID="714330df0c20aa3bfc7b40e2a4078e9aa8b4e2a5fd215957b901f4e2faf10feb" Oct 30 01:39:55.832593 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:55.832569 6572 scope.go:122] "RemoveContainer" containerID="eddd24cf5e63f63d05db0be8ffc9b9ca70e40000914b1c8b3bfd12c8b09b59f4" Oct 30 01:39:56.238435 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:56.238371 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-bf96646b8-n6q7h" podUID="c56b43cf-29cd-4f22-b8e1-48feaaea009b" containerName="payload-processing" containerID="cri-o://a6474b848c0df9bb38ee81935affb80b68cfce0f790433b3060afad030628fc7" gracePeriod=25 Oct 30 01:39:56.270295 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:56.270249 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-6b6795975-2hdzq" podUID="3c3467ed-9f98-4311-9943-f97b7c9f2ed6" containerName="payload-pre-processing" containerID="cri-o://42dadc6e12777500b208c90fc732164aca4b53b21d9623fed93ae2274b9adde8" gracePeriod=25 Oct 30 01:39:56.548590 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:56.548564 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-bf96646b8-n6q7h" Oct 30 01:39:56.616930 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:56.616899 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-6b6795975-2hdzq" Oct 30 01:39:56.660160 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:56.660130 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c56b43cf-29cd-4f22-b8e1-48feaaea009b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56b43cf-29cd-4f22-b8e1-48feaaea009b-plugins-config-volume\") pod \"c56b43cf-29cd-4f22-b8e1-48feaaea009b\" (UID: \"c56b43cf-29cd-4f22-b8e1-48feaaea009b\") " Oct 30 01:39:56.660365 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:56.660331 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c56b43cf-29cd-4f22-b8e1-48feaaea009b-kube-api-access-fzgnv\" (UniqueName: \"kubernetes.io/projected/c56b43cf-29cd-4f22-b8e1-48feaaea009b-kube-api-access-fzgnv\") pod \"c56b43cf-29cd-4f22-b8e1-48feaaea009b\" (UID: \"c56b43cf-29cd-4f22-b8e1-48feaaea009b\") " Oct 30 01:39:56.661047 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:56.660555 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c56b43cf-29cd-4f22-b8e1-48feaaea009b-plugins-config-volume" pod "c56b43cf-29cd-4f22-b8e1-48feaaea009b" (UID: "c56b43cf-29cd-4f22-b8e1-48feaaea009b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:39:56.662918 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:56.662762 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56b43cf-29cd-4f22-b8e1-48feaaea009b-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:56.664510 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:56.664482 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c56b43cf-29cd-4f22-b8e1-48feaaea009b-kube-api-access-fzgnv" pod "c56b43cf-29cd-4f22-b8e1-48feaaea009b" (UID: "c56b43cf-29cd-4f22-b8e1-48feaaea009b"). InnerVolumeSpecName "kube-api-access-fzgnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:39:56.676371 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:56.676061 6572 generic.go:356] "Generic (PLEG): container finished" podID="c56b43cf-29cd-4f22-b8e1-48feaaea009b" containerID="a6474b848c0df9bb38ee81935affb80b68cfce0f790433b3060afad030628fc7" exitCode=0 Oct 30 01:39:56.676371 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:56.676115 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-bf96646b8-n6q7h" Oct 30 01:39:56.676371 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:56.676145 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-bf96646b8-n6q7h" event={"ID":"c56b43cf-29cd-4f22-b8e1-48feaaea009b","Type":"ContainerDied","Data":"a6474b848c0df9bb38ee81935affb80b68cfce0f790433b3060afad030628fc7"} Oct 30 01:39:56.676371 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:56.676206 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-bf96646b8-n6q7h" event={"ID":"c56b43cf-29cd-4f22-b8e1-48feaaea009b","Type":"ContainerDied","Data":"1bda6a9c862a1dd326e3500db14e528903cf261fb3a79587b2d660559f139255"} Oct 30 01:39:56.676371 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:56.676234 6572 scope.go:122] "RemoveContainer" containerID="a6474b848c0df9bb38ee81935affb80b68cfce0f790433b3060afad030628fc7" Oct 30 01:39:56.679584 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:56.679557 6572 generic.go:356] "Generic (PLEG): container finished" podID="3c3467ed-9f98-4311-9943-f97b7c9f2ed6" containerID="42dadc6e12777500b208c90fc732164aca4b53b21d9623fed93ae2274b9adde8" exitCode=0 Oct 30 01:39:56.679717 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:56.679640 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-6b6795975-2hdzq" event={"ID":"3c3467ed-9f98-4311-9943-f97b7c9f2ed6","Type":"ContainerDied","Data":"42dadc6e12777500b208c90fc732164aca4b53b21d9623fed93ae2274b9adde8"} Oct 30 01:39:56.679717 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:56.679687 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-6b6795975-2hdzq" event={"ID":"3c3467ed-9f98-4311-9943-f97b7c9f2ed6","Type":"ContainerDied","Data":"71fdf1e7f04380d0907ec90f1d0879390b94b3da2ea5538641808bca61bbe4ba"} Oct 30 01:39:56.679860 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:56.679845 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-6b6795975-2hdzq" Oct 30 01:39:56.682325 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:56.682298 6572 generic.go:356] "Generic (PLEG): container finished" podID="46ee12a0-1673-4ba6-b16d-2be18884f588" containerID="e860ab11401c78a99f213c39f97a5f308e3db9759277e120bc1d6e01cabf44df" exitCode=0 Oct 30 01:39:56.682449 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:56.682392 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1050396f-r58zb" event={"ID":"46ee12a0-1673-4ba6-b16d-2be18884f588","Type":"ContainerDied","Data":"e860ab11401c78a99f213c39f97a5f308e3db9759277e120bc1d6e01cabf44df"} Oct 30 01:39:56.694575 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:56.694548 6572 scope.go:122] "RemoveContainer" containerID="a6474b848c0df9bb38ee81935affb80b68cfce0f790433b3060afad030628fc7" Oct 30 01:39:56.695008 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:39:56.694973 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6474b848c0df9bb38ee81935affb80b68cfce0f790433b3060afad030628fc7\": container with ID starting with a6474b848c0df9bb38ee81935affb80b68cfce0f790433b3060afad030628fc7 not found: ID does not exist" containerID="a6474b848c0df9bb38ee81935affb80b68cfce0f790433b3060afad030628fc7" Oct 30 01:39:56.695105 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:56.695022 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6474b848c0df9bb38ee81935affb80b68cfce0f790433b3060afad030628fc7"} err="failed to get container status \"a6474b848c0df9bb38ee81935affb80b68cfce0f790433b3060afad030628fc7\": rpc error: code = NotFound desc = could not find container \"a6474b848c0df9bb38ee81935affb80b68cfce0f790433b3060afad030628fc7\": container with ID starting with a6474b848c0df9bb38ee81935affb80b68cfce0f790433b3060afad030628fc7 not found: ID does not exist" Oct 30 01:39:56.695105 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:56.695044 6572 scope.go:122] "RemoveContainer" containerID="42dadc6e12777500b208c90fc732164aca4b53b21d9623fed93ae2274b9adde8" Oct 30 01:39:56.707365 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:56.707348 6572 scope.go:122] "RemoveContainer" containerID="42dadc6e12777500b208c90fc732164aca4b53b21d9623fed93ae2274b9adde8" Oct 30 01:39:56.707710 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:39:56.707689 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42dadc6e12777500b208c90fc732164aca4b53b21d9623fed93ae2274b9adde8\": container with ID starting with 42dadc6e12777500b208c90fc732164aca4b53b21d9623fed93ae2274b9adde8 not found: ID does not exist" containerID="42dadc6e12777500b208c90fc732164aca4b53b21d9623fed93ae2274b9adde8" Oct 30 01:39:56.707796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:56.707717 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42dadc6e12777500b208c90fc732164aca4b53b21d9623fed93ae2274b9adde8"} err="failed to get container status \"42dadc6e12777500b208c90fc732164aca4b53b21d9623fed93ae2274b9adde8\": rpc error: code = NotFound desc = could not find container \"42dadc6e12777500b208c90fc732164aca4b53b21d9623fed93ae2274b9adde8\": container with ID starting with 42dadc6e12777500b208c90fc732164aca4b53b21d9623fed93ae2274b9adde8 not found: ID does not exist" Oct 30 01:39:56.728810 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:56.728775 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-bf96646b8-n6q7h"] Oct 30 01:39:56.732651 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:56.732621 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-bf96646b8-n6q7h"] Oct 30 01:39:56.763204 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:56.763114 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c3467ed-9f98-4311-9943-f97b7c9f2ed6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c3467ed-9f98-4311-9943-f97b7c9f2ed6-plugins-config-volume\") pod \"3c3467ed-9f98-4311-9943-f97b7c9f2ed6\" (UID: \"3c3467ed-9f98-4311-9943-f97b7c9f2ed6\") " Oct 30 01:39:56.763428 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:56.763411 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c3467ed-9f98-4311-9943-f97b7c9f2ed6-kube-api-access-vxcv9\" (UniqueName: \"kubernetes.io/projected/3c3467ed-9f98-4311-9943-f97b7c9f2ed6-kube-api-access-vxcv9\") pod \"3c3467ed-9f98-4311-9943-f97b7c9f2ed6\" (UID: \"3c3467ed-9f98-4311-9943-f97b7c9f2ed6\") " Oct 30 01:39:56.763499 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:56.763476 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c3467ed-9f98-4311-9943-f97b7c9f2ed6-plugins-config-volume" pod "3c3467ed-9f98-4311-9943-f97b7c9f2ed6" (UID: "3c3467ed-9f98-4311-9943-f97b7c9f2ed6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:39:56.763900 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:56.763880 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzgnv\" (UniqueName: \"kubernetes.io/projected/c56b43cf-29cd-4f22-b8e1-48feaaea009b-kube-api-access-fzgnv\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:56.763992 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:56.763903 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c3467ed-9f98-4311-9943-f97b7c9f2ed6-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:56.765500 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:56.765479 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3467ed-9f98-4311-9943-f97b7c9f2ed6-kube-api-access-vxcv9" pod "3c3467ed-9f98-4311-9943-f97b7c9f2ed6" (UID: "3c3467ed-9f98-4311-9943-f97b7c9f2ed6"). InnerVolumeSpecName "kube-api-access-vxcv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:39:56.864834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:56.864793 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxcv9\" (UniqueName: \"kubernetes.io/projected/3c3467ed-9f98-4311-9943-f97b7c9f2ed6-kube-api-access-vxcv9\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:57.025226 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:57.025133 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-6b6795975-2hdzq"] Oct 30 01:39:57.029867 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:57.029834 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-6b6795975-2hdzq"] Oct 30 01:39:57.521289 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:57.521252 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:57.861473 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:57.861440 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1050396f-r58zb" Oct 30 01:39:57.982471 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:57.982436 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46ee12a0-1673-4ba6-b16d-2be18884f588-kube-api-access-lnmqs\" (UniqueName: \"kubernetes.io/projected/46ee12a0-1673-4ba6-b16d-2be18884f588-kube-api-access-lnmqs\") pod \"46ee12a0-1673-4ba6-b16d-2be18884f588\" (UID: \"46ee12a0-1673-4ba6-b16d-2be18884f588\") " Oct 30 01:39:57.982644 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:57.982606 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46ee12a0-1673-4ba6-b16d-2be18884f588-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46ee12a0-1673-4ba6-b16d-2be18884f588-maas-api-xks-ca\") pod \"46ee12a0-1673-4ba6-b16d-2be18884f588\" (UID: \"46ee12a0-1673-4ba6-b16d-2be18884f588\") " Oct 30 01:39:57.982695 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:57.982680 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46ee12a0-1673-4ba6-b16d-2be18884f588-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46ee12a0-1673-4ba6-b16d-2be18884f588-maas-api-openshift-ca\") pod \"46ee12a0-1673-4ba6-b16d-2be18884f588\" (UID: \"46ee12a0-1673-4ba6-b16d-2be18884f588\") " Oct 30 01:39:57.983088 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:57.983053 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46ee12a0-1673-4ba6-b16d-2be18884f588-maas-api-openshift-ca" pod "46ee12a0-1673-4ba6-b16d-2be18884f588" (UID: "46ee12a0-1673-4ba6-b16d-2be18884f588"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:39:57.983313 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:57.983294 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46ee12a0-1673-4ba6-b16d-2be18884f588-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:57.984802 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:57.984776 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46ee12a0-1673-4ba6-b16d-2be18884f588-maas-api-xks-ca" pod "46ee12a0-1673-4ba6-b16d-2be18884f588" (UID: "46ee12a0-1673-4ba6-b16d-2be18884f588"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:39:57.984924 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:57.984904 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ee12a0-1673-4ba6-b16d-2be18884f588-kube-api-access-lnmqs" pod "46ee12a0-1673-4ba6-b16d-2be18884f588" (UID: "46ee12a0-1673-4ba6-b16d-2be18884f588"). InnerVolumeSpecName "kube-api-access-lnmqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:39:58.085246 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:58.085155 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnmqs\" (UniqueName: \"kubernetes.io/projected/46ee12a0-1673-4ba6-b16d-2be18884f588-kube-api-access-lnmqs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:58.085246 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:58.085189 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46ee12a0-1673-4ba6-b16d-2be18884f588-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:39:58.542699 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:58.542667 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c3467ed-9f98-4311-9943-f97b7c9f2ed6" path="/var/lib/kubelet/pods/3c3467ed-9f98-4311-9943-f97b7c9f2ed6/volumes" Oct 30 01:39:58.543132 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:58.543117 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c56b43cf-29cd-4f22-b8e1-48feaaea009b" path="/var/lib/kubelet/pods/c56b43cf-29cd-4f22-b8e1-48feaaea009b/volumes" Oct 30 01:39:58.701760 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:58.701720 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1050396f-r58zb" Oct 30 01:39:58.701951 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:58.701785 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1050396f-r58zb" event={"ID":"46ee12a0-1673-4ba6-b16d-2be18884f588","Type":"ContainerDied","Data":"def02e1eea0deaf4d09eada03d4c133ac34952fc3497c66f926e90f74fb8df40"} Oct 30 01:39:58.701951 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:58.701842 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="def02e1eea0deaf4d09eada03d4c133ac34952fc3497c66f926e90f74fb8df40" Oct 30 01:39:59.443547 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:59.443513 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-587d749d6d-nd7r5"] Oct 30 01:39:59.475605 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:59.475571 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5c97978cff-n6khx"] Oct 30 01:39:59.670417 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:59.670381 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:59.888132 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:39:59.888047 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:01.637706 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:01.637670 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:03.575977 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:03.575937 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:04.443881 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:04.443839 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-587d749d6d-nd7r5" podUID="dac431a9-b17e-4b64-bad9-98cccf1e244b" containerName="payload-processing" containerID="cri-o://bff63ad3fe909c7d59a6a86ae6a2deb8754e6d15537cf4090350ae9924ccc680" gracePeriod=25 Oct 30 01:40:04.476179 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:04.476135 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5c97978cff-n6khx" podUID="3c145c57-8343-42ed-9a5f-5c333ad1f598" containerName="payload-pre-processing" containerID="cri-o://2b38acca3f81dafbb2652e9b0e333762bee1e380b08d862c96410b877c655d17" gracePeriod=25 Oct 30 01:40:04.533520 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:04.533486 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:04.748584 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:04.748547 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-587d749d6d-nd7r5" Oct 30 01:40:04.754293 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:04.754259 6572 generic.go:356] "Generic (PLEG): container finished" podID="dac431a9-b17e-4b64-bad9-98cccf1e244b" containerID="bff63ad3fe909c7d59a6a86ae6a2deb8754e6d15537cf4090350ae9924ccc680" exitCode=0 Oct 30 01:40:04.754459 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:04.754311 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-587d749d6d-nd7r5" Oct 30 01:40:04.754459 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:04.754340 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-587d749d6d-nd7r5" event={"ID":"dac431a9-b17e-4b64-bad9-98cccf1e244b","Type":"ContainerDied","Data":"bff63ad3fe909c7d59a6a86ae6a2deb8754e6d15537cf4090350ae9924ccc680"} Oct 30 01:40:04.754459 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:04.754404 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-587d749d6d-nd7r5" event={"ID":"dac431a9-b17e-4b64-bad9-98cccf1e244b","Type":"ContainerDied","Data":"2fd12fbed8c003e8b361e0bbc62439680e664915ced93f4a8a8b9eb3240fc293"} Oct 30 01:40:04.754459 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:04.754435 6572 scope.go:122] "RemoveContainer" containerID="bff63ad3fe909c7d59a6a86ae6a2deb8754e6d15537cf4090350ae9924ccc680" Oct 30 01:40:04.756850 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:04.756819 6572 generic.go:356] "Generic (PLEG): container finished" podID="3c145c57-8343-42ed-9a5f-5c333ad1f598" containerID="2b38acca3f81dafbb2652e9b0e333762bee1e380b08d862c96410b877c655d17" exitCode=0 Oct 30 01:40:04.757011 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:04.756984 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5c97978cff-n6khx" event={"ID":"3c145c57-8343-42ed-9a5f-5c333ad1f598","Type":"ContainerDied","Data":"2b38acca3f81dafbb2652e9b0e333762bee1e380b08d862c96410b877c655d17"} Oct 30 01:40:04.768759 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:04.768713 6572 scope.go:122] "RemoveContainer" containerID="bff63ad3fe909c7d59a6a86ae6a2deb8754e6d15537cf4090350ae9924ccc680" Oct 30 01:40:04.769135 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:40:04.769115 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bff63ad3fe909c7d59a6a86ae6a2deb8754e6d15537cf4090350ae9924ccc680\": container with ID starting with bff63ad3fe909c7d59a6a86ae6a2deb8754e6d15537cf4090350ae9924ccc680 not found: ID does not exist" containerID="bff63ad3fe909c7d59a6a86ae6a2deb8754e6d15537cf4090350ae9924ccc680" Oct 30 01:40:04.769206 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:04.769151 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bff63ad3fe909c7d59a6a86ae6a2deb8754e6d15537cf4090350ae9924ccc680"} err="failed to get container status \"bff63ad3fe909c7d59a6a86ae6a2deb8754e6d15537cf4090350ae9924ccc680\": rpc error: code = NotFound desc = could not find container \"bff63ad3fe909c7d59a6a86ae6a2deb8754e6d15537cf4090350ae9924ccc680\": container with ID starting with bff63ad3fe909c7d59a6a86ae6a2deb8754e6d15537cf4090350ae9924ccc680 not found: ID does not exist" Oct 30 01:40:04.773735 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:04.773709 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dac431a9-b17e-4b64-bad9-98cccf1e244b-kube-api-access-tvzbd\" (UniqueName: \"kubernetes.io/projected/dac431a9-b17e-4b64-bad9-98cccf1e244b-kube-api-access-tvzbd\") pod \"dac431a9-b17e-4b64-bad9-98cccf1e244b\" (UID: \"dac431a9-b17e-4b64-bad9-98cccf1e244b\") " Oct 30 01:40:04.773907 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:04.773892 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dac431a9-b17e-4b64-bad9-98cccf1e244b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dac431a9-b17e-4b64-bad9-98cccf1e244b-plugins-config-volume\") pod \"dac431a9-b17e-4b64-bad9-98cccf1e244b\" (UID: \"dac431a9-b17e-4b64-bad9-98cccf1e244b\") " Oct 30 01:40:04.774304 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:04.774280 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dac431a9-b17e-4b64-bad9-98cccf1e244b-plugins-config-volume" pod "dac431a9-b17e-4b64-bad9-98cccf1e244b" (UID: "dac431a9-b17e-4b64-bad9-98cccf1e244b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:40:04.776091 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:04.776068 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dac431a9-b17e-4b64-bad9-98cccf1e244b-kube-api-access-tvzbd" pod "dac431a9-b17e-4b64-bad9-98cccf1e244b" (UID: "dac431a9-b17e-4b64-bad9-98cccf1e244b"). InnerVolumeSpecName "kube-api-access-tvzbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:40:04.804863 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:04.804829 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5c97978cff-n6khx" Oct 30 01:40:04.875407 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:04.875367 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c145c57-8343-42ed-9a5f-5c333ad1f598-kube-api-access-spc4m\" (UniqueName: \"kubernetes.io/projected/3c145c57-8343-42ed-9a5f-5c333ad1f598-kube-api-access-spc4m\") pod \"3c145c57-8343-42ed-9a5f-5c333ad1f598\" (UID: \"3c145c57-8343-42ed-9a5f-5c333ad1f598\") " Oct 30 01:40:04.875597 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:04.875584 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c145c57-8343-42ed-9a5f-5c333ad1f598-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c145c57-8343-42ed-9a5f-5c333ad1f598-plugins-config-volume\") pod \"3c145c57-8343-42ed-9a5f-5c333ad1f598\" (UID: \"3c145c57-8343-42ed-9a5f-5c333ad1f598\") " Oct 30 01:40:04.876044 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:04.876009 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c145c57-8343-42ed-9a5f-5c333ad1f598-plugins-config-volume" pod "3c145c57-8343-42ed-9a5f-5c333ad1f598" (UID: "3c145c57-8343-42ed-9a5f-5c333ad1f598"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:40:04.876158 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:04.876089 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dac431a9-b17e-4b64-bad9-98cccf1e244b-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:40:04.876158 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:04.876104 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvzbd\" (UniqueName: \"kubernetes.io/projected/dac431a9-b17e-4b64-bad9-98cccf1e244b-kube-api-access-tvzbd\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:40:04.877739 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:04.877716 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c145c57-8343-42ed-9a5f-5c333ad1f598-kube-api-access-spc4m" pod "3c145c57-8343-42ed-9a5f-5c333ad1f598" (UID: "3c145c57-8343-42ed-9a5f-5c333ad1f598"). InnerVolumeSpecName "kube-api-access-spc4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:40:04.978180 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:04.978080 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c145c57-8343-42ed-9a5f-5c333ad1f598-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:40:04.978180 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:04.978118 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spc4m\" (UniqueName: \"kubernetes.io/projected/3c145c57-8343-42ed-9a5f-5c333ad1f598-kube-api-access-spc4m\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:40:05.124775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:05.115417 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-587d749d6d-nd7r5"] Oct 30 01:40:05.160770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:05.160594 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-587d749d6d-nd7r5"] Oct 30 01:40:05.650036 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:05.649998 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:05.769340 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:05.769259 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5c97978cff-n6khx" Oct 30 01:40:05.769840 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:05.769391 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5c97978cff-n6khx" event={"ID":"3c145c57-8343-42ed-9a5f-5c333ad1f598","Type":"ContainerDied","Data":"bf049505231d6196a7eddb97b093bc17fdf9c710e62d1f0e9dbc1fd37ffd8345"} Oct 30 01:40:05.769840 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:05.769471 6572 scope.go:122] "RemoveContainer" containerID="2b38acca3f81dafbb2652e9b0e333762bee1e380b08d862c96410b877c655d17" Oct 30 01:40:05.822593 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:05.822529 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5c97978cff-n6khx"] Oct 30 01:40:05.833464 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:05.833429 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5c97978cff-n6khx"] Oct 30 01:40:05.845872 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:05.845838 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:06.544871 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:06.544835 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c145c57-8343-42ed-9a5f-5c333ad1f598" path="/var/lib/kubelet/pods/3c145c57-8343-42ed-9a5f-5c333ad1f598/volumes" Oct 30 01:40:06.545304 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:06.545290 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dac431a9-b17e-4b64-bad9-98cccf1e244b" path="/var/lib/kubelet/pods/dac431a9-b17e-4b64-bad9-98cccf1e244b/volumes" Oct 30 01:40:08.184682 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:08.184580 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:08.395770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:08.392702 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:08.637210 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:08.637053 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:09.976823 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:09.975843 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" podUID="cb20484a-51e2-4581-a629-e92d4234784e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:40:09.994787 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:09.990259 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:10.007625 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:10.007585 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" podUID="317505ef-238c-458e-a74e-1198806b94d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:40:10.200173 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:10.200128 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:10.529772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:10.524510 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:10.974843 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:10.974804 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:11.785727 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:11.785672 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:11.862669 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:11.862635 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-5cf4874fcf-4594l"] Oct 30 01:40:11.863679 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:11.863651 6572 cpu_manager.go:414] "containerMap: removing container" podUID="c56b43cf-29cd-4f22-b8e1-48feaaea009b" containerName="payload-processing" Oct 30 01:40:11.863679 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:11.863677 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c56b43cf-29cd-4f22-b8e1-48feaaea009b" containerName="payload-processing" Oct 30 01:40:11.863884 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:11.863709 6572 cpu_manager.go:414] "containerMap: removing container" podUID="46ee12a0-1673-4ba6-b16d-2be18884f588" containerName="revoke-keys" Oct 30 01:40:11.863884 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:11.863717 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46ee12a0-1673-4ba6-b16d-2be18884f588" containerName="revoke-keys" Oct 30 01:40:11.863884 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:11.863768 6572 cpu_manager.go:414] "containerMap: removing container" podUID="dac431a9-b17e-4b64-bad9-98cccf1e244b" containerName="payload-processing" Oct 30 01:40:11.863884 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:11.863777 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dac431a9-b17e-4b64-bad9-98cccf1e244b" containerName="payload-processing" Oct 30 01:40:11.863884 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:11.863799 6572 cpu_manager.go:414] "containerMap: removing container" podUID="3c3467ed-9f98-4311-9943-f97b7c9f2ed6" containerName="payload-pre-processing" Oct 30 01:40:11.863884 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:11.863806 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c3467ed-9f98-4311-9943-f97b7c9f2ed6" containerName="payload-pre-processing" Oct 30 01:40:11.863884 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:11.863828 6572 cpu_manager.go:414] "containerMap: removing container" podUID="3c145c57-8343-42ed-9a5f-5c333ad1f598" containerName="payload-pre-processing" Oct 30 01:40:11.863884 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:11.863835 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c145c57-8343-42ed-9a5f-5c333ad1f598" containerName="payload-pre-processing" Oct 30 01:40:11.864126 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:11.863978 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="dac431a9-b17e-4b64-bad9-98cccf1e244b" containerName="payload-processing" Oct 30 01:40:11.864126 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:11.864001 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c3467ed-9f98-4311-9943-f97b7c9f2ed6" containerName="payload-pre-processing" Oct 30 01:40:11.864126 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:11.864013 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c145c57-8343-42ed-9a5f-5c333ad1f598" containerName="payload-pre-processing" Oct 30 01:40:11.864126 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:11.864027 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="c56b43cf-29cd-4f22-b8e1-48feaaea009b" containerName="payload-processing" Oct 30 01:40:11.864126 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:11.864042 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="46ee12a0-1673-4ba6-b16d-2be18884f588" containerName="revoke-keys" Oct 30 01:40:11.869245 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:11.869219 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-5cf4874fcf-4594l" Oct 30 01:40:11.886966 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:11.886921 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-5cf4874fcf-4594l"] Oct 30 01:40:11.963770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:11.956095 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2ac18f\"" Oct 30 01:40:11.963770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:11.956254 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2ac18f-dockercfg-gnfqr\"" Oct 30 01:40:12.099443 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:12.099322 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh7ck\" (UniqueName: \"kubernetes.io/projected/dd5dbedf-dafc-4737-804b-035d10ede5dd-kube-api-access-gh7ck\") pod \"payload-pre-processing-e2e-ipp-a-2ac18f-5cf4874fcf-4594l\" (UID: \"dd5dbedf-dafc-4737-804b-035d10ede5dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-5cf4874fcf-4594l" Oct 30 01:40:12.100044 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:12.099999 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd5dbedf-dafc-4737-804b-035d10ede5dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2ac18f-5cf4874fcf-4594l\" (UID: \"dd5dbedf-dafc-4737-804b-035d10ede5dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-5cf4874fcf-4594l" Oct 30 01:40:12.104560 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:12.104315 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:12.202356 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:12.202321 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd5dbedf-dafc-4737-804b-035d10ede5dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2ac18f-5cf4874fcf-4594l\" (UID: \"dd5dbedf-dafc-4737-804b-035d10ede5dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-5cf4874fcf-4594l" Oct 30 01:40:12.202558 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:12.202374 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh7ck\" (UniqueName: \"kubernetes.io/projected/dd5dbedf-dafc-4737-804b-035d10ede5dd-kube-api-access-gh7ck\") pod \"payload-pre-processing-e2e-ipp-a-2ac18f-5cf4874fcf-4594l\" (UID: \"dd5dbedf-dafc-4737-804b-035d10ede5dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-5cf4874fcf-4594l" Oct 30 01:40:12.203066 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:12.203040 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd5dbedf-dafc-4737-804b-035d10ede5dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2ac18f-5cf4874fcf-4594l\" (UID: \"dd5dbedf-dafc-4737-804b-035d10ede5dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-5cf4874fcf-4594l" Oct 30 01:40:12.225804 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:12.225732 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh7ck\" (UniqueName: \"kubernetes.io/projected/dd5dbedf-dafc-4737-804b-035d10ede5dd-kube-api-access-gh7ck\") pod \"payload-pre-processing-e2e-ipp-a-2ac18f-5cf4874fcf-4594l\" (UID: \"dd5dbedf-dafc-4737-804b-035d10ede5dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-5cf4874fcf-4594l" Oct 30 01:40:12.486510 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:12.486473 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-5cf4874fcf-4594l" Oct 30 01:40:12.787045 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:12.787011 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-5cf4874fcf-4594l"] Oct 30 01:40:12.789240 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:40:12.789208 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd5dbedf_dafc_4737_804b_035d10ede5dd.slice/crio-b63d3c0bf773092590423d3c6856668766a7850260254a9c5c791cf2b8c0f9be WatchSource:0}: Error finding container b63d3c0bf773092590423d3c6856668766a7850260254a9c5c791cf2b8c0f9be: Status 404 returned error can't find the container with id b63d3c0bf773092590423d3c6856668766a7850260254a9c5c791cf2b8c0f9be Oct 30 01:40:13.117218 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:13.117128 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-5cf4874fcf-4594l" event={"ID":"dd5dbedf-dafc-4737-804b-035d10ede5dd","Type":"ContainerStarted","Data":"945fc2f297e844381569ca2e03e7158ffe1c15c27ee8a62d351da631a48f5252"} Oct 30 01:40:13.117218 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:13.117183 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-5cf4874fcf-4594l" event={"ID":"dd5dbedf-dafc-4737-804b-035d10ede5dd","Type":"ContainerStarted","Data":"b63d3c0bf773092590423d3c6856668766a7850260254a9c5c791cf2b8c0f9be"} Oct 30 01:40:13.117431 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:13.117333 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-5cf4874fcf-4594l" Oct 30 01:40:13.163651 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:13.163589 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-5cf4874fcf-4594l" podStartSLOduration=2.163573472 podStartE2EDuration="2.163573472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:40:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:40:13.161191783 +0000 UTC m=+1338.737617187" watchObservedRunningTime="2026-10-30 01:40:13.163573472 +0000 UTC m=+1338.739998874" Oct 30 01:40:13.448317 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:13.448274 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:14.692484 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:40:14.692445 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:40:15.277614 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:15.277563 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:15.473593 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:15.473554 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:16.145906 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:16.145871 6572 generic.go:356] "Generic (PLEG): container finished" podID="bcef90f8-7163-4fcc-94df-f10f46f8aa56" containerID="8deeafbbc128ac76469ef1bb3adcabb7ddf26e84a96ef395a10f33c865e21d20" exitCode=0 Oct 30 01:40:16.146311 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:16.145942 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" event={"ID":"bcef90f8-7163-4fcc-94df-f10f46f8aa56","Type":"ContainerDied","Data":"8deeafbbc128ac76469ef1bb3adcabb7ddf26e84a96ef395a10f33c865e21d20"} Oct 30 01:40:16.146488 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:16.146468 6572 scope.go:122] "RemoveContainer" containerID="8deeafbbc128ac76469ef1bb3adcabb7ddf26e84a96ef395a10f33c865e21d20" Oct 30 01:40:16.148176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:16.148141 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:40:16.148301 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:16.148225 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:40:16.148301 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:16.148275 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:40:17.170561 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:17.168325 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" event={"ID":"bcef90f8-7163-4fcc-94df-f10f46f8aa56","Type":"ContainerStarted","Data":"d2d7d6c71f3aead958c8ceb36166d0d985afd4ce3dd95e83cd822112b599b1bd"} Oct 30 01:40:17.261783 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:17.257597 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:17.520834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:17.518645 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:40:17.520834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:17.518861 6572 patch_prober.go:29] interesting pod/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 01:40:17.520834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:17.518927 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" podUID="bcef90f8-7163-4fcc-94df-f10f46f8aa56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 01:40:18.288487 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:18.288450 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:40:18.526773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:18.526246 6572 patch_prober.go:29] interesting pod/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 01:40:18.526773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:18.526325 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" podUID="bcef90f8-7163-4fcc-94df-f10f46f8aa56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 01:40:19.522772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:19.520626 6572 patch_prober.go:29] interesting pod/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 01:40:19.522772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:19.520684 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" podUID="bcef90f8-7163-4fcc-94df-f10f46f8aa56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 01:40:19.705709 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:19.705674 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:20.528772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:20.525202 6572 patch_prober.go:29] interesting pod/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 01:40:20.528772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:20.525270 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" podUID="bcef90f8-7163-4fcc-94df-f10f46f8aa56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 01:40:20.948914 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:20.948877 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0db534-5d88d59d4b-z4jpk"] Oct 30 01:40:21.170223 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:21.170039 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:21.390787 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:21.390623 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:21.481367 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:21.481327 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-5c6888c49f-sm8xt"] Oct 30 01:40:21.485865 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:21.485841 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-5c6888c49f-sm8xt" Oct 30 01:40:21.520771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:21.502129 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-fc51e9\"" Oct 30 01:40:21.523949 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:21.523912 6572 patch_prober.go:29] interesting pod/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 01:40:21.524111 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:21.523973 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" podUID="bcef90f8-7163-4fcc-94df-f10f46f8aa56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 01:40:21.534930 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:21.534898 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-5c6888c49f-sm8xt"] Oct 30 01:40:21.561682 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:21.561646 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbbct\" (UniqueName: \"kubernetes.io/projected/918a8572-3c60-4673-94a6-6d26854cb598-kube-api-access-xbbct\") pod \"payload-pre-processing-e2e-ipp-b-fc51e9-5c6888c49f-sm8xt\" (UID: \"918a8572-3c60-4673-94a6-6d26854cb598\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-5c6888c49f-sm8xt" Oct 30 01:40:21.561884 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:21.561725 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918a8572-3c60-4673-94a6-6d26854cb598-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fc51e9-5c6888c49f-sm8xt\" (UID: \"918a8572-3c60-4673-94a6-6d26854cb598\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-5c6888c49f-sm8xt" Oct 30 01:40:21.643821 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:21.643403 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-84d4cc64b-bdmc7"] Oct 30 01:40:21.656430 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:21.650146 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-84d4cc64b-bdmc7" Oct 30 01:40:21.656430 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:21.650047 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-84d4cc64b-bdmc7"] Oct 30 01:40:21.699770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:21.699516 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbbct\" (UniqueName: \"kubernetes.io/projected/918a8572-3c60-4673-94a6-6d26854cb598-kube-api-access-xbbct\") pod \"payload-pre-processing-e2e-ipp-b-fc51e9-5c6888c49f-sm8xt\" (UID: \"918a8572-3c60-4673-94a6-6d26854cb598\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-5c6888c49f-sm8xt" Oct 30 01:40:21.699770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:21.699588 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918a8572-3c60-4673-94a6-6d26854cb598-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fc51e9-5c6888c49f-sm8xt\" (UID: \"918a8572-3c60-4673-94a6-6d26854cb598\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-5c6888c49f-sm8xt" Oct 30 01:40:21.699770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:21.699694 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g45dl\" (UniqueName: \"kubernetes.io/projected/39556bd2-cdd6-4875-b55f-771f7f60195f-kube-api-access-g45dl\") pod \"payload-processing-e2e-ipp-b-fc51e9-84d4cc64b-bdmc7\" (UID: \"39556bd2-cdd6-4875-b55f-771f7f60195f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-84d4cc64b-bdmc7" Oct 30 01:40:21.700102 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:21.699839 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39556bd2-cdd6-4875-b55f-771f7f60195f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-fc51e9-84d4cc64b-bdmc7\" (UID: \"39556bd2-cdd6-4875-b55f-771f7f60195f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-84d4cc64b-bdmc7" Oct 30 01:40:21.743772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:21.712865 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918a8572-3c60-4673-94a6-6d26854cb598-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fc51e9-5c6888c49f-sm8xt\" (UID: \"918a8572-3c60-4673-94a6-6d26854cb598\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-5c6888c49f-sm8xt" Oct 30 01:40:21.776125 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:21.775969 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbbct\" (UniqueName: \"kubernetes.io/projected/918a8572-3c60-4673-94a6-6d26854cb598-kube-api-access-xbbct\") pod \"payload-pre-processing-e2e-ipp-b-fc51e9-5c6888c49f-sm8xt\" (UID: \"918a8572-3c60-4673-94a6-6d26854cb598\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-5c6888c49f-sm8xt" Oct 30 01:40:21.779287 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:21.779196 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-fc51e9-dockercfg-q7lzj\"" Oct 30 01:40:21.814812 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:21.803186 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39556bd2-cdd6-4875-b55f-771f7f60195f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-fc51e9-84d4cc64b-bdmc7\" (UID: \"39556bd2-cdd6-4875-b55f-771f7f60195f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-84d4cc64b-bdmc7" Oct 30 01:40:21.814812 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:21.803394 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g45dl\" (UniqueName: \"kubernetes.io/projected/39556bd2-cdd6-4875-b55f-771f7f60195f-kube-api-access-g45dl\") pod \"payload-processing-e2e-ipp-b-fc51e9-84d4cc64b-bdmc7\" (UID: \"39556bd2-cdd6-4875-b55f-771f7f60195f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-84d4cc64b-bdmc7" Oct 30 01:40:21.814812 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:21.805080 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39556bd2-cdd6-4875-b55f-771f7f60195f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-fc51e9-84d4cc64b-bdmc7\" (UID: \"39556bd2-cdd6-4875-b55f-771f7f60195f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-84d4cc64b-bdmc7" Oct 30 01:40:21.841330 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:21.841302 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-5c6888c49f-sm8xt" Oct 30 01:40:21.901971 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:21.901893 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g45dl\" (UniqueName: \"kubernetes.io/projected/39556bd2-cdd6-4875-b55f-771f7f60195f-kube-api-access-g45dl\") pod \"payload-processing-e2e-ipp-b-fc51e9-84d4cc64b-bdmc7\" (UID: \"39556bd2-cdd6-4875-b55f-771f7f60195f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-84d4cc64b-bdmc7" Oct 30 01:40:21.933451 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:21.933419 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-84d4cc64b-bdmc7" Oct 30 01:40:22.273613 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:40:22.273578 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod918a8572_3c60_4673_94a6_6d26854cb598.slice/crio-78a1b4b6b0488adc3ae877b24a22c225280c7f91eec60e6364fbc2b46fe0b39d WatchSource:0}: Error finding container 78a1b4b6b0488adc3ae877b24a22c225280c7f91eec60e6364fbc2b46fe0b39d: Status 404 returned error can't find the container with id 78a1b4b6b0488adc3ae877b24a22c225280c7f91eec60e6364fbc2b46fe0b39d Oct 30 01:40:22.275929 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:22.275567 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-5c6888c49f-sm8xt"] Oct 30 01:40:22.344536 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:40:22.344505 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39556bd2_cdd6_4875_b55f_771f7f60195f.slice/crio-81e3e9a1ad7a08e5c194623640db7f7263c241d97ec711a16e424248ca004980 WatchSource:0}: Error finding container 81e3e9a1ad7a08e5c194623640db7f7263c241d97ec711a16e424248ca004980: Status 404 returned error can't find the container with id 81e3e9a1ad7a08e5c194623640db7f7263c241d97ec711a16e424248ca004980 Oct 30 01:40:22.351581 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:22.351550 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-84d4cc64b-bdmc7"] Oct 30 01:40:22.526838 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:22.523118 6572 patch_prober.go:29] interesting pod/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 01:40:22.526838 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:22.523177 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" podUID="bcef90f8-7163-4fcc-94df-f10f46f8aa56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 01:40:22.605781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:22.602873 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:23.251823 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:23.251786 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-84d4cc64b-bdmc7" event={"ID":"39556bd2-cdd6-4875-b55f-771f7f60195f","Type":"ContainerStarted","Data":"92ebf9229257ffdc759f0be6210d33c36a523f97ef4289b63be45168009cc866"} Oct 30 01:40:23.252029 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:23.251845 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-84d4cc64b-bdmc7" event={"ID":"39556bd2-cdd6-4875-b55f-771f7f60195f","Type":"ContainerStarted","Data":"81e3e9a1ad7a08e5c194623640db7f7263c241d97ec711a16e424248ca004980"} Oct 30 01:40:23.252029 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:23.251876 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-84d4cc64b-bdmc7" Oct 30 01:40:23.253485 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:23.253458 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-5c6888c49f-sm8xt" event={"ID":"918a8572-3c60-4673-94a6-6d26854cb598","Type":"ContainerStarted","Data":"9f258f6030c49cfa97d6a3c0d6df1b50833ea4e2cafd30ccaca7d188ca13d309"} Oct 30 01:40:23.253627 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:23.253498 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-5c6888c49f-sm8xt" event={"ID":"918a8572-3c60-4673-94a6-6d26854cb598","Type":"ContainerStarted","Data":"78a1b4b6b0488adc3ae877b24a22c225280c7f91eec60e6364fbc2b46fe0b39d"} Oct 30 01:40:23.253627 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:23.253580 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-5c6888c49f-sm8xt" Oct 30 01:40:23.294353 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:23.294292 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-84d4cc64b-bdmc7" podStartSLOduration=2.294274623 podStartE2EDuration="2.294274623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:40:23.292419035 +0000 UTC m=+1348.868844450" watchObservedRunningTime="2026-10-30 01:40:23.294274623 +0000 UTC m=+1348.870700029" Oct 30 01:40:23.349561 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:23.349505 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-5c6888c49f-sm8xt" podStartSLOduration=2.349489659 podStartE2EDuration="2.349489659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:40:23.34690644 +0000 UTC m=+1348.923331843" watchObservedRunningTime="2026-10-30 01:40:23.349489659 +0000 UTC m=+1348.925915107" Oct 30 01:40:23.537829 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:23.537065 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:40:23.540034 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:23.539610 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:40:24.134362 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:24.134332 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-5cf4874fcf-4594l" Oct 30 01:40:25.158409 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:25.158360 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:25.950158 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:25.950108 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db534-5d88d59d4b-z4jpk" podUID="a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196" containerName="payload-processing" containerID="cri-o://7351daed5dc085d4868fd0d0f35ae9d6cbf5d3da08ac1846df9d3601918811ce" gracePeriod=25 Oct 30 01:40:26.266695 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:26.266671 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db534-5d88d59d4b-z4jpk" Oct 30 01:40:26.278339 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:26.278310 6572 generic.go:356] "Generic (PLEG): container finished" podID="a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196" containerID="7351daed5dc085d4868fd0d0f35ae9d6cbf5d3da08ac1846df9d3601918811ce" exitCode=0 Oct 30 01:40:26.278462 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:26.278342 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db534-5d88d59d4b-z4jpk" Oct 30 01:40:26.278462 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:26.278356 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db534-5d88d59d4b-z4jpk" event={"ID":"a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196","Type":"ContainerDied","Data":"7351daed5dc085d4868fd0d0f35ae9d6cbf5d3da08ac1846df9d3601918811ce"} Oct 30 01:40:26.278462 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:26.278425 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db534-5d88d59d4b-z4jpk" event={"ID":"a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196","Type":"ContainerDied","Data":"a80c304efd4fa10879b9dce40469f66d67b0ac5f74fb398351c51671ccdcf191"} Oct 30 01:40:26.278462 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:26.278448 6572 scope.go:122] "RemoveContainer" containerID="7351daed5dc085d4868fd0d0f35ae9d6cbf5d3da08ac1846df9d3601918811ce" Oct 30 01:40:26.289359 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:26.289339 6572 scope.go:122] "RemoveContainer" containerID="7351daed5dc085d4868fd0d0f35ae9d6cbf5d3da08ac1846df9d3601918811ce" Oct 30 01:40:26.289621 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:40:26.289602 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7351daed5dc085d4868fd0d0f35ae9d6cbf5d3da08ac1846df9d3601918811ce\": container with ID starting with 7351daed5dc085d4868fd0d0f35ae9d6cbf5d3da08ac1846df9d3601918811ce not found: ID does not exist" containerID="7351daed5dc085d4868fd0d0f35ae9d6cbf5d3da08ac1846df9d3601918811ce" Oct 30 01:40:26.289683 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:26.289628 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7351daed5dc085d4868fd0d0f35ae9d6cbf5d3da08ac1846df9d3601918811ce"} err="failed to get container status \"7351daed5dc085d4868fd0d0f35ae9d6cbf5d3da08ac1846df9d3601918811ce\": rpc error: code = NotFound desc = could not find container \"7351daed5dc085d4868fd0d0f35ae9d6cbf5d3da08ac1846df9d3601918811ce\": container with ID starting with 7351daed5dc085d4868fd0d0f35ae9d6cbf5d3da08ac1846df9d3601918811ce not found: ID does not exist" Oct 30 01:40:26.352457 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:26.352416 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196-plugins-config-volume\") pod \"a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196\" (UID: \"a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196\") " Oct 30 01:40:26.352673 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:26.352655 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196-kube-api-access-966l9\" (UniqueName: \"kubernetes.io/projected/a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196-kube-api-access-966l9\") pod \"a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196\" (UID: \"a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196\") " Oct 30 01:40:26.352966 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:26.352938 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196-plugins-config-volume" pod "a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196" (UID: "a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:40:26.353138 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:26.353113 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:40:26.354984 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:26.354957 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196-kube-api-access-966l9" pod "a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196" (UID: "a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196"). InnerVolumeSpecName "kube-api-access-966l9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:40:26.454501 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:26.454468 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-966l9\" (UniqueName: \"kubernetes.io/projected/a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196-kube-api-access-966l9\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:40:26.532714 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:26.532458 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:26.672774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:26.672193 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0db534-5d88d59d4b-z4jpk"] Oct 30 01:40:26.678164 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:26.677813 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0db534-5d88d59d4b-z4jpk"] Oct 30 01:40:26.730806 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:26.729382 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:26.994517 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:26.994485 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-75f96f6674-gz2rx"] Oct 30 01:40:26.995211 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:26.995194 6572 cpu_manager.go:414] "containerMap: removing container" podUID="a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196" containerName="payload-processing" Oct 30 01:40:26.995270 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:26.995213 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196" containerName="payload-processing" Oct 30 01:40:26.995338 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:26.995329 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196" containerName="payload-processing" Oct 30 01:40:27.000304 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:27.000286 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-75f96f6674-gz2rx" Oct 30 01:40:27.037773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:27.029925 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d8eb06\"" Oct 30 01:40:27.037773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:27.030165 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d8eb06-dockercfg-dkggb\"" Oct 30 01:40:27.041767 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:27.040071 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-75f96f6674-gz2rx"] Oct 30 01:40:27.164999 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:27.164964 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5b5b9f459-njx29"] Oct 30 01:40:27.169236 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:27.169216 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5b5b9f459-njx29" Oct 30 01:40:27.176086 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:27.176034 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwl7b\" (UniqueName: \"kubernetes.io/projected/84354b2e-c89c-486d-adb1-0204a5f120c1-kube-api-access-lwl7b\") pod \"payload-processing-e2e-ipp-b-d8eb06-75f96f6674-gz2rx\" (UID: \"84354b2e-c89c-486d-adb1-0204a5f120c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-75f96f6674-gz2rx" Oct 30 01:40:27.176215 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:27.176180 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84354b2e-c89c-486d-adb1-0204a5f120c1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d8eb06-75f96f6674-gz2rx\" (UID: \"84354b2e-c89c-486d-adb1-0204a5f120c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-75f96f6674-gz2rx" Oct 30 01:40:27.195528 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:27.195497 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5b5b9f459-njx29"] Oct 30 01:40:27.277735 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:27.277609 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04eef61c-0c54-4a8a-b1b8-da3d30e4af77-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d8eb06-5b5b9f459-njx29\" (UID: \"04eef61c-0c54-4a8a-b1b8-da3d30e4af77\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5b5b9f459-njx29" Oct 30 01:40:27.278821 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:27.277734 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwl7b\" (UniqueName: \"kubernetes.io/projected/84354b2e-c89c-486d-adb1-0204a5f120c1-kube-api-access-lwl7b\") pod \"payload-processing-e2e-ipp-b-d8eb06-75f96f6674-gz2rx\" (UID: \"84354b2e-c89c-486d-adb1-0204a5f120c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-75f96f6674-gz2rx" Oct 30 01:40:27.278821 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:27.277867 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84354b2e-c89c-486d-adb1-0204a5f120c1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d8eb06-75f96f6674-gz2rx\" (UID: \"84354b2e-c89c-486d-adb1-0204a5f120c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-75f96f6674-gz2rx" Oct 30 01:40:27.278821 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:27.277948 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ddxk\" (UniqueName: \"kubernetes.io/projected/04eef61c-0c54-4a8a-b1b8-da3d30e4af77-kube-api-access-2ddxk\") pod \"payload-pre-processing-e2e-ipp-b-d8eb06-5b5b9f459-njx29\" (UID: \"04eef61c-0c54-4a8a-b1b8-da3d30e4af77\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5b5b9f459-njx29" Oct 30 01:40:27.278821 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:27.278666 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84354b2e-c89c-486d-adb1-0204a5f120c1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d8eb06-75f96f6674-gz2rx\" (UID: \"84354b2e-c89c-486d-adb1-0204a5f120c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-75f96f6674-gz2rx" Oct 30 01:40:27.297357 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:27.297327 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwl7b\" (UniqueName: \"kubernetes.io/projected/84354b2e-c89c-486d-adb1-0204a5f120c1-kube-api-access-lwl7b\") pod \"payload-processing-e2e-ipp-b-d8eb06-75f96f6674-gz2rx\" (UID: \"84354b2e-c89c-486d-adb1-0204a5f120c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-75f96f6674-gz2rx" Oct 30 01:40:27.331007 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:27.330973 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-75f96f6674-gz2rx" Oct 30 01:40:27.379853 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:27.379812 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04eef61c-0c54-4a8a-b1b8-da3d30e4af77-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d8eb06-5b5b9f459-njx29\" (UID: \"04eef61c-0c54-4a8a-b1b8-da3d30e4af77\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5b5b9f459-njx29" Oct 30 01:40:27.380042 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:27.380011 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ddxk\" (UniqueName: \"kubernetes.io/projected/04eef61c-0c54-4a8a-b1b8-da3d30e4af77-kube-api-access-2ddxk\") pod \"payload-pre-processing-e2e-ipp-b-d8eb06-5b5b9f459-njx29\" (UID: \"04eef61c-0c54-4a8a-b1b8-da3d30e4af77\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5b5b9f459-njx29" Oct 30 01:40:27.380635 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:27.380611 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04eef61c-0c54-4a8a-b1b8-da3d30e4af77-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d8eb06-5b5b9f459-njx29\" (UID: \"04eef61c-0c54-4a8a-b1b8-da3d30e4af77\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5b5b9f459-njx29" Oct 30 01:40:27.433784 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:27.422430 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ddxk\" (UniqueName: \"kubernetes.io/projected/04eef61c-0c54-4a8a-b1b8-da3d30e4af77-kube-api-access-2ddxk\") pod \"payload-pre-processing-e2e-ipp-b-d8eb06-5b5b9f459-njx29\" (UID: \"04eef61c-0c54-4a8a-b1b8-da3d30e4af77\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5b5b9f459-njx29" Oct 30 01:40:27.484064 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:27.484031 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5b5b9f459-njx29" Oct 30 01:40:27.647326 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:27.647252 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-75f96f6674-gz2rx"] Oct 30 01:40:27.651090 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:40:27.650864 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84354b2e_c89c_486d_adb1_0204a5f120c1.slice/crio-143611ba0e40a09f265a1886754c79c18a655a82fa6fc3b630546960c5eaac5c WatchSource:0}: Error finding container 143611ba0e40a09f265a1886754c79c18a655a82fa6fc3b630546960c5eaac5c: Status 404 returned error can't find the container with id 143611ba0e40a09f265a1886754c79c18a655a82fa6fc3b630546960c5eaac5c Oct 30 01:40:27.829435 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:27.829248 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:27.830284 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:27.830252 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5b5b9f459-njx29"] Oct 30 01:40:27.831142 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:40:27.831104 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04eef61c_0c54_4a8a_b1b8_da3d30e4af77.slice/crio-28ed9e98d4d0a46643829764c66848d017acc18d3d9261a8eea5d5b2648f68b4 WatchSource:0}: Error finding container 28ed9e98d4d0a46643829764c66848d017acc18d3d9261a8eea5d5b2648f68b4: Status 404 returned error can't find the container with id 28ed9e98d4d0a46643829764c66848d017acc18d3d9261a8eea5d5b2648f68b4 Oct 30 01:40:28.298192 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:28.298145 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-75f96f6674-gz2rx" event={"ID":"84354b2e-c89c-486d-adb1-0204a5f120c1","Type":"ContainerStarted","Data":"39c350de8c7a448567c5374af4b214773c0cd2f973801423c6a8a241998a4a1a"} Oct 30 01:40:28.298683 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:28.298249 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-75f96f6674-gz2rx" event={"ID":"84354b2e-c89c-486d-adb1-0204a5f120c1","Type":"ContainerStarted","Data":"143611ba0e40a09f265a1886754c79c18a655a82fa6fc3b630546960c5eaac5c"} Oct 30 01:40:28.298683 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:28.298620 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-75f96f6674-gz2rx" Oct 30 01:40:28.300181 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:28.300154 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5b5b9f459-njx29" event={"ID":"04eef61c-0c54-4a8a-b1b8-da3d30e4af77","Type":"ContainerStarted","Data":"8d1676e928ac97094c000512eab9d17606d4b8213770876cb4b6e71e46f946d8"} Oct 30 01:40:28.300326 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:28.300195 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5b5b9f459-njx29" event={"ID":"04eef61c-0c54-4a8a-b1b8-da3d30e4af77","Type":"ContainerStarted","Data":"28ed9e98d4d0a46643829764c66848d017acc18d3d9261a8eea5d5b2648f68b4"} Oct 30 01:40:28.300326 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:28.300215 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5b5b9f459-njx29" Oct 30 01:40:28.339273 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:28.339189 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-75f96f6674-gz2rx" podStartSLOduration=2.339169311 podStartE2EDuration="2.339169311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:40:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:40:28.337658396 +0000 UTC m=+1353.914083806" watchObservedRunningTime="2026-10-30 01:40:28.339169311 +0000 UTC m=+1353.915594714" Oct 30 01:40:28.376194 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:28.375855 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5b5b9f459-njx29" podStartSLOduration=1.37583557 podStartE2EDuration="1.37583557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:40:28.375317554 +0000 UTC m=+1353.951742956" watchObservedRunningTime="2026-10-30 01:40:28.37583557 +0000 UTC m=+1353.952260972" Oct 30 01:40:28.544359 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:28.544322 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196" path="/var/lib/kubelet/pods/a17631e3-0dc4-4ce0-bfc1-6f6c6b02f196/volumes" Oct 30 01:40:29.488097 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:29.488063 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:31.620412 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:31.615551 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:32.338100 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:32.338066 6572 generic.go:356] "Generic (PLEG): container finished" podID="bfd2951a-980f-4a1a-bb49-20e1d9fce44d" containerID="cabbabac34140d1126a01d4d0a722cd69a31736a775ed126cbfff1e32c6c74d2" exitCode=0 Oct 30 01:40:32.338279 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:32.338139 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" event={"ID":"bfd2951a-980f-4a1a-bb49-20e1d9fce44d","Type":"ContainerDied","Data":"cabbabac34140d1126a01d4d0a722cd69a31736a775ed126cbfff1e32c6c74d2"} Oct 30 01:40:32.338614 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:32.338595 6572 scope.go:122] "RemoveContainer" containerID="cabbabac34140d1126a01d4d0a722cd69a31736a775ed126cbfff1e32c6c74d2" Oct 30 01:40:32.340202 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:32.340173 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:40:32.340275 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:32.340242 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:40:32.340313 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:32.340274 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:40:32.451162 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:32.451089 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:32.633273 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:32.633184 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:33.224305 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:33.224272 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:33.349388 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:33.349355 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" event={"ID":"bfd2951a-980f-4a1a-bb49-20e1d9fce44d","Type":"ContainerStarted","Data":"cae2e61e4343e7b48804ce1812c41a9ec81782033838c25e75712fdd6963f4dc"} Oct 30 01:40:33.952803 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:33.952728 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:40:33.953306 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:33.952931 6572 patch_prober.go:29] interesting pod/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 01:40:33.953306 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:33.953001 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" podUID="bfd2951a-980f-4a1a-bb49-20e1d9fce44d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 01:40:34.265651 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:34.265577 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-5c6888c49f-sm8xt" Oct 30 01:40:34.265808 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:34.265657 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-84d4cc64b-bdmc7" Oct 30 01:40:34.889224 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:34.889185 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:34.952839 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:34.952803 6572 patch_prober.go:29] interesting pod/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 01:40:34.953313 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:34.952861 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" podUID="bfd2951a-980f-4a1a-bb49-20e1d9fce44d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 01:40:35.952911 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:35.952874 6572 patch_prober.go:29] interesting pod/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 01:40:35.953331 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:35.952951 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" podUID="bfd2951a-980f-4a1a-bb49-20e1d9fce44d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 01:40:36.576093 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:36.576055 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:36.952943 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:36.952905 6572 patch_prober.go:29] interesting pod/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 01:40:36.953365 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:36.952981 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" podUID="bfd2951a-980f-4a1a-bb49-20e1d9fce44d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 01:40:37.952803 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:37.952767 6572 patch_prober.go:29] interesting pod/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 01:40:37.953033 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:37.952825 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" podUID="bfd2951a-980f-4a1a-bb49-20e1d9fce44d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 01:40:38.286699 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:38.286609 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:38.957439 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:38.957408 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:40:38.957880 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:38.957806 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:40:38.958424 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:38.958405 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:40:39.313244 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:39.313162 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-75f96f6674-gz2rx" Oct 30 01:40:39.313244 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:39.313233 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5b5b9f459-njx29" Oct 30 01:40:39.459598 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:39.459565 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19a5f1-fjl9d"] Oct 30 01:40:39.462888 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:39.462865 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19a5f1-fjl9d" Oct 30 01:40:39.477725 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:39.477698 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:40:39.477894 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:39.477787 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:40:39.481519 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:39.481491 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19a5f1-fjl9d"] Oct 30 01:40:39.541505 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:39.541476 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:40:39.541675 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:39.541541 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:40:39.571138 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:39.571057 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kz76\" (UniqueName: \"kubernetes.io/projected/fef4ac06-3bbf-4fbe-8de2-caae3e7fa017-kube-api-access-9kz76\") pod \"maas-api-revoke-keys-e2e-ipp-a-19a5f1-fjl9d\" (UID: \"fef4ac06-3bbf-4fbe-8de2-caae3e7fa017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19a5f1-fjl9d" Oct 30 01:40:39.571301 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:39.571142 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fef4ac06-3bbf-4fbe-8de2-caae3e7fa017-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-19a5f1-fjl9d\" (UID: \"fef4ac06-3bbf-4fbe-8de2-caae3e7fa017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19a5f1-fjl9d" Oct 30 01:40:39.571369 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:39.571351 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fef4ac06-3bbf-4fbe-8de2-caae3e7fa017-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-19a5f1-fjl9d\" (UID: \"fef4ac06-3bbf-4fbe-8de2-caae3e7fa017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19a5f1-fjl9d" Oct 30 01:40:39.673316 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:39.673268 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kz76\" (UniqueName: \"kubernetes.io/projected/fef4ac06-3bbf-4fbe-8de2-caae3e7fa017-kube-api-access-9kz76\") pod \"maas-api-revoke-keys-e2e-ipp-a-19a5f1-fjl9d\" (UID: \"fef4ac06-3bbf-4fbe-8de2-caae3e7fa017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19a5f1-fjl9d" Oct 30 01:40:39.674138 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:39.674110 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fef4ac06-3bbf-4fbe-8de2-caae3e7fa017-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-19a5f1-fjl9d\" (UID: \"fef4ac06-3bbf-4fbe-8de2-caae3e7fa017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19a5f1-fjl9d" Oct 30 01:40:39.674781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:39.674739 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fef4ac06-3bbf-4fbe-8de2-caae3e7fa017-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-19a5f1-fjl9d\" (UID: \"fef4ac06-3bbf-4fbe-8de2-caae3e7fa017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19a5f1-fjl9d" Oct 30 01:40:39.676233 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:39.676183 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fef4ac06-3bbf-4fbe-8de2-caae3e7fa017-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-19a5f1-fjl9d\" (UID: \"fef4ac06-3bbf-4fbe-8de2-caae3e7fa017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19a5f1-fjl9d" Oct 30 01:40:39.700263 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:39.692174 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fef4ac06-3bbf-4fbe-8de2-caae3e7fa017-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-19a5f1-fjl9d\" (UID: \"fef4ac06-3bbf-4fbe-8de2-caae3e7fa017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19a5f1-fjl9d" Oct 30 01:40:39.731550 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:39.731508 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kz76\" (UniqueName: \"kubernetes.io/projected/fef4ac06-3bbf-4fbe-8de2-caae3e7fa017-kube-api-access-9kz76\") pod \"maas-api-revoke-keys-e2e-ipp-a-19a5f1-fjl9d\" (UID: \"fef4ac06-3bbf-4fbe-8de2-caae3e7fa017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19a5f1-fjl9d" Oct 30 01:40:39.780772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:39.779078 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19a5f1-fjl9d" Oct 30 01:40:40.095378 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:40.095350 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19a5f1-fjl9d"] Oct 30 01:40:40.096257 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:40:40.096217 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfef4ac06_3bbf_4fbe_8de2_caae3e7fa017.slice/crio-5c5b1159bbbe15553a9cc75dfea0acf4b51db5ba175da33372939896500ac929 WatchSource:0}: Error finding container 5c5b1159bbbe15553a9cc75dfea0acf4b51db5ba175da33372939896500ac929: Status 404 returned error can't find the container with id 5c5b1159bbbe15553a9cc75dfea0acf4b51db5ba175da33372939896500ac929 Oct 30 01:40:40.110821 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:40.110741 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:40.302719 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:40.302682 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:40.426382 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:40.426354 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19a5f1-fjl9d" event={"ID":"fef4ac06-3bbf-4fbe-8de2-caae3e7fa017","Type":"ContainerStarted","Data":"5c5b1159bbbe15553a9cc75dfea0acf4b51db5ba175da33372939896500ac929"} Oct 30 01:40:40.465080 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:40.464770 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19a5f1-fjl9d" podStartSLOduration=1.464737368 podStartE2EDuration="1.464737368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:40:40.464237311 +0000 UTC m=+1366.040662713" watchObservedRunningTime="2026-10-30 01:40:40.464737368 +0000 UTC m=+1366.041162769" Oct 30 01:40:41.436216 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:41.436183 6572 generic.go:356] "Generic (PLEG): container finished" podID="fef4ac06-3bbf-4fbe-8de2-caae3e7fa017" containerID="996d63491e9c5e2f791889c7b994eafc1f0873a1b3e2e12d5abef087348815ae" exitCode=0 Oct 30 01:40:41.436647 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:41.436270 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19a5f1-fjl9d" event={"ID":"fef4ac06-3bbf-4fbe-8de2-caae3e7fa017","Type":"ContainerDied","Data":"996d63491e9c5e2f791889c7b994eafc1f0873a1b3e2e12d5abef087348815ae"} Oct 30 01:40:41.595471 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:41.595416 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:42.118859 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:42.118823 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:42.644812 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:42.644784 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19a5f1-fjl9d" Oct 30 01:40:42.679356 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:42.679318 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:42.720033 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:42.718717 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fef4ac06-3bbf-4fbe-8de2-caae3e7fa017-kube-api-access-9kz76\" (UniqueName: \"kubernetes.io/projected/fef4ac06-3bbf-4fbe-8de2-caae3e7fa017-kube-api-access-9kz76\") pod \"fef4ac06-3bbf-4fbe-8de2-caae3e7fa017\" (UID: \"fef4ac06-3bbf-4fbe-8de2-caae3e7fa017\") " Oct 30 01:40:42.720033 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:42.719018 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fef4ac06-3bbf-4fbe-8de2-caae3e7fa017-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fef4ac06-3bbf-4fbe-8de2-caae3e7fa017-maas-api-openshift-ca\") pod \"fef4ac06-3bbf-4fbe-8de2-caae3e7fa017\" (UID: \"fef4ac06-3bbf-4fbe-8de2-caae3e7fa017\") " Oct 30 01:40:42.720033 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:42.719088 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fef4ac06-3bbf-4fbe-8de2-caae3e7fa017-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fef4ac06-3bbf-4fbe-8de2-caae3e7fa017-maas-api-xks-ca\") pod \"fef4ac06-3bbf-4fbe-8de2-caae3e7fa017\" (UID: \"fef4ac06-3bbf-4fbe-8de2-caae3e7fa017\") " Oct 30 01:40:42.720941 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:42.720915 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fef4ac06-3bbf-4fbe-8de2-caae3e7fa017-maas-api-openshift-ca" pod "fef4ac06-3bbf-4fbe-8de2-caae3e7fa017" (UID: "fef4ac06-3bbf-4fbe-8de2-caae3e7fa017"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:40:42.722328 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:42.722299 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fef4ac06-3bbf-4fbe-8de2-caae3e7fa017-maas-api-xks-ca" pod "fef4ac06-3bbf-4fbe-8de2-caae3e7fa017" (UID: "fef4ac06-3bbf-4fbe-8de2-caae3e7fa017"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:40:42.722625 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:42.722606 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fef4ac06-3bbf-4fbe-8de2-caae3e7fa017-kube-api-access-9kz76" pod "fef4ac06-3bbf-4fbe-8de2-caae3e7fa017" (UID: "fef4ac06-3bbf-4fbe-8de2-caae3e7fa017"). InnerVolumeSpecName "kube-api-access-9kz76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:40:42.820781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:42.820733 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fef4ac06-3bbf-4fbe-8de2-caae3e7fa017-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:40:42.820781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:42.820780 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fef4ac06-3bbf-4fbe-8de2-caae3e7fa017-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:40:42.820990 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:42.820796 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kz76\" (UniqueName: \"kubernetes.io/projected/fef4ac06-3bbf-4fbe-8de2-caae3e7fa017-kube-api-access-9kz76\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:40:42.878176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:42.878135 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:43.021538 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:43.021430 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d9ede0-rj4zc"] Oct 30 01:40:43.023487 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:43.023444 6572 cpu_manager.go:414] "containerMap: removing container" podUID="fef4ac06-3bbf-4fbe-8de2-caae3e7fa017" containerName="revoke-keys" Oct 30 01:40:43.023655 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:43.023494 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fef4ac06-3bbf-4fbe-8de2-caae3e7fa017" containerName="revoke-keys" Oct 30 01:40:43.023874 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:43.023854 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="fef4ac06-3bbf-4fbe-8de2-caae3e7fa017" containerName="revoke-keys" Oct 30 01:40:43.029692 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:43.029655 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d9ede0-rj4zc" Oct 30 01:40:43.126822 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:43.096303 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d9ede0-rj4zc"] Oct 30 01:40:43.136064 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:43.129244 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ae2f8aa-149d-4f1d-953d-884ae912487a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d9ede0-rj4zc\" (UID: \"6ae2f8aa-149d-4f1d-953d-884ae912487a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d9ede0-rj4zc" Oct 30 01:40:43.136501 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:43.136471 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrfv5\" (UniqueName: \"kubernetes.io/projected/6ae2f8aa-149d-4f1d-953d-884ae912487a-kube-api-access-vrfv5\") pod \"maas-api-revoke-keys-e2e-ipp-a-d9ede0-rj4zc\" (UID: \"6ae2f8aa-149d-4f1d-953d-884ae912487a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d9ede0-rj4zc" Oct 30 01:40:43.138677 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:43.138648 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ae2f8aa-149d-4f1d-953d-884ae912487a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d9ede0-rj4zc\" (UID: \"6ae2f8aa-149d-4f1d-953d-884ae912487a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d9ede0-rj4zc" Oct 30 01:40:43.241486 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:43.241450 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ae2f8aa-149d-4f1d-953d-884ae912487a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d9ede0-rj4zc\" (UID: \"6ae2f8aa-149d-4f1d-953d-884ae912487a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d9ede0-rj4zc" Oct 30 01:40:43.241706 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:43.241642 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrfv5\" (UniqueName: \"kubernetes.io/projected/6ae2f8aa-149d-4f1d-953d-884ae912487a-kube-api-access-vrfv5\") pod \"maas-api-revoke-keys-e2e-ipp-a-d9ede0-rj4zc\" (UID: \"6ae2f8aa-149d-4f1d-953d-884ae912487a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d9ede0-rj4zc" Oct 30 01:40:43.241706 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:43.241700 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ae2f8aa-149d-4f1d-953d-884ae912487a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d9ede0-rj4zc\" (UID: \"6ae2f8aa-149d-4f1d-953d-884ae912487a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d9ede0-rj4zc" Oct 30 01:40:43.242401 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:43.242375 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ae2f8aa-149d-4f1d-953d-884ae912487a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d9ede0-rj4zc\" (UID: \"6ae2f8aa-149d-4f1d-953d-884ae912487a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d9ede0-rj4zc" Oct 30 01:40:43.244036 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:43.244008 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ae2f8aa-149d-4f1d-953d-884ae912487a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d9ede0-rj4zc\" (UID: \"6ae2f8aa-149d-4f1d-953d-884ae912487a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d9ede0-rj4zc" Oct 30 01:40:43.260444 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:43.260413 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrfv5\" (UniqueName: \"kubernetes.io/projected/6ae2f8aa-149d-4f1d-953d-884ae912487a-kube-api-access-vrfv5\") pod \"maas-api-revoke-keys-e2e-ipp-a-d9ede0-rj4zc\" (UID: \"6ae2f8aa-149d-4f1d-953d-884ae912487a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d9ede0-rj4zc" Oct 30 01:40:43.456256 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:43.456228 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19a5f1-fjl9d" Oct 30 01:40:43.456444 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:43.456269 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19a5f1-fjl9d" event={"ID":"fef4ac06-3bbf-4fbe-8de2-caae3e7fa017","Type":"ContainerDied","Data":"5c5b1159bbbe15553a9cc75dfea0acf4b51db5ba175da33372939896500ac929"} Oct 30 01:40:43.456444 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:43.456321 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c5b1159bbbe15553a9cc75dfea0acf4b51db5ba175da33372939896500ac929" Oct 30 01:40:43.463645 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:43.463618 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d9ede0-rj4zc" Oct 30 01:40:43.763083 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:43.763051 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d9ede0-rj4zc"] Oct 30 01:40:43.764305 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:40:43.764274 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae2f8aa_149d_4f1d_953d_884ae912487a.slice/crio-9c033994bd5de98f77b9f545564ed55c242e68850ce5d06425cf6aa91113785b WatchSource:0}: Error finding container 9c033994bd5de98f77b9f545564ed55c242e68850ce5d06425cf6aa91113785b: Status 404 returned error can't find the container with id 9c033994bd5de98f77b9f545564ed55c242e68850ce5d06425cf6aa91113785b Oct 30 01:40:44.467182 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:44.467142 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d9ede0-rj4zc" event={"ID":"6ae2f8aa-149d-4f1d-953d-884ae912487a","Type":"ContainerStarted","Data":"e9dd87355e8fda4cd0bf8a1c2d0761e7bcbaed9127e8353b7dbe57dc7b3100a1"} Oct 30 01:40:44.467359 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:44.467206 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d9ede0-rj4zc" event={"ID":"6ae2f8aa-149d-4f1d-953d-884ae912487a","Type":"ContainerStarted","Data":"9c033994bd5de98f77b9f545564ed55c242e68850ce5d06425cf6aa91113785b"} Oct 30 01:40:44.507219 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:44.507165 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d9ede0-rj4zc" podStartSLOduration=2.50715006 podStartE2EDuration="2.50715006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:40:44.503092884 +0000 UTC m=+1370.079518286" watchObservedRunningTime="2026-10-30 01:40:44.50715006 +0000 UTC m=+1370.083575462" Oct 30 01:40:44.605103 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:44.605064 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:44.687808 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:40:44.687734 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:40:45.477823 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:45.477788 6572 generic.go:356] "Generic (PLEG): container finished" podID="6ae2f8aa-149d-4f1d-953d-884ae912487a" containerID="e9dd87355e8fda4cd0bf8a1c2d0761e7bcbaed9127e8353b7dbe57dc7b3100a1" exitCode=0 Oct 30 01:40:45.478281 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:45.477876 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d9ede0-rj4zc" event={"ID":"6ae2f8aa-149d-4f1d-953d-884ae912487a","Type":"ContainerDied","Data":"e9dd87355e8fda4cd0bf8a1c2d0761e7bcbaed9127e8353b7dbe57dc7b3100a1"} Oct 30 01:40:46.546902 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:46.546865 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:46.684197 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:46.684159 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d9ede0-rj4zc" Oct 30 01:40:46.794489 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:46.794389 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ae2f8aa-149d-4f1d-953d-884ae912487a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ae2f8aa-149d-4f1d-953d-884ae912487a-maas-api-xks-ca\") pod \"6ae2f8aa-149d-4f1d-953d-884ae912487a\" (UID: \"6ae2f8aa-149d-4f1d-953d-884ae912487a\") " Oct 30 01:40:46.794489 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:46.794464 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ae2f8aa-149d-4f1d-953d-884ae912487a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ae2f8aa-149d-4f1d-953d-884ae912487a-maas-api-openshift-ca\") pod \"6ae2f8aa-149d-4f1d-953d-884ae912487a\" (UID: \"6ae2f8aa-149d-4f1d-953d-884ae912487a\") " Oct 30 01:40:46.794701 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:46.794507 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ae2f8aa-149d-4f1d-953d-884ae912487a-kube-api-access-vrfv5\" (UniqueName: \"kubernetes.io/projected/6ae2f8aa-149d-4f1d-953d-884ae912487a-kube-api-access-vrfv5\") pod \"6ae2f8aa-149d-4f1d-953d-884ae912487a\" (UID: \"6ae2f8aa-149d-4f1d-953d-884ae912487a\") " Oct 30 01:40:46.794981 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:46.794947 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ae2f8aa-149d-4f1d-953d-884ae912487a-maas-api-openshift-ca" pod "6ae2f8aa-149d-4f1d-953d-884ae912487a" (UID: "6ae2f8aa-149d-4f1d-953d-884ae912487a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:40:46.796677 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:46.796650 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ae2f8aa-149d-4f1d-953d-884ae912487a-maas-api-xks-ca" pod "6ae2f8aa-149d-4f1d-953d-884ae912487a" (UID: "6ae2f8aa-149d-4f1d-953d-884ae912487a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:40:46.796804 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:46.796739 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae2f8aa-149d-4f1d-953d-884ae912487a-kube-api-access-vrfv5" pod "6ae2f8aa-149d-4f1d-953d-884ae912487a" (UID: "6ae2f8aa-149d-4f1d-953d-884ae912487a"). InnerVolumeSpecName "kube-api-access-vrfv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:40:46.896676 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:46.896634 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ae2f8aa-149d-4f1d-953d-884ae912487a-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:40:46.896676 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:46.896668 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ae2f8aa-149d-4f1d-953d-884ae912487a-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:40:46.896676 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:46.896678 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrfv5\" (UniqueName: \"kubernetes.io/projected/6ae2f8aa-149d-4f1d-953d-884ae912487a-kube-api-access-vrfv5\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:40:47.497167 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:47.497134 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d9ede0-rj4zc" Oct 30 01:40:47.497371 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:47.497156 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d9ede0-rj4zc" event={"ID":"6ae2f8aa-149d-4f1d-953d-884ae912487a","Type":"ContainerDied","Data":"9c033994bd5de98f77b9f545564ed55c242e68850ce5d06425cf6aa91113785b"} Oct 30 01:40:47.497371 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:47.497247 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c033994bd5de98f77b9f545564ed55c242e68850ce5d06425cf6aa91113785b" Oct 30 01:40:47.841240 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:47.841140 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:48.022788 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:48.022723 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:48.328496 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:48.328375 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0db534-5cb9dc6664-bphz5"] Oct 30 01:40:48.329375 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:48.329341 6572 cpu_manager.go:414] "containerMap: removing container" podUID="6ae2f8aa-149d-4f1d-953d-884ae912487a" containerName="revoke-keys" Oct 30 01:40:48.329375 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:48.329371 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ae2f8aa-149d-4f1d-953d-884ae912487a" containerName="revoke-keys" Oct 30 01:40:48.329591 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:48.329557 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ae2f8aa-149d-4f1d-953d-884ae912487a" containerName="revoke-keys" Oct 30 01:40:48.333846 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:48.333816 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db534-5cb9dc6664-bphz5" Oct 30 01:40:48.354184 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:48.354146 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0db534-5cb9dc6664-bphz5"] Oct 30 01:40:48.364305 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:48.361568 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-0db534\"" Oct 30 01:40:48.364305 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:48.361978 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-0db534-dockercfg-h29p2\"" Oct 30 01:40:48.521426 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:48.521383 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvx2n\" (UniqueName: \"kubernetes.io/projected/18295ccd-e011-41e0-8077-7abea7fce67f-kube-api-access-nvx2n\") pod \"payload-processing-e2e-ipp-b-0db534-5cb9dc6664-bphz5\" (UID: \"18295ccd-e011-41e0-8077-7abea7fce67f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0db534-5cb9dc6664-bphz5" Oct 30 01:40:48.521604 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:48.521439 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18295ccd-e011-41e0-8077-7abea7fce67f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0db534-5cb9dc6664-bphz5\" (UID: \"18295ccd-e011-41e0-8077-7abea7fce67f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0db534-5cb9dc6664-bphz5" Oct 30 01:40:48.623374 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:48.623280 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvx2n\" (UniqueName: \"kubernetes.io/projected/18295ccd-e011-41e0-8077-7abea7fce67f-kube-api-access-nvx2n\") pod \"payload-processing-e2e-ipp-b-0db534-5cb9dc6664-bphz5\" (UID: \"18295ccd-e011-41e0-8077-7abea7fce67f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0db534-5cb9dc6664-bphz5" Oct 30 01:40:48.623374 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:48.623327 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18295ccd-e011-41e0-8077-7abea7fce67f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0db534-5cb9dc6664-bphz5\" (UID: \"18295ccd-e011-41e0-8077-7abea7fce67f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0db534-5cb9dc6664-bphz5" Oct 30 01:40:48.624198 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:48.624170 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18295ccd-e011-41e0-8077-7abea7fce67f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0db534-5cb9dc6664-bphz5\" (UID: \"18295ccd-e011-41e0-8077-7abea7fce67f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0db534-5cb9dc6664-bphz5" Oct 30 01:40:48.642212 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:48.642173 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvx2n\" (UniqueName: \"kubernetes.io/projected/18295ccd-e011-41e0-8077-7abea7fce67f-kube-api-access-nvx2n\") pod \"payload-processing-e2e-ipp-b-0db534-5cb9dc6664-bphz5\" (UID: \"18295ccd-e011-41e0-8077-7abea7fce67f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0db534-5cb9dc6664-bphz5" Oct 30 01:40:48.668829 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:48.668794 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db534-5cb9dc6664-bphz5" Oct 30 01:40:48.970981 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:48.970949 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0db534-5cb9dc6664-bphz5"] Oct 30 01:40:48.972312 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:40:48.972280 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18295ccd_e011_41e0_8077_7abea7fce67f.slice/crio-1c04997a926657b633effe1a947bcbe9073543d93a21e37e79fd182887cd0970 WatchSource:0}: Error finding container 1c04997a926657b633effe1a947bcbe9073543d93a21e37e79fd182887cd0970: Status 404 returned error can't find the container with id 1c04997a926657b633effe1a947bcbe9073543d93a21e37e79fd182887cd0970 Oct 30 01:40:49.024500 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:49.024315 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:49.516118 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:49.516082 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db534-5cb9dc6664-bphz5" event={"ID":"18295ccd-e011-41e0-8077-7abea7fce67f","Type":"ContainerStarted","Data":"114fa08b80935aac2f6d562c6a906d16b31dd96fe71f238607bd6cec251e5ef4"} Oct 30 01:40:49.516303 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:49.516143 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db534-5cb9dc6664-bphz5" event={"ID":"18295ccd-e011-41e0-8077-7abea7fce67f","Type":"ContainerStarted","Data":"1c04997a926657b633effe1a947bcbe9073543d93a21e37e79fd182887cd0970"} Oct 30 01:40:49.516303 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:49.516169 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db534-5cb9dc6664-bphz5" Oct 30 01:40:49.558691 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:49.558641 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db534-5cb9dc6664-bphz5" podStartSLOduration=1.558628109 podStartE2EDuration="1.558628109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:40:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:40:49.555079726 +0000 UTC m=+1375.131505127" watchObservedRunningTime="2026-10-30 01:40:49.558628109 +0000 UTC m=+1375.135053530" Oct 30 01:40:50.805875 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:50.805828 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:51.346887 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:51.346849 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:53.509631 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:53.509595 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:55.697793 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:55.694461 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:55.960128 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:55.960055 6572 scope.go:122] "RemoveContainer" containerID="6ebf2030e9184d67ebe2950149cb181e6f31529567072bcce6b2134d69432ac2" Oct 30 01:40:55.970991 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:55.970970 6572 scope.go:122] "RemoveContainer" containerID="2cd0d5c12766664f2414990d2b10bae1b3f0eac35bffee23b37cb74fa5d4a1f2" Oct 30 01:40:55.981345 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:55.981320 6572 scope.go:122] "RemoveContainer" containerID="21aac4872ccbb34e54c8c6ba970def3af1182ef36a8b61b6db67b8648422b01e" Oct 30 01:40:55.991252 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:55.991230 6572 scope.go:122] "RemoveContainer" containerID="176a7d6064b2dba41e33018b0bfe33ce4d6e03e4aa6284d57e4f8dbdbca2b4a3" Oct 30 01:40:56.000380 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:56.000359 6572 scope.go:122] "RemoveContainer" containerID="c8c5ff6ac86ef6f7a5a645beb5b799b973284153af0e62e5470172dd14991fa1" Oct 30 01:40:58.014182 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:58.014145 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:59.443451 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:59.443417 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:59.460988 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:59.460963 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-7dd7645496-mmljt"] Oct 30 01:40:59.461169 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:59.461146 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-7dd7645496-mmljt" podUID="773200f9-3e31-41f4-817f-d6fb5b97e12f" containerName="payload-processing" containerID="cri-o://36c139386af68504ca1aa9f945c2dda21376f92deb16e15fb4cdd5d7fd090b8a" gracePeriod=30 Oct 30 01:40:59.467434 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:40:59.467343 6572 helpers.go:246] readString: Failed to read "/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod773200f9_3e31_41f4_817f_d6fb5b97e12f.slice/crio-36c139386af68504ca1aa9f945c2dda21376f92deb16e15fb4cdd5d7fd090b8a.scope/cpu.max": read /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod773200f9_3e31_41f4_817f_d6fb5b97e12f.slice/crio-36c139386af68504ca1aa9f945c2dda21376f92deb16e15fb4cdd5d7fd090b8a.scope/cpu.max: no such device Oct 30 01:40:59.477290 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:59.477265 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7dc7f7bb59-mc2jj"] Oct 30 01:40:59.477587 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:59.477561 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7dc7f7bb59-mc2jj" podUID="00aac51a-0f35-4f98-95cc-8ee9a919a3fe" containerName="payload-pre-processing" containerID="cri-o://f5117d5f02a2b64be2b08783dd9d90886692168c5b21d5077057c2fb8c74d3b9" gracePeriod=30 Oct 30 01:40:59.543380 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:40:59.543298 6572 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00aac51a_0f35_4f98_95cc_8ee9a919a3fe.slice/crio-f5117d5f02a2b64be2b08783dd9d90886692168c5b21d5077057c2fb8c74d3b9.scope\": RecentStats: unable to find data in memory cache]" Oct 30 01:40:59.634125 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:59.634090 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:59.645693 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:59.645481 6572 generic.go:356] "Generic (PLEG): container finished" podID="00aac51a-0f35-4f98-95cc-8ee9a919a3fe" containerID="f5117d5f02a2b64be2b08783dd9d90886692168c5b21d5077057c2fb8c74d3b9" exitCode=0 Oct 30 01:40:59.645693 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:59.645550 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7dc7f7bb59-mc2jj" event={"ID":"00aac51a-0f35-4f98-95cc-8ee9a919a3fe","Type":"ContainerDied","Data":"f5117d5f02a2b64be2b08783dd9d90886692168c5b21d5077057c2fb8c74d3b9"} Oct 30 01:40:59.649369 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:59.649219 6572 generic.go:356] "Generic (PLEG): container finished" podID="773200f9-3e31-41f4-817f-d6fb5b97e12f" containerID="36c139386af68504ca1aa9f945c2dda21376f92deb16e15fb4cdd5d7fd090b8a" exitCode=0 Oct 30 01:40:59.649369 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:59.649323 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-7dd7645496-mmljt" event={"ID":"773200f9-3e31-41f4-817f-d6fb5b97e12f","Type":"ContainerDied","Data":"36c139386af68504ca1aa9f945c2dda21376f92deb16e15fb4cdd5d7fd090b8a"} Oct 30 01:40:59.805271 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:59.805248 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-7dd7645496-mmljt" Oct 30 01:40:59.810349 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:59.810323 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:59.813879 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:59.813862 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7dc7f7bb59-mc2jj" Oct 30 01:40:59.941557 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:59.940311 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00aac51a-0f35-4f98-95cc-8ee9a919a3fe-kube-api-access-42fcg\" (UniqueName: \"kubernetes.io/projected/00aac51a-0f35-4f98-95cc-8ee9a919a3fe-kube-api-access-42fcg\") pod \"00aac51a-0f35-4f98-95cc-8ee9a919a3fe\" (UID: \"00aac51a-0f35-4f98-95cc-8ee9a919a3fe\") " Oct 30 01:40:59.941557 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:59.940411 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/773200f9-3e31-41f4-817f-d6fb5b97e12f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/773200f9-3e31-41f4-817f-d6fb5b97e12f-plugins-config-volume\") pod \"773200f9-3e31-41f4-817f-d6fb5b97e12f\" (UID: \"773200f9-3e31-41f4-817f-d6fb5b97e12f\") " Oct 30 01:40:59.941557 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:59.940452 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00aac51a-0f35-4f98-95cc-8ee9a919a3fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00aac51a-0f35-4f98-95cc-8ee9a919a3fe-plugins-config-volume\") pod \"00aac51a-0f35-4f98-95cc-8ee9a919a3fe\" (UID: \"00aac51a-0f35-4f98-95cc-8ee9a919a3fe\") " Oct 30 01:40:59.941557 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:59.940545 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/773200f9-3e31-41f4-817f-d6fb5b97e12f-kube-api-access-qngpp\" (UniqueName: \"kubernetes.io/projected/773200f9-3e31-41f4-817f-d6fb5b97e12f-kube-api-access-qngpp\") pod \"773200f9-3e31-41f4-817f-d6fb5b97e12f\" (UID: \"773200f9-3e31-41f4-817f-d6fb5b97e12f\") " Oct 30 01:40:59.942607 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:59.942564 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00aac51a-0f35-4f98-95cc-8ee9a919a3fe-plugins-config-volume" pod "00aac51a-0f35-4f98-95cc-8ee9a919a3fe" (UID: "00aac51a-0f35-4f98-95cc-8ee9a919a3fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:40:59.943548 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:59.943524 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/773200f9-3e31-41f4-817f-d6fb5b97e12f-plugins-config-volume" pod "773200f9-3e31-41f4-817f-d6fb5b97e12f" (UID: "773200f9-3e31-41f4-817f-d6fb5b97e12f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:40:59.952695 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:59.952640 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00aac51a-0f35-4f98-95cc-8ee9a919a3fe-kube-api-access-42fcg" pod "00aac51a-0f35-4f98-95cc-8ee9a919a3fe" (UID: "00aac51a-0f35-4f98-95cc-8ee9a919a3fe"). InnerVolumeSpecName "kube-api-access-42fcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:40:59.952986 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:40:59.952957 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/773200f9-3e31-41f4-817f-d6fb5b97e12f-kube-api-access-qngpp" pod "773200f9-3e31-41f4-817f-d6fb5b97e12f" (UID: "773200f9-3e31-41f4-817f-d6fb5b97e12f"). InnerVolumeSpecName "kube-api-access-qngpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:41:00.042144 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:00.042062 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42fcg\" (UniqueName: \"kubernetes.io/projected/00aac51a-0f35-4f98-95cc-8ee9a919a3fe-kube-api-access-42fcg\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:41:00.042144 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:00.042091 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/773200f9-3e31-41f4-817f-d6fb5b97e12f-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:41:00.042144 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:00.042102 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00aac51a-0f35-4f98-95cc-8ee9a919a3fe-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:41:00.042144 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:00.042112 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qngpp\" (UniqueName: \"kubernetes.io/projected/773200f9-3e31-41f4-817f-d6fb5b97e12f-kube-api-access-qngpp\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:41:00.074209 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:00.074174 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:00.191865 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:00.191837 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19a5f1-fjl9d"] Oct 30 01:41:00.195882 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:00.195857 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-19a5f1-fjl9d"] Oct 30 01:41:00.267026 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:00.266993 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:00.527338 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:00.527312 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db534-5cb9dc6664-bphz5" Oct 30 01:41:00.543960 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:00.543934 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fef4ac06-3bbf-4fbe-8de2-caae3e7fa017" path="/var/lib/kubelet/pods/fef4ac06-3bbf-4fbe-8de2-caae3e7fa017/volumes" Oct 30 01:41:00.659114 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:00.659087 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7dc7f7bb59-mc2jj" Oct 30 01:41:00.659304 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:00.659125 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7dc7f7bb59-mc2jj" event={"ID":"00aac51a-0f35-4f98-95cc-8ee9a919a3fe","Type":"ContainerDied","Data":"8c68ebd4a872358bb9dc738b79c2946887536728a5fc81e667bf1ff8a3dfd66f"} Oct 30 01:41:00.659304 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:00.659194 6572 scope.go:122] "RemoveContainer" containerID="f5117d5f02a2b64be2b08783dd9d90886692168c5b21d5077057c2fb8c74d3b9" Oct 30 01:41:00.660912 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:00.660895 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-7dd7645496-mmljt" Oct 30 01:41:00.661040 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:00.660929 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-7dd7645496-mmljt" event={"ID":"773200f9-3e31-41f4-817f-d6fb5b97e12f","Type":"ContainerDied","Data":"c374accc81a7cc255a82f2a49f5c460f34b25059ca989f59bcf0d33a7eda7ef2"} Oct 30 01:41:00.669869 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:00.669850 6572 scope.go:122] "RemoveContainer" containerID="36c139386af68504ca1aa9f945c2dda21376f92deb16e15fb4cdd5d7fd090b8a" Oct 30 01:41:00.709496 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:00.709462 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-7dd7645496-mmljt"] Oct 30 01:41:00.717408 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:00.717379 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-7dd7645496-mmljt"] Oct 30 01:41:00.749644 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:00.749553 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7dc7f7bb59-mc2jj"] Oct 30 01:41:00.753417 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:00.753392 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-19a5f1-7dc7f7bb59-mc2jj"] Oct 30 01:41:00.769018 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:00.768990 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:01.979441 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:01.979409 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:02.542666 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:02.542622 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00aac51a-0f35-4f98-95cc-8ee9a919a3fe" path="/var/lib/kubelet/pods/00aac51a-0f35-4f98-95cc-8ee9a919a3fe/volumes" Oct 30 01:41:02.543173 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:02.543150 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="773200f9-3e31-41f4-817f-d6fb5b97e12f" path="/var/lib/kubelet/pods/773200f9-3e31-41f4-817f-d6fb5b97e12f/volumes" Oct 30 01:41:03.629922 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:03.629883 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:04.134707 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:04.134673 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7bdcc-dhxrh"] Oct 30 01:41:04.135322 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:04.135307 6572 cpu_manager.go:414] "containerMap: removing container" podUID="773200f9-3e31-41f4-817f-d6fb5b97e12f" containerName="payload-processing" Oct 30 01:41:04.135370 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:04.135323 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="773200f9-3e31-41f4-817f-d6fb5b97e12f" containerName="payload-processing" Oct 30 01:41:04.135370 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:04.135356 6572 cpu_manager.go:414] "containerMap: removing container" podUID="00aac51a-0f35-4f98-95cc-8ee9a919a3fe" containerName="payload-pre-processing" Oct 30 01:41:04.135370 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:04.135361 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00aac51a-0f35-4f98-95cc-8ee9a919a3fe" containerName="payload-pre-processing" Oct 30 01:41:04.135500 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:04.135489 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="773200f9-3e31-41f4-817f-d6fb5b97e12f" containerName="payload-processing" Oct 30 01:41:04.135535 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:04.135509 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="00aac51a-0f35-4f98-95cc-8ee9a919a3fe" containerName="payload-pre-processing" Oct 30 01:41:04.141074 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:04.141053 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7bdcc-dhxrh" Oct 30 01:41:04.156545 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:04.156521 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7bdcc-dhxrh"] Oct 30 01:41:04.163478 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:04.163453 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:41:04.163613 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:04.163503 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:41:04.163613 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:04.163517 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:41:04.163613 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:04.163575 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:41:04.295057 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:04.295028 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f988585-a8a7-4316-b9fb-d40788522236-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f7bdcc-dhxrh\" (UID: \"4f988585-a8a7-4316-b9fb-d40788522236\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7bdcc-dhxrh" Oct 30 01:41:04.295225 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:04.295063 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdh4g\" (UniqueName: \"kubernetes.io/projected/4f988585-a8a7-4316-b9fb-d40788522236-kube-api-access-sdh4g\") pod \"maas-api-revoke-keys-e2e-ipp-a-f7bdcc-dhxrh\" (UID: \"4f988585-a8a7-4316-b9fb-d40788522236\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7bdcc-dhxrh" Oct 30 01:41:04.295269 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:04.295225 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f988585-a8a7-4316-b9fb-d40788522236-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f7bdcc-dhxrh\" (UID: \"4f988585-a8a7-4316-b9fb-d40788522236\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7bdcc-dhxrh" Oct 30 01:41:04.397659 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:04.397592 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f988585-a8a7-4316-b9fb-d40788522236-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f7bdcc-dhxrh\" (UID: \"4f988585-a8a7-4316-b9fb-d40788522236\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7bdcc-dhxrh" Oct 30 01:41:04.397659 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:04.397632 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdh4g\" (UniqueName: \"kubernetes.io/projected/4f988585-a8a7-4316-b9fb-d40788522236-kube-api-access-sdh4g\") pod \"maas-api-revoke-keys-e2e-ipp-a-f7bdcc-dhxrh\" (UID: \"4f988585-a8a7-4316-b9fb-d40788522236\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7bdcc-dhxrh" Oct 30 01:41:04.398068 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:04.397709 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f988585-a8a7-4316-b9fb-d40788522236-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f7bdcc-dhxrh\" (UID: \"4f988585-a8a7-4316-b9fb-d40788522236\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7bdcc-dhxrh" Oct 30 01:41:04.398367 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:04.398345 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f988585-a8a7-4316-b9fb-d40788522236-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f7bdcc-dhxrh\" (UID: \"4f988585-a8a7-4316-b9fb-d40788522236\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7bdcc-dhxrh" Oct 30 01:41:04.399935 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:04.399918 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f988585-a8a7-4316-b9fb-d40788522236-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f7bdcc-dhxrh\" (UID: \"4f988585-a8a7-4316-b9fb-d40788522236\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7bdcc-dhxrh" Oct 30 01:41:04.420663 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:04.420639 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdh4g\" (UniqueName: \"kubernetes.io/projected/4f988585-a8a7-4316-b9fb-d40788522236-kube-api-access-sdh4g\") pod \"maas-api-revoke-keys-e2e-ipp-a-f7bdcc-dhxrh\" (UID: \"4f988585-a8a7-4316-b9fb-d40788522236\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7bdcc-dhxrh" Oct 30 01:41:04.454438 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:04.454411 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7bdcc-dhxrh" Oct 30 01:41:04.654866 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:04.654769 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:04.751145 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:41:04.751118 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f988585_a8a7_4316_b9fb_d40788522236.slice/crio-8c2a33cc10018ceb5edd3110c13f86367c13db9d0ab6e029a6b031df59cc2c87 WatchSource:0}: Error finding container 8c2a33cc10018ceb5edd3110c13f86367c13db9d0ab6e029a6b031df59cc2c87: Status 404 returned error can't find the container with id 8c2a33cc10018ceb5edd3110c13f86367c13db9d0ab6e029a6b031df59cc2c87 Oct 30 01:41:04.751864 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:04.751844 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7bdcc-dhxrh"] Oct 30 01:41:05.474966 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:05.474927 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:05.702358 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:05.702327 6572 generic.go:356] "Generic (PLEG): container finished" podID="4f988585-a8a7-4316-b9fb-d40788522236" containerID="8d6eb7b6247f6462e7bad9fa1295d693a2c24278c52154180cb8a3301e062341" exitCode=0 Oct 30 01:41:05.702730 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:05.702406 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7bdcc-dhxrh" event={"ID":"4f988585-a8a7-4316-b9fb-d40788522236","Type":"ContainerDied","Data":"8d6eb7b6247f6462e7bad9fa1295d693a2c24278c52154180cb8a3301e062341"} Oct 30 01:41:05.702730 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:05.702459 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7bdcc-dhxrh" event={"ID":"4f988585-a8a7-4316-b9fb-d40788522236","Type":"ContainerStarted","Data":"8c2a33cc10018ceb5edd3110c13f86367c13db9d0ab6e029a6b031df59cc2c87"} Oct 30 01:41:06.868076 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:06.868049 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7bdcc-dhxrh" Oct 30 01:41:06.931107 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:06.931077 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f988585-a8a7-4316-b9fb-d40788522236-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f988585-a8a7-4316-b9fb-d40788522236-maas-api-xks-ca\") pod \"4f988585-a8a7-4316-b9fb-d40788522236\" (UID: \"4f988585-a8a7-4316-b9fb-d40788522236\") " Oct 30 01:41:06.931272 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:06.931149 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f988585-a8a7-4316-b9fb-d40788522236-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f988585-a8a7-4316-b9fb-d40788522236-maas-api-openshift-ca\") pod \"4f988585-a8a7-4316-b9fb-d40788522236\" (UID: \"4f988585-a8a7-4316-b9fb-d40788522236\") " Oct 30 01:41:06.931336 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:06.931302 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f988585-a8a7-4316-b9fb-d40788522236-kube-api-access-sdh4g\" (UniqueName: \"kubernetes.io/projected/4f988585-a8a7-4316-b9fb-d40788522236-kube-api-access-sdh4g\") pod \"4f988585-a8a7-4316-b9fb-d40788522236\" (UID: \"4f988585-a8a7-4316-b9fb-d40788522236\") " Oct 30 01:41:06.931538 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:06.931510 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f988585-a8a7-4316-b9fb-d40788522236-maas-api-openshift-ca" pod "4f988585-a8a7-4316-b9fb-d40788522236" (UID: "4f988585-a8a7-4316-b9fb-d40788522236"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:41:06.931782 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:06.931762 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f988585-a8a7-4316-b9fb-d40788522236-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:41:06.933175 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:06.933153 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f988585-a8a7-4316-b9fb-d40788522236-maas-api-xks-ca" pod "4f988585-a8a7-4316-b9fb-d40788522236" (UID: "4f988585-a8a7-4316-b9fb-d40788522236"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:41:06.933341 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:06.933321 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f988585-a8a7-4316-b9fb-d40788522236-kube-api-access-sdh4g" pod "4f988585-a8a7-4316-b9fb-d40788522236" (UID: "4f988585-a8a7-4316-b9fb-d40788522236"). InnerVolumeSpecName "kube-api-access-sdh4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:41:06.996629 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:06.996526 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:07.033071 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:07.033038 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdh4g\" (UniqueName: \"kubernetes.io/projected/4f988585-a8a7-4316-b9fb-d40788522236-kube-api-access-sdh4g\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:41:07.033071 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:07.033067 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f988585-a8a7-4316-b9fb-d40788522236-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:41:07.192448 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:07.192409 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:07.421855 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:07.406868 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:07.719863 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:07.719742 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7bdcc-dhxrh" Oct 30 01:41:07.719863 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:07.719776 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7bdcc-dhxrh" event={"ID":"4f988585-a8a7-4316-b9fb-d40788522236","Type":"ContainerDied","Data":"8c2a33cc10018ceb5edd3110c13f86367c13db9d0ab6e029a6b031df59cc2c87"} Oct 30 01:41:07.720101 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:07.719923 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c2a33cc10018ceb5edd3110c13f86367c13db9d0ab6e029a6b031df59cc2c87" Oct 30 01:41:07.838950 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:07.838919 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc51e9-ps5m4"] Oct 30 01:41:07.839537 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:07.839524 6572 cpu_manager.go:414] "containerMap: removing container" podUID="4f988585-a8a7-4316-b9fb-d40788522236" containerName="revoke-keys" Oct 30 01:41:07.839583 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:07.839538 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f988585-a8a7-4316-b9fb-d40788522236" containerName="revoke-keys" Oct 30 01:41:07.839667 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:07.839657 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f988585-a8a7-4316-b9fb-d40788522236" containerName="revoke-keys" Oct 30 01:41:07.843423 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:07.843403 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc51e9-ps5m4" Oct 30 01:41:07.857704 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:07.857682 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:41:07.858011 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:07.857991 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:41:07.862081 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:07.862060 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc51e9-ps5m4"] Oct 30 01:41:07.869101 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:07.869067 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:41:07.869101 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:07.869081 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:41:07.945289 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:07.945257 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8e808e8-7f20-4aa5-94aa-5a29f952d269-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fc51e9-ps5m4\" (UID: \"d8e808e8-7f20-4aa5-94aa-5a29f952d269\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc51e9-ps5m4" Oct 30 01:41:07.945428 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:07.945407 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8e808e8-7f20-4aa5-94aa-5a29f952d269-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fc51e9-ps5m4\" (UID: \"d8e808e8-7f20-4aa5-94aa-5a29f952d269\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc51e9-ps5m4" Oct 30 01:41:07.945517 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:07.945493 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlwx5\" (UniqueName: \"kubernetes.io/projected/d8e808e8-7f20-4aa5-94aa-5a29f952d269-kube-api-access-rlwx5\") pod \"maas-api-revoke-keys-e2e-ipp-b-fc51e9-ps5m4\" (UID: \"d8e808e8-7f20-4aa5-94aa-5a29f952d269\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc51e9-ps5m4" Oct 30 01:41:08.047065 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:08.046992 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8e808e8-7f20-4aa5-94aa-5a29f952d269-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fc51e9-ps5m4\" (UID: \"d8e808e8-7f20-4aa5-94aa-5a29f952d269\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc51e9-ps5m4" Oct 30 01:41:08.047204 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:08.047083 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8e808e8-7f20-4aa5-94aa-5a29f952d269-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fc51e9-ps5m4\" (UID: \"d8e808e8-7f20-4aa5-94aa-5a29f952d269\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc51e9-ps5m4" Oct 30 01:41:08.047204 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:08.047106 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlwx5\" (UniqueName: \"kubernetes.io/projected/d8e808e8-7f20-4aa5-94aa-5a29f952d269-kube-api-access-rlwx5\") pod \"maas-api-revoke-keys-e2e-ipp-b-fc51e9-ps5m4\" (UID: \"d8e808e8-7f20-4aa5-94aa-5a29f952d269\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc51e9-ps5m4" Oct 30 01:41:08.047795 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:08.047771 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8e808e8-7f20-4aa5-94aa-5a29f952d269-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fc51e9-ps5m4\" (UID: \"d8e808e8-7f20-4aa5-94aa-5a29f952d269\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc51e9-ps5m4" Oct 30 01:41:08.049371 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:08.049354 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8e808e8-7f20-4aa5-94aa-5a29f952d269-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fc51e9-ps5m4\" (UID: \"d8e808e8-7f20-4aa5-94aa-5a29f952d269\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc51e9-ps5m4" Oct 30 01:41:08.066174 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:08.066145 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlwx5\" (UniqueName: \"kubernetes.io/projected/d8e808e8-7f20-4aa5-94aa-5a29f952d269-kube-api-access-rlwx5\") pod \"maas-api-revoke-keys-e2e-ipp-b-fc51e9-ps5m4\" (UID: \"d8e808e8-7f20-4aa5-94aa-5a29f952d269\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc51e9-ps5m4" Oct 30 01:41:08.157686 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:08.157653 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc51e9-ps5m4" Oct 30 01:41:08.459344 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:41:08.459315 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8e808e8_7f20_4aa5_94aa_5a29f952d269.slice/crio-919e146cdea33c0d90051e1f9ddf97e727926e54a64e5546ac9c57190b1a09d7 WatchSource:0}: Error finding container 919e146cdea33c0d90051e1f9ddf97e727926e54a64e5546ac9c57190b1a09d7: Status 404 returned error can't find the container with id 919e146cdea33c0d90051e1f9ddf97e727926e54a64e5546ac9c57190b1a09d7 Oct 30 01:41:08.459606 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:08.459579 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc51e9-ps5m4"] Oct 30 01:41:08.530146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:08.530120 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:08.732311 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:08.732285 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc51e9-ps5m4" event={"ID":"d8e808e8-7f20-4aa5-94aa-5a29f952d269","Type":"ContainerStarted","Data":"237e7c18b521415c115447c553699fba789fbbabd9ade9a7938126921c65ca9f"} Oct 30 01:41:08.732445 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:08.732332 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc51e9-ps5m4" event={"ID":"d8e808e8-7f20-4aa5-94aa-5a29f952d269","Type":"ContainerStarted","Data":"919e146cdea33c0d90051e1f9ddf97e727926e54a64e5546ac9c57190b1a09d7"} Oct 30 01:41:08.768888 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:08.768840 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc51e9-ps5m4" podStartSLOduration=1.76882639 podStartE2EDuration="1.76882639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:41:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:41:08.767955649 +0000 UTC m=+1394.344381053" watchObservedRunningTime="2026-10-30 01:41:08.76882639 +0000 UTC m=+1394.345251836" Oct 30 01:41:08.775700 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:08.775679 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:09.742636 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:09.742603 6572 generic.go:356] "Generic (PLEG): container finished" podID="d8e808e8-7f20-4aa5-94aa-5a29f952d269" containerID="237e7c18b521415c115447c553699fba789fbbabd9ade9a7938126921c65ca9f" exitCode=0 Oct 30 01:41:09.743011 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:09.742660 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc51e9-ps5m4" event={"ID":"d8e808e8-7f20-4aa5-94aa-5a29f952d269","Type":"ContainerDied","Data":"237e7c18b521415c115447c553699fba789fbbabd9ade9a7938126921c65ca9f"} Oct 30 01:41:09.837199 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:09.837162 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:10.924093 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:10.924068 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc51e9-ps5m4" Oct 30 01:41:10.986252 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:10.986211 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8e808e8-7f20-4aa5-94aa-5a29f952d269-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8e808e8-7f20-4aa5-94aa-5a29f952d269-maas-api-openshift-ca\") pod \"d8e808e8-7f20-4aa5-94aa-5a29f952d269\" (UID: \"d8e808e8-7f20-4aa5-94aa-5a29f952d269\") " Oct 30 01:41:10.986826 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:10.986283 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8e808e8-7f20-4aa5-94aa-5a29f952d269-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8e808e8-7f20-4aa5-94aa-5a29f952d269-maas-api-xks-ca\") pod \"d8e808e8-7f20-4aa5-94aa-5a29f952d269\" (UID: \"d8e808e8-7f20-4aa5-94aa-5a29f952d269\") " Oct 30 01:41:10.986826 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:10.986641 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8e808e8-7f20-4aa5-94aa-5a29f952d269-kube-api-access-rlwx5\" (UniqueName: \"kubernetes.io/projected/d8e808e8-7f20-4aa5-94aa-5a29f952d269-kube-api-access-rlwx5\") pod \"d8e808e8-7f20-4aa5-94aa-5a29f952d269\" (UID: \"d8e808e8-7f20-4aa5-94aa-5a29f952d269\") " Oct 30 01:41:10.987017 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:10.986941 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8e808e8-7f20-4aa5-94aa-5a29f952d269-maas-api-openshift-ca" pod "d8e808e8-7f20-4aa5-94aa-5a29f952d269" (UID: "d8e808e8-7f20-4aa5-94aa-5a29f952d269"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:41:10.987632 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:10.987605 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8e808e8-7f20-4aa5-94aa-5a29f952d269-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:41:10.991032 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:10.989088 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8e808e8-7f20-4aa5-94aa-5a29f952d269-kube-api-access-rlwx5" pod "d8e808e8-7f20-4aa5-94aa-5a29f952d269" (UID: "d8e808e8-7f20-4aa5-94aa-5a29f952d269"). InnerVolumeSpecName "kube-api-access-rlwx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:41:10.991493 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:10.991455 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8e808e8-7f20-4aa5-94aa-5a29f952d269-maas-api-xks-ca" pod "d8e808e8-7f20-4aa5-94aa-5a29f952d269" (UID: "d8e808e8-7f20-4aa5-94aa-5a29f952d269"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:41:11.089276 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:11.089233 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlwx5\" (UniqueName: \"kubernetes.io/projected/d8e808e8-7f20-4aa5-94aa-5a29f952d269-kube-api-access-rlwx5\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:41:11.089276 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:11.089272 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8e808e8-7f20-4aa5-94aa-5a29f952d269-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:41:11.642923 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:11.642887 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:11.761505 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:11.761472 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc51e9-ps5m4" Oct 30 01:41:11.761505 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:11.761481 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc51e9-ps5m4" event={"ID":"d8e808e8-7f20-4aa5-94aa-5a29f952d269","Type":"ContainerDied","Data":"919e146cdea33c0d90051e1f9ddf97e727926e54a64e5546ac9c57190b1a09d7"} Oct 30 01:41:11.761691 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:11.761540 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="919e146cdea33c0d90051e1f9ddf97e727926e54a64e5546ac9c57190b1a09d7" Oct 30 01:41:13.921971 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:13.921930 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:14.686793 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:41:14.686732 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:41:15.754256 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:15.754207 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:16.307173 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:16.307138 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-5cf4874fcf-4594l"] Oct 30 01:41:16.515298 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:16.515263 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:16.713126 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:16.713094 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:16.917565 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:16.917507 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:18.047500 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:18.047465 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:20.040908 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.040873 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:20.056066 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.056039 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-84d4cc64b-bdmc7"] Oct 30 01:41:20.056293 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.056251 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-84d4cc64b-bdmc7" podUID="39556bd2-cdd6-4875-b55f-771f7f60195f" containerName="payload-processing" containerID="cri-o://92ebf9229257ffdc759f0be6210d33c36a523f97ef4289b63be45168009cc866" gracePeriod=30 Oct 30 01:41:20.089082 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.089045 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-5c6888c49f-sm8xt"] Oct 30 01:41:20.089305 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.089280 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-5c6888c49f-sm8xt" podUID="918a8572-3c60-4673-94a6-6d26854cb598" containerName="payload-pre-processing" containerID="cri-o://9f258f6030c49cfa97d6a3c0d6df1b50833ea4e2cafd30ccaca7d188ca13d309" gracePeriod=30 Oct 30 01:41:20.253112 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.253061 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:20.400291 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.400267 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-84d4cc64b-bdmc7" Oct 30 01:41:20.429533 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.429511 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-5c6888c49f-sm8xt" Oct 30 01:41:20.445953 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.445928 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:20.517169 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.517139 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-cbfd98f6-9nnfl"] Oct 30 01:41:20.517355 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.517336 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-cbfd98f6-9nnfl" podUID="85fef8a6-8af8-46e7-b994-475da53e403b" containerName="payload-processing" containerID="cri-o://29471aa1d401da2551b86bb256ba879ba7c43be1fd062a5c986a7b479a5edc5b" gracePeriod=30 Oct 30 01:41:20.519467 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.519147 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39556bd2-cdd6-4875-b55f-771f7f60195f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39556bd2-cdd6-4875-b55f-771f7f60195f-plugins-config-volume\") pod \"39556bd2-cdd6-4875-b55f-771f7f60195f\" (UID: \"39556bd2-cdd6-4875-b55f-771f7f60195f\") " Oct 30 01:41:20.519467 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.519189 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39556bd2-cdd6-4875-b55f-771f7f60195f-kube-api-access-g45dl\" (UniqueName: \"kubernetes.io/projected/39556bd2-cdd6-4875-b55f-771f7f60195f-kube-api-access-g45dl\") pod \"39556bd2-cdd6-4875-b55f-771f7f60195f\" (UID: \"39556bd2-cdd6-4875-b55f-771f7f60195f\") " Oct 30 01:41:20.519467 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.519284 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/918a8572-3c60-4673-94a6-6d26854cb598-kube-api-access-xbbct\" (UniqueName: \"kubernetes.io/projected/918a8572-3c60-4673-94a6-6d26854cb598-kube-api-access-xbbct\") pod \"918a8572-3c60-4673-94a6-6d26854cb598\" (UID: \"918a8572-3c60-4673-94a6-6d26854cb598\") " Oct 30 01:41:20.519467 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.519342 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/918a8572-3c60-4673-94a6-6d26854cb598-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918a8572-3c60-4673-94a6-6d26854cb598-plugins-config-volume\") pod \"918a8572-3c60-4673-94a6-6d26854cb598\" (UID: \"918a8572-3c60-4673-94a6-6d26854cb598\") " Oct 30 01:41:20.519849 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.519826 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/918a8572-3c60-4673-94a6-6d26854cb598-plugins-config-volume" pod "918a8572-3c60-4673-94a6-6d26854cb598" (UID: "918a8572-3c60-4673-94a6-6d26854cb598"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:41:20.520063 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.520047 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918a8572-3c60-4673-94a6-6d26854cb598-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:41:20.520148 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.520063 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39556bd2-cdd6-4875-b55f-771f7f60195f-plugins-config-volume" pod "39556bd2-cdd6-4875-b55f-771f7f60195f" (UID: "39556bd2-cdd6-4875-b55f-771f7f60195f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:41:20.521944 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.521917 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39556bd2-cdd6-4875-b55f-771f7f60195f-kube-api-access-g45dl" pod "39556bd2-cdd6-4875-b55f-771f7f60195f" (UID: "39556bd2-cdd6-4875-b55f-771f7f60195f"). InnerVolumeSpecName "kube-api-access-g45dl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:41:20.522042 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.521980 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/918a8572-3c60-4673-94a6-6d26854cb598-kube-api-access-xbbct" pod "918a8572-3c60-4673-94a6-6d26854cb598" (UID: "918a8572-3c60-4673-94a6-6d26854cb598"). InnerVolumeSpecName "kube-api-access-xbbct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:41:20.544692 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.544668 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-96888c8c9-wzmnm"] Oct 30 01:41:20.545061 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.545032 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-96888c8c9-wzmnm" podUID="5349aa56-fc17-4286-9390-3a68ff65542f" containerName="payload-pre-processing" containerID="cri-o://4688126a5999c457b849494fb010714e75976499dcf538eb0dfc55f6e9633d0d" gracePeriod=30 Oct 30 01:41:20.621130 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.621055 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39556bd2-cdd6-4875-b55f-771f7f60195f-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:41:20.621130 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.621088 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g45dl\" (UniqueName: \"kubernetes.io/projected/39556bd2-cdd6-4875-b55f-771f7f60195f-kube-api-access-g45dl\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:41:20.621130 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.621099 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbbct\" (UniqueName: \"kubernetes.io/projected/918a8572-3c60-4673-94a6-6d26854cb598-kube-api-access-xbbct\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:41:20.638606 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.638579 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:20.683845 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.683785 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc51e9-ps5m4"] Oct 30 01:41:20.687550 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.687531 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fc51e9-ps5m4"] Oct 30 01:41:20.815379 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.815353 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-cbfd98f6-9nnfl" Oct 30 01:41:20.826490 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.826453 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:20.832270 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.832244 6572 generic.go:356] "Generic (PLEG): container finished" podID="85fef8a6-8af8-46e7-b994-475da53e403b" containerID="29471aa1d401da2551b86bb256ba879ba7c43be1fd062a5c986a7b479a5edc5b" exitCode=0 Oct 30 01:41:20.832400 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.832284 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-cbfd98f6-9nnfl" Oct 30 01:41:20.832400 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.832322 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-cbfd98f6-9nnfl" event={"ID":"85fef8a6-8af8-46e7-b994-475da53e403b","Type":"ContainerDied","Data":"29471aa1d401da2551b86bb256ba879ba7c43be1fd062a5c986a7b479a5edc5b"} Oct 30 01:41:20.832400 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.832379 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-cbfd98f6-9nnfl" event={"ID":"85fef8a6-8af8-46e7-b994-475da53e403b","Type":"ContainerDied","Data":"c50b39d2338178382b0ebfc465f5627b4b1e1d0859d66441a86a6032daa75e5a"} Oct 30 01:41:20.832526 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.832410 6572 scope.go:122] "RemoveContainer" containerID="29471aa1d401da2551b86bb256ba879ba7c43be1fd062a5c986a7b479a5edc5b" Oct 30 01:41:20.834400 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.834380 6572 generic.go:356] "Generic (PLEG): container finished" podID="5349aa56-fc17-4286-9390-3a68ff65542f" containerID="4688126a5999c457b849494fb010714e75976499dcf538eb0dfc55f6e9633d0d" exitCode=0 Oct 30 01:41:20.834525 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.834458 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-96888c8c9-wzmnm" event={"ID":"5349aa56-fc17-4286-9390-3a68ff65542f","Type":"ContainerDied","Data":"4688126a5999c457b849494fb010714e75976499dcf538eb0dfc55f6e9633d0d"} Oct 30 01:41:20.836123 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.836099 6572 generic.go:356] "Generic (PLEG): container finished" podID="918a8572-3c60-4673-94a6-6d26854cb598" containerID="9f258f6030c49cfa97d6a3c0d6df1b50833ea4e2cafd30ccaca7d188ca13d309" exitCode=0 Oct 30 01:41:20.836235 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.836134 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-5c6888c49f-sm8xt" Oct 30 01:41:20.836235 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.836190 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-5c6888c49f-sm8xt" event={"ID":"918a8572-3c60-4673-94a6-6d26854cb598","Type":"ContainerDied","Data":"9f258f6030c49cfa97d6a3c0d6df1b50833ea4e2cafd30ccaca7d188ca13d309"} Oct 30 01:41:20.836311 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.836253 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-5c6888c49f-sm8xt" event={"ID":"918a8572-3c60-4673-94a6-6d26854cb598","Type":"ContainerDied","Data":"78a1b4b6b0488adc3ae877b24a22c225280c7f91eec60e6364fbc2b46fe0b39d"} Oct 30 01:41:20.837814 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.837792 6572 generic.go:356] "Generic (PLEG): container finished" podID="39556bd2-cdd6-4875-b55f-771f7f60195f" containerID="92ebf9229257ffdc759f0be6210d33c36a523f97ef4289b63be45168009cc866" exitCode=0 Oct 30 01:41:20.837953 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.837837 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-84d4cc64b-bdmc7" Oct 30 01:41:20.837953 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.837886 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-84d4cc64b-bdmc7" event={"ID":"39556bd2-cdd6-4875-b55f-771f7f60195f","Type":"ContainerDied","Data":"92ebf9229257ffdc759f0be6210d33c36a523f97ef4289b63be45168009cc866"} Oct 30 01:41:20.838116 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.837948 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-84d4cc64b-bdmc7" event={"ID":"39556bd2-cdd6-4875-b55f-771f7f60195f","Type":"ContainerDied","Data":"81e3e9a1ad7a08e5c194623640db7f7263c241d97ec711a16e424248ca004980"} Oct 30 01:41:20.846340 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.846318 6572 scope.go:122] "RemoveContainer" containerID="29471aa1d401da2551b86bb256ba879ba7c43be1fd062a5c986a7b479a5edc5b" Oct 30 01:41:20.846612 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:41:20.846589 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29471aa1d401da2551b86bb256ba879ba7c43be1fd062a5c986a7b479a5edc5b\": container with ID starting with 29471aa1d401da2551b86bb256ba879ba7c43be1fd062a5c986a7b479a5edc5b not found: ID does not exist" containerID="29471aa1d401da2551b86bb256ba879ba7c43be1fd062a5c986a7b479a5edc5b" Oct 30 01:41:20.846693 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.846617 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29471aa1d401da2551b86bb256ba879ba7c43be1fd062a5c986a7b479a5edc5b"} err="failed to get container status \"29471aa1d401da2551b86bb256ba879ba7c43be1fd062a5c986a7b479a5edc5b\": rpc error: code = NotFound desc = could not find container \"29471aa1d401da2551b86bb256ba879ba7c43be1fd062a5c986a7b479a5edc5b\": container with ID starting with 29471aa1d401da2551b86bb256ba879ba7c43be1fd062a5c986a7b479a5edc5b not found: ID does not exist" Oct 30 01:41:20.846693 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.846635 6572 scope.go:122] "RemoveContainer" containerID="9f258f6030c49cfa97d6a3c0d6df1b50833ea4e2cafd30ccaca7d188ca13d309" Oct 30 01:41:20.861023 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.861001 6572 scope.go:122] "RemoveContainer" containerID="9f258f6030c49cfa97d6a3c0d6df1b50833ea4e2cafd30ccaca7d188ca13d309" Oct 30 01:41:20.861354 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:41:20.861293 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f258f6030c49cfa97d6a3c0d6df1b50833ea4e2cafd30ccaca7d188ca13d309\": container with ID starting with 9f258f6030c49cfa97d6a3c0d6df1b50833ea4e2cafd30ccaca7d188ca13d309 not found: ID does not exist" containerID="9f258f6030c49cfa97d6a3c0d6df1b50833ea4e2cafd30ccaca7d188ca13d309" Oct 30 01:41:20.861448 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.861366 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f258f6030c49cfa97d6a3c0d6df1b50833ea4e2cafd30ccaca7d188ca13d309"} err="failed to get container status \"9f258f6030c49cfa97d6a3c0d6df1b50833ea4e2cafd30ccaca7d188ca13d309\": rpc error: code = NotFound desc = could not find container \"9f258f6030c49cfa97d6a3c0d6df1b50833ea4e2cafd30ccaca7d188ca13d309\": container with ID starting with 9f258f6030c49cfa97d6a3c0d6df1b50833ea4e2cafd30ccaca7d188ca13d309 not found: ID does not exist" Oct 30 01:41:20.861448 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.861389 6572 scope.go:122] "RemoveContainer" containerID="92ebf9229257ffdc759f0be6210d33c36a523f97ef4289b63be45168009cc866" Oct 30 01:41:20.863697 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.863676 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-96888c8c9-wzmnm" Oct 30 01:41:20.874091 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.874070 6572 scope.go:122] "RemoveContainer" containerID="92ebf9229257ffdc759f0be6210d33c36a523f97ef4289b63be45168009cc866" Oct 30 01:41:20.874358 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:41:20.874341 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92ebf9229257ffdc759f0be6210d33c36a523f97ef4289b63be45168009cc866\": container with ID starting with 92ebf9229257ffdc759f0be6210d33c36a523f97ef4289b63be45168009cc866 not found: ID does not exist" containerID="92ebf9229257ffdc759f0be6210d33c36a523f97ef4289b63be45168009cc866" Oct 30 01:41:20.874422 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.874367 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92ebf9229257ffdc759f0be6210d33c36a523f97ef4289b63be45168009cc866"} err="failed to get container status \"92ebf9229257ffdc759f0be6210d33c36a523f97ef4289b63be45168009cc866\": rpc error: code = NotFound desc = could not find container \"92ebf9229257ffdc759f0be6210d33c36a523f97ef4289b63be45168009cc866\": container with ID starting with 92ebf9229257ffdc759f0be6210d33c36a523f97ef4289b63be45168009cc866 not found: ID does not exist" Oct 30 01:41:20.904011 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.903975 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-84d4cc64b-bdmc7"] Oct 30 01:41:20.963885 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.939785 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5349aa56-fc17-4286-9390-3a68ff65542f-kube-api-access-n685q\" (UniqueName: \"kubernetes.io/projected/5349aa56-fc17-4286-9390-3a68ff65542f-kube-api-access-n685q\") pod \"5349aa56-fc17-4286-9390-3a68ff65542f\" (UID: \"5349aa56-fc17-4286-9390-3a68ff65542f\") " Oct 30 01:41:20.963885 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.939874 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85fef8a6-8af8-46e7-b994-475da53e403b-kube-api-access-dscrr\" (UniqueName: \"kubernetes.io/projected/85fef8a6-8af8-46e7-b994-475da53e403b-kube-api-access-dscrr\") pod \"85fef8a6-8af8-46e7-b994-475da53e403b\" (UID: \"85fef8a6-8af8-46e7-b994-475da53e403b\") " Oct 30 01:41:20.963885 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.940053 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85fef8a6-8af8-46e7-b994-475da53e403b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85fef8a6-8af8-46e7-b994-475da53e403b-plugins-config-volume\") pod \"85fef8a6-8af8-46e7-b994-475da53e403b\" (UID: \"85fef8a6-8af8-46e7-b994-475da53e403b\") " Oct 30 01:41:20.963885 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.940147 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5349aa56-fc17-4286-9390-3a68ff65542f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5349aa56-fc17-4286-9390-3a68ff65542f-plugins-config-volume\") pod \"5349aa56-fc17-4286-9390-3a68ff65542f\" (UID: \"5349aa56-fc17-4286-9390-3a68ff65542f\") " Oct 30 01:41:20.963885 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.954030 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5349aa56-fc17-4286-9390-3a68ff65542f-plugins-config-volume" pod "5349aa56-fc17-4286-9390-3a68ff65542f" (UID: "5349aa56-fc17-4286-9390-3a68ff65542f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:41:20.979617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.979559 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85fef8a6-8af8-46e7-b994-475da53e403b-plugins-config-volume" pod "85fef8a6-8af8-46e7-b994-475da53e403b" (UID: "85fef8a6-8af8-46e7-b994-475da53e403b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:41:20.980307 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.980236 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fc51e9-84d4cc64b-bdmc7"] Oct 30 01:41:21.002767 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.989808 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-5c6888c49f-sm8xt"] Oct 30 01:41:21.002767 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:20.995654 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fc51e9-5c6888c49f-sm8xt"] Oct 30 01:41:21.010766 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:21.005310 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5349aa56-fc17-4286-9390-3a68ff65542f-kube-api-access-n685q" pod "5349aa56-fc17-4286-9390-3a68ff65542f" (UID: "5349aa56-fc17-4286-9390-3a68ff65542f"). InnerVolumeSpecName "kube-api-access-n685q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:41:21.028771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:21.027925 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85fef8a6-8af8-46e7-b994-475da53e403b-kube-api-access-dscrr" pod "85fef8a6-8af8-46e7-b994-475da53e403b" (UID: "85fef8a6-8af8-46e7-b994-475da53e403b"). InnerVolumeSpecName "kube-api-access-dscrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:41:21.042412 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:21.042163 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dscrr\" (UniqueName: \"kubernetes.io/projected/85fef8a6-8af8-46e7-b994-475da53e403b-kube-api-access-dscrr\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:41:21.042412 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:21.042194 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85fef8a6-8af8-46e7-b994-475da53e403b-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:41:21.042412 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:21.042211 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5349aa56-fc17-4286-9390-3a68ff65542f-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:41:21.043059 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:21.042224 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n685q\" (UniqueName: \"kubernetes.io/projected/5349aa56-fc17-4286-9390-3a68ff65542f-kube-api-access-n685q\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:41:21.179499 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:21.179466 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-cbfd98f6-9nnfl"] Oct 30 01:41:21.183665 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:21.183638 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d9ede0-cbfd98f6-9nnfl"] Oct 30 01:41:21.268901 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:21.268866 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:21.308105 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:21.308051 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-5cf4874fcf-4594l" podUID="dd5dbedf-dafc-4737-804b-035d10ede5dd" containerName="payload-pre-processing" containerID="cri-o://945fc2f297e844381569ca2e03e7158ffe1c15c27ee8a62d351da631a48f5252" gracePeriod=25 Oct 30 01:41:21.326421 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:21.326196 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d9ede0-rj4zc"] Oct 30 01:41:21.339252 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:21.335720 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d9ede0-rj4zc"] Oct 30 01:41:21.467223 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:21.467182 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:21.635561 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:21.635531 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-5cf4874fcf-4594l" Oct 30 01:41:21.721499 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:21.721411 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:21.751985 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:21.751953 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd5dbedf-dafc-4737-804b-035d10ede5dd-kube-api-access-gh7ck\" (UniqueName: \"kubernetes.io/projected/dd5dbedf-dafc-4737-804b-035d10ede5dd-kube-api-access-gh7ck\") pod \"dd5dbedf-dafc-4737-804b-035d10ede5dd\" (UID: \"dd5dbedf-dafc-4737-804b-035d10ede5dd\") " Oct 30 01:41:21.752166 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:21.752031 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd5dbedf-dafc-4737-804b-035d10ede5dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd5dbedf-dafc-4737-804b-035d10ede5dd-plugins-config-volume\") pod \"dd5dbedf-dafc-4737-804b-035d10ede5dd\" (UID: \"dd5dbedf-dafc-4737-804b-035d10ede5dd\") " Oct 30 01:41:21.752463 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:21.752436 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd5dbedf-dafc-4737-804b-035d10ede5dd-plugins-config-volume" pod "dd5dbedf-dafc-4737-804b-035d10ede5dd" (UID: "dd5dbedf-dafc-4737-804b-035d10ede5dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:41:21.752702 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:21.752684 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd5dbedf-dafc-4737-804b-035d10ede5dd-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:41:21.754537 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:21.754499 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd5dbedf-dafc-4737-804b-035d10ede5dd-kube-api-access-gh7ck" pod "dd5dbedf-dafc-4737-804b-035d10ede5dd" (UID: "dd5dbedf-dafc-4737-804b-035d10ede5dd"). InnerVolumeSpecName "kube-api-access-gh7ck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:41:21.855037 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:21.854916 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh7ck\" (UniqueName: \"kubernetes.io/projected/dd5dbedf-dafc-4737-804b-035d10ede5dd-kube-api-access-gh7ck\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:41:21.867770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:21.866347 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-96888c8c9-wzmnm" event={"ID":"5349aa56-fc17-4286-9390-3a68ff65542f","Type":"ContainerDied","Data":"be2d51149dddce7fcd5a716b455e717849de2a5138e2dac4a5a896eeda75eb2f"} Oct 30 01:41:21.867770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:21.866417 6572 scope.go:122] "RemoveContainer" containerID="4688126a5999c457b849494fb010714e75976499dcf538eb0dfc55f6e9633d0d" Oct 30 01:41:21.867770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:21.866599 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-96888c8c9-wzmnm" Oct 30 01:41:21.879369 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:21.879339 6572 generic.go:356] "Generic (PLEG): container finished" podID="dd5dbedf-dafc-4737-804b-035d10ede5dd" containerID="945fc2f297e844381569ca2e03e7158ffe1c15c27ee8a62d351da631a48f5252" exitCode=0 Oct 30 01:41:21.879640 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:21.879619 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-5cf4874fcf-4594l" event={"ID":"dd5dbedf-dafc-4737-804b-035d10ede5dd","Type":"ContainerDied","Data":"945fc2f297e844381569ca2e03e7158ffe1c15c27ee8a62d351da631a48f5252"} Oct 30 01:41:21.879851 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:21.879827 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-5cf4874fcf-4594l" event={"ID":"dd5dbedf-dafc-4737-804b-035d10ede5dd","Type":"ContainerDied","Data":"b63d3c0bf773092590423d3c6856668766a7850260254a9c5c791cf2b8c0f9be"} Oct 30 01:41:21.880021 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:21.880008 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-5cf4874fcf-4594l" Oct 30 01:41:21.889763 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:21.889689 6572 scope.go:122] "RemoveContainer" containerID="945fc2f297e844381569ca2e03e7158ffe1c15c27ee8a62d351da631a48f5252" Oct 30 01:41:21.906362 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:21.906340 6572 scope.go:122] "RemoveContainer" containerID="945fc2f297e844381569ca2e03e7158ffe1c15c27ee8a62d351da631a48f5252" Oct 30 01:41:21.906723 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:41:21.906705 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"945fc2f297e844381569ca2e03e7158ffe1c15c27ee8a62d351da631a48f5252\": container with ID starting with 945fc2f297e844381569ca2e03e7158ffe1c15c27ee8a62d351da631a48f5252 not found: ID does not exist" containerID="945fc2f297e844381569ca2e03e7158ffe1c15c27ee8a62d351da631a48f5252" Oct 30 01:41:21.906819 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:21.906732 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"945fc2f297e844381569ca2e03e7158ffe1c15c27ee8a62d351da631a48f5252"} err="failed to get container status \"945fc2f297e844381569ca2e03e7158ffe1c15c27ee8a62d351da631a48f5252\": rpc error: code = NotFound desc = could not find container \"945fc2f297e844381569ca2e03e7158ffe1c15c27ee8a62d351da631a48f5252\": container with ID starting with 945fc2f297e844381569ca2e03e7158ffe1c15c27ee8a62d351da631a48f5252 not found: ID does not exist" Oct 30 01:41:21.924487 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:21.924464 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:21.937205 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:21.937180 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-5cf4874fcf-4594l"] Oct 30 01:41:21.948113 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:21.948078 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-5cf4874fcf-4594l"] Oct 30 01:41:21.975895 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:21.975833 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-96888c8c9-wzmnm"] Oct 30 01:41:21.982711 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:21.982688 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d9ede0-96888c8c9-wzmnm"] Oct 30 01:41:22.547855 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:22.547824 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39556bd2-cdd6-4875-b55f-771f7f60195f" path="/var/lib/kubelet/pods/39556bd2-cdd6-4875-b55f-771f7f60195f/volumes" Oct 30 01:41:22.548260 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:22.548246 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5349aa56-fc17-4286-9390-3a68ff65542f" path="/var/lib/kubelet/pods/5349aa56-fc17-4286-9390-3a68ff65542f/volumes" Oct 30 01:41:22.548655 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:22.548639 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ae2f8aa-149d-4f1d-953d-884ae912487a" path="/var/lib/kubelet/pods/6ae2f8aa-149d-4f1d-953d-884ae912487a/volumes" Oct 30 01:41:22.549346 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:22.549330 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85fef8a6-8af8-46e7-b994-475da53e403b" path="/var/lib/kubelet/pods/85fef8a6-8af8-46e7-b994-475da53e403b/volumes" Oct 30 01:41:22.549704 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:22.549690 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="918a8572-3c60-4673-94a6-6d26854cb598" path="/var/lib/kubelet/pods/918a8572-3c60-4673-94a6-6d26854cb598/volumes" Oct 30 01:41:22.550085 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:22.550072 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8e808e8-7f20-4aa5-94aa-5a29f952d269" path="/var/lib/kubelet/pods/d8e808e8-7f20-4aa5-94aa-5a29f952d269/volumes" Oct 30 01:41:22.550449 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:22.550434 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd5dbedf-dafc-4737-804b-035d10ede5dd" path="/var/lib/kubelet/pods/dd5dbedf-dafc-4737-804b-035d10ede5dd/volumes" Oct 30 01:41:23.894556 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:23.894522 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:25.664539 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:25.664506 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:27.361960 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:27.361924 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:28.074976 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:28.074930 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:28.280052 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:28.280012 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:28.485771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:28.485044 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:28.785488 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:28.785385 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:29.002790 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:28.998353 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:29.220818 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:29.218709 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:29.415933 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:29.415894 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:29.587332 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:29.587251 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8eb06-ldzz6"] Oct 30 01:41:29.587931 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:29.587915 6572 cpu_manager.go:414] "containerMap: removing container" podUID="d8e808e8-7f20-4aa5-94aa-5a29f952d269" containerName="revoke-keys" Oct 30 01:41:29.587982 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:29.587934 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8e808e8-7f20-4aa5-94aa-5a29f952d269" containerName="revoke-keys" Oct 30 01:41:29.587982 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:29.587945 6572 cpu_manager.go:414] "containerMap: removing container" podUID="dd5dbedf-dafc-4737-804b-035d10ede5dd" containerName="payload-pre-processing" Oct 30 01:41:29.587982 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:29.587952 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd5dbedf-dafc-4737-804b-035d10ede5dd" containerName="payload-pre-processing" Oct 30 01:41:29.587982 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:29.587962 6572 cpu_manager.go:414] "containerMap: removing container" podUID="39556bd2-cdd6-4875-b55f-771f7f60195f" containerName="payload-processing" Oct 30 01:41:29.587982 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:29.587967 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39556bd2-cdd6-4875-b55f-771f7f60195f" containerName="payload-processing" Oct 30 01:41:29.587982 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:29.587981 6572 cpu_manager.go:414] "containerMap: removing container" podUID="85fef8a6-8af8-46e7-b994-475da53e403b" containerName="payload-processing" Oct 30 01:41:29.588177 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:29.587986 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85fef8a6-8af8-46e7-b994-475da53e403b" containerName="payload-processing" Oct 30 01:41:29.588177 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:29.587998 6572 cpu_manager.go:414] "containerMap: removing container" podUID="5349aa56-fc17-4286-9390-3a68ff65542f" containerName="payload-pre-processing" Oct 30 01:41:29.588177 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:29.588003 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5349aa56-fc17-4286-9390-3a68ff65542f" containerName="payload-pre-processing" Oct 30 01:41:29.588177 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:29.588017 6572 cpu_manager.go:414] "containerMap: removing container" podUID="918a8572-3c60-4673-94a6-6d26854cb598" containerName="payload-pre-processing" Oct 30 01:41:29.588177 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:29.588022 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="918a8572-3c60-4673-94a6-6d26854cb598" containerName="payload-pre-processing" Oct 30 01:41:29.588177 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:29.588108 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8e808e8-7f20-4aa5-94aa-5a29f952d269" containerName="revoke-keys" Oct 30 01:41:29.588177 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:29.588118 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd5dbedf-dafc-4737-804b-035d10ede5dd" containerName="payload-pre-processing" Oct 30 01:41:29.588177 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:29.588124 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="39556bd2-cdd6-4875-b55f-771f7f60195f" containerName="payload-processing" Oct 30 01:41:29.588177 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:29.588132 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="85fef8a6-8af8-46e7-b994-475da53e403b" containerName="payload-processing" Oct 30 01:41:29.588177 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:29.588139 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="5349aa56-fc17-4286-9390-3a68ff65542f" containerName="payload-pre-processing" Oct 30 01:41:29.588177 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:29.588145 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="918a8572-3c60-4673-94a6-6d26854cb598" containerName="payload-pre-processing" Oct 30 01:41:29.593205 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:29.593184 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8eb06-ldzz6" Oct 30 01:41:29.612959 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:29.612925 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8eb06-ldzz6"] Oct 30 01:41:29.620176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:29.620149 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:41:29.620176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:29.620176 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:41:29.620482 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:29.620286 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:41:29.620482 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:29.620306 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:41:29.693598 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:29.693555 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dppdp\" (UniqueName: \"kubernetes.io/projected/2e41b81b-9c53-4789-89f0-b0b6ba165ffc-kube-api-access-dppdp\") pod \"maas-api-revoke-keys-e2e-ipp-b-d8eb06-ldzz6\" (UID: \"2e41b81b-9c53-4789-89f0-b0b6ba165ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8eb06-ldzz6" Oct 30 01:41:29.693842 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:29.693606 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e41b81b-9c53-4789-89f0-b0b6ba165ffc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d8eb06-ldzz6\" (UID: \"2e41b81b-9c53-4789-89f0-b0b6ba165ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8eb06-ldzz6" Oct 30 01:41:29.693897 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:29.693857 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e41b81b-9c53-4789-89f0-b0b6ba165ffc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d8eb06-ldzz6\" (UID: \"2e41b81b-9c53-4789-89f0-b0b6ba165ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8eb06-ldzz6" Oct 30 01:41:29.795048 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:29.795016 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e41b81b-9c53-4789-89f0-b0b6ba165ffc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d8eb06-ldzz6\" (UID: \"2e41b81b-9c53-4789-89f0-b0b6ba165ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8eb06-ldzz6" Oct 30 01:41:29.795223 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:29.795116 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dppdp\" (UniqueName: \"kubernetes.io/projected/2e41b81b-9c53-4789-89f0-b0b6ba165ffc-kube-api-access-dppdp\") pod \"maas-api-revoke-keys-e2e-ipp-b-d8eb06-ldzz6\" (UID: \"2e41b81b-9c53-4789-89f0-b0b6ba165ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8eb06-ldzz6" Oct 30 01:41:29.795223 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:29.795145 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e41b81b-9c53-4789-89f0-b0b6ba165ffc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d8eb06-ldzz6\" (UID: \"2e41b81b-9c53-4789-89f0-b0b6ba165ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8eb06-ldzz6" Oct 30 01:41:29.795882 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:29.795859 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e41b81b-9c53-4789-89f0-b0b6ba165ffc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d8eb06-ldzz6\" (UID: \"2e41b81b-9c53-4789-89f0-b0b6ba165ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8eb06-ldzz6" Oct 30 01:41:29.797486 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:29.797467 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e41b81b-9c53-4789-89f0-b0b6ba165ffc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d8eb06-ldzz6\" (UID: \"2e41b81b-9c53-4789-89f0-b0b6ba165ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8eb06-ldzz6" Oct 30 01:41:29.814081 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:29.814054 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dppdp\" (UniqueName: \"kubernetes.io/projected/2e41b81b-9c53-4789-89f0-b0b6ba165ffc-kube-api-access-dppdp\") pod \"maas-api-revoke-keys-e2e-ipp-b-d8eb06-ldzz6\" (UID: \"2e41b81b-9c53-4789-89f0-b0b6ba165ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8eb06-ldzz6" Oct 30 01:41:29.906978 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:29.906887 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8eb06-ldzz6" Oct 30 01:41:30.010858 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.010482 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:30.219841 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.219816 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8eb06-ldzz6"] Oct 30 01:41:30.221246 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:41:30.221220 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e41b81b_9c53_4789_89f0_b0b6ba165ffc.slice/crio-7f7196501f63f21e1dff15d9abc4f2e9c0328f3604bdf0fbd596a28f6bf2af6c WatchSource:0}: Error finding container 7f7196501f63f21e1dff15d9abc4f2e9c0328f3604bdf0fbd596a28f6bf2af6c: Status 404 returned error can't find the container with id 7f7196501f63f21e1dff15d9abc4f2e9c0328f3604bdf0fbd596a28f6bf2af6c Oct 30 01:41:30.312953 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.312911 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb"] Oct 30 01:41:30.319178 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.319152 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:41:30.342576 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.342248 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-3fde1c-openshift-default-dockercfg-kdj69\"" Oct 30 01:41:30.345018 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.344990 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb"] Oct 30 01:41:30.403545 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.403505 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-847q6\" (UniqueName: \"kubernetes.io/projected/05ffea96-f1dc-407d-ad3e-df11499478ce-kube-api-access-847q6\") pod \"e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb\" (UID: \"05ffea96-f1dc-407d-ad3e-df11499478ce\") " pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:41:30.403743 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.403569 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05ffea96-f1dc-407d-ad3e-df11499478ce-istio-envoy\") pod \"e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb\" (UID: \"05ffea96-f1dc-407d-ad3e-df11499478ce\") " pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:41:30.403743 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.403663 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05ffea96-f1dc-407d-ad3e-df11499478ce-istio-token\") pod \"e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb\" (UID: \"05ffea96-f1dc-407d-ad3e-df11499478ce\") " pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:41:30.403743 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.403693 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ffea96-f1dc-407d-ad3e-df11499478ce-workload-socket\") pod \"e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb\" (UID: \"05ffea96-f1dc-407d-ad3e-df11499478ce\") " pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:41:30.403953 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.403827 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05ffea96-f1dc-407d-ad3e-df11499478ce-workload-certs\") pod \"e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb\" (UID: \"05ffea96-f1dc-407d-ad3e-df11499478ce\") " pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:41:30.403953 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.403857 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05ffea96-f1dc-407d-ad3e-df11499478ce-istiod-ca-cert\") pod \"e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb\" (UID: \"05ffea96-f1dc-407d-ad3e-df11499478ce\") " pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:41:30.403953 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.403881 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05ffea96-f1dc-407d-ad3e-df11499478ce-istio-data\") pod \"e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb\" (UID: \"05ffea96-f1dc-407d-ad3e-df11499478ce\") " pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:41:30.404110 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.403981 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05ffea96-f1dc-407d-ad3e-df11499478ce-istio-podinfo\") pod \"e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb\" (UID: \"05ffea96-f1dc-407d-ad3e-df11499478ce\") " pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:41:30.404110 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.404013 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ffea96-f1dc-407d-ad3e-df11499478ce-credential-socket\") pod \"e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb\" (UID: \"05ffea96-f1dc-407d-ad3e-df11499478ce\") " pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:41:30.488837 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.479985 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:30.513780 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.511971 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05ffea96-f1dc-407d-ad3e-df11499478ce-istio-envoy\") pod \"e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb\" (UID: \"05ffea96-f1dc-407d-ad3e-df11499478ce\") " pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:41:30.513780 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.512074 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05ffea96-f1dc-407d-ad3e-df11499478ce-istio-token\") pod \"e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb\" (UID: \"05ffea96-f1dc-407d-ad3e-df11499478ce\") " pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:41:30.513780 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.512116 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ffea96-f1dc-407d-ad3e-df11499478ce-workload-socket\") pod \"e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb\" (UID: \"05ffea96-f1dc-407d-ad3e-df11499478ce\") " pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:41:30.513780 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.512269 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05ffea96-f1dc-407d-ad3e-df11499478ce-workload-certs\") pod \"e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb\" (UID: \"05ffea96-f1dc-407d-ad3e-df11499478ce\") " pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:41:30.513780 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.512327 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05ffea96-f1dc-407d-ad3e-df11499478ce-istiod-ca-cert\") pod \"e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb\" (UID: \"05ffea96-f1dc-407d-ad3e-df11499478ce\") " pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:41:30.513780 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.512380 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05ffea96-f1dc-407d-ad3e-df11499478ce-istio-data\") pod \"e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb\" (UID: \"05ffea96-f1dc-407d-ad3e-df11499478ce\") " pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:41:30.513780 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.512432 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05ffea96-f1dc-407d-ad3e-df11499478ce-istio-podinfo\") pod \"e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb\" (UID: \"05ffea96-f1dc-407d-ad3e-df11499478ce\") " pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:41:30.513780 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.512493 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ffea96-f1dc-407d-ad3e-df11499478ce-credential-socket\") pod \"e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb\" (UID: \"05ffea96-f1dc-407d-ad3e-df11499478ce\") " pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:41:30.513780 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.512586 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-847q6\" (UniqueName: \"kubernetes.io/projected/05ffea96-f1dc-407d-ad3e-df11499478ce-kube-api-access-847q6\") pod \"e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb\" (UID: \"05ffea96-f1dc-407d-ad3e-df11499478ce\") " pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:41:30.514422 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.514101 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05ffea96-f1dc-407d-ad3e-df11499478ce-istiod-ca-cert\") pod \"e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb\" (UID: \"05ffea96-f1dc-407d-ad3e-df11499478ce\") " pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:41:30.519779 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.514594 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ffea96-f1dc-407d-ad3e-df11499478ce-workload-socket\") pod \"e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb\" (UID: \"05ffea96-f1dc-407d-ad3e-df11499478ce\") " pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:41:30.519779 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.514852 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05ffea96-f1dc-407d-ad3e-df11499478ce-workload-certs\") pod \"e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb\" (UID: \"05ffea96-f1dc-407d-ad3e-df11499478ce\") " pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:41:30.519779 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.515437 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05ffea96-f1dc-407d-ad3e-df11499478ce-istio-data\") pod \"e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb\" (UID: \"05ffea96-f1dc-407d-ad3e-df11499478ce\") " pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:41:30.519779 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.515660 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ffea96-f1dc-407d-ad3e-df11499478ce-credential-socket\") pod \"e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb\" (UID: \"05ffea96-f1dc-407d-ad3e-df11499478ce\") " pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:41:30.542827 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.532094 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05ffea96-f1dc-407d-ad3e-df11499478ce-istio-envoy\") pod \"e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb\" (UID: \"05ffea96-f1dc-407d-ad3e-df11499478ce\") " pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:41:30.548508 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.548475 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05ffea96-f1dc-407d-ad3e-df11499478ce-istio-token\") pod \"e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb\" (UID: \"05ffea96-f1dc-407d-ad3e-df11499478ce\") " pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:41:30.557977 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.554607 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05ffea96-f1dc-407d-ad3e-df11499478ce-istio-podinfo\") pod \"e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb\" (UID: \"05ffea96-f1dc-407d-ad3e-df11499478ce\") " pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:41:30.558563 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.558533 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-847q6\" (UniqueName: \"kubernetes.io/projected/05ffea96-f1dc-407d-ad3e-df11499478ce-kube-api-access-847q6\") pod \"e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb\" (UID: \"05ffea96-f1dc-407d-ad3e-df11499478ce\") " pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:41:30.643792 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.643733 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:41:30.673167 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.673076 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:30.961209 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.961182 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb"] Oct 30 01:41:30.962016 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:41:30.961981 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05ffea96_f1dc_407d_ad3e_df11499478ce.slice/crio-69a7ffa663f7caed4d64eabff9c01305d2cf33b6a54c25db5b80498b809fb02e WatchSource:0}: Error finding container 69a7ffa663f7caed4d64eabff9c01305d2cf33b6a54c25db5b80498b809fb02e: Status 404 returned error can't find the container with id 69a7ffa663f7caed4d64eabff9c01305d2cf33b6a54c25db5b80498b809fb02e Oct 30 01:41:30.962095 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.962015 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8eb06-ldzz6" event={"ID":"2e41b81b-9c53-4789-89f0-b0b6ba165ffc","Type":"ContainerStarted","Data":"f225611cfa0dfdd95882eb70ebd04f8b7025ef330a9f62674862f0c64072bf81"} Oct 30 01:41:30.962095 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.962066 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8eb06-ldzz6" event={"ID":"2e41b81b-9c53-4789-89f0-b0b6ba165ffc","Type":"ContainerStarted","Data":"7f7196501f63f21e1dff15d9abc4f2e9c0328f3604bdf0fbd596a28f6bf2af6c"} Oct 30 01:41:30.965761 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.965713 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:41:30.965855 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.965816 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:41:30.965895 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:30.965861 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:41:31.001400 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:31.001345 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8eb06-ldzz6" podStartSLOduration=2.001326372 podStartE2EDuration="2.001326372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:41:31.000040556 +0000 UTC m=+1416.576465979" watchObservedRunningTime="2026-10-30 01:41:31.001326372 +0000 UTC m=+1416.577751776" Oct 30 01:41:31.133204 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:31.133145 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:31.973002 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:31.972969 6572 generic.go:356] "Generic (PLEG): container finished" podID="2e41b81b-9c53-4789-89f0-b0b6ba165ffc" containerID="f225611cfa0dfdd95882eb70ebd04f8b7025ef330a9f62674862f0c64072bf81" exitCode=0 Oct 30 01:41:31.973514 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:31.973061 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8eb06-ldzz6" event={"ID":"2e41b81b-9c53-4789-89f0-b0b6ba165ffc","Type":"ContainerDied","Data":"f225611cfa0dfdd95882eb70ebd04f8b7025ef330a9f62674862f0c64072bf81"} Oct 30 01:41:31.975016 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:31.974994 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" event={"ID":"05ffea96-f1dc-407d-ad3e-df11499478ce","Type":"ContainerStarted","Data":"4edac2164303c8213d07e4a4d7a3e68dd8017177ee7776fd842bdc85561ff29d"} Oct 30 01:41:31.975181 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:31.975038 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" event={"ID":"05ffea96-f1dc-407d-ad3e-df11499478ce","Type":"ContainerStarted","Data":"69a7ffa663f7caed4d64eabff9c01305d2cf33b6a54c25db5b80498b809fb02e"} Oct 30 01:41:32.055935 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:32.055884 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" podStartSLOduration=2.055867788 podStartE2EDuration="2.055867788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:41:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:41:32.052082566 +0000 UTC m=+1417.628507967" watchObservedRunningTime="2026-10-30 01:41:32.055867788 +0000 UTC m=+1417.632293189" Oct 30 01:41:32.644198 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:32.644157 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:41:32.650609 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:32.650581 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:41:32.988601 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:32.988556 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:41:32.993508 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:32.993481 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:41:33.006794 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:33.006089 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:33.162542 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:33.162517 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8eb06-ldzz6" Oct 30 01:41:33.249467 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:33.249376 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e41b81b-9c53-4789-89f0-b0b6ba165ffc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e41b81b-9c53-4789-89f0-b0b6ba165ffc-maas-api-openshift-ca\") pod \"2e41b81b-9c53-4789-89f0-b0b6ba165ffc\" (UID: \"2e41b81b-9c53-4789-89f0-b0b6ba165ffc\") " Oct 30 01:41:33.249634 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:33.249495 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e41b81b-9c53-4789-89f0-b0b6ba165ffc-kube-api-access-dppdp\" (UniqueName: \"kubernetes.io/projected/2e41b81b-9c53-4789-89f0-b0b6ba165ffc-kube-api-access-dppdp\") pod \"2e41b81b-9c53-4789-89f0-b0b6ba165ffc\" (UID: \"2e41b81b-9c53-4789-89f0-b0b6ba165ffc\") " Oct 30 01:41:33.249634 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:33.249530 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e41b81b-9c53-4789-89f0-b0b6ba165ffc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e41b81b-9c53-4789-89f0-b0b6ba165ffc-maas-api-xks-ca\") pod \"2e41b81b-9c53-4789-89f0-b0b6ba165ffc\" (UID: \"2e41b81b-9c53-4789-89f0-b0b6ba165ffc\") " Oct 30 01:41:33.249845 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:33.249726 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e41b81b-9c53-4789-89f0-b0b6ba165ffc-maas-api-openshift-ca" pod "2e41b81b-9c53-4789-89f0-b0b6ba165ffc" (UID: "2e41b81b-9c53-4789-89f0-b0b6ba165ffc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:41:33.250194 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:33.250174 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e41b81b-9c53-4789-89f0-b0b6ba165ffc-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:41:33.251702 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:33.251679 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e41b81b-9c53-4789-89f0-b0b6ba165ffc-kube-api-access-dppdp" pod "2e41b81b-9c53-4789-89f0-b0b6ba165ffc" (UID: "2e41b81b-9c53-4789-89f0-b0b6ba165ffc"). InnerVolumeSpecName "kube-api-access-dppdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:41:33.251816 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:33.251738 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e41b81b-9c53-4789-89f0-b0b6ba165ffc-maas-api-xks-ca" pod "2e41b81b-9c53-4789-89f0-b0b6ba165ffc" (UID: "2e41b81b-9c53-4789-89f0-b0b6ba165ffc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:41:33.351405 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:33.351370 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dppdp\" (UniqueName: \"kubernetes.io/projected/2e41b81b-9c53-4789-89f0-b0b6ba165ffc-kube-api-access-dppdp\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:41:33.351405 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:33.351402 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e41b81b-9c53-4789-89f0-b0b6ba165ffc-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:41:33.999578 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:33.999546 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8eb06-ldzz6" Oct 30 01:41:34.000120 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:33.999573 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8eb06-ldzz6" event={"ID":"2e41b81b-9c53-4789-89f0-b0b6ba165ffc","Type":"ContainerDied","Data":"7f7196501f63f21e1dff15d9abc4f2e9c0328f3604bdf0fbd596a28f6bf2af6c"} Oct 30 01:41:34.000120 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:33.999633 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f7196501f63f21e1dff15d9abc4f2e9c0328f3604bdf0fbd596a28f6bf2af6c" Oct 30 01:41:34.572667 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:34.572634 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:34.818943 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:34.818908 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:35.658016 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:35.657985 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:36.267692 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:36.267654 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:36.289243 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:36.289207 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3fde1c-58cf94cd5b-nq9g2"] Oct 30 01:41:36.289965 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:36.289936 6572 cpu_manager.go:414] "containerMap: removing container" podUID="2e41b81b-9c53-4789-89f0-b0b6ba165ffc" containerName="revoke-keys" Oct 30 01:41:36.289965 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:36.289964 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e41b81b-9c53-4789-89f0-b0b6ba165ffc" containerName="revoke-keys" Oct 30 01:41:36.290145 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:36.290114 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e41b81b-9c53-4789-89f0-b0b6ba165ffc" containerName="revoke-keys" Oct 30 01:41:36.293933 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:36.293914 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3fde1c-58cf94cd5b-nq9g2" Oct 30 01:41:36.314076 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:36.314045 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3fde1c-58cf94cd5b-nq9g2"] Oct 30 01:41:36.321137 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:36.321103 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-3fde1c\"" Oct 30 01:41:36.388903 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:36.388862 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bb426f0-f37a-4660-819b-fa2bf8bb961c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3fde1c-58cf94cd5b-nq9g2\" (UID: \"3bb426f0-f37a-4660-819b-fa2bf8bb961c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3fde1c-58cf94cd5b-nq9g2" Oct 30 01:41:36.389170 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:36.389144 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24p2b\" (UniqueName: \"kubernetes.io/projected/3bb426f0-f37a-4660-819b-fa2bf8bb961c-kube-api-access-24p2b\") pod \"payload-processing-e2e-praxis-a-3fde1c-58cf94cd5b-nq9g2\" (UID: \"3bb426f0-f37a-4660-819b-fa2bf8bb961c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3fde1c-58cf94cd5b-nq9g2" Oct 30 01:41:36.475980 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:36.475881 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:36.518565 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:36.515850 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bb426f0-f37a-4660-819b-fa2bf8bb961c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3fde1c-58cf94cd5b-nq9g2\" (UID: \"3bb426f0-f37a-4660-819b-fa2bf8bb961c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3fde1c-58cf94cd5b-nq9g2" Oct 30 01:41:36.518565 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:36.516282 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bb426f0-f37a-4660-819b-fa2bf8bb961c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3fde1c-58cf94cd5b-nq9g2\" (UID: \"3bb426f0-f37a-4660-819b-fa2bf8bb961c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3fde1c-58cf94cd5b-nq9g2" Oct 30 01:41:36.518565 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:36.516684 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24p2b\" (UniqueName: \"kubernetes.io/projected/3bb426f0-f37a-4660-819b-fa2bf8bb961c-kube-api-access-24p2b\") pod \"payload-processing-e2e-praxis-a-3fde1c-58cf94cd5b-nq9g2\" (UID: \"3bb426f0-f37a-4660-819b-fa2bf8bb961c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3fde1c-58cf94cd5b-nq9g2" Oct 30 01:41:36.541170 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:36.541141 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24p2b\" (UniqueName: \"kubernetes.io/projected/3bb426f0-f37a-4660-819b-fa2bf8bb961c-kube-api-access-24p2b\") pod \"payload-processing-e2e-praxis-a-3fde1c-58cf94cd5b-nq9g2\" (UID: \"3bb426f0-f37a-4660-819b-fa2bf8bb961c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3fde1c-58cf94cd5b-nq9g2" Oct 30 01:41:36.608995 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:36.608963 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3fde1c-58cf94cd5b-nq9g2" Oct 30 01:41:36.920112 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:36.920059 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3fde1c-58cf94cd5b-nq9g2"] Oct 30 01:41:36.921306 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:41:36.921272 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb426f0_f37a_4660_819b_fa2bf8bb961c.slice/crio-4849bcf2d87067d596a2b6b2d0899ae33415425c8dece1e2c3d11a4ba39b3baf WatchSource:0}: Error finding container 4849bcf2d87067d596a2b6b2d0899ae33415425c8dece1e2c3d11a4ba39b3baf: Status 404 returned error can't find the container with id 4849bcf2d87067d596a2b6b2d0899ae33415425c8dece1e2c3d11a4ba39b3baf Oct 30 01:41:37.027842 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:37.027808 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3fde1c-58cf94cd5b-nq9g2" event={"ID":"3bb426f0-f37a-4660-819b-fa2bf8bb961c","Type":"ContainerStarted","Data":"43f2f6294ee8f63aafb414ba2b48c77d00788a2b44dbaae7ebc9c9b2caa641ac"} Oct 30 01:41:37.028042 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:37.027866 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3fde1c-58cf94cd5b-nq9g2" event={"ID":"3bb426f0-f37a-4660-819b-fa2bf8bb961c","Type":"ContainerStarted","Data":"4849bcf2d87067d596a2b6b2d0899ae33415425c8dece1e2c3d11a4ba39b3baf"} Oct 30 01:41:37.028042 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:37.028027 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-3fde1c-58cf94cd5b-nq9g2" Oct 30 01:41:37.088064 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:37.083627 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:37.126774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:37.119327 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-3fde1c-58cf94cd5b-nq9g2" podStartSLOduration=1.119303463 podStartE2EDuration="1.119303463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:41:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:41:37.06629005 +0000 UTC m=+1422.642715452" watchObservedRunningTime="2026-10-30 01:41:37.119303463 +0000 UTC m=+1422.695728866" Oct 30 01:41:37.277406 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:37.277306 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:37.377453 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:37.377408 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-c564cf647-d674t"] Oct 30 01:41:37.392782 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:37.392125 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-c564cf647-d674t" Oct 30 01:41:37.424372 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:37.422723 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-c564cf647-d674t"] Oct 30 01:41:37.429768 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:37.424627 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2ac18f\"" Oct 30 01:41:37.429768 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:37.424767 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2ac18f-dockercfg-mfq7h\"" Oct 30 01:41:37.488547 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:37.487844 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53166fd9-5f8c-45c4-a69a-c919a1bf0201-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2ac18f-c564cf647-d674t\" (UID: \"53166fd9-5f8c-45c4-a69a-c919a1bf0201\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-c564cf647-d674t" Oct 30 01:41:37.488547 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:37.488114 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kskbc\" (UniqueName: \"kubernetes.io/projected/53166fd9-5f8c-45c4-a69a-c919a1bf0201-kube-api-access-kskbc\") pod \"payload-pre-processing-e2e-ipp-a-2ac18f-c564cf647-d674t\" (UID: \"53166fd9-5f8c-45c4-a69a-c919a1bf0201\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-c564cf647-d674t" Oct 30 01:41:37.589433 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:37.589324 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kskbc\" (UniqueName: \"kubernetes.io/projected/53166fd9-5f8c-45c4-a69a-c919a1bf0201-kube-api-access-kskbc\") pod \"payload-pre-processing-e2e-ipp-a-2ac18f-c564cf647-d674t\" (UID: \"53166fd9-5f8c-45c4-a69a-c919a1bf0201\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-c564cf647-d674t" Oct 30 01:41:37.589629 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:37.589497 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53166fd9-5f8c-45c4-a69a-c919a1bf0201-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2ac18f-c564cf647-d674t\" (UID: \"53166fd9-5f8c-45c4-a69a-c919a1bf0201\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-c564cf647-d674t" Oct 30 01:41:37.590575 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:37.590543 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53166fd9-5f8c-45c4-a69a-c919a1bf0201-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2ac18f-c564cf647-d674t\" (UID: \"53166fd9-5f8c-45c4-a69a-c919a1bf0201\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-c564cf647-d674t" Oct 30 01:41:37.612482 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:37.612445 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kskbc\" (UniqueName: \"kubernetes.io/projected/53166fd9-5f8c-45c4-a69a-c919a1bf0201-kube-api-access-kskbc\") pod \"payload-pre-processing-e2e-ipp-a-2ac18f-c564cf647-d674t\" (UID: \"53166fd9-5f8c-45c4-a69a-c919a1bf0201\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-c564cf647-d674t" Oct 30 01:41:37.766783 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:37.766727 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-c564cf647-d674t" Oct 30 01:41:37.911496 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:37.911455 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:38.080043 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:38.080009 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-c564cf647-d674t"] Oct 30 01:41:38.080365 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:41:38.080345 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53166fd9_5f8c_45c4_a69a_c919a1bf0201.slice/crio-9d880ae945aa5e7e8058dcd67b8654ff3b6a05b012171c7c5fba37ad3c311ad2 WatchSource:0}: Error finding container 9d880ae945aa5e7e8058dcd67b8654ff3b6a05b012171c7c5fba37ad3c311ad2: Status 404 returned error can't find the container with id 9d880ae945aa5e7e8058dcd67b8654ff3b6a05b012171c7c5fba37ad3c311ad2 Oct 30 01:41:38.105865 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:38.105830 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:38.282864 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:38.282736 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:38.659493 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:38.659452 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:38.699239 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:38.699205 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7bdcc-dhxrh"] Oct 30 01:41:38.710311 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:38.710272 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7bdcc-dhxrh"] Oct 30 01:41:38.858429 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:38.857874 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:39.047926 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:39.047881 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-c564cf647-d674t" event={"ID":"53166fd9-5f8c-45c4-a69a-c919a1bf0201","Type":"ContainerStarted","Data":"52f9bef11d73f4d2db7e350b7aca7d80e9e776fc72c29db8a59c44e778d3124f"} Oct 30 01:41:39.048128 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:39.047951 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-c564cf647-d674t" event={"ID":"53166fd9-5f8c-45c4-a69a-c919a1bf0201","Type":"ContainerStarted","Data":"9d880ae945aa5e7e8058dcd67b8654ff3b6a05b012171c7c5fba37ad3c311ad2"} Oct 30 01:41:39.048128 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:39.047983 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-c564cf647-d674t" Oct 30 01:41:39.049890 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:39.049863 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:39.089846 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:39.089792 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-c564cf647-d674t" podStartSLOduration=2.089775832 podStartE2EDuration="2.089775832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:41:39.085538928 +0000 UTC m=+1424.661964324" watchObservedRunningTime="2026-10-30 01:41:39.089775832 +0000 UTC m=+1424.666201234" Oct 30 01:41:39.275085 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:39.275050 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:40.542959 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:40.542926 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f988585-a8a7-4316-b9fb-d40788522236" path="/var/lib/kubelet/pods/4f988585-a8a7-4316-b9fb-d40788522236/volumes" Oct 30 01:41:40.880554 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:40.880459 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:43.115694 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:43.115656 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:44.687974 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:41:44.687937 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:41:45.078869 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:45.078743 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:45.087801 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:45.087768 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-75f96f6674-gz2rx"] Oct 30 01:41:45.088018 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:45.087993 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-75f96f6674-gz2rx" podUID="84354b2e-c89c-486d-adb1-0204a5f120c1" containerName="payload-processing" containerID="cri-o://39c350de8c7a448567c5374af4b214773c0cd2f973801423c6a8a241998a4a1a" gracePeriod=30 Oct 30 01:41:45.109209 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:45.109134 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5b5b9f459-njx29"] Oct 30 01:41:45.111827 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:45.109377 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5b5b9f459-njx29" podUID="04eef61c-0c54-4a8a-b1b8-da3d30e4af77" containerName="payload-pre-processing" containerID="cri-o://8d1676e928ac97094c000512eab9d17606d4b8213770876cb4b6e71e46f946d8" gracePeriod=30 Oct 30 01:41:45.279863 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:45.279684 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:45.460161 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:45.460125 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-75f96f6674-gz2rx" Oct 30 01:41:45.463540 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:45.463511 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5b5b9f459-njx29" Oct 30 01:41:45.481549 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:45.481514 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:45.632242 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:45.632133 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04eef61c-0c54-4a8a-b1b8-da3d30e4af77-kube-api-access-2ddxk\" (UniqueName: \"kubernetes.io/projected/04eef61c-0c54-4a8a-b1b8-da3d30e4af77-kube-api-access-2ddxk\") pod \"04eef61c-0c54-4a8a-b1b8-da3d30e4af77\" (UID: \"04eef61c-0c54-4a8a-b1b8-da3d30e4af77\") " Oct 30 01:41:45.632469 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:45.632446 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04eef61c-0c54-4a8a-b1b8-da3d30e4af77-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04eef61c-0c54-4a8a-b1b8-da3d30e4af77-plugins-config-volume\") pod \"04eef61c-0c54-4a8a-b1b8-da3d30e4af77\" (UID: \"04eef61c-0c54-4a8a-b1b8-da3d30e4af77\") " Oct 30 01:41:45.632772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:45.632694 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84354b2e-c89c-486d-adb1-0204a5f120c1-kube-api-access-lwl7b\" (UniqueName: \"kubernetes.io/projected/84354b2e-c89c-486d-adb1-0204a5f120c1-kube-api-access-lwl7b\") pod \"84354b2e-c89c-486d-adb1-0204a5f120c1\" (UID: \"84354b2e-c89c-486d-adb1-0204a5f120c1\") " Oct 30 01:41:45.632934 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:45.632790 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84354b2e-c89c-486d-adb1-0204a5f120c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84354b2e-c89c-486d-adb1-0204a5f120c1-plugins-config-volume\") pod \"84354b2e-c89c-486d-adb1-0204a5f120c1\" (UID: \"84354b2e-c89c-486d-adb1-0204a5f120c1\") " Oct 30 01:41:45.633224 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:45.633195 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04eef61c-0c54-4a8a-b1b8-da3d30e4af77-plugins-config-volume" pod "04eef61c-0c54-4a8a-b1b8-da3d30e4af77" (UID: "04eef61c-0c54-4a8a-b1b8-da3d30e4af77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:41:45.633380 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:45.633355 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04eef61c-0c54-4a8a-b1b8-da3d30e4af77-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:41:45.634134 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:45.634110 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84354b2e-c89c-486d-adb1-0204a5f120c1-plugins-config-volume" pod "84354b2e-c89c-486d-adb1-0204a5f120c1" (UID: "84354b2e-c89c-486d-adb1-0204a5f120c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:41:45.637269 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:45.637238 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04eef61c-0c54-4a8a-b1b8-da3d30e4af77-kube-api-access-2ddxk" pod "04eef61c-0c54-4a8a-b1b8-da3d30e4af77" (UID: "04eef61c-0c54-4a8a-b1b8-da3d30e4af77"). InnerVolumeSpecName "kube-api-access-2ddxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:41:45.638405 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:45.638380 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84354b2e-c89c-486d-adb1-0204a5f120c1-kube-api-access-lwl7b" pod "84354b2e-c89c-486d-adb1-0204a5f120c1" (UID: "84354b2e-c89c-486d-adb1-0204a5f120c1"). InnerVolumeSpecName "kube-api-access-lwl7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:41:45.675385 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:45.675346 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:45.734546 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:45.734507 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwl7b\" (UniqueName: \"kubernetes.io/projected/84354b2e-c89c-486d-adb1-0204a5f120c1-kube-api-access-lwl7b\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:41:45.734546 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:45.734540 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84354b2e-c89c-486d-adb1-0204a5f120c1-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:41:45.734546 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:45.734549 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ddxk\" (UniqueName: \"kubernetes.io/projected/04eef61c-0c54-4a8a-b1b8-da3d30e4af77-kube-api-access-2ddxk\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:41:45.808584 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:45.808547 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8eb06-ldzz6"] Oct 30 01:41:45.813317 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:45.813290 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d8eb06-ldzz6"] Oct 30 01:41:45.866802 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:45.866759 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:45.984975 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:45.984944 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db534-g5gxs"] Oct 30 01:41:45.985556 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:45.985541 6572 cpu_manager.go:414] "containerMap: removing container" podUID="04eef61c-0c54-4a8a-b1b8-da3d30e4af77" containerName="payload-pre-processing" Oct 30 01:41:45.985603 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:45.985557 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04eef61c-0c54-4a8a-b1b8-da3d30e4af77" containerName="payload-pre-processing" Oct 30 01:41:45.985603 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:45.985581 6572 cpu_manager.go:414] "containerMap: removing container" podUID="84354b2e-c89c-486d-adb1-0204a5f120c1" containerName="payload-processing" Oct 30 01:41:45.985603 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:45.985587 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84354b2e-c89c-486d-adb1-0204a5f120c1" containerName="payload-processing" Oct 30 01:41:45.985719 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:45.985709 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="84354b2e-c89c-486d-adb1-0204a5f120c1" containerName="payload-processing" Oct 30 01:41:45.985776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:45.985729 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="04eef61c-0c54-4a8a-b1b8-da3d30e4af77" containerName="payload-pre-processing" Oct 30 01:41:45.989456 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:45.989437 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db534-g5gxs" Oct 30 01:41:46.003689 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.003661 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:41:46.003689 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.003661 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:41:46.007105 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.007080 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db534-g5gxs"] Oct 30 01:41:46.027941 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.027919 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:41:46.028063 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.028026 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:41:46.038809 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.038786 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c8c3eba-77d5-4ea2-814f-99a29cc9b07f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0db534-g5gxs\" (UID: \"3c8c3eba-77d5-4ea2-814f-99a29cc9b07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db534-g5gxs" Oct 30 01:41:46.038919 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.038840 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c8c3eba-77d5-4ea2-814f-99a29cc9b07f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0db534-g5gxs\" (UID: \"3c8c3eba-77d5-4ea2-814f-99a29cc9b07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db534-g5gxs" Oct 30 01:41:46.038965 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.038951 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp2vb\" (UniqueName: \"kubernetes.io/projected/3c8c3eba-77d5-4ea2-814f-99a29cc9b07f-kube-api-access-tp2vb\") pod \"maas-api-revoke-keys-e2e-ipp-b-0db534-g5gxs\" (UID: \"3c8c3eba-77d5-4ea2-814f-99a29cc9b07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db534-g5gxs" Oct 30 01:41:46.053012 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.052986 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:46.114630 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.114601 6572 generic.go:356] "Generic (PLEG): container finished" podID="04eef61c-0c54-4a8a-b1b8-da3d30e4af77" containerID="8d1676e928ac97094c000512eab9d17606d4b8213770876cb4b6e71e46f946d8" exitCode=0 Oct 30 01:41:46.114819 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.114650 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5b5b9f459-njx29" Oct 30 01:41:46.114819 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.114690 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5b5b9f459-njx29" event={"ID":"04eef61c-0c54-4a8a-b1b8-da3d30e4af77","Type":"ContainerDied","Data":"8d1676e928ac97094c000512eab9d17606d4b8213770876cb4b6e71e46f946d8"} Oct 30 01:41:46.114819 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.114785 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5b5b9f459-njx29" event={"ID":"04eef61c-0c54-4a8a-b1b8-da3d30e4af77","Type":"ContainerDied","Data":"28ed9e98d4d0a46643829764c66848d017acc18d3d9261a8eea5d5b2648f68b4"} Oct 30 01:41:46.114960 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.114833 6572 scope.go:122] "RemoveContainer" containerID="8d1676e928ac97094c000512eab9d17606d4b8213770876cb4b6e71e46f946d8" Oct 30 01:41:46.121646 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.121605 6572 generic.go:356] "Generic (PLEG): container finished" podID="84354b2e-c89c-486d-adb1-0204a5f120c1" containerID="39c350de8c7a448567c5374af4b214773c0cd2f973801423c6a8a241998a4a1a" exitCode=0 Oct 30 01:41:46.121831 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.121674 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-75f96f6674-gz2rx" Oct 30 01:41:46.121831 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.121701 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-75f96f6674-gz2rx" event={"ID":"84354b2e-c89c-486d-adb1-0204a5f120c1","Type":"ContainerDied","Data":"39c350de8c7a448567c5374af4b214773c0cd2f973801423c6a8a241998a4a1a"} Oct 30 01:41:46.121831 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.121794 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-75f96f6674-gz2rx" event={"ID":"84354b2e-c89c-486d-adb1-0204a5f120c1","Type":"ContainerDied","Data":"143611ba0e40a09f265a1886754c79c18a655a82fa6fc3b630546960c5eaac5c"} Oct 30 01:41:46.134104 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.134081 6572 scope.go:122] "RemoveContainer" containerID="8d1676e928ac97094c000512eab9d17606d4b8213770876cb4b6e71e46f946d8" Oct 30 01:41:46.134389 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:41:46.134369 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d1676e928ac97094c000512eab9d17606d4b8213770876cb4b6e71e46f946d8\": container with ID starting with 8d1676e928ac97094c000512eab9d17606d4b8213770876cb4b6e71e46f946d8 not found: ID does not exist" containerID="8d1676e928ac97094c000512eab9d17606d4b8213770876cb4b6e71e46f946d8" Oct 30 01:41:46.134505 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.134397 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d1676e928ac97094c000512eab9d17606d4b8213770876cb4b6e71e46f946d8"} err="failed to get container status \"8d1676e928ac97094c000512eab9d17606d4b8213770876cb4b6e71e46f946d8\": rpc error: code = NotFound desc = could not find container \"8d1676e928ac97094c000512eab9d17606d4b8213770876cb4b6e71e46f946d8\": container with ID starting with 8d1676e928ac97094c000512eab9d17606d4b8213770876cb4b6e71e46f946d8 not found: ID does not exist" Oct 30 01:41:46.134505 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.134414 6572 scope.go:122] "RemoveContainer" containerID="39c350de8c7a448567c5374af4b214773c0cd2f973801423c6a8a241998a4a1a" Oct 30 01:41:46.139905 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.139884 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c8c3eba-77d5-4ea2-814f-99a29cc9b07f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0db534-g5gxs\" (UID: \"3c8c3eba-77d5-4ea2-814f-99a29cc9b07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db534-g5gxs" Oct 30 01:41:46.140019 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.139929 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c8c3eba-77d5-4ea2-814f-99a29cc9b07f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0db534-g5gxs\" (UID: \"3c8c3eba-77d5-4ea2-814f-99a29cc9b07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db534-g5gxs" Oct 30 01:41:46.140019 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.139999 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp2vb\" (UniqueName: \"kubernetes.io/projected/3c8c3eba-77d5-4ea2-814f-99a29cc9b07f-kube-api-access-tp2vb\") pod \"maas-api-revoke-keys-e2e-ipp-b-0db534-g5gxs\" (UID: \"3c8c3eba-77d5-4ea2-814f-99a29cc9b07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db534-g5gxs" Oct 30 01:41:46.140513 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.140491 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c8c3eba-77d5-4ea2-814f-99a29cc9b07f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0db534-g5gxs\" (UID: \"3c8c3eba-77d5-4ea2-814f-99a29cc9b07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db534-g5gxs" Oct 30 01:41:46.142428 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.142405 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c8c3eba-77d5-4ea2-814f-99a29cc9b07f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0db534-g5gxs\" (UID: \"3c8c3eba-77d5-4ea2-814f-99a29cc9b07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db534-g5gxs" Oct 30 01:41:46.145054 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.145033 6572 scope.go:122] "RemoveContainer" containerID="39c350de8c7a448567c5374af4b214773c0cd2f973801423c6a8a241998a4a1a" Oct 30 01:41:46.145322 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:41:46.145299 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39c350de8c7a448567c5374af4b214773c0cd2f973801423c6a8a241998a4a1a\": container with ID starting with 39c350de8c7a448567c5374af4b214773c0cd2f973801423c6a8a241998a4a1a not found: ID does not exist" containerID="39c350de8c7a448567c5374af4b214773c0cd2f973801423c6a8a241998a4a1a" Oct 30 01:41:46.145395 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.145326 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39c350de8c7a448567c5374af4b214773c0cd2f973801423c6a8a241998a4a1a"} err="failed to get container status \"39c350de8c7a448567c5374af4b214773c0cd2f973801423c6a8a241998a4a1a\": rpc error: code = NotFound desc = could not find container \"39c350de8c7a448567c5374af4b214773c0cd2f973801423c6a8a241998a4a1a\": container with ID starting with 39c350de8c7a448567c5374af4b214773c0cd2f973801423c6a8a241998a4a1a not found: ID does not exist" Oct 30 01:41:46.158326 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.158306 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp2vb\" (UniqueName: \"kubernetes.io/projected/3c8c3eba-77d5-4ea2-814f-99a29cc9b07f-kube-api-access-tp2vb\") pod \"maas-api-revoke-keys-e2e-ipp-b-0db534-g5gxs\" (UID: \"3c8c3eba-77d5-4ea2-814f-99a29cc9b07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db534-g5gxs" Oct 30 01:41:46.165696 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.165670 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5b5b9f459-njx29"] Oct 30 01:41:46.169486 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.169463 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d8eb06-5b5b9f459-njx29"] Oct 30 01:41:46.215683 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.215640 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-75f96f6674-gz2rx"] Oct 30 01:41:46.218798 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.218774 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d8eb06-75f96f6674-gz2rx"] Oct 30 01:41:46.303416 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.303335 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db534-g5gxs" Oct 30 01:41:46.407978 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.407931 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:46.543119 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.543087 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04eef61c-0c54-4a8a-b1b8-da3d30e4af77" path="/var/lib/kubelet/pods/04eef61c-0c54-4a8a-b1b8-da3d30e4af77/volumes" Oct 30 01:41:46.543558 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.543543 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e41b81b-9c53-4789-89f0-b0b6ba165ffc" path="/var/lib/kubelet/pods/2e41b81b-9c53-4789-89f0-b0b6ba165ffc/volumes" Oct 30 01:41:46.543949 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.543935 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84354b2e-c89c-486d-adb1-0204a5f120c1" path="/var/lib/kubelet/pods/84354b2e-c89c-486d-adb1-0204a5f120c1/volumes" Oct 30 01:41:46.607716 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:46.607684 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db534-g5gxs"] Oct 30 01:41:46.608075 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:41:46.608045 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c8c3eba_77d5_4ea2_814f_99a29cc9b07f.slice/crio-80d4a10036fa8276841e80f5a2accc62185ea1e4a767cb7fc03e63e811247b38 WatchSource:0}: Error finding container 80d4a10036fa8276841e80f5a2accc62185ea1e4a767cb7fc03e63e811247b38: Status 404 returned error can't find the container with id 80d4a10036fa8276841e80f5a2accc62185ea1e4a767cb7fc03e63e811247b38 Oct 30 01:41:47.132464 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:47.132423 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db534-g5gxs" event={"ID":"3c8c3eba-77d5-4ea2-814f-99a29cc9b07f","Type":"ContainerStarted","Data":"bbeb048e14d009062948fea6dffcc284be9d88aad6fa6723c75df81a6c6bf5ad"} Oct 30 01:41:47.132977 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:47.132484 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db534-g5gxs" event={"ID":"3c8c3eba-77d5-4ea2-814f-99a29cc9b07f","Type":"ContainerStarted","Data":"80d4a10036fa8276841e80f5a2accc62185ea1e4a767cb7fc03e63e811247b38"} Oct 30 01:41:47.178228 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:47.178100 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db534-g5gxs" podStartSLOduration=2.178077782 podStartE2EDuration="2.178077782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:41:47.17434905 +0000 UTC m=+1432.750774453" watchObservedRunningTime="2026-10-30 01:41:47.178077782 +0000 UTC m=+1432.754503184" Oct 30 01:41:47.796978 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:47.796894 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:48.039194 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:48.039162 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-3fde1c-58cf94cd5b-nq9g2" Oct 30 01:41:48.143338 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:48.143252 6572 generic.go:356] "Generic (PLEG): container finished" podID="3c8c3eba-77d5-4ea2-814f-99a29cc9b07f" containerID="bbeb048e14d009062948fea6dffcc284be9d88aad6fa6723c75df81a6c6bf5ad" exitCode=0 Oct 30 01:41:48.143338 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:48.143301 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db534-g5gxs" event={"ID":"3c8c3eba-77d5-4ea2-814f-99a29cc9b07f","Type":"ContainerDied","Data":"bbeb048e14d009062948fea6dffcc284be9d88aad6fa6723c75df81a6c6bf5ad"} Oct 30 01:41:49.312020 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:49.311993 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db534-g5gxs" Oct 30 01:41:49.383994 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:49.383954 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c8c3eba-77d5-4ea2-814f-99a29cc9b07f-kube-api-access-tp2vb\" (UniqueName: \"kubernetes.io/projected/3c8c3eba-77d5-4ea2-814f-99a29cc9b07f-kube-api-access-tp2vb\") pod \"3c8c3eba-77d5-4ea2-814f-99a29cc9b07f\" (UID: \"3c8c3eba-77d5-4ea2-814f-99a29cc9b07f\") " Oct 30 01:41:49.384250 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:49.384104 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c8c3eba-77d5-4ea2-814f-99a29cc9b07f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c8c3eba-77d5-4ea2-814f-99a29cc9b07f-maas-api-xks-ca\") pod \"3c8c3eba-77d5-4ea2-814f-99a29cc9b07f\" (UID: \"3c8c3eba-77d5-4ea2-814f-99a29cc9b07f\") " Oct 30 01:41:49.384250 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:49.384209 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c8c3eba-77d5-4ea2-814f-99a29cc9b07f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c8c3eba-77d5-4ea2-814f-99a29cc9b07f-maas-api-openshift-ca\") pod \"3c8c3eba-77d5-4ea2-814f-99a29cc9b07f\" (UID: \"3c8c3eba-77d5-4ea2-814f-99a29cc9b07f\") " Oct 30 01:41:49.384972 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:49.384936 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c8c3eba-77d5-4ea2-814f-99a29cc9b07f-maas-api-openshift-ca" pod "3c8c3eba-77d5-4ea2-814f-99a29cc9b07f" (UID: "3c8c3eba-77d5-4ea2-814f-99a29cc9b07f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:41:49.386986 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:49.386951 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c8c3eba-77d5-4ea2-814f-99a29cc9b07f-maas-api-xks-ca" pod "3c8c3eba-77d5-4ea2-814f-99a29cc9b07f" (UID: "3c8c3eba-77d5-4ea2-814f-99a29cc9b07f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:41:49.389241 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:49.389212 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c8c3eba-77d5-4ea2-814f-99a29cc9b07f-kube-api-access-tp2vb" pod "3c8c3eba-77d5-4ea2-814f-99a29cc9b07f" (UID: "3c8c3eba-77d5-4ea2-814f-99a29cc9b07f"). InnerVolumeSpecName "kube-api-access-tp2vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:41:49.478684 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:49.475530 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:49.485822 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:49.485788 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c8c3eba-77d5-4ea2-814f-99a29cc9b07f-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:41:49.485822 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:49.485821 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp2vb\" (UniqueName: \"kubernetes.io/projected/3c8c3eba-77d5-4ea2-814f-99a29cc9b07f-kube-api-access-tp2vb\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:41:49.485822 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:49.485831 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c8c3eba-77d5-4ea2-814f-99a29cc9b07f-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:41:50.060664 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:50.060634 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-c564cf647-d674t" Oct 30 01:41:50.165334 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:50.165295 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db534-g5gxs" event={"ID":"3c8c3eba-77d5-4ea2-814f-99a29cc9b07f","Type":"ContainerDied","Data":"80d4a10036fa8276841e80f5a2accc62185ea1e4a767cb7fc03e63e811247b38"} Oct 30 01:41:50.165525 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:50.165359 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80d4a10036fa8276841e80f5a2accc62185ea1e4a767cb7fc03e63e811247b38" Oct 30 01:41:50.165525 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:50.165427 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db534-g5gxs" Oct 30 01:41:51.311690 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:51.311634 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:52.338772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:52.338623 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:52.527135 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:52.527097 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:52.729012 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:52.728123 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:53.043864 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:53.043737 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:53.797410 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:53.797018 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:55.460967 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:55.460929 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:56.109931 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:56.109173 6572 scope.go:122] "RemoveContainer" containerID="5668577a077c96238c98a58812de34e2bf8ca0f0520c77e4c162bda5e68a0581" Oct 30 01:41:56.127090 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:56.127066 6572 scope.go:122] "RemoveContainer" containerID="85e6280a559adae7f53d83ae19e1c621f6a6c9d00ba6192a22b737f332e1cdb3" Oct 30 01:41:56.146788 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:56.146758 6572 scope.go:122] "RemoveContainer" containerID="e0c789a00bb3ceddc35b1b1bb1947ccdd094099ec19ae7be6d03afde7d9328f5" Oct 30 01:41:57.563059 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:57.563027 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:57.763279 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:57.763240 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:58.684978 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:58.684940 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:58.874323 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:58.874287 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:59.078136 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:59.078048 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:59.305783 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:59.305719 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:59.773358 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:41:59.773320 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:00.625192 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:00.625157 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:00.714418 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:00.714378 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-5cd7b66fdc-qxr68"] Oct 30 01:42:00.715029 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:00.715014 6572 cpu_manager.go:414] "containerMap: removing container" podUID="3c8c3eba-77d5-4ea2-814f-99a29cc9b07f" containerName="revoke-keys" Oct 30 01:42:00.715085 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:00.715030 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c8c3eba-77d5-4ea2-814f-99a29cc9b07f" containerName="revoke-keys" Oct 30 01:42:00.715150 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:00.715141 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c8c3eba-77d5-4ea2-814f-99a29cc9b07f" containerName="revoke-keys" Oct 30 01:42:00.720700 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:00.720678 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-5cd7b66fdc-qxr68" Oct 30 01:42:00.739429 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:00.739400 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-5cd7b66fdc-qxr68"] Oct 30 01:42:00.753185 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:00.753157 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ea0fc4-dockercfg-nshf2\"" Oct 30 01:42:00.800666 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:00.800625 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ea0fc4\"" Oct 30 01:42:00.805828 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:00.801386 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82wlq\" (UniqueName: \"kubernetes.io/projected/e7ec7abc-07c7-4248-8d21-632673feeece-kube-api-access-82wlq\") pod \"payload-pre-processing-e2e-ipp-b-ea0fc4-5cd7b66fdc-qxr68\" (UID: \"e7ec7abc-07c7-4248-8d21-632673feeece\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-5cd7b66fdc-qxr68" Oct 30 01:42:00.805828 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:00.801478 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7ec7abc-07c7-4248-8d21-632673feeece-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ea0fc4-5cd7b66fdc-qxr68\" (UID: \"e7ec7abc-07c7-4248-8d21-632673feeece\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-5cd7b66fdc-qxr68" Oct 30 01:42:00.849857 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:00.849822 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:00.902649 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:00.902547 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82wlq\" (UniqueName: \"kubernetes.io/projected/e7ec7abc-07c7-4248-8d21-632673feeece-kube-api-access-82wlq\") pod \"payload-pre-processing-e2e-ipp-b-ea0fc4-5cd7b66fdc-qxr68\" (UID: \"e7ec7abc-07c7-4248-8d21-632673feeece\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-5cd7b66fdc-qxr68" Oct 30 01:42:00.902649 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:00.902645 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7ec7abc-07c7-4248-8d21-632673feeece-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ea0fc4-5cd7b66fdc-qxr68\" (UID: \"e7ec7abc-07c7-4248-8d21-632673feeece\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-5cd7b66fdc-qxr68" Oct 30 01:42:00.903642 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:00.903617 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7ec7abc-07c7-4248-8d21-632673feeece-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ea0fc4-5cd7b66fdc-qxr68\" (UID: \"e7ec7abc-07c7-4248-8d21-632673feeece\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-5cd7b66fdc-qxr68" Oct 30 01:42:00.921826 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:00.921785 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82wlq\" (UniqueName: \"kubernetes.io/projected/e7ec7abc-07c7-4248-8d21-632673feeece-kube-api-access-82wlq\") pod \"payload-pre-processing-e2e-ipp-b-ea0fc4-5cd7b66fdc-qxr68\" (UID: \"e7ec7abc-07c7-4248-8d21-632673feeece\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-5cd7b66fdc-qxr68" Oct 30 01:42:01.037215 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:01.037181 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-5cd7b66fdc-qxr68" Oct 30 01:42:01.338699 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:01.338665 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-5cd7b66fdc-qxr68"] Oct 30 01:42:01.340300 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:42:01.340268 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7ec7abc_07c7_4248_8d21_632673feeece.slice/crio-8d73b17e11dbbd5efacd5ca12868b2404ce5741528e65a2c8210bb67fa30ccc1 WatchSource:0}: Error finding container 8d73b17e11dbbd5efacd5ca12868b2404ce5741528e65a2c8210bb67fa30ccc1: Status 404 returned error can't find the container with id 8d73b17e11dbbd5efacd5ca12868b2404ce5741528e65a2c8210bb67fa30ccc1 Oct 30 01:42:02.120955 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:02.117440 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:02.270105 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:02.270068 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-5cd7b66fdc-qxr68" event={"ID":"e7ec7abc-07c7-4248-8d21-632673feeece","Type":"ContainerStarted","Data":"9e923dd131c4a3e95bf6b480fd3ac4272ba4f7ddf8035db2f3030bb3a1dd1c1c"} Oct 30 01:42:02.270320 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:02.270164 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-5cd7b66fdc-qxr68" Oct 30 01:42:02.270320 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:02.270188 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-5cd7b66fdc-qxr68" event={"ID":"e7ec7abc-07c7-4248-8d21-632673feeece","Type":"ContainerStarted","Data":"8d73b17e11dbbd5efacd5ca12868b2404ce5741528e65a2c8210bb67fa30ccc1"} Oct 30 01:42:02.312544 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:02.312485 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-5cd7b66fdc-qxr68" podStartSLOduration=2.312469392 podStartE2EDuration="2.312469392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:42:02.308927499 +0000 UTC m=+1447.885352902" watchObservedRunningTime="2026-10-30 01:42:02.312469392 +0000 UTC m=+1447.888894793" Oct 30 01:42:02.322087 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:02.322059 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:02.338257 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:02.338218 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c22e6b-7cd96f6bdc-p9l2f"] Oct 30 01:42:02.342576 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:02.342558 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c22e6b-7cd96f6bdc-p9l2f" Oct 30 01:42:02.360086 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:02.360055 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-c22e6b\"" Oct 30 01:42:02.360259 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:02.360235 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-c22e6b-dockercfg-w5xbb\"" Oct 30 01:42:02.361875 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:02.361848 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c22e6b-7cd96f6bdc-p9l2f"] Oct 30 01:42:02.422156 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:02.422117 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q94sn\" (UniqueName: \"kubernetes.io/projected/fac5d647-98c6-4ac2-97ba-c66b93d7673e-kube-api-access-q94sn\") pod \"payload-pre-processing-e2e-praxis-a-c22e6b-7cd96f6bdc-p9l2f\" (UID: \"fac5d647-98c6-4ac2-97ba-c66b93d7673e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c22e6b-7cd96f6bdc-p9l2f" Oct 30 01:42:02.422329 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:02.422179 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fac5d647-98c6-4ac2-97ba-c66b93d7673e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c22e6b-7cd96f6bdc-p9l2f\" (UID: \"fac5d647-98c6-4ac2-97ba-c66b93d7673e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c22e6b-7cd96f6bdc-p9l2f" Oct 30 01:42:02.454482 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:02.454428 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c22e6b-6486d6dc9b-gkwvs"] Oct 30 01:42:02.486464 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:02.477117 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c22e6b-6486d6dc9b-gkwvs" Oct 30 01:42:02.493793 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:02.492268 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c22e6b-6486d6dc9b-gkwvs"] Oct 30 01:42:02.542772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:02.542253 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q94sn\" (UniqueName: \"kubernetes.io/projected/fac5d647-98c6-4ac2-97ba-c66b93d7673e-kube-api-access-q94sn\") pod \"payload-pre-processing-e2e-praxis-a-c22e6b-7cd96f6bdc-p9l2f\" (UID: \"fac5d647-98c6-4ac2-97ba-c66b93d7673e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c22e6b-7cd96f6bdc-p9l2f" Oct 30 01:42:02.542772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:02.542349 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fac5d647-98c6-4ac2-97ba-c66b93d7673e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c22e6b-7cd96f6bdc-p9l2f\" (UID: \"fac5d647-98c6-4ac2-97ba-c66b93d7673e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c22e6b-7cd96f6bdc-p9l2f" Oct 30 01:42:02.570801 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:02.543318 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fac5d647-98c6-4ac2-97ba-c66b93d7673e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c22e6b-7cd96f6bdc-p9l2f\" (UID: \"fac5d647-98c6-4ac2-97ba-c66b93d7673e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c22e6b-7cd96f6bdc-p9l2f" Oct 30 01:42:02.570801 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:02.547849 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:02.599688 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:02.599648 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q94sn\" (UniqueName: \"kubernetes.io/projected/fac5d647-98c6-4ac2-97ba-c66b93d7673e-kube-api-access-q94sn\") pod \"payload-pre-processing-e2e-praxis-a-c22e6b-7cd96f6bdc-p9l2f\" (UID: \"fac5d647-98c6-4ac2-97ba-c66b93d7673e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c22e6b-7cd96f6bdc-p9l2f" Oct 30 01:42:02.644086 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:02.644039 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d1710d2-4436-4f5b-bfb9-82bd8210f7b3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c22e6b-6486d6dc9b-gkwvs\" (UID: \"0d1710d2-4436-4f5b-bfb9-82bd8210f7b3\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c22e6b-6486d6dc9b-gkwvs" Oct 30 01:42:02.644362 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:02.644340 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tq97\" (UniqueName: \"kubernetes.io/projected/0d1710d2-4436-4f5b-bfb9-82bd8210f7b3-kube-api-access-6tq97\") pod \"payload-processing-e2e-praxis-a-c22e6b-6486d6dc9b-gkwvs\" (UID: \"0d1710d2-4436-4f5b-bfb9-82bd8210f7b3\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c22e6b-6486d6dc9b-gkwvs" Oct 30 01:42:02.656773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:02.656725 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c22e6b-7cd96f6bdc-p9l2f" Oct 30 01:42:02.746572 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:02.746536 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tq97\" (UniqueName: \"kubernetes.io/projected/0d1710d2-4436-4f5b-bfb9-82bd8210f7b3-kube-api-access-6tq97\") pod \"payload-processing-e2e-praxis-a-c22e6b-6486d6dc9b-gkwvs\" (UID: \"0d1710d2-4436-4f5b-bfb9-82bd8210f7b3\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c22e6b-6486d6dc9b-gkwvs" Oct 30 01:42:02.746760 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:02.746701 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d1710d2-4436-4f5b-bfb9-82bd8210f7b3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c22e6b-6486d6dc9b-gkwvs\" (UID: \"0d1710d2-4436-4f5b-bfb9-82bd8210f7b3\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c22e6b-6486d6dc9b-gkwvs" Oct 30 01:42:02.747709 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:02.747682 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d1710d2-4436-4f5b-bfb9-82bd8210f7b3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c22e6b-6486d6dc9b-gkwvs\" (UID: \"0d1710d2-4436-4f5b-bfb9-82bd8210f7b3\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c22e6b-6486d6dc9b-gkwvs" Oct 30 01:42:02.765926 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:02.765902 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tq97\" (UniqueName: \"kubernetes.io/projected/0d1710d2-4436-4f5b-bfb9-82bd8210f7b3-kube-api-access-6tq97\") pod \"payload-processing-e2e-praxis-a-c22e6b-6486d6dc9b-gkwvs\" (UID: \"0d1710d2-4436-4f5b-bfb9-82bd8210f7b3\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c22e6b-6486d6dc9b-gkwvs" Oct 30 01:42:02.905078 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:02.905040 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c22e6b-6486d6dc9b-gkwvs" Oct 30 01:42:02.954465 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:02.954434 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c22e6b-7cd96f6bdc-p9l2f"] Oct 30 01:42:02.955026 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:42:02.954994 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfac5d647_98c6_4ac2_97ba_c66b93d7673e.slice/crio-c2ef06b0f6be875e32b2814c4236e00972b09dce2b70406f923d2b86aedcf82b WatchSource:0}: Error finding container c2ef06b0f6be875e32b2814c4236e00972b09dce2b70406f923d2b86aedcf82b: Status 404 returned error can't find the container with id c2ef06b0f6be875e32b2814c4236e00972b09dce2b70406f923d2b86aedcf82b Oct 30 01:42:03.207569 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:03.207542 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c22e6b-6486d6dc9b-gkwvs"] Oct 30 01:42:03.208887 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:42:03.208858 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d1710d2_4436_4f5b_bfb9_82bd8210f7b3.slice/crio-a8c458f701b46bb8373aa8b7572a506fd805904dadd8092e2488719a2e930cf2 WatchSource:0}: Error finding container a8c458f701b46bb8373aa8b7572a506fd805904dadd8092e2488719a2e930cf2: Status 404 returned error can't find the container with id a8c458f701b46bb8373aa8b7572a506fd805904dadd8092e2488719a2e930cf2 Oct 30 01:42:03.280648 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:03.280603 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c22e6b-6486d6dc9b-gkwvs" event={"ID":"0d1710d2-4436-4f5b-bfb9-82bd8210f7b3","Type":"ContainerStarted","Data":"22a40e88b3d0cf06e5201ae0ffe987ffb1b120dd4917e9d16134148d9c6046d4"} Oct 30 01:42:03.280831 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:03.280691 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-c22e6b-6486d6dc9b-gkwvs" Oct 30 01:42:03.280831 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:03.280730 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c22e6b-6486d6dc9b-gkwvs" event={"ID":"0d1710d2-4436-4f5b-bfb9-82bd8210f7b3","Type":"ContainerStarted","Data":"a8c458f701b46bb8373aa8b7572a506fd805904dadd8092e2488719a2e930cf2"} Oct 30 01:42:03.282469 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:03.282441 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c22e6b-7cd96f6bdc-p9l2f" event={"ID":"fac5d647-98c6-4ac2-97ba-c66b93d7673e","Type":"ContainerStarted","Data":"41f3da1b4d7fd19e1d2d60fef826cca2e824c57b7b33ed52c044058f59689dba"} Oct 30 01:42:03.282585 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:03.282489 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c22e6b-7cd96f6bdc-p9l2f" event={"ID":"fac5d647-98c6-4ac2-97ba-c66b93d7673e","Type":"ContainerStarted","Data":"c2ef06b0f6be875e32b2814c4236e00972b09dce2b70406f923d2b86aedcf82b"} Oct 30 01:42:03.282676 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:03.282658 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c22e6b-7cd96f6bdc-p9l2f" Oct 30 01:42:03.318587 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:03.318531 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-c22e6b-6486d6dc9b-gkwvs" podStartSLOduration=1.3185158590000001 podStartE2EDuration="1.318515859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:42:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:42:03.316477975 +0000 UTC m=+1448.892903373" watchObservedRunningTime="2026-10-30 01:42:03.318515859 +0000 UTC m=+1448.894941262" Oct 30 01:42:03.355091 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:03.355033 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c22e6b-7cd96f6bdc-p9l2f" podStartSLOduration=1.355019172 podStartE2EDuration="1.355019172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:42:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:42:03.353322115 +0000 UTC m=+1448.929747517" watchObservedRunningTime="2026-10-30 01:42:03.355019172 +0000 UTC m=+1448.931444574" Oct 30 01:42:03.568868 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:03.568699 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:03.579317 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:03.579274 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0db534-5cb9dc6664-bphz5"] Oct 30 01:42:03.579516 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:03.579490 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db534-5cb9dc6664-bphz5" podUID="18295ccd-e011-41e0-8077-7abea7fce67f" containerName="payload-processing" containerID="cri-o://114fa08b80935aac2f6d562c6a906d16b31dd96fe71f238607bd6cec251e5ef4" gracePeriod=30 Oct 30 01:42:03.772667 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:03.772634 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:03.860954 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:03.860929 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db534-5cb9dc6664-bphz5" Oct 30 01:42:03.997795 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:03.996179 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:03.997795 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:03.996851 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18295ccd-e011-41e0-8077-7abea7fce67f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18295ccd-e011-41e0-8077-7abea7fce67f-plugins-config-volume\") pod \"18295ccd-e011-41e0-8077-7abea7fce67f\" (UID: \"18295ccd-e011-41e0-8077-7abea7fce67f\") " Oct 30 01:42:03.997795 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:03.996920 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18295ccd-e011-41e0-8077-7abea7fce67f-kube-api-access-nvx2n\" (UniqueName: \"kubernetes.io/projected/18295ccd-e011-41e0-8077-7abea7fce67f-kube-api-access-nvx2n\") pod \"18295ccd-e011-41e0-8077-7abea7fce67f\" (UID: \"18295ccd-e011-41e0-8077-7abea7fce67f\") " Oct 30 01:42:04.006458 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:03.999147 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18295ccd-e011-41e0-8077-7abea7fce67f-plugins-config-volume" pod "18295ccd-e011-41e0-8077-7abea7fce67f" (UID: "18295ccd-e011-41e0-8077-7abea7fce67f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:42:04.006458 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:04.004300 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18295ccd-e011-41e0-8077-7abea7fce67f-kube-api-access-nvx2n" pod "18295ccd-e011-41e0-8077-7abea7fce67f" (UID: "18295ccd-e011-41e0-8077-7abea7fce67f"). InnerVolumeSpecName "kube-api-access-nvx2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:42:04.098604 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:04.098522 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18295ccd-e011-41e0-8077-7abea7fce67f-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:42:04.098604 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:04.098551 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvx2n\" (UniqueName: \"kubernetes.io/projected/18295ccd-e011-41e0-8077-7abea7fce67f-kube-api-access-nvx2n\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:42:04.201897 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:04.201858 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:04.254889 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:04.254854 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db534-g5gxs"] Oct 30 01:42:04.260270 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:04.260237 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0db534-g5gxs"] Oct 30 01:42:04.293660 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:04.293628 6572 generic.go:356] "Generic (PLEG): container finished" podID="18295ccd-e011-41e0-8077-7abea7fce67f" containerID="114fa08b80935aac2f6d562c6a906d16b31dd96fe71f238607bd6cec251e5ef4" exitCode=0 Oct 30 01:42:04.294728 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:04.293711 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db534-5cb9dc6664-bphz5" Oct 30 01:42:04.294728 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:04.293728 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db534-5cb9dc6664-bphz5" event={"ID":"18295ccd-e011-41e0-8077-7abea7fce67f","Type":"ContainerDied","Data":"114fa08b80935aac2f6d562c6a906d16b31dd96fe71f238607bd6cec251e5ef4"} Oct 30 01:42:04.294728 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:04.293843 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0db534-5cb9dc6664-bphz5" event={"ID":"18295ccd-e011-41e0-8077-7abea7fce67f","Type":"ContainerDied","Data":"1c04997a926657b633effe1a947bcbe9073543d93a21e37e79fd182887cd0970"} Oct 30 01:42:04.294728 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:04.293885 6572 scope.go:122] "RemoveContainer" containerID="114fa08b80935aac2f6d562c6a906d16b31dd96fe71f238607bd6cec251e5ef4" Oct 30 01:42:04.308390 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:04.308372 6572 scope.go:122] "RemoveContainer" containerID="114fa08b80935aac2f6d562c6a906d16b31dd96fe71f238607bd6cec251e5ef4" Oct 30 01:42:04.308755 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:42:04.308707 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"114fa08b80935aac2f6d562c6a906d16b31dd96fe71f238607bd6cec251e5ef4\": container with ID starting with 114fa08b80935aac2f6d562c6a906d16b31dd96fe71f238607bd6cec251e5ef4 not found: ID does not exist" containerID="114fa08b80935aac2f6d562c6a906d16b31dd96fe71f238607bd6cec251e5ef4" Oct 30 01:42:04.308832 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:04.308794 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"114fa08b80935aac2f6d562c6a906d16b31dd96fe71f238607bd6cec251e5ef4"} err="failed to get container status \"114fa08b80935aac2f6d562c6a906d16b31dd96fe71f238607bd6cec251e5ef4\": rpc error: code = NotFound desc = could not find container \"114fa08b80935aac2f6d562c6a906d16b31dd96fe71f238607bd6cec251e5ef4\": container with ID starting with 114fa08b80935aac2f6d562c6a906d16b31dd96fe71f238607bd6cec251e5ef4 not found: ID does not exist" Oct 30 01:42:04.346328 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:04.346291 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0db534-5cb9dc6664-bphz5"] Oct 30 01:42:04.349990 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:04.349930 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0db534-5cb9dc6664-bphz5"] Oct 30 01:42:04.392221 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:04.392177 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:04.544192 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:04.544154 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18295ccd-e011-41e0-8077-7abea7fce67f" path="/var/lib/kubelet/pods/18295ccd-e011-41e0-8077-7abea7fce67f/volumes" Oct 30 01:42:04.544642 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:04.544627 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c8c3eba-77d5-4ea2-814f-99a29cc9b07f" path="/var/lib/kubelet/pods/3c8c3eba-77d5-4ea2-814f-99a29cc9b07f/volumes" Oct 30 01:42:04.573589 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:04.573552 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:04.860329 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:04.860290 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:05.057954 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:05.057913 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:06.917315 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:06.917274 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:08.571672 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:08.571638 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:10.257448 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:10.257416 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:11.096148 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:11.096119 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:11.305330 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:11.305296 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:11.329307 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:11.329272 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv"] Oct 30 01:42:11.331890 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:11.331852 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" podUID="bcef90f8-7163-4fcc-94df-f10f46f8aa56" containerName="istio-proxy" containerID="cri-o://d2d7d6c71f3aead958c8ceb36166d0d985afd4ce3dd95e83cd822112b599b1bd" gracePeriod=30 Oct 30 01:42:11.492386 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:11.492354 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:12.019379 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:12.019342 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:12.991070 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:12.991031 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:13.285282 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:13.285205 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-5cd7b66fdc-qxr68" Oct 30 01:42:13.642477 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:13.642379 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:13.857312 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:13.857279 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:14.073601 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:14.073566 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:14.297050 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:14.297014 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-c22e6b-6486d6dc9b-gkwvs" Oct 30 01:42:14.297458 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:14.297441 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c22e6b-7cd96f6bdc-p9l2f" Oct 30 01:42:14.689402 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:42:14.689363 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:42:15.738056 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:15.737986 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:16.616459 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:16.616377 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:42:16.694289 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:16.694241 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcef90f8-7163-4fcc-94df-f10f46f8aa56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcef90f8-7163-4fcc-94df-f10f46f8aa56-workload-socket\") pod \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\" (UID: \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\") " Oct 30 01:42:16.694289 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:16.694287 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcef90f8-7163-4fcc-94df-f10f46f8aa56-istio-token\" (UniqueName: \"kubernetes.io/projected/bcef90f8-7163-4fcc-94df-f10f46f8aa56-istio-token\") pod \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\" (UID: \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\") " Oct 30 01:42:16.694570 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:16.694354 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcef90f8-7163-4fcc-94df-f10f46f8aa56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcef90f8-7163-4fcc-94df-f10f46f8aa56-istiod-ca-cert\") pod \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\" (UID: \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\") " Oct 30 01:42:16.694570 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:16.694416 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcef90f8-7163-4fcc-94df-f10f46f8aa56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcef90f8-7163-4fcc-94df-f10f46f8aa56-workload-certs\") pod \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\" (UID: \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\") " Oct 30 01:42:16.694570 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:16.694472 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcef90f8-7163-4fcc-94df-f10f46f8aa56-kube-api-access-vwcgh\" (UniqueName: \"kubernetes.io/projected/bcef90f8-7163-4fcc-94df-f10f46f8aa56-kube-api-access-vwcgh\") pod \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\" (UID: \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\") " Oct 30 01:42:16.694570 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:16.694501 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcef90f8-7163-4fcc-94df-f10f46f8aa56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcef90f8-7163-4fcc-94df-f10f46f8aa56-istio-envoy\") pod \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\" (UID: \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\") " Oct 30 01:42:16.694570 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:16.694521 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcef90f8-7163-4fcc-94df-f10f46f8aa56-workload-socket" pod "bcef90f8-7163-4fcc-94df-f10f46f8aa56" (UID: "bcef90f8-7163-4fcc-94df-f10f46f8aa56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:42:16.694570 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:16.694541 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bcef90f8-7163-4fcc-94df-f10f46f8aa56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcef90f8-7163-4fcc-94df-f10f46f8aa56-istio-podinfo\") pod \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\" (UID: \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\") " Oct 30 01:42:16.694960 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:16.694613 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcef90f8-7163-4fcc-94df-f10f46f8aa56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcef90f8-7163-4fcc-94df-f10f46f8aa56-credential-socket\") pod \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\" (UID: \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\") " Oct 30 01:42:16.694960 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:16.694645 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcef90f8-7163-4fcc-94df-f10f46f8aa56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcef90f8-7163-4fcc-94df-f10f46f8aa56-istio-data\") pod \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\" (UID: \"bcef90f8-7163-4fcc-94df-f10f46f8aa56\") " Oct 30 01:42:16.694960 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:16.694692 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcef90f8-7163-4fcc-94df-f10f46f8aa56-workload-certs" pod "bcef90f8-7163-4fcc-94df-f10f46f8aa56" (UID: "bcef90f8-7163-4fcc-94df-f10f46f8aa56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:42:16.694960 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:16.694912 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcef90f8-7163-4fcc-94df-f10f46f8aa56-istio-data" pod "bcef90f8-7163-4fcc-94df-f10f46f8aa56" (UID: "bcef90f8-7163-4fcc-94df-f10f46f8aa56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:42:16.695414 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:16.695302 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcef90f8-7163-4fcc-94df-f10f46f8aa56-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:42:16.695414 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:16.695330 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcef90f8-7163-4fcc-94df-f10f46f8aa56-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:42:16.695414 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:16.695343 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcef90f8-7163-4fcc-94df-f10f46f8aa56-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:42:16.695674 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:16.695634 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcef90f8-7163-4fcc-94df-f10f46f8aa56-credential-socket" pod "bcef90f8-7163-4fcc-94df-f10f46f8aa56" (UID: "bcef90f8-7163-4fcc-94df-f10f46f8aa56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:42:16.696031 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:16.695969 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcef90f8-7163-4fcc-94df-f10f46f8aa56-istiod-ca-cert" pod "bcef90f8-7163-4fcc-94df-f10f46f8aa56" (UID: "bcef90f8-7163-4fcc-94df-f10f46f8aa56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:42:16.697552 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:16.697529 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcef90f8-7163-4fcc-94df-f10f46f8aa56-istio-token" pod "bcef90f8-7163-4fcc-94df-f10f46f8aa56" (UID: "bcef90f8-7163-4fcc-94df-f10f46f8aa56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:42:16.697706 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:16.697685 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcef90f8-7163-4fcc-94df-f10f46f8aa56-istio-envoy" pod "bcef90f8-7163-4fcc-94df-f10f46f8aa56" (UID: "bcef90f8-7163-4fcc-94df-f10f46f8aa56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:42:16.698326 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:16.698287 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bcef90f8-7163-4fcc-94df-f10f46f8aa56-istio-podinfo" pod "bcef90f8-7163-4fcc-94df-f10f46f8aa56" (UID: "bcef90f8-7163-4fcc-94df-f10f46f8aa56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:42:16.698326 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:16.698294 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcef90f8-7163-4fcc-94df-f10f46f8aa56-kube-api-access-vwcgh" pod "bcef90f8-7163-4fcc-94df-f10f46f8aa56" (UID: "bcef90f8-7163-4fcc-94df-f10f46f8aa56"). InnerVolumeSpecName "kube-api-access-vwcgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:42:16.796272 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:16.796221 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcef90f8-7163-4fcc-94df-f10f46f8aa56-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:42:16.796272 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:16.796265 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwcgh\" (UniqueName: \"kubernetes.io/projected/bcef90f8-7163-4fcc-94df-f10f46f8aa56-kube-api-access-vwcgh\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:42:16.796272 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:16.796284 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcef90f8-7163-4fcc-94df-f10f46f8aa56-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:42:16.796812 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:16.796296 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcef90f8-7163-4fcc-94df-f10f46f8aa56-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:42:16.796812 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:16.796308 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcef90f8-7163-4fcc-94df-f10f46f8aa56-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:42:16.796812 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:16.796319 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcef90f8-7163-4fcc-94df-f10f46f8aa56-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:42:17.423699 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:17.423663 6572 generic.go:356] "Generic (PLEG): container finished" podID="bcef90f8-7163-4fcc-94df-f10f46f8aa56" containerID="d2d7d6c71f3aead958c8ceb36166d0d985afd4ce3dd95e83cd822112b599b1bd" exitCode=0 Oct 30 01:42:17.423948 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:17.423715 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" event={"ID":"bcef90f8-7163-4fcc-94df-f10f46f8aa56","Type":"ContainerDied","Data":"d2d7d6c71f3aead958c8ceb36166d0d985afd4ce3dd95e83cd822112b599b1bd"} Oct 30 01:42:17.423948 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:17.423727 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" Oct 30 01:42:17.423948 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:17.423772 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv" event={"ID":"bcef90f8-7163-4fcc-94df-f10f46f8aa56","Type":"ContainerDied","Data":"71200f73d4c4ab9bcaef2831339755e3786a2a9c4083b7be08487c6e4efdd6b5"} Oct 30 01:42:17.423948 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:17.423794 6572 scope.go:122] "RemoveContainer" containerID="d2d7d6c71f3aead958c8ceb36166d0d985afd4ce3dd95e83cd822112b599b1bd" Oct 30 01:42:17.436210 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:17.436182 6572 scope.go:122] "RemoveContainer" containerID="8deeafbbc128ac76469ef1bb3adcabb7ddf26e84a96ef395a10f33c865e21d20" Oct 30 01:42:17.448059 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:17.448032 6572 scope.go:122] "RemoveContainer" containerID="d2d7d6c71f3aead958c8ceb36166d0d985afd4ce3dd95e83cd822112b599b1bd" Oct 30 01:42:17.448466 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:42:17.448446 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2d7d6c71f3aead958c8ceb36166d0d985afd4ce3dd95e83cd822112b599b1bd\": container with ID starting with d2d7d6c71f3aead958c8ceb36166d0d985afd4ce3dd95e83cd822112b599b1bd not found: ID does not exist" containerID="d2d7d6c71f3aead958c8ceb36166d0d985afd4ce3dd95e83cd822112b599b1bd" Oct 30 01:42:17.448528 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:17.448478 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2d7d6c71f3aead958c8ceb36166d0d985afd4ce3dd95e83cd822112b599b1bd"} err="failed to get container status \"d2d7d6c71f3aead958c8ceb36166d0d985afd4ce3dd95e83cd822112b599b1bd\": rpc error: code = NotFound desc = could not find container \"d2d7d6c71f3aead958c8ceb36166d0d985afd4ce3dd95e83cd822112b599b1bd\": container with ID starting with d2d7d6c71f3aead958c8ceb36166d0d985afd4ce3dd95e83cd822112b599b1bd not found: ID does not exist" Oct 30 01:42:17.448528 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:17.448499 6572 scope.go:122] "RemoveContainer" containerID="8deeafbbc128ac76469ef1bb3adcabb7ddf26e84a96ef395a10f33c865e21d20" Oct 30 01:42:17.448865 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:42:17.448838 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8deeafbbc128ac76469ef1bb3adcabb7ddf26e84a96ef395a10f33c865e21d20\": container with ID starting with 8deeafbbc128ac76469ef1bb3adcabb7ddf26e84a96ef395a10f33c865e21d20 not found: ID does not exist" containerID="8deeafbbc128ac76469ef1bb3adcabb7ddf26e84a96ef395a10f33c865e21d20" Oct 30 01:42:17.449054 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:17.449022 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8deeafbbc128ac76469ef1bb3adcabb7ddf26e84a96ef395a10f33c865e21d20"} err="failed to get container status \"8deeafbbc128ac76469ef1bb3adcabb7ddf26e84a96ef395a10f33c865e21d20\": rpc error: code = NotFound desc = could not find container \"8deeafbbc128ac76469ef1bb3adcabb7ddf26e84a96ef395a10f33c865e21d20\": container with ID starting with 8deeafbbc128ac76469ef1bb3adcabb7ddf26e84a96ef395a10f33c865e21d20 not found: ID does not exist" Oct 30 01:42:17.487054 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:17.487017 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv"] Oct 30 01:42:17.497885 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:17.497848 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-0db534-openshift-default-7f7f5cd64f-k5cgv"] Oct 30 01:42:17.628346 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:17.628308 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:17.922907 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:17.922872 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:18.227046 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:18.224374 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:18.422966 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:18.422934 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:18.543575 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:18.543484 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcef90f8-7163-4fcc-94df-f10f46f8aa56" path="/var/lib/kubelet/pods/bcef90f8-7163-4fcc-94df-f10f46f8aa56/volumes" Oct 30 01:42:19.290030 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:19.289996 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:19.306273 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:19.306238 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0ae258-7889569bbb-k695k"] Oct 30 01:42:19.306908 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:19.306890 6572 cpu_manager.go:414] "containerMap: removing container" podUID="18295ccd-e011-41e0-8077-7abea7fce67f" containerName="payload-processing" Oct 30 01:42:19.306996 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:19.306909 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18295ccd-e011-41e0-8077-7abea7fce67f" containerName="payload-processing" Oct 30 01:42:19.306996 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:19.306938 6572 cpu_manager.go:414] "containerMap: removing container" podUID="bcef90f8-7163-4fcc-94df-f10f46f8aa56" containerName="istio-proxy" Oct 30 01:42:19.306996 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:19.306945 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcef90f8-7163-4fcc-94df-f10f46f8aa56" containerName="istio-proxy" Oct 30 01:42:19.306996 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:19.306984 6572 cpu_manager.go:414] "containerMap: removing container" podUID="bcef90f8-7163-4fcc-94df-f10f46f8aa56" containerName="istio-proxy" Oct 30 01:42:19.306996 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:19.306992 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcef90f8-7163-4fcc-94df-f10f46f8aa56" containerName="istio-proxy" Oct 30 01:42:19.307239 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:19.307160 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcef90f8-7163-4fcc-94df-f10f46f8aa56" containerName="istio-proxy" Oct 30 01:42:19.307239 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:19.307181 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcef90f8-7163-4fcc-94df-f10f46f8aa56" containerName="istio-proxy" Oct 30 01:42:19.307239 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:19.307201 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="18295ccd-e011-41e0-8077-7abea7fce67f" containerName="payload-processing" Oct 30 01:42:19.312179 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:19.312150 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0ae258-7889569bbb-k695k" Oct 30 01:42:19.340170 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:19.340139 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-0ae258\"" Oct 30 01:42:19.340614 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:19.340305 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-0ae258-dockercfg-x54h6\"" Oct 30 01:42:19.342816 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:19.342793 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0ae258-7889569bbb-k695k"] Oct 30 01:42:19.444583 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:19.444542 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b13d20c-ba82-40b2-bffe-47e7ea20c922-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-0ae258-7889569bbb-k695k\" (UID: \"5b13d20c-ba82-40b2-bffe-47e7ea20c922\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0ae258-7889569bbb-k695k" Oct 30 01:42:19.446716 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:19.446669 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r42wk\" (UniqueName: \"kubernetes.io/projected/5b13d20c-ba82-40b2-bffe-47e7ea20c922-kube-api-access-r42wk\") pod \"payload-processing-e2e-praxis-a-0ae258-7889569bbb-k695k\" (UID: \"5b13d20c-ba82-40b2-bffe-47e7ea20c922\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0ae258-7889569bbb-k695k" Oct 30 01:42:19.494110 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:19.494070 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:19.549989 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:19.549874 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r42wk\" (UniqueName: \"kubernetes.io/projected/5b13d20c-ba82-40b2-bffe-47e7ea20c922-kube-api-access-r42wk\") pod \"payload-processing-e2e-praxis-a-0ae258-7889569bbb-k695k\" (UID: \"5b13d20c-ba82-40b2-bffe-47e7ea20c922\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0ae258-7889569bbb-k695k" Oct 30 01:42:19.550368 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:19.550340 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b13d20c-ba82-40b2-bffe-47e7ea20c922-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-0ae258-7889569bbb-k695k\" (UID: \"5b13d20c-ba82-40b2-bffe-47e7ea20c922\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0ae258-7889569bbb-k695k" Oct 30 01:42:19.551770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:19.551351 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b13d20c-ba82-40b2-bffe-47e7ea20c922-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-0ae258-7889569bbb-k695k\" (UID: \"5b13d20c-ba82-40b2-bffe-47e7ea20c922\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0ae258-7889569bbb-k695k" Oct 30 01:42:19.569570 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:19.569533 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r42wk\" (UniqueName: \"kubernetes.io/projected/5b13d20c-ba82-40b2-bffe-47e7ea20c922-kube-api-access-r42wk\") pod \"payload-processing-e2e-praxis-a-0ae258-7889569bbb-k695k\" (UID: \"5b13d20c-ba82-40b2-bffe-47e7ea20c922\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0ae258-7889569bbb-k695k" Oct 30 01:42:19.627964 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:19.627930 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0ae258-7889569bbb-k695k" Oct 30 01:42:19.940974 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:19.940942 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0ae258-7889569bbb-k695k"] Oct 30 01:42:19.942045 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:42:19.942011 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b13d20c_ba82_40b2_bffe_47e7ea20c922.slice/crio-c90e5786277c3d19645fd9214325f68e8105a2c9b7267f59b7251e9155847221 WatchSource:0}: Error finding container c90e5786277c3d19645fd9214325f68e8105a2c9b7267f59b7251e9155847221: Status 404 returned error can't find the container with id c90e5786277c3d19645fd9214325f68e8105a2c9b7267f59b7251e9155847221 Oct 30 01:42:20.047487 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:20.047441 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:20.223952 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:20.223861 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:20.407933 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:20.407894 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:20.455967 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:20.455918 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0ae258-7889569bbb-k695k" event={"ID":"5b13d20c-ba82-40b2-bffe-47e7ea20c922","Type":"ContainerStarted","Data":"2bb011c0be0f492e595654773c1ebf7898538c99073dcc2ce923b78f21b924f2"} Oct 30 01:42:20.455967 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:20.455972 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0ae258-7889569bbb-k695k" event={"ID":"5b13d20c-ba82-40b2-bffe-47e7ea20c922","Type":"ContainerStarted","Data":"c90e5786277c3d19645fd9214325f68e8105a2c9b7267f59b7251e9155847221"} Oct 30 01:42:20.456235 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:20.456058 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-0ae258-7889569bbb-k695k" Oct 30 01:42:20.496109 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:20.495974 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-0ae258-7889569bbb-k695k" podStartSLOduration=1.495954805 podStartE2EDuration="1.495954805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:42:20.493011782 +0000 UTC m=+1466.069437183" watchObservedRunningTime="2026-10-30 01:42:20.495954805 +0000 UTC m=+1466.072380207" Oct 30 01:42:21.858002 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:21.857961 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:23.641660 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:23.641625 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:25.456205 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:25.456163 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:26.105405 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:26.105362 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:26.618444 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:26.618408 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:26.923983 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:26.923947 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2"] Oct 30 01:42:26.929506 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:26.929481 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:42:26.955950 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:26.955914 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2"] Oct 30 01:42:26.960382 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:26.960354 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-c9d174-openshift-default-dockercfg-dfnrm\"" Oct 30 01:42:26.985362 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:26.985319 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgqgf\" (UniqueName: \"kubernetes.io/projected/98cd042c-e590-4f84-8acc-220357883bef-kube-api-access-zgqgf\") pod \"e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2\" (UID: \"98cd042c-e590-4f84-8acc-220357883bef\") " pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:42:26.985576 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:26.985480 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cd042c-e590-4f84-8acc-220357883bef-workload-socket\") pod \"e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2\" (UID: \"98cd042c-e590-4f84-8acc-220357883bef\") " pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:42:26.985649 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:26.985588 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98cd042c-e590-4f84-8acc-220357883bef-istio-envoy\") pod \"e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2\" (UID: \"98cd042c-e590-4f84-8acc-220357883bef\") " pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:42:26.985714 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:26.985664 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98cd042c-e590-4f84-8acc-220357883bef-istio-data\") pod \"e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2\" (UID: \"98cd042c-e590-4f84-8acc-220357883bef\") " pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:42:26.985804 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:26.985778 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98cd042c-e590-4f84-8acc-220357883bef-istio-podinfo\") pod \"e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2\" (UID: \"98cd042c-e590-4f84-8acc-220357883bef\") " pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:42:26.985872 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:26.985821 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cd042c-e590-4f84-8acc-220357883bef-credential-socket\") pod \"e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2\" (UID: \"98cd042c-e590-4f84-8acc-220357883bef\") " pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:42:26.985872 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:26.985841 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98cd042c-e590-4f84-8acc-220357883bef-workload-certs\") pod \"e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2\" (UID: \"98cd042c-e590-4f84-8acc-220357883bef\") " pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:42:26.985983 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:26.985958 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98cd042c-e590-4f84-8acc-220357883bef-istiod-ca-cert\") pod \"e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2\" (UID: \"98cd042c-e590-4f84-8acc-220357883bef\") " pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:42:26.986043 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:26.985999 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98cd042c-e590-4f84-8acc-220357883bef-istio-token\") pod \"e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2\" (UID: \"98cd042c-e590-4f84-8acc-220357883bef\") " pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:42:27.099786 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:27.089595 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98cd042c-e590-4f84-8acc-220357883bef-istiod-ca-cert\") pod \"e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2\" (UID: \"98cd042c-e590-4f84-8acc-220357883bef\") " pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:42:27.100130 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:27.100099 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98cd042c-e590-4f84-8acc-220357883bef-istio-token\") pod \"e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2\" (UID: \"98cd042c-e590-4f84-8acc-220357883bef\") " pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:42:27.100293 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:27.100274 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgqgf\" (UniqueName: \"kubernetes.io/projected/98cd042c-e590-4f84-8acc-220357883bef-kube-api-access-zgqgf\") pod \"e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2\" (UID: \"98cd042c-e590-4f84-8acc-220357883bef\") " pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:42:27.100458 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:27.093613 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98cd042c-e590-4f84-8acc-220357883bef-istiod-ca-cert\") pod \"e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2\" (UID: \"98cd042c-e590-4f84-8acc-220357883bef\") " pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:42:27.103776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:27.100567 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cd042c-e590-4f84-8acc-220357883bef-workload-socket\") pod \"e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2\" (UID: \"98cd042c-e590-4f84-8acc-220357883bef\") " pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:42:27.104050 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:27.104027 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98cd042c-e590-4f84-8acc-220357883bef-istio-envoy\") pod \"e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2\" (UID: \"98cd042c-e590-4f84-8acc-220357883bef\") " pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:42:27.111942 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:27.111896 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98cd042c-e590-4f84-8acc-220357883bef-istio-data\") pod \"e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2\" (UID: \"98cd042c-e590-4f84-8acc-220357883bef\") " pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:42:27.112649 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:27.112627 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98cd042c-e590-4f84-8acc-220357883bef-istio-podinfo\") pod \"e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2\" (UID: \"98cd042c-e590-4f84-8acc-220357883bef\") " pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:42:27.127930 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:27.127893 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cd042c-e590-4f84-8acc-220357883bef-credential-socket\") pod \"e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2\" (UID: \"98cd042c-e590-4f84-8acc-220357883bef\") " pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:42:27.128146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:27.128129 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98cd042c-e590-4f84-8acc-220357883bef-workload-certs\") pod \"e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2\" (UID: \"98cd042c-e590-4f84-8acc-220357883bef\") " pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:42:27.128492 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:27.123350 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98cd042c-e590-4f84-8acc-220357883bef-istio-podinfo\") pod \"e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2\" (UID: \"98cd042c-e590-4f84-8acc-220357883bef\") " pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:42:27.128628 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:27.104727 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cd042c-e590-4f84-8acc-220357883bef-workload-socket\") pod \"e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2\" (UID: \"98cd042c-e590-4f84-8acc-220357883bef\") " pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:42:27.129081 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:27.129057 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cd042c-e590-4f84-8acc-220357883bef-credential-socket\") pod \"e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2\" (UID: \"98cd042c-e590-4f84-8acc-220357883bef\") " pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:42:27.129221 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:27.112504 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98cd042c-e590-4f84-8acc-220357883bef-istio-data\") pod \"e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2\" (UID: \"98cd042c-e590-4f84-8acc-220357883bef\") " pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:42:27.129581 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:27.129559 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98cd042c-e590-4f84-8acc-220357883bef-workload-certs\") pod \"e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2\" (UID: \"98cd042c-e590-4f84-8acc-220357883bef\") " pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:42:27.158768 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:27.136670 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:27.159537 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:27.159502 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98cd042c-e590-4f84-8acc-220357883bef-istio-envoy\") pod \"e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2\" (UID: \"98cd042c-e590-4f84-8acc-220357883bef\") " pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:42:27.160445 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:27.160382 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgqgf\" (UniqueName: \"kubernetes.io/projected/98cd042c-e590-4f84-8acc-220357883bef-kube-api-access-zgqgf\") pod \"e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2\" (UID: \"98cd042c-e590-4f84-8acc-220357883bef\") " pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:42:27.161002 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:27.160981 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98cd042c-e590-4f84-8acc-220357883bef-istio-token\") pod \"e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2\" (UID: \"98cd042c-e590-4f84-8acc-220357883bef\") " pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:42:27.246858 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:27.246735 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:42:27.333943 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:27.333609 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:27.596881 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:27.596834 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2"] Oct 30 01:42:27.598556 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:42:27.598507 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98cd042c_e590_4f84_8acc_220357883bef.slice/crio-cfdc96ec140ef5c34fb7b4f3ff7f5aecd59c4e0b6d9dc0a8573bd9f96460df92 WatchSource:0}: Error finding container cfdc96ec140ef5c34fb7b4f3ff7f5aecd59c4e0b6d9dc0a8573bd9f96460df92: Status 404 returned error can't find the container with id cfdc96ec140ef5c34fb7b4f3ff7f5aecd59c4e0b6d9dc0a8573bd9f96460df92 Oct 30 01:42:27.602084 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:27.602026 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:42:27.602261 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:27.602131 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:42:27.602261 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:27.602184 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:42:28.138336 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:28.138300 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:28.550683 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:28.550646 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" event={"ID":"98cd042c-e590-4f84-8acc-220357883bef","Type":"ContainerStarted","Data":"1ab66c1173fb183401523059288f77755e92e96d4c5f3f58813475a22a26c788"} Oct 30 01:42:28.550886 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:28.550704 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" event={"ID":"98cd042c-e590-4f84-8acc-220357883bef","Type":"ContainerStarted","Data":"cfdc96ec140ef5c34fb7b4f3ff7f5aecd59c4e0b6d9dc0a8573bd9f96460df92"} Oct 30 01:42:28.623205 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:28.623132 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" podStartSLOduration=2.623106257 podStartE2EDuration="2.623106257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:42:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:42:28.601311339 +0000 UTC m=+1474.177736741" watchObservedRunningTime="2026-10-30 01:42:28.623106257 +0000 UTC m=+1474.199531659" Oct 30 01:42:29.247243 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:29.247198 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:42:29.253386 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:29.253358 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:42:29.558558 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:29.558471 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:42:29.559601 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:29.559580 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:42:30.326543 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:30.316948 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:31.144291 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:31.144257 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:31.397339 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:31.397257 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:31.474844 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:31.474815 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-0ae258-7889569bbb-k695k" Oct 30 01:42:32.197828 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:32.197797 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-6b3dbd-6fbdd58c6d-5xdjd_7d90b45f-4155-4aee-a02e-b7a69295a8a7/payload-processing/0.log" Oct 30 01:42:32.421508 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:32.421478 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:32.805368 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:32.805336 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:33.021118 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:33.021081 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:33.830479 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:33.830440 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3dbd-ngqkb"] Oct 30 01:42:33.837032 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:33.837012 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3dbd-ngqkb" Oct 30 01:42:33.852685 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:33.852662 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3dbd-ngqkb"] Oct 30 01:42:33.859671 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:33.859647 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:42:33.859890 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:33.859692 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:42:33.859890 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:33.859731 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:42:33.862436 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:33.862415 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:42:33.969071 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:33.969039 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56c4f9e0-ecab-47ec-aea2-3f5ab302d779-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6b3dbd-ngqkb\" (UID: \"56c4f9e0-ecab-47ec-aea2-3f5ab302d779\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3dbd-ngqkb" Oct 30 01:42:33.969303 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:33.969108 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56c4f9e0-ecab-47ec-aea2-3f5ab302d779-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6b3dbd-ngqkb\" (UID: \"56c4f9e0-ecab-47ec-aea2-3f5ab302d779\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3dbd-ngqkb" Oct 30 01:42:33.969303 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:33.969233 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z859\" (UniqueName: \"kubernetes.io/projected/56c4f9e0-ecab-47ec-aea2-3f5ab302d779-kube-api-access-5z859\") pod \"maas-api-revoke-keys-e2e-ipp-a-6b3dbd-ngqkb\" (UID: \"56c4f9e0-ecab-47ec-aea2-3f5ab302d779\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3dbd-ngqkb" Oct 30 01:42:34.030001 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:34.029951 6572 patch_prober.go:29] interesting pod/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:42:34.030196 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:34.030030 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" podUID="05ffea96-f1dc-407d-ad3e-df11499478ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:42:34.070434 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:34.070373 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56c4f9e0-ecab-47ec-aea2-3f5ab302d779-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6b3dbd-ngqkb\" (UID: \"56c4f9e0-ecab-47ec-aea2-3f5ab302d779\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3dbd-ngqkb" Oct 30 01:42:34.070650 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:34.070472 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56c4f9e0-ecab-47ec-aea2-3f5ab302d779-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6b3dbd-ngqkb\" (UID: \"56c4f9e0-ecab-47ec-aea2-3f5ab302d779\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3dbd-ngqkb" Oct 30 01:42:34.070650 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:34.070584 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z859\" (UniqueName: \"kubernetes.io/projected/56c4f9e0-ecab-47ec-aea2-3f5ab302d779-kube-api-access-5z859\") pod \"maas-api-revoke-keys-e2e-ipp-a-6b3dbd-ngqkb\" (UID: \"56c4f9e0-ecab-47ec-aea2-3f5ab302d779\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3dbd-ngqkb" Oct 30 01:42:34.071248 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:34.071221 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56c4f9e0-ecab-47ec-aea2-3f5ab302d779-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6b3dbd-ngqkb\" (UID: \"56c4f9e0-ecab-47ec-aea2-3f5ab302d779\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3dbd-ngqkb" Oct 30 01:42:34.072997 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:34.072968 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56c4f9e0-ecab-47ec-aea2-3f5ab302d779-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6b3dbd-ngqkb\" (UID: \"56c4f9e0-ecab-47ec-aea2-3f5ab302d779\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3dbd-ngqkb" Oct 30 01:42:34.089157 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:34.089088 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z859\" (UniqueName: \"kubernetes.io/projected/56c4f9e0-ecab-47ec-aea2-3f5ab302d779-kube-api-access-5z859\") pod \"maas-api-revoke-keys-e2e-ipp-a-6b3dbd-ngqkb\" (UID: \"56c4f9e0-ecab-47ec-aea2-3f5ab302d779\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3dbd-ngqkb" Oct 30 01:42:34.154256 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:34.154216 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3dbd-ngqkb" Oct 30 01:42:34.307619 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:34.307588 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:34.462834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:34.462792 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3dbd-ngqkb"] Oct 30 01:42:34.464363 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:42:34.464332 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56c4f9e0_ecab_47ec_aea2_3f5ab302d779.slice/crio-0751a24066999b00bb74598e95c9ff32a212ce211f1038e04de81b76c4733294 WatchSource:0}: Error finding container 0751a24066999b00bb74598e95c9ff32a212ce211f1038e04de81b76c4733294: Status 404 returned error can't find the container with id 0751a24066999b00bb74598e95c9ff32a212ce211f1038e04de81b76c4733294 Oct 30 01:42:34.611252 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:34.611212 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3dbd-ngqkb" event={"ID":"56c4f9e0-ecab-47ec-aea2-3f5ab302d779","Type":"ContainerStarted","Data":"0751a24066999b00bb74598e95c9ff32a212ce211f1038e04de81b76c4733294"} Oct 30 01:42:35.034692 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:35.034657 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28af3258-qjrpd"] Oct 30 01:42:35.040940 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:35.040915 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28af3258-qjrpd"] Oct 30 01:42:35.620738 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:35.620695 6572 generic.go:356] "Generic (PLEG): container finished" podID="56c4f9e0-ecab-47ec-aea2-3f5ab302d779" containerID="0361785c962212fac6a82f52919e9aec51b7e5407379f13ab6de25e8779413b0" exitCode=0 Oct 30 01:42:35.620942 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:35.620792 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3dbd-ngqkb" event={"ID":"56c4f9e0-ecab-47ec-aea2-3f5ab302d779","Type":"ContainerDied","Data":"0361785c962212fac6a82f52919e9aec51b7e5407379f13ab6de25e8779413b0"} Oct 30 01:42:35.970185 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:35.967123 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:36.543114 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:36.543080 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="456ee785-1d45-499c-810d-68121a7a2f2d" path="/var/lib/kubelet/pods/456ee785-1d45-499c-810d-68121a7a2f2d/volumes" Oct 30 01:42:36.630196 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:36.630116 6572 generic.go:356] "Generic (PLEG): container finished" podID="05ffea96-f1dc-407d-ad3e-df11499478ce" containerID="4edac2164303c8213d07e4a4d7a3e68dd8017177ee7776fd842bdc85561ff29d" exitCode=0 Oct 30 01:42:36.630375 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:36.630192 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" event={"ID":"05ffea96-f1dc-407d-ad3e-df11499478ce","Type":"ContainerDied","Data":"4edac2164303c8213d07e4a4d7a3e68dd8017177ee7776fd842bdc85561ff29d"} Oct 30 01:42:36.631287 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:36.630671 6572 scope.go:122] "RemoveContainer" containerID="4edac2164303c8213d07e4a4d7a3e68dd8017177ee7776fd842bdc85561ff29d" Oct 30 01:42:36.632626 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:36.632587 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:42:36.632775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:36.632666 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:42:36.632775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:36.632714 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:42:36.861769 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:36.848147 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3dbd-ngqkb" Oct 30 01:42:36.921769 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:36.919898 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56c4f9e0-ecab-47ec-aea2-3f5ab302d779-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56c4f9e0-ecab-47ec-aea2-3f5ab302d779-maas-api-openshift-ca\") pod \"56c4f9e0-ecab-47ec-aea2-3f5ab302d779\" (UID: \"56c4f9e0-ecab-47ec-aea2-3f5ab302d779\") " Oct 30 01:42:36.921769 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:36.919982 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56c4f9e0-ecab-47ec-aea2-3f5ab302d779-kube-api-access-5z859\" (UniqueName: \"kubernetes.io/projected/56c4f9e0-ecab-47ec-aea2-3f5ab302d779-kube-api-access-5z859\") pod \"56c4f9e0-ecab-47ec-aea2-3f5ab302d779\" (UID: \"56c4f9e0-ecab-47ec-aea2-3f5ab302d779\") " Oct 30 01:42:36.921769 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:36.920157 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56c4f9e0-ecab-47ec-aea2-3f5ab302d779-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56c4f9e0-ecab-47ec-aea2-3f5ab302d779-maas-api-xks-ca\") pod \"56c4f9e0-ecab-47ec-aea2-3f5ab302d779\" (UID: \"56c4f9e0-ecab-47ec-aea2-3f5ab302d779\") " Oct 30 01:42:36.922067 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:36.921863 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56c4f9e0-ecab-47ec-aea2-3f5ab302d779-maas-api-openshift-ca" pod "56c4f9e0-ecab-47ec-aea2-3f5ab302d779" (UID: "56c4f9e0-ecab-47ec-aea2-3f5ab302d779"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:42:36.935775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:36.924842 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56c4f9e0-ecab-47ec-aea2-3f5ab302d779-maas-api-xks-ca" pod "56c4f9e0-ecab-47ec-aea2-3f5ab302d779" (UID: "56c4f9e0-ecab-47ec-aea2-3f5ab302d779"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:42:36.935775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:36.926662 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56c4f9e0-ecab-47ec-aea2-3f5ab302d779-kube-api-access-5z859" pod "56c4f9e0-ecab-47ec-aea2-3f5ab302d779" (UID: "56c4f9e0-ecab-47ec-aea2-3f5ab302d779"). InnerVolumeSpecName "kube-api-access-5z859". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:42:37.021830 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:37.021599 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56c4f9e0-ecab-47ec-aea2-3f5ab302d779-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:42:37.021830 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:37.021628 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z859\" (UniqueName: \"kubernetes.io/projected/56c4f9e0-ecab-47ec-aea2-3f5ab302d779-kube-api-access-5z859\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:42:37.021830 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:37.021639 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56c4f9e0-ecab-47ec-aea2-3f5ab302d779-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:42:37.648058 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:37.648024 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3dbd-ngqkb" event={"ID":"56c4f9e0-ecab-47ec-aea2-3f5ab302d779","Type":"ContainerDied","Data":"0751a24066999b00bb74598e95c9ff32a212ce211f1038e04de81b76c4733294"} Oct 30 01:42:37.648441 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:37.648075 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0751a24066999b00bb74598e95c9ff32a212ce211f1038e04de81b76c4733294" Oct 30 01:42:37.648441 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:37.648124 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3dbd-ngqkb" Oct 30 01:42:37.656491 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:37.656457 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" event={"ID":"05ffea96-f1dc-407d-ad3e-df11499478ce","Type":"ContainerStarted","Data":"c33e6852a52071dc259863931dd9455f22fafa846a304082da471671bc0d167f"} Oct 30 01:42:37.869456 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:37.869353 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:38.644584 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:38.644544 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:42:38.644843 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:38.644765 6572 patch_prober.go:29] interesting pod/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 30 01:42:38.644843 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:38.644819 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" podUID="05ffea96-f1dc-407d-ad3e-df11499478ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 30 01:42:39.644190 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:39.644149 6572 patch_prober.go:29] interesting pod/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 30 01:42:39.644595 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:39.644209 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" podUID="05ffea96-f1dc-407d-ad3e-df11499478ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 30 01:42:39.958457 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:39.958420 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:40.644858 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:40.644819 6572 patch_prober.go:29] interesting pod/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 30 01:42:40.645300 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:40.644871 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" podUID="05ffea96-f1dc-407d-ad3e-df11499478ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 30 01:42:41.644452 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:41.644419 6572 patch_prober.go:29] interesting pod/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 30 01:42:41.644638 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:41.644467 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" podUID="05ffea96-f1dc-407d-ad3e-df11499478ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 30 01:42:42.416770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:42.412113 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:42.644510 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:42.644472 6572 patch_prober.go:29] interesting pod/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 30 01:42:42.644698 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:42.644539 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" podUID="05ffea96-f1dc-407d-ad3e-df11499478ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 30 01:42:42.674140 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:42.674048 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:42.869067 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:42.869030 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:43.650283 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:43.650250 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:42:43.650824 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:43.650652 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:42:43.651596 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:43.651577 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:42:44.504675 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:44.504645 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:44.688214 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:42:44.688169 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:42:46.233192 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:46.233134 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:48.235051 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:48.235015 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:49.411827 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:49.411791 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-5cd7b66fdc-qxr68"] Oct 30 01:42:49.476398 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:49.476362 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:49.704214 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:49.704128 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:49.898691 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:49.898657 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:50.753682 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:50.753642 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:50.936490 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:50.936456 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:51.265770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:51.263277 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:52.013941 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:52.013904 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:53.928388 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:53.928338 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:54.412297 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:54.412261 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-5cd7b66fdc-qxr68" podUID="e7ec7abc-07c7-4248-8d21-632673feeece" containerName="payload-pre-processing" containerID="cri-o://9e923dd131c4a3e95bf6b480fd3ac4272ba4f7ddf8035db2f3030bb3a1dd1c1c" gracePeriod=25 Oct 30 01:42:54.708033 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:54.708010 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-5cd7b66fdc-qxr68" Oct 30 01:42:54.798025 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:54.797979 6572 generic.go:356] "Generic (PLEG): container finished" podID="e7ec7abc-07c7-4248-8d21-632673feeece" containerID="9e923dd131c4a3e95bf6b480fd3ac4272ba4f7ddf8035db2f3030bb3a1dd1c1c" exitCode=0 Oct 30 01:42:54.798793 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:54.798045 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-5cd7b66fdc-qxr68" event={"ID":"e7ec7abc-07c7-4248-8d21-632673feeece","Type":"ContainerDied","Data":"9e923dd131c4a3e95bf6b480fd3ac4272ba4f7ddf8035db2f3030bb3a1dd1c1c"} Oct 30 01:42:54.798793 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:54.798088 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-5cd7b66fdc-qxr68" Oct 30 01:42:54.798793 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:54.798102 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-5cd7b66fdc-qxr68" event={"ID":"e7ec7abc-07c7-4248-8d21-632673feeece","Type":"ContainerDied","Data":"8d73b17e11dbbd5efacd5ca12868b2404ce5741528e65a2c8210bb67fa30ccc1"} Oct 30 01:42:54.798793 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:54.798125 6572 scope.go:122] "RemoveContainer" containerID="9e923dd131c4a3e95bf6b480fd3ac4272ba4f7ddf8035db2f3030bb3a1dd1c1c" Oct 30 01:42:54.809593 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:54.809563 6572 scope.go:122] "RemoveContainer" containerID="9e923dd131c4a3e95bf6b480fd3ac4272ba4f7ddf8035db2f3030bb3a1dd1c1c" Oct 30 01:42:54.809947 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:42:54.809923 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e923dd131c4a3e95bf6b480fd3ac4272ba4f7ddf8035db2f3030bb3a1dd1c1c\": container with ID starting with 9e923dd131c4a3e95bf6b480fd3ac4272ba4f7ddf8035db2f3030bb3a1dd1c1c not found: ID does not exist" containerID="9e923dd131c4a3e95bf6b480fd3ac4272ba4f7ddf8035db2f3030bb3a1dd1c1c" Oct 30 01:42:54.810029 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:54.809959 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e923dd131c4a3e95bf6b480fd3ac4272ba4f7ddf8035db2f3030bb3a1dd1c1c"} err="failed to get container status \"9e923dd131c4a3e95bf6b480fd3ac4272ba4f7ddf8035db2f3030bb3a1dd1c1c\": rpc error: code = NotFound desc = could not find container \"9e923dd131c4a3e95bf6b480fd3ac4272ba4f7ddf8035db2f3030bb3a1dd1c1c\": container with ID starting with 9e923dd131c4a3e95bf6b480fd3ac4272ba4f7ddf8035db2f3030bb3a1dd1c1c not found: ID does not exist" Oct 30 01:42:54.830907 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:54.830882 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7ec7abc-07c7-4248-8d21-632673feeece-kube-api-access-82wlq\" (UniqueName: \"kubernetes.io/projected/e7ec7abc-07c7-4248-8d21-632673feeece-kube-api-access-82wlq\") pod \"e7ec7abc-07c7-4248-8d21-632673feeece\" (UID: \"e7ec7abc-07c7-4248-8d21-632673feeece\") " Oct 30 01:42:54.831023 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:54.830975 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7ec7abc-07c7-4248-8d21-632673feeece-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7ec7abc-07c7-4248-8d21-632673feeece-plugins-config-volume\") pod \"e7ec7abc-07c7-4248-8d21-632673feeece\" (UID: \"e7ec7abc-07c7-4248-8d21-632673feeece\") " Oct 30 01:42:54.831279 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:54.831258 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7ec7abc-07c7-4248-8d21-632673feeece-plugins-config-volume" pod "e7ec7abc-07c7-4248-8d21-632673feeece" (UID: "e7ec7abc-07c7-4248-8d21-632673feeece"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:42:54.831479 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:54.831466 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7ec7abc-07c7-4248-8d21-632673feeece-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:42:54.832909 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:54.832890 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7ec7abc-07c7-4248-8d21-632673feeece-kube-api-access-82wlq" pod "e7ec7abc-07c7-4248-8d21-632673feeece" (UID: "e7ec7abc-07c7-4248-8d21-632673feeece"). InnerVolumeSpecName "kube-api-access-82wlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:42:54.933276 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:54.933202 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82wlq\" (UniqueName: \"kubernetes.io/projected/e7ec7abc-07c7-4248-8d21-632673feeece-kube-api-access-82wlq\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:42:55.142506 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:55.142469 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-5cd7b66fdc-qxr68"] Oct 30 01:42:55.153608 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:55.153582 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-5cd7b66fdc-qxr68"] Oct 30 01:42:55.630023 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:55.629990 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9477f07e-mfh26"] Oct 30 01:42:55.630588 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:55.630574 6572 cpu_manager.go:414] "containerMap: removing container" podUID="56c4f9e0-ecab-47ec-aea2-3f5ab302d779" containerName="revoke-keys" Oct 30 01:42:55.630641 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:55.630589 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56c4f9e0-ecab-47ec-aea2-3f5ab302d779" containerName="revoke-keys" Oct 30 01:42:55.630641 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:55.630633 6572 cpu_manager.go:414] "containerMap: removing container" podUID="e7ec7abc-07c7-4248-8d21-632673feeece" containerName="payload-pre-processing" Oct 30 01:42:55.630711 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:55.630639 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7ec7abc-07c7-4248-8d21-632673feeece" containerName="payload-pre-processing" Oct 30 01:42:55.630765 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:55.630731 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7ec7abc-07c7-4248-8d21-632673feeece" containerName="payload-pre-processing" Oct 30 01:42:55.630765 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:55.630741 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="56c4f9e0-ecab-47ec-aea2-3f5ab302d779" containerName="revoke-keys" Oct 30 01:42:55.635644 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:55.635623 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9477f07e-mfh26" Oct 30 01:42:55.653617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:55.653593 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9477f07e-mfh26"] Oct 30 01:42:55.660606 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:55.660558 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:42:55.660606 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:55.660585 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:42:55.660859 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:55.660695 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:42:55.661335 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:55.661316 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:42:55.744780 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:55.744079 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806c6aa2-26f5-460b-be1e-e7112f9b679d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9477f07e-mfh26\" (UID: \"806c6aa2-26f5-460b-be1e-e7112f9b679d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9477f07e-mfh26" Oct 30 01:42:55.744780 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:55.744246 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806c6aa2-26f5-460b-be1e-e7112f9b679d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9477f07e-mfh26\" (UID: \"806c6aa2-26f5-460b-be1e-e7112f9b679d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9477f07e-mfh26" Oct 30 01:42:55.744780 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:55.744356 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdcjf\" (UniqueName: \"kubernetes.io/projected/806c6aa2-26f5-460b-be1e-e7112f9b679d-kube-api-access-gdcjf\") pod \"maas-api-revoke-sub-9477f07e-mfh26\" (UID: \"806c6aa2-26f5-460b-be1e-e7112f9b679d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9477f07e-mfh26" Oct 30 01:42:55.847653 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:55.847601 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806c6aa2-26f5-460b-be1e-e7112f9b679d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9477f07e-mfh26\" (UID: \"806c6aa2-26f5-460b-be1e-e7112f9b679d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9477f07e-mfh26" Oct 30 01:42:55.847918 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:55.847774 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806c6aa2-26f5-460b-be1e-e7112f9b679d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9477f07e-mfh26\" (UID: \"806c6aa2-26f5-460b-be1e-e7112f9b679d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9477f07e-mfh26" Oct 30 01:42:55.847918 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:55.847869 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdcjf\" (UniqueName: \"kubernetes.io/projected/806c6aa2-26f5-460b-be1e-e7112f9b679d-kube-api-access-gdcjf\") pod \"maas-api-revoke-sub-9477f07e-mfh26\" (UID: \"806c6aa2-26f5-460b-be1e-e7112f9b679d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9477f07e-mfh26" Oct 30 01:42:55.849551 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:55.849149 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806c6aa2-26f5-460b-be1e-e7112f9b679d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9477f07e-mfh26\" (UID: \"806c6aa2-26f5-460b-be1e-e7112f9b679d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9477f07e-mfh26" Oct 30 01:42:55.877719 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:55.877681 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdcjf\" (UniqueName: \"kubernetes.io/projected/806c6aa2-26f5-460b-be1e-e7112f9b679d-kube-api-access-gdcjf\") pod \"maas-api-revoke-sub-9477f07e-mfh26\" (UID: \"806c6aa2-26f5-460b-be1e-e7112f9b679d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9477f07e-mfh26" Oct 30 01:42:55.878334 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:55.878309 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806c6aa2-26f5-460b-be1e-e7112f9b679d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9477f07e-mfh26\" (UID: \"806c6aa2-26f5-460b-be1e-e7112f9b679d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9477f07e-mfh26" Oct 30 01:42:55.950937 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:55.950899 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9477f07e-mfh26" Oct 30 01:42:56.227955 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:56.227864 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:56.245322 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:56.245297 6572 scope.go:122] "RemoveContainer" containerID="fa17973dd266f07f6fbe84cee9209f1b8126a6c7dc0a1b173465fa5995d007a6" Oct 30 01:42:56.258863 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:56.258818 6572 scope.go:122] "RemoveContainer" containerID="9cb8fac76441e5ac79431475acb6101dc64ec9f22432800a78e7ebba6d83bad2" Oct 30 01:42:56.279032 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:56.277365 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9477f07e-mfh26"] Oct 30 01:42:56.452782 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:56.452653 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:56.544432 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:56.544357 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7ec7abc-07c7-4248-8d21-632673feeece" path="/var/lib/kubelet/pods/e7ec7abc-07c7-4248-8d21-632673feeece/volumes" Oct 30 01:42:56.838801 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:56.838681 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9477f07e-mfh26" event={"ID":"806c6aa2-26f5-460b-be1e-e7112f9b679d","Type":"ContainerStarted","Data":"fb0c84f36f550373e03817d193f8da2ac1c307e95d9385fbf7de1d4303d0715b"} Oct 30 01:42:56.838801 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:56.838738 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9477f07e-mfh26" event={"ID":"806c6aa2-26f5-460b-be1e-e7112f9b679d","Type":"ContainerStarted","Data":"504e7e7d8e86318a007e0e800370faa8d5421781f0268c89a39d10a38bc55117"} Oct 30 01:42:56.897930 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:56.897876 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9477f07e-mfh26" podStartSLOduration=1.897862737 podStartE2EDuration="1.897862737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:42:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:42:56.895484232 +0000 UTC m=+1502.471909635" watchObservedRunningTime="2026-10-30 01:42:56.897862737 +0000 UTC m=+1502.474288139" Oct 30 01:42:57.848604 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:57.848570 6572 generic.go:356] "Generic (PLEG): container finished" podID="806c6aa2-26f5-460b-be1e-e7112f9b679d" containerID="fb0c84f36f550373e03817d193f8da2ac1c307e95d9385fbf7de1d4303d0715b" exitCode=0 Oct 30 01:42:57.849060 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:57.848659 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9477f07e-mfh26" event={"ID":"806c6aa2-26f5-460b-be1e-e7112f9b679d","Type":"ContainerDied","Data":"fb0c84f36f550373e03817d193f8da2ac1c307e95d9385fbf7de1d4303d0715b"} Oct 30 01:42:58.437527 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:58.437489 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:59.012624 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:59.012595 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9477f07e-mfh26" Oct 30 01:42:59.101213 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:59.101177 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/806c6aa2-26f5-460b-be1e-e7112f9b679d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806c6aa2-26f5-460b-be1e-e7112f9b679d-maas-api-xks-ca\") pod \"806c6aa2-26f5-460b-be1e-e7112f9b679d\" (UID: \"806c6aa2-26f5-460b-be1e-e7112f9b679d\") " Oct 30 01:42:59.101440 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:59.101282 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/806c6aa2-26f5-460b-be1e-e7112f9b679d-kube-api-access-gdcjf\" (UniqueName: \"kubernetes.io/projected/806c6aa2-26f5-460b-be1e-e7112f9b679d-kube-api-access-gdcjf\") pod \"806c6aa2-26f5-460b-be1e-e7112f9b679d\" (UID: \"806c6aa2-26f5-460b-be1e-e7112f9b679d\") " Oct 30 01:42:59.101440 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:59.101427 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/806c6aa2-26f5-460b-be1e-e7112f9b679d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806c6aa2-26f5-460b-be1e-e7112f9b679d-maas-api-openshift-ca\") pod \"806c6aa2-26f5-460b-be1e-e7112f9b679d\" (UID: \"806c6aa2-26f5-460b-be1e-e7112f9b679d\") " Oct 30 01:42:59.102034 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:59.101933 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/806c6aa2-26f5-460b-be1e-e7112f9b679d-maas-api-openshift-ca" pod "806c6aa2-26f5-460b-be1e-e7112f9b679d" (UID: "806c6aa2-26f5-460b-be1e-e7112f9b679d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:42:59.102169 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:59.102153 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/806c6aa2-26f5-460b-be1e-e7112f9b679d-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:42:59.103524 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:59.103494 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/806c6aa2-26f5-460b-be1e-e7112f9b679d-maas-api-xks-ca" pod "806c6aa2-26f5-460b-be1e-e7112f9b679d" (UID: "806c6aa2-26f5-460b-be1e-e7112f9b679d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:42:59.103620 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:59.103553 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/806c6aa2-26f5-460b-be1e-e7112f9b679d-kube-api-access-gdcjf" pod "806c6aa2-26f5-460b-be1e-e7112f9b679d" (UID: "806c6aa2-26f5-460b-be1e-e7112f9b679d"). InnerVolumeSpecName "kube-api-access-gdcjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:42:59.203256 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:59.203220 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/806c6aa2-26f5-460b-be1e-e7112f9b679d-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:42:59.203256 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:59.203253 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdcjf\" (UniqueName: \"kubernetes.io/projected/806c6aa2-26f5-460b-be1e-e7112f9b679d-kube-api-access-gdcjf\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:42:59.303587 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:59.303541 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:59.520458 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:59.520425 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:59.875776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:59.875671 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9477f07e-mfh26" Oct 30 01:42:59.875776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:59.875693 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9477f07e-mfh26" event={"ID":"806c6aa2-26f5-460b-be1e-e7112f9b679d","Type":"ContainerDied","Data":"504e7e7d8e86318a007e0e800370faa8d5421781f0268c89a39d10a38bc55117"} Oct 30 01:42:59.875988 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:42:59.875812 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="504e7e7d8e86318a007e0e800370faa8d5421781f0268c89a39d10a38bc55117" Oct 30 01:43:00.291501 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:00.291459 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:02.036436 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:02.036400 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:03.736950 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:03.736913 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:04.876336 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:04.876301 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:05.505126 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:05.505078 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:07.235291 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:07.235258 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:09.510323 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:09.510291 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:10.988911 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:10.988803 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:11.178039 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:11.178002 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:11.441444 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:11.441409 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:12.926062 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:12.926029 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:13.142665 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:13.142630 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:13.300801 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:13.300698 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-9fc8b47c5-wxtb4"] Oct 30 01:43:13.301409 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:13.301392 6572 cpu_manager.go:414] "containerMap: removing container" podUID="806c6aa2-26f5-460b-be1e-e7112f9b679d" containerName="revoke-keys" Oct 30 01:43:13.301453 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:13.301411 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="806c6aa2-26f5-460b-be1e-e7112f9b679d" containerName="revoke-keys" Oct 30 01:43:13.301513 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:13.301504 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="806c6aa2-26f5-460b-be1e-e7112f9b679d" containerName="revoke-keys" Oct 30 01:43:13.305375 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:13.305351 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-9fc8b47c5-wxtb4" Oct 30 01:43:13.321532 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:13.321499 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ea0fc4\"" Oct 30 01:43:13.338223 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:13.338193 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-9fc8b47c5-wxtb4"] Oct 30 01:43:13.390863 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:13.390827 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-648f7c96c4-8jp52"] Oct 30 01:43:13.396299 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:13.396271 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-648f7c96c4-8jp52" Oct 30 01:43:13.572405 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:13.572304 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8kmt\" (UniqueName: \"kubernetes.io/projected/69aaaf90-0480-4fe7-bf6b-313ddb1f0feb-kube-api-access-s8kmt\") pod \"payload-pre-processing-e2e-ipp-b-ea0fc4-9fc8b47c5-wxtb4\" (UID: \"69aaaf90-0480-4fe7-bf6b-313ddb1f0feb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-9fc8b47c5-wxtb4" Oct 30 01:43:13.572588 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:13.572520 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69aaaf90-0480-4fe7-bf6b-313ddb1f0feb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ea0fc4-9fc8b47c5-wxtb4\" (UID: \"69aaaf90-0480-4fe7-bf6b-313ddb1f0feb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-9fc8b47c5-wxtb4" Oct 30 01:43:13.577673 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:13.574816 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ea0fc4-dockercfg-8m2k6\"" Oct 30 01:43:13.595474 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:13.595442 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-648f7c96c4-8jp52"] Oct 30 01:43:13.674159 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:13.674121 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/864805b2-2538-4ca4-89c8-f4b5e756c0a4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ea0fc4-648f7c96c4-8jp52\" (UID: \"864805b2-2538-4ca4-89c8-f4b5e756c0a4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-648f7c96c4-8jp52" Oct 30 01:43:13.674358 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:13.674185 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8kmt\" (UniqueName: \"kubernetes.io/projected/69aaaf90-0480-4fe7-bf6b-313ddb1f0feb-kube-api-access-s8kmt\") pod \"payload-pre-processing-e2e-ipp-b-ea0fc4-9fc8b47c5-wxtb4\" (UID: \"69aaaf90-0480-4fe7-bf6b-313ddb1f0feb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-9fc8b47c5-wxtb4" Oct 30 01:43:13.674411 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:13.674382 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w976n\" (UniqueName: \"kubernetes.io/projected/864805b2-2538-4ca4-89c8-f4b5e756c0a4-kube-api-access-w976n\") pod \"payload-processing-e2e-ipp-b-ea0fc4-648f7c96c4-8jp52\" (UID: \"864805b2-2538-4ca4-89c8-f4b5e756c0a4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-648f7c96c4-8jp52" Oct 30 01:43:13.674451 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:13.674433 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69aaaf90-0480-4fe7-bf6b-313ddb1f0feb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ea0fc4-9fc8b47c5-wxtb4\" (UID: \"69aaaf90-0480-4fe7-bf6b-313ddb1f0feb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-9fc8b47c5-wxtb4" Oct 30 01:43:13.675200 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:13.675179 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69aaaf90-0480-4fe7-bf6b-313ddb1f0feb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ea0fc4-9fc8b47c5-wxtb4\" (UID: \"69aaaf90-0480-4fe7-bf6b-313ddb1f0feb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-9fc8b47c5-wxtb4" Oct 30 01:43:13.693407 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:13.693372 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8kmt\" (UniqueName: \"kubernetes.io/projected/69aaaf90-0480-4fe7-bf6b-313ddb1f0feb-kube-api-access-s8kmt\") pod \"payload-pre-processing-e2e-ipp-b-ea0fc4-9fc8b47c5-wxtb4\" (UID: \"69aaaf90-0480-4fe7-bf6b-313ddb1f0feb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-9fc8b47c5-wxtb4" Oct 30 01:43:13.759555 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:13.759464 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:13.776212 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:13.776170 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/864805b2-2538-4ca4-89c8-f4b5e756c0a4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ea0fc4-648f7c96c4-8jp52\" (UID: \"864805b2-2538-4ca4-89c8-f4b5e756c0a4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-648f7c96c4-8jp52" Oct 30 01:43:13.776707 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:13.776673 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w976n\" (UniqueName: \"kubernetes.io/projected/864805b2-2538-4ca4-89c8-f4b5e756c0a4-kube-api-access-w976n\") pod \"payload-processing-e2e-ipp-b-ea0fc4-648f7c96c4-8jp52\" (UID: \"864805b2-2538-4ca4-89c8-f4b5e756c0a4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-648f7c96c4-8jp52" Oct 30 01:43:13.777263 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:13.777238 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/864805b2-2538-4ca4-89c8-f4b5e756c0a4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ea0fc4-648f7c96c4-8jp52\" (UID: \"864805b2-2538-4ca4-89c8-f4b5e756c0a4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-648f7c96c4-8jp52" Oct 30 01:43:13.795806 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:13.795702 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w976n\" (UniqueName: \"kubernetes.io/projected/864805b2-2538-4ca4-89c8-f4b5e756c0a4-kube-api-access-w976n\") pod \"payload-processing-e2e-ipp-b-ea0fc4-648f7c96c4-8jp52\" (UID: \"864805b2-2538-4ca4-89c8-f4b5e756c0a4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-648f7c96c4-8jp52" Oct 30 01:43:13.865057 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:13.864969 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-648f7c96c4-8jp52" Oct 30 01:43:13.920216 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:13.920173 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-9fc8b47c5-wxtb4" Oct 30 01:43:14.193336 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:14.193302 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-648f7c96c4-8jp52"] Oct 30 01:43:14.199260 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:43:14.199226 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod864805b2_2538_4ca4_89c8_f4b5e756c0a4.slice/crio-4daca96da2a4a53019676f58e32878c4d974057bf892ae6ec6d2b2e8c247972a WatchSource:0}: Error finding container 4daca96da2a4a53019676f58e32878c4d974057bf892ae6ec6d2b2e8c247972a: Status 404 returned error can't find the container with id 4daca96da2a4a53019676f58e32878c4d974057bf892ae6ec6d2b2e8c247972a Oct 30 01:43:14.203594 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:14.203560 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:14.254179 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:14.254147 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-9fc8b47c5-wxtb4"] Oct 30 01:43:14.259169 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:43:14.259139 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69aaaf90_0480_4fe7_bf6b_313ddb1f0feb.slice/crio-77043ebf4b468cb68004e79251207cce7e1949341d59c9fb3beadcb553876ba7 WatchSource:0}: Error finding container 77043ebf4b468cb68004e79251207cce7e1949341d59c9fb3beadcb553876ba7: Status 404 returned error can't find the container with id 77043ebf4b468cb68004e79251207cce7e1949341d59c9fb3beadcb553876ba7 Oct 30 01:43:14.689839 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:43:14.689799 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:43:14.963895 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:14.963858 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:15.089369 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:15.089329 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-9fc8b47c5-wxtb4" event={"ID":"69aaaf90-0480-4fe7-bf6b-313ddb1f0feb","Type":"ContainerStarted","Data":"9f222c3a7682b4762b8eab2603a68ef36d034a686910c1c553550932de42babc"} Oct 30 01:43:15.089599 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:15.089394 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-9fc8b47c5-wxtb4" event={"ID":"69aaaf90-0480-4fe7-bf6b-313ddb1f0feb","Type":"ContainerStarted","Data":"77043ebf4b468cb68004e79251207cce7e1949341d59c9fb3beadcb553876ba7"} Oct 30 01:43:15.089599 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:15.089560 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-9fc8b47c5-wxtb4" Oct 30 01:43:15.091333 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:15.091308 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-648f7c96c4-8jp52" event={"ID":"864805b2-2538-4ca4-89c8-f4b5e756c0a4","Type":"ContainerStarted","Data":"324018c10e751618c08e963be7cfdc08b9bcb62428cec9788b2ee8cdb9f2727a"} Oct 30 01:43:15.091476 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:15.091356 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-648f7c96c4-8jp52" Oct 30 01:43:15.091476 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:15.091374 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-648f7c96c4-8jp52" event={"ID":"864805b2-2538-4ca4-89c8-f4b5e756c0a4","Type":"ContainerStarted","Data":"4daca96da2a4a53019676f58e32878c4d974057bf892ae6ec6d2b2e8c247972a"} Oct 30 01:43:15.127572 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:15.127511 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-9fc8b47c5-wxtb4" podStartSLOduration=2.127494934 podStartE2EDuration="2.127494934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:43:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:43:15.126186465 +0000 UTC m=+1520.702611877" watchObservedRunningTime="2026-10-30 01:43:15.127494934 +0000 UTC m=+1520.703920335" Oct 30 01:43:15.169047 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:15.168985 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-648f7c96c4-8jp52" podStartSLOduration=2.168965665 podStartE2EDuration="2.168965665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:43:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:43:15.16445864 +0000 UTC m=+1520.740884043" watchObservedRunningTime="2026-10-30 01:43:15.168965665 +0000 UTC m=+1520.745391066" Oct 30 01:43:15.532780 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:15.531734 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:15.726523 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:15.726481 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:16.087162 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:16.087117 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:17.927393 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:17.927355 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:19.333725 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:19.333686 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:19.537241 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:19.537201 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:20.439774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:20.436382 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:21.026051 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:21.026014 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:21.290838 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:21.278815 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:22.363352 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:22.363304 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:22.599967 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:22.599927 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:23.880373 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:23.880337 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:23.898649 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:23.898606 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6fbdd58c6d-5xdjd"] Oct 30 01:43:23.898883 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:23.898858 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6fbdd58c6d-5xdjd" podUID="7d90b45f-4155-4aee-a02e-b7a69295a8a7" containerName="payload-processing" containerID="cri-o://8fc13abda953cd92747a44b22ea2cee04a467286c4ca36f7fc25f79babe51707" gracePeriod=30 Oct 30 01:43:23.942262 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:23.942228 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-85bd8d7b8c-bhtcj"] Oct 30 01:43:23.942468 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:23.942435 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-85bd8d7b8c-bhtcj" podUID="75596710-fb32-477b-93e7-37f13e5b0099" containerName="payload-pre-processing" containerID="cri-o://4720f94b9237b7ebbe152d5e48f77cf1bf32264f4c3418c6c9933b1e2caa010a" gracePeriod=30 Oct 30 01:43:24.064615 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:24.064453 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:24.183397 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:24.181577 6572 generic.go:356] "Generic (PLEG): container finished" podID="7d90b45f-4155-4aee-a02e-b7a69295a8a7" containerID="8fc13abda953cd92747a44b22ea2cee04a467286c4ca36f7fc25f79babe51707" exitCode=0 Oct 30 01:43:24.183397 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:24.181782 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6fbdd58c6d-5xdjd" event={"ID":"7d90b45f-4155-4aee-a02e-b7a69295a8a7","Type":"ContainerDied","Data":"8fc13abda953cd92747a44b22ea2cee04a467286c4ca36f7fc25f79babe51707"} Oct 30 01:43:24.184430 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:24.184068 6572 generic.go:356] "Generic (PLEG): container finished" podID="75596710-fb32-477b-93e7-37f13e5b0099" containerID="4720f94b9237b7ebbe152d5e48f77cf1bf32264f4c3418c6c9933b1e2caa010a" exitCode=0 Oct 30 01:43:24.184430 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:24.184318 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-85bd8d7b8c-bhtcj" event={"ID":"75596710-fb32-477b-93e7-37f13e5b0099","Type":"ContainerDied","Data":"4720f94b9237b7ebbe152d5e48f77cf1bf32264f4c3418c6c9933b1e2caa010a"} Oct 30 01:43:24.389845 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:24.385114 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:24.479233 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:24.479189 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6fbdd58c6d-5xdjd" Oct 30 01:43:24.482408 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:24.482384 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-85bd8d7b8c-bhtcj" Oct 30 01:43:24.489924 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:24.489892 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75596710-fb32-477b-93e7-37f13e5b0099-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75596710-fb32-477b-93e7-37f13e5b0099-plugins-config-volume\") pod \"75596710-fb32-477b-93e7-37f13e5b0099\" (UID: \"75596710-fb32-477b-93e7-37f13e5b0099\") " Oct 30 01:43:24.490087 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:24.489940 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75596710-fb32-477b-93e7-37f13e5b0099-kube-api-access-2xrc8\" (UniqueName: \"kubernetes.io/projected/75596710-fb32-477b-93e7-37f13e5b0099-kube-api-access-2xrc8\") pod \"75596710-fb32-477b-93e7-37f13e5b0099\" (UID: \"75596710-fb32-477b-93e7-37f13e5b0099\") " Oct 30 01:43:24.490087 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:24.490054 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d90b45f-4155-4aee-a02e-b7a69295a8a7-kube-api-access-dkjtz\" (UniqueName: \"kubernetes.io/projected/7d90b45f-4155-4aee-a02e-b7a69295a8a7-kube-api-access-dkjtz\") pod \"7d90b45f-4155-4aee-a02e-b7a69295a8a7\" (UID: \"7d90b45f-4155-4aee-a02e-b7a69295a8a7\") " Oct 30 01:43:24.490234 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:24.490216 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d90b45f-4155-4aee-a02e-b7a69295a8a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d90b45f-4155-4aee-a02e-b7a69295a8a7-plugins-config-volume\") pod \"7d90b45f-4155-4aee-a02e-b7a69295a8a7\" (UID: \"7d90b45f-4155-4aee-a02e-b7a69295a8a7\") " Oct 30 01:43:24.490324 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:24.490303 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75596710-fb32-477b-93e7-37f13e5b0099-plugins-config-volume" pod "75596710-fb32-477b-93e7-37f13e5b0099" (UID: "75596710-fb32-477b-93e7-37f13e5b0099"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:43:24.490760 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:24.490721 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75596710-fb32-477b-93e7-37f13e5b0099-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:24.490879 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:24.490799 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d90b45f-4155-4aee-a02e-b7a69295a8a7-plugins-config-volume" pod "7d90b45f-4155-4aee-a02e-b7a69295a8a7" (UID: "7d90b45f-4155-4aee-a02e-b7a69295a8a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:43:24.492350 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:24.492321 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75596710-fb32-477b-93e7-37f13e5b0099-kube-api-access-2xrc8" pod "75596710-fb32-477b-93e7-37f13e5b0099" (UID: "75596710-fb32-477b-93e7-37f13e5b0099"). InnerVolumeSpecName "kube-api-access-2xrc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:43:24.492710 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:24.492678 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d90b45f-4155-4aee-a02e-b7a69295a8a7-kube-api-access-dkjtz" pod "7d90b45f-4155-4aee-a02e-b7a69295a8a7" (UID: "7d90b45f-4155-4aee-a02e-b7a69295a8a7"). InnerVolumeSpecName "kube-api-access-dkjtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:43:24.553904 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:24.553704 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3dbd-ngqkb"] Oct 30 01:43:24.555614 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:24.555590 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6b3dbd-ngqkb"] Oct 30 01:43:24.584983 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:24.579065 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:24.592613 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:24.592574 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xrc8\" (UniqueName: \"kubernetes.io/projected/75596710-fb32-477b-93e7-37f13e5b0099-kube-api-access-2xrc8\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:24.592613 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:24.592613 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkjtz\" (UniqueName: \"kubernetes.io/projected/7d90b45f-4155-4aee-a02e-b7a69295a8a7-kube-api-access-dkjtz\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:24.592999 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:24.592629 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d90b45f-4155-4aee-a02e-b7a69295a8a7-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:24.766977 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:24.766944 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:25.148724 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:25.148613 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:25.194928 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:25.194894 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-85bd8d7b8c-bhtcj" Oct 30 01:43:25.194928 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:25.194919 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-85bd8d7b8c-bhtcj" event={"ID":"75596710-fb32-477b-93e7-37f13e5b0099","Type":"ContainerDied","Data":"7e3714fb74c66f82498464122ae4a8a5556d0f3f6d6774414f5184a177200404"} Oct 30 01:43:25.195198 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:25.194980 6572 scope.go:122] "RemoveContainer" containerID="4720f94b9237b7ebbe152d5e48f77cf1bf32264f4c3418c6c9933b1e2caa010a" Oct 30 01:43:25.196655 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:25.196634 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6fbdd58c6d-5xdjd" Oct 30 01:43:25.196655 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:25.196647 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6fbdd58c6d-5xdjd" event={"ID":"7d90b45f-4155-4aee-a02e-b7a69295a8a7","Type":"ContainerDied","Data":"1b540790c3f26379176ae718d988c8df29180ed0003815a6c96724125882ffcf"} Oct 30 01:43:25.232419 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:25.231977 6572 scope.go:122] "RemoveContainer" containerID="8fc13abda953cd92747a44b22ea2cee04a467286c4ca36f7fc25f79babe51707" Oct 30 01:43:25.255055 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:25.255016 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6fbdd58c6d-5xdjd"] Oct 30 01:43:25.266138 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:25.266103 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6fbdd58c6d-5xdjd"] Oct 30 01:43:25.290928 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:25.290893 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-85bd8d7b8c-bhtcj"] Oct 30 01:43:25.294683 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:25.294650 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-85bd8d7b8c-bhtcj"] Oct 30 01:43:25.388999 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:25.388944 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-85bd8d7b8c-bhtcj" podUID="75596710-fb32-477b-93e7-37f13e5b0099" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.153:9004: i/o timeout" Oct 30 01:43:26.103572 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:26.103538 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-648f7c96c4-8jp52" Oct 30 01:43:26.103770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:26.103617 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-9fc8b47c5-wxtb4" Oct 30 01:43:26.545005 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:26.544971 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56c4f9e0-ecab-47ec-aea2-3f5ab302d779" path="/var/lib/kubelet/pods/56c4f9e0-ecab-47ec-aea2-3f5ab302d779/volumes" Oct 30 01:43:26.545460 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:26.545444 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75596710-fb32-477b-93e7-37f13e5b0099" path="/var/lib/kubelet/pods/75596710-fb32-477b-93e7-37f13e5b0099/volumes" Oct 30 01:43:26.545848 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:26.545833 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d90b45f-4155-4aee-a02e-b7a69295a8a7" path="/var/lib/kubelet/pods/7d90b45f-4155-4aee-a02e-b7a69295a8a7/volumes" Oct 30 01:43:26.726776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:26.714674 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:27.159112 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:27.159079 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:27.688219 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:27.688187 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf"] Oct 30 01:43:27.688920 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:27.688888 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" podUID="6b50de5b-2782-45a9-8b90-44e3508013f8" containerName="istio-proxy" containerID="cri-o://15b6f1a4c55c7c82910ce7775a4ab791b96e3f519793e8c5b6fb46c53c74ba9f" gracePeriod=30 Oct 30 01:43:28.656795 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:28.656686 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:29.570578 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:29.570538 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:29.762563 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:29.762529 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:30.099948 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:30.099906 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3fde1c-ftk89"] Oct 30 01:43:30.100941 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:30.100916 6572 cpu_manager.go:414] "containerMap: removing container" podUID="7d90b45f-4155-4aee-a02e-b7a69295a8a7" containerName="payload-processing" Oct 30 01:43:30.101014 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:30.100945 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d90b45f-4155-4aee-a02e-b7a69295a8a7" containerName="payload-processing" Oct 30 01:43:30.101014 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:30.101010 6572 cpu_manager.go:414] "containerMap: removing container" podUID="75596710-fb32-477b-93e7-37f13e5b0099" containerName="payload-pre-processing" Oct 30 01:43:30.101084 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:30.101018 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75596710-fb32-477b-93e7-37f13e5b0099" containerName="payload-pre-processing" Oct 30 01:43:30.101216 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:30.101201 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d90b45f-4155-4aee-a02e-b7a69295a8a7" containerName="payload-processing" Oct 30 01:43:30.101253 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:30.101232 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="75596710-fb32-477b-93e7-37f13e5b0099" containerName="payload-pre-processing" Oct 30 01:43:30.107271 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:30.107245 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3fde1c-ftk89" Oct 30 01:43:30.126244 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:30.126212 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:43:30.126809 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:30.126771 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3fde1c-ftk89"] Oct 30 01:43:30.132045 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:30.132007 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:43:30.132045 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:30.132007 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:43:30.137616 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:30.137577 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:43:30.197178 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:30.197137 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8b2k\" (UniqueName: \"kubernetes.io/projected/a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c-kube-api-access-k8b2k\") pod \"maas-api-revoke-keys-e2e-praxis-a-3fde1c-ftk89\" (UID: \"a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3fde1c-ftk89" Oct 30 01:43:30.197358 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:30.197206 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3fde1c-ftk89\" (UID: \"a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3fde1c-ftk89" Oct 30 01:43:30.197441 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:30.197412 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3fde1c-ftk89\" (UID: \"a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3fde1c-ftk89" Oct 30 01:43:30.231518 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:30.231471 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:30.298475 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:30.298442 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3fde1c-ftk89\" (UID: \"a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3fde1c-ftk89" Oct 30 01:43:30.298646 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:30.298601 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8b2k\" (UniqueName: \"kubernetes.io/projected/a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c-kube-api-access-k8b2k\") pod \"maas-api-revoke-keys-e2e-praxis-a-3fde1c-ftk89\" (UID: \"a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3fde1c-ftk89" Oct 30 01:43:30.298713 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:30.298698 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3fde1c-ftk89\" (UID: \"a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3fde1c-ftk89" Oct 30 01:43:30.299384 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:30.299353 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3fde1c-ftk89\" (UID: \"a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3fde1c-ftk89" Oct 30 01:43:30.301909 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:30.301859 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3fde1c-ftk89\" (UID: \"a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3fde1c-ftk89" Oct 30 01:43:30.318412 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:30.318381 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8b2k\" (UniqueName: \"kubernetes.io/projected/a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c-kube-api-access-k8b2k\") pod \"maas-api-revoke-keys-e2e-praxis-a-3fde1c-ftk89\" (UID: \"a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3fde1c-ftk89" Oct 30 01:43:30.424229 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:30.424193 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3fde1c-ftk89" Oct 30 01:43:30.426296 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:30.426266 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:30.600977 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:30.600934 6572 patch_prober.go:29] interesting pod/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:43:30.601429 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:30.601007 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" podUID="98cd042c-e590-4f84-8acc-220357883bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:43:30.748402 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:43:30.748357 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7a0f4cd_9a7d_4108_aeb3_a5ba4ced935c.slice/crio-0ad034105704ac40023d245c44a26f9a590018d6acfae30a4bb37eeada7fe4d0 WatchSource:0}: Error finding container 0ad034105704ac40023d245c44a26f9a590018d6acfae30a4bb37eeada7fe4d0: Status 404 returned error can't find the container with id 0ad034105704ac40023d245c44a26f9a590018d6acfae30a4bb37eeada7fe4d0 Oct 30 01:43:30.749915 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:30.749886 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3fde1c-ftk89"] Oct 30 01:43:31.291550 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:31.291448 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3fde1c-ftk89" event={"ID":"a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c","Type":"ContainerStarted","Data":"8ed4abecda950aa8d35439877ccc030f41033276b85146100db4a9d4674565bd"} Oct 30 01:43:31.291550 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:31.291514 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3fde1c-ftk89" event={"ID":"a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c","Type":"ContainerStarted","Data":"0ad034105704ac40023d245c44a26f9a590018d6acfae30a4bb37eeada7fe4d0"} Oct 30 01:43:31.330239 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:31.330176 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3fde1c-ftk89" podStartSLOduration=1.330159304 podStartE2EDuration="1.330159304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:43:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:43:31.32845162 +0000 UTC m=+1536.904877027" watchObservedRunningTime="2026-10-30 01:43:31.330159304 +0000 UTC m=+1536.906584706" Oct 30 01:43:31.991875 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:31.991833 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2ac18f-wl9j6"] Oct 30 01:43:31.995988 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:31.995967 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2ac18f-wl9j6" Oct 30 01:43:32.015088 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:32.015049 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2ac18f-wl9j6"] Oct 30 01:43:32.102018 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:32.101908 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:32.127113 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:32.127073 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51763c04-1938-4d21-b5af-2f6529c64e6e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2ac18f-wl9j6\" (UID: \"51763c04-1938-4d21-b5af-2f6529c64e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2ac18f-wl9j6" Oct 30 01:43:32.127322 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:32.127147 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51763c04-1938-4d21-b5af-2f6529c64e6e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2ac18f-wl9j6\" (UID: \"51763c04-1938-4d21-b5af-2f6529c64e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2ac18f-wl9j6" Oct 30 01:43:32.127372 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:32.127318 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s8hn\" (UniqueName: \"kubernetes.io/projected/51763c04-1938-4d21-b5af-2f6529c64e6e-kube-api-access-4s8hn\") pod \"maas-api-revoke-keys-e2e-ipp-a-2ac18f-wl9j6\" (UID: \"51763c04-1938-4d21-b5af-2f6529c64e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2ac18f-wl9j6" Oct 30 01:43:32.228446 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:32.228414 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s8hn\" (UniqueName: \"kubernetes.io/projected/51763c04-1938-4d21-b5af-2f6529c64e6e-kube-api-access-4s8hn\") pod \"maas-api-revoke-keys-e2e-ipp-a-2ac18f-wl9j6\" (UID: \"51763c04-1938-4d21-b5af-2f6529c64e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2ac18f-wl9j6" Oct 30 01:43:32.228633 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:32.228540 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51763c04-1938-4d21-b5af-2f6529c64e6e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2ac18f-wl9j6\" (UID: \"51763c04-1938-4d21-b5af-2f6529c64e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2ac18f-wl9j6" Oct 30 01:43:32.228633 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:32.228585 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51763c04-1938-4d21-b5af-2f6529c64e6e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2ac18f-wl9j6\" (UID: \"51763c04-1938-4d21-b5af-2f6529c64e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2ac18f-wl9j6" Oct 30 01:43:32.229283 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:32.229256 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51763c04-1938-4d21-b5af-2f6529c64e6e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2ac18f-wl9j6\" (UID: \"51763c04-1938-4d21-b5af-2f6529c64e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2ac18f-wl9j6" Oct 30 01:43:32.231110 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:32.231086 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51763c04-1938-4d21-b5af-2f6529c64e6e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2ac18f-wl9j6\" (UID: \"51763c04-1938-4d21-b5af-2f6529c64e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2ac18f-wl9j6" Oct 30 01:43:32.247073 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:32.247046 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s8hn\" (UniqueName: \"kubernetes.io/projected/51763c04-1938-4d21-b5af-2f6529c64e6e-kube-api-access-4s8hn\") pod \"maas-api-revoke-keys-e2e-ipp-a-2ac18f-wl9j6\" (UID: \"51763c04-1938-4d21-b5af-2f6529c64e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2ac18f-wl9j6" Oct 30 01:43:32.310992 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:32.310960 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2ac18f-wl9j6" Oct 30 01:43:32.310992 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:32.310972 6572 generic.go:356] "Generic (PLEG): container finished" podID="98cd042c-e590-4f84-8acc-220357883bef" containerID="1ab66c1173fb183401523059288f77755e92e96d4c5f3f58813475a22a26c788" exitCode=0 Oct 30 01:43:32.311210 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:32.311050 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" event={"ID":"98cd042c-e590-4f84-8acc-220357883bef","Type":"ContainerDied","Data":"1ab66c1173fb183401523059288f77755e92e96d4c5f3f58813475a22a26c788"} Oct 30 01:43:32.311555 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:32.311534 6572 scope.go:122] "RemoveContainer" containerID="1ab66c1173fb183401523059288f77755e92e96d4c5f3f58813475a22a26c788" Oct 30 01:43:32.313075 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:32.313053 6572 generic.go:356] "Generic (PLEG): container finished" podID="a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c" containerID="8ed4abecda950aa8d35439877ccc030f41033276b85146100db4a9d4674565bd" exitCode=0 Oct 30 01:43:32.313180 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:32.313122 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3fde1c-ftk89" event={"ID":"a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c","Type":"ContainerDied","Data":"8ed4abecda950aa8d35439877ccc030f41033276b85146100db4a9d4674565bd"} Oct 30 01:43:32.313325 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:32.313302 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:43:32.313376 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:32.313364 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:43:32.313417 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:32.313397 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:43:32.634166 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:43:32.634126 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51763c04_1938_4d21_b5af_2f6529c64e6e.slice/crio-075df30d8904a946dedbd81b75ca23abd846c6e6d72a6fb61e457ed4c1313e0d WatchSource:0}: Error finding container 075df30d8904a946dedbd81b75ca23abd846c6e6d72a6fb61e457ed4c1313e0d: Status 404 returned error can't find the container with id 075df30d8904a946dedbd81b75ca23abd846c6e6d72a6fb61e457ed4c1313e0d Oct 30 01:43:32.634630 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:32.634605 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2ac18f-wl9j6"] Oct 30 01:43:32.939627 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:32.939603 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" Oct 30 01:43:33.040976 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.040943 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b50de5b-2782-45a9-8b90-44e3508013f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b50de5b-2782-45a9-8b90-44e3508013f8-istio-envoy\") pod \"6b50de5b-2782-45a9-8b90-44e3508013f8\" (UID: \"6b50de5b-2782-45a9-8b90-44e3508013f8\") " Oct 30 01:43:33.041429 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.041004 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b50de5b-2782-45a9-8b90-44e3508013f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b50de5b-2782-45a9-8b90-44e3508013f8-istiod-ca-cert\") pod \"6b50de5b-2782-45a9-8b90-44e3508013f8\" (UID: \"6b50de5b-2782-45a9-8b90-44e3508013f8\") " Oct 30 01:43:33.041429 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.041052 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b50de5b-2782-45a9-8b90-44e3508013f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b50de5b-2782-45a9-8b90-44e3508013f8-istio-data\") pod \"6b50de5b-2782-45a9-8b90-44e3508013f8\" (UID: \"6b50de5b-2782-45a9-8b90-44e3508013f8\") " Oct 30 01:43:33.041429 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.041077 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b50de5b-2782-45a9-8b90-44e3508013f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b50de5b-2782-45a9-8b90-44e3508013f8-workload-socket\") pod \"6b50de5b-2782-45a9-8b90-44e3508013f8\" (UID: \"6b50de5b-2782-45a9-8b90-44e3508013f8\") " Oct 30 01:43:33.041429 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.041107 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b50de5b-2782-45a9-8b90-44e3508013f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b50de5b-2782-45a9-8b90-44e3508013f8-workload-certs\") pod \"6b50de5b-2782-45a9-8b90-44e3508013f8\" (UID: \"6b50de5b-2782-45a9-8b90-44e3508013f8\") " Oct 30 01:43:33.041429 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.041234 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b50de5b-2782-45a9-8b90-44e3508013f8-istio-token\" (UniqueName: \"kubernetes.io/projected/6b50de5b-2782-45a9-8b90-44e3508013f8-istio-token\") pod \"6b50de5b-2782-45a9-8b90-44e3508013f8\" (UID: \"6b50de5b-2782-45a9-8b90-44e3508013f8\") " Oct 30 01:43:33.041429 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.041290 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b50de5b-2782-45a9-8b90-44e3508013f8-kube-api-access-2qss2\" (UniqueName: \"kubernetes.io/projected/6b50de5b-2782-45a9-8b90-44e3508013f8-kube-api-access-2qss2\") pod \"6b50de5b-2782-45a9-8b90-44e3508013f8\" (UID: \"6b50de5b-2782-45a9-8b90-44e3508013f8\") " Oct 30 01:43:33.041429 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.041332 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b50de5b-2782-45a9-8b90-44e3508013f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b50de5b-2782-45a9-8b90-44e3508013f8-credential-socket\") pod \"6b50de5b-2782-45a9-8b90-44e3508013f8\" (UID: \"6b50de5b-2782-45a9-8b90-44e3508013f8\") " Oct 30 01:43:33.041429 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.041388 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b50de5b-2782-45a9-8b90-44e3508013f8-istio-data" pod "6b50de5b-2782-45a9-8b90-44e3508013f8" (UID: "6b50de5b-2782-45a9-8b90-44e3508013f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:33.041429 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.041402 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b50de5b-2782-45a9-8b90-44e3508013f8-workload-socket" pod "6b50de5b-2782-45a9-8b90-44e3508013f8" (UID: "6b50de5b-2782-45a9-8b90-44e3508013f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:33.041944 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.041470 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b50de5b-2782-45a9-8b90-44e3508013f8-istiod-ca-cert" pod "6b50de5b-2782-45a9-8b90-44e3508013f8" (UID: "6b50de5b-2782-45a9-8b90-44e3508013f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:43:33.041944 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.041518 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b50de5b-2782-45a9-8b90-44e3508013f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b50de5b-2782-45a9-8b90-44e3508013f8-istio-podinfo\") pod \"6b50de5b-2782-45a9-8b90-44e3508013f8\" (UID: \"6b50de5b-2782-45a9-8b90-44e3508013f8\") " Oct 30 01:43:33.041944 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.041695 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b50de5b-2782-45a9-8b90-44e3508013f8-credential-socket" pod "6b50de5b-2782-45a9-8b90-44e3508013f8" (UID: "6b50de5b-2782-45a9-8b90-44e3508013f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:33.041944 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.041716 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b50de5b-2782-45a9-8b90-44e3508013f8-workload-certs" pod "6b50de5b-2782-45a9-8b90-44e3508013f8" (UID: "6b50de5b-2782-45a9-8b90-44e3508013f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:33.042183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.042160 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b50de5b-2782-45a9-8b90-44e3508013f8-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:33.042249 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.042187 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b50de5b-2782-45a9-8b90-44e3508013f8-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:33.042249 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.042201 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b50de5b-2782-45a9-8b90-44e3508013f8-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:33.042249 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.042216 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b50de5b-2782-45a9-8b90-44e3508013f8-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:33.042249 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.042230 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b50de5b-2782-45a9-8b90-44e3508013f8-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:33.043885 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.043857 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b50de5b-2782-45a9-8b90-44e3508013f8-istio-envoy" pod "6b50de5b-2782-45a9-8b90-44e3508013f8" (UID: "6b50de5b-2782-45a9-8b90-44e3508013f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:33.043968 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.043911 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b50de5b-2782-45a9-8b90-44e3508013f8-istio-podinfo" pod "6b50de5b-2782-45a9-8b90-44e3508013f8" (UID: "6b50de5b-2782-45a9-8b90-44e3508013f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:43:33.044016 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.043969 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b50de5b-2782-45a9-8b90-44e3508013f8-kube-api-access-2qss2" pod "6b50de5b-2782-45a9-8b90-44e3508013f8" (UID: "6b50de5b-2782-45a9-8b90-44e3508013f8"). InnerVolumeSpecName "kube-api-access-2qss2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:43:33.044016 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.044000 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b50de5b-2782-45a9-8b90-44e3508013f8-istio-token" pod "6b50de5b-2782-45a9-8b90-44e3508013f8" (UID: "6b50de5b-2782-45a9-8b90-44e3508013f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:43:33.143815 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.143717 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b50de5b-2782-45a9-8b90-44e3508013f8-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:33.143815 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.143817 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qss2\" (UniqueName: \"kubernetes.io/projected/6b50de5b-2782-45a9-8b90-44e3508013f8-kube-api-access-2qss2\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:33.143815 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.143832 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b50de5b-2782-45a9-8b90-44e3508013f8-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:33.144106 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.143842 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b50de5b-2782-45a9-8b90-44e3508013f8-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:33.340583 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.340540 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" event={"ID":"98cd042c-e590-4f84-8acc-220357883bef","Type":"ContainerStarted","Data":"7b7e97c2fb8ce21981618ce287f92a3a95843b1fc3c880d33c56ba354061375e"} Oct 30 01:43:33.342819 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.342780 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2ac18f-wl9j6" event={"ID":"51763c04-1938-4d21-b5af-2f6529c64e6e","Type":"ContainerStarted","Data":"2c52c569095a5c908c4a92c928c0caca17ac0870e2967605ba5b7de4b69e8fb7"} Oct 30 01:43:33.343001 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.342851 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2ac18f-wl9j6" event={"ID":"51763c04-1938-4d21-b5af-2f6529c64e6e","Type":"ContainerStarted","Data":"075df30d8904a946dedbd81b75ca23abd846c6e6d72a6fb61e457ed4c1313e0d"} Oct 30 01:43:33.344768 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.344705 6572 generic.go:356] "Generic (PLEG): container finished" podID="6b50de5b-2782-45a9-8b90-44e3508013f8" containerID="15b6f1a4c55c7c82910ce7775a4ab791b96e3f519793e8c5b6fb46c53c74ba9f" exitCode=0 Oct 30 01:43:33.344949 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.344798 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" event={"ID":"6b50de5b-2782-45a9-8b90-44e3508013f8","Type":"ContainerDied","Data":"15b6f1a4c55c7c82910ce7775a4ab791b96e3f519793e8c5b6fb46c53c74ba9f"} Oct 30 01:43:33.344949 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.344858 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" Oct 30 01:43:33.344949 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.344862 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf" event={"ID":"6b50de5b-2782-45a9-8b90-44e3508013f8","Type":"ContainerDied","Data":"edffcd27ccb93b205779b1698ca8dcfa7dedfe14e68e2f5f4ef8bdf94da0b4ec"} Oct 30 01:43:33.344949 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.344890 6572 scope.go:122] "RemoveContainer" containerID="15b6f1a4c55c7c82910ce7775a4ab791b96e3f519793e8c5b6fb46c53c74ba9f" Oct 30 01:43:33.359699 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.359673 6572 scope.go:122] "RemoveContainer" containerID="15b6f1a4c55c7c82910ce7775a4ab791b96e3f519793e8c5b6fb46c53c74ba9f" Oct 30 01:43:33.360115 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:43:33.360086 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15b6f1a4c55c7c82910ce7775a4ab791b96e3f519793e8c5b6fb46c53c74ba9f\": container with ID starting with 15b6f1a4c55c7c82910ce7775a4ab791b96e3f519793e8c5b6fb46c53c74ba9f not found: ID does not exist" containerID="15b6f1a4c55c7c82910ce7775a4ab791b96e3f519793e8c5b6fb46c53c74ba9f" Oct 30 01:43:33.360232 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.360123 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15b6f1a4c55c7c82910ce7775a4ab791b96e3f519793e8c5b6fb46c53c74ba9f"} err="failed to get container status \"15b6f1a4c55c7c82910ce7775a4ab791b96e3f519793e8c5b6fb46c53c74ba9f\": rpc error: code = NotFound desc = could not find container \"15b6f1a4c55c7c82910ce7775a4ab791b96e3f519793e8c5b6fb46c53c74ba9f\": container with ID starting with 15b6f1a4c55c7c82910ce7775a4ab791b96e3f519793e8c5b6fb46c53c74ba9f not found: ID does not exist" Oct 30 01:43:33.449357 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.448783 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf"] Oct 30 01:43:33.450316 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.450287 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-cfxpf"] Oct 30 01:43:33.474736 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.473914 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2ac18f-wl9j6" podStartSLOduration=2.473897132 podStartE2EDuration="2.473897132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:43:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:43:33.472370162 +0000 UTC m=+1539.048795578" watchObservedRunningTime="2026-10-30 01:43:33.473897132 +0000 UTC m=+1539.050322533" Oct 30 01:43:33.563443 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.563408 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3fde1c-ftk89" Oct 30 01:43:33.651323 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.651290 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c-maas-api-xks-ca\") pod \"a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c\" (UID: \"a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c\") " Oct 30 01:43:33.651507 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.651393 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c-maas-api-openshift-ca\") pod \"a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c\" (UID: \"a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c\") " Oct 30 01:43:33.651582 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.651565 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c-kube-api-access-k8b2k\" (UniqueName: \"kubernetes.io/projected/a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c-kube-api-access-k8b2k\") pod \"a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c\" (UID: \"a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c\") " Oct 30 01:43:33.651810 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.651780 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c-maas-api-openshift-ca" pod "a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c" (UID: "a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:43:33.652351 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.652319 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:33.653657 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.653633 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c-maas-api-xks-ca" pod "a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c" (UID: "a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:43:33.653796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.653771 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c-kube-api-access-k8b2k" pod "a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c" (UID: "a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c"). InnerVolumeSpecName "kube-api-access-k8b2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:43:33.754132 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.754089 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8b2k\" (UniqueName: \"kubernetes.io/projected/a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c-kube-api-access-k8b2k\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:33.754132 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.754123 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:33.894811 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:33.894637 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:34.247214 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:34.247162 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:43:34.248254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:34.248225 6572 patch_prober.go:29] interesting pod/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 30 01:43:34.248346 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:34.248293 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" podUID="98cd042c-e590-4f84-8acc-220357883bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 30 01:43:34.354261 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:34.354222 6572 generic.go:356] "Generic (PLEG): container finished" podID="51763c04-1938-4d21-b5af-2f6529c64e6e" containerID="2c52c569095a5c908c4a92c928c0caca17ac0870e2967605ba5b7de4b69e8fb7" exitCode=0 Oct 30 01:43:34.354453 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:34.354311 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2ac18f-wl9j6" event={"ID":"51763c04-1938-4d21-b5af-2f6529c64e6e","Type":"ContainerDied","Data":"2c52c569095a5c908c4a92c928c0caca17ac0870e2967605ba5b7de4b69e8fb7"} Oct 30 01:43:34.355881 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:34.355859 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3fde1c-ftk89" Oct 30 01:43:34.355998 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:34.355885 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3fde1c-ftk89" event={"ID":"a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c","Type":"ContainerDied","Data":"0ad034105704ac40023d245c44a26f9a590018d6acfae30a4bb37eeada7fe4d0"} Oct 30 01:43:34.355998 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:34.355924 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ad034105704ac40023d245c44a26f9a590018d6acfae30a4bb37eeada7fe4d0" Oct 30 01:43:34.546163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:34.546083 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b50de5b-2782-45a9-8b90-44e3508013f8" path="/var/lib/kubelet/pods/6b50de5b-2782-45a9-8b90-44e3508013f8/volumes" Oct 30 01:43:35.248081 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:35.248037 6572 patch_prober.go:29] interesting pod/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 30 01:43:35.248468 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:35.248122 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" podUID="98cd042c-e590-4f84-8acc-220357883bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 30 01:43:35.586187 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:35.586156 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2ac18f-wl9j6" Oct 30 01:43:35.686773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:35.686458 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:35.722770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:35.707287 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51763c04-1938-4d21-b5af-2f6529c64e6e-kube-api-access-4s8hn\" (UniqueName: \"kubernetes.io/projected/51763c04-1938-4d21-b5af-2f6529c64e6e-kube-api-access-4s8hn\") pod \"51763c04-1938-4d21-b5af-2f6529c64e6e\" (UID: \"51763c04-1938-4d21-b5af-2f6529c64e6e\") " Oct 30 01:43:35.722770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:35.707467 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51763c04-1938-4d21-b5af-2f6529c64e6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51763c04-1938-4d21-b5af-2f6529c64e6e-maas-api-xks-ca\") pod \"51763c04-1938-4d21-b5af-2f6529c64e6e\" (UID: \"51763c04-1938-4d21-b5af-2f6529c64e6e\") " Oct 30 01:43:35.722770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:35.707597 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51763c04-1938-4d21-b5af-2f6529c64e6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51763c04-1938-4d21-b5af-2f6529c64e6e-maas-api-openshift-ca\") pod \"51763c04-1938-4d21-b5af-2f6529c64e6e\" (UID: \"51763c04-1938-4d21-b5af-2f6529c64e6e\") " Oct 30 01:43:35.733763 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:35.730141 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51763c04-1938-4d21-b5af-2f6529c64e6e-maas-api-openshift-ca" pod "51763c04-1938-4d21-b5af-2f6529c64e6e" (UID: "51763c04-1938-4d21-b5af-2f6529c64e6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:43:35.769383 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:35.768924 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51763c04-1938-4d21-b5af-2f6529c64e6e-kube-api-access-4s8hn" pod "51763c04-1938-4d21-b5af-2f6529c64e6e" (UID: "51763c04-1938-4d21-b5af-2f6529c64e6e"). InnerVolumeSpecName "kube-api-access-4s8hn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:43:35.788916 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:35.788879 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51763c04-1938-4d21-b5af-2f6529c64e6e-maas-api-xks-ca" pod "51763c04-1938-4d21-b5af-2f6529c64e6e" (UID: "51763c04-1938-4d21-b5af-2f6529c64e6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:43:35.813935 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:35.813845 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s8hn\" (UniqueName: \"kubernetes.io/projected/51763c04-1938-4d21-b5af-2f6529c64e6e-kube-api-access-4s8hn\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:35.813935 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:35.813886 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51763c04-1938-4d21-b5af-2f6529c64e6e-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:35.813935 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:35.813903 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51763c04-1938-4d21-b5af-2f6529c64e6e-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:36.247382 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:36.247342 6572 patch_prober.go:29] interesting pod/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 30 01:43:36.247581 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:36.247398 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" podUID="98cd042c-e590-4f84-8acc-220357883bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 30 01:43:36.376272 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:36.376236 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2ac18f-wl9j6" event={"ID":"51763c04-1938-4d21-b5af-2f6529c64e6e","Type":"ContainerDied","Data":"075df30d8904a946dedbd81b75ca23abd846c6e6d72a6fb61e457ed4c1313e0d"} Oct 30 01:43:36.376667 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:36.376285 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="075df30d8904a946dedbd81b75ca23abd846c6e6d72a6fb61e457ed4c1313e0d" Oct 30 01:43:36.376667 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:36.376253 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2ac18f-wl9j6" Oct 30 01:43:37.247144 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:37.247106 6572 patch_prober.go:29] interesting pod/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 30 01:43:37.247461 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:37.247171 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" podUID="98cd042c-e590-4f84-8acc-220357883bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 30 01:43:37.866587 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:37.866552 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:38.247241 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:38.247205 6572 patch_prober.go:29] interesting pod/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 30 01:43:38.247437 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:38.247257 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" podUID="98cd042c-e590-4f84-8acc-220357883bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 30 01:43:38.897826 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:38.897788 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:39.112403 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:39.112363 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:39.251735 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:39.251696 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:43:39.252189 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:39.252176 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:43:39.252977 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:39.252959 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:43:39.547865 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:39.547733 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:40.498384 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:40.498350 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3fde1c-58cf94cd5b-nq9g2"] Oct 30 01:43:40.720946 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:40.720912 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:40.916352 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:40.916317 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:41.875867 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:41.875819 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n"] Oct 30 01:43:41.876386 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:41.876108 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" podUID="d7243362-b004-4393-8a15-8ea62c257b6d" containerName="istio-proxy" containerID="cri-o://9f775cffb5e8cb2436d7c7f18b3586d3a5d4ce1fc4e66ebed075b12a97a6b65c" gracePeriod=30 Oct 30 01:43:41.910029 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:41.909990 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:43.303127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:43.303093 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:43.738421 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:43.738375 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:43.923518 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:43.923484 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-c564cf647-d674t"] Oct 30 01:43:43.923721 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:43.923687 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-c564cf647-d674t" podUID="53166fd9-5f8c-45c4-a69a-c919a1bf0201" containerName="payload-pre-processing" containerID="cri-o://52f9bef11d73f4d2db7e350b7aca7d80e9e776fc72c29db8a59c44e778d3124f" gracePeriod=30 Oct 30 01:43:43.965796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:43.965742 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:44.102083 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:44.102013 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3fde1c-ftk89"] Oct 30 01:43:44.107166 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:44.107127 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3fde1c-ftk89"] Oct 30 01:43:44.163422 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:44.163382 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:44.208739 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:44.208714 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-c564cf647-d674t" Oct 30 01:43:44.283814 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:44.283688 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53166fd9-5f8c-45c4-a69a-c919a1bf0201-kube-api-access-kskbc\" (UniqueName: \"kubernetes.io/projected/53166fd9-5f8c-45c4-a69a-c919a1bf0201-kube-api-access-kskbc\") pod \"53166fd9-5f8c-45c4-a69a-c919a1bf0201\" (UID: \"53166fd9-5f8c-45c4-a69a-c919a1bf0201\") " Oct 30 01:43:44.284391 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:44.284330 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53166fd9-5f8c-45c4-a69a-c919a1bf0201-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53166fd9-5f8c-45c4-a69a-c919a1bf0201-plugins-config-volume\") pod \"53166fd9-5f8c-45c4-a69a-c919a1bf0201\" (UID: \"53166fd9-5f8c-45c4-a69a-c919a1bf0201\") " Oct 30 01:43:44.285336 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:44.284795 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53166fd9-5f8c-45c4-a69a-c919a1bf0201-plugins-config-volume" pod "53166fd9-5f8c-45c4-a69a-c919a1bf0201" (UID: "53166fd9-5f8c-45c4-a69a-c919a1bf0201"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:43:44.287019 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:44.286931 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53166fd9-5f8c-45c4-a69a-c919a1bf0201-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:44.287455 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:44.287421 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53166fd9-5f8c-45c4-a69a-c919a1bf0201-kube-api-access-kskbc" pod "53166fd9-5f8c-45c4-a69a-c919a1bf0201" (UID: "53166fd9-5f8c-45c4-a69a-c919a1bf0201"). InnerVolumeSpecName "kube-api-access-kskbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:43:44.390570 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:44.390531 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kskbc\" (UniqueName: \"kubernetes.io/projected/53166fd9-5f8c-45c4-a69a-c919a1bf0201-kube-api-access-kskbc\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:44.488345 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:44.488306 6572 generic.go:356] "Generic (PLEG): container finished" podID="53166fd9-5f8c-45c4-a69a-c919a1bf0201" containerID="52f9bef11d73f4d2db7e350b7aca7d80e9e776fc72c29db8a59c44e778d3124f" exitCode=0 Oct 30 01:43:44.488575 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:44.488360 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-c564cf647-d674t" Oct 30 01:43:44.488575 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:44.488393 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-c564cf647-d674t" event={"ID":"53166fd9-5f8c-45c4-a69a-c919a1bf0201","Type":"ContainerDied","Data":"52f9bef11d73f4d2db7e350b7aca7d80e9e776fc72c29db8a59c44e778d3124f"} Oct 30 01:43:44.488575 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:44.488486 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-c564cf647-d674t" event={"ID":"53166fd9-5f8c-45c4-a69a-c919a1bf0201","Type":"ContainerDied","Data":"9d880ae945aa5e7e8058dcd67b8654ff3b6a05b012171c7c5fba37ad3c311ad2"} Oct 30 01:43:44.488575 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:44.488513 6572 scope.go:122] "RemoveContainer" containerID="52f9bef11d73f4d2db7e350b7aca7d80e9e776fc72c29db8a59c44e778d3124f" Oct 30 01:43:44.500681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:44.500654 6572 scope.go:122] "RemoveContainer" containerID="52f9bef11d73f4d2db7e350b7aca7d80e9e776fc72c29db8a59c44e778d3124f" Oct 30 01:43:44.501046 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:43:44.501021 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52f9bef11d73f4d2db7e350b7aca7d80e9e776fc72c29db8a59c44e778d3124f\": container with ID starting with 52f9bef11d73f4d2db7e350b7aca7d80e9e776fc72c29db8a59c44e778d3124f not found: ID does not exist" containerID="52f9bef11d73f4d2db7e350b7aca7d80e9e776fc72c29db8a59c44e778d3124f" Oct 30 01:43:44.501142 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:44.501056 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52f9bef11d73f4d2db7e350b7aca7d80e9e776fc72c29db8a59c44e778d3124f"} err="failed to get container status \"52f9bef11d73f4d2db7e350b7aca7d80e9e776fc72c29db8a59c44e778d3124f\": rpc error: code = NotFound desc = could not find container \"52f9bef11d73f4d2db7e350b7aca7d80e9e776fc72c29db8a59c44e778d3124f\": container with ID starting with 52f9bef11d73f4d2db7e350b7aca7d80e9e776fc72c29db8a59c44e778d3124f not found: ID does not exist" Oct 30 01:43:44.544433 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:44.544349 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c" path="/var/lib/kubelet/pods/a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c/volumes" Oct 30 01:43:44.544707 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:44.544692 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-c564cf647-d674t"] Oct 30 01:43:44.551033 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:44.551001 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2ac18f-c564cf647-d674t"] Oct 30 01:43:44.620332 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:44.620283 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:44.688672 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:43:44.688632 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:43:44.698329 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:44.698299 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2ac18f-wl9j6"] Oct 30 01:43:44.701497 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:44.701470 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2ac18f-wl9j6"] Oct 30 01:43:44.826966 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:44.826856 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:45.031630 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:45.031598 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:45.310447 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:45.310407 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:45.498915 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:45.498878 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-3fde1c-58cf94cd5b-nq9g2" podUID="3bb426f0-f37a-4660-819b-fa2bf8bb961c" containerName="payload-processing" containerID="cri-o://43f2f6294ee8f63aafb414ba2b48c77d00788a2b44dbaae7ebc9c9b2caa641ac" gracePeriod=25 Oct 30 01:43:45.764273 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:45.764236 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3fde1c-58cf94cd5b-nq9g2" Oct 30 01:43:45.916470 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:45.915018 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bb426f0-f37a-4660-819b-fa2bf8bb961c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bb426f0-f37a-4660-819b-fa2bf8bb961c-plugins-config-volume\") pod \"3bb426f0-f37a-4660-819b-fa2bf8bb961c\" (UID: \"3bb426f0-f37a-4660-819b-fa2bf8bb961c\") " Oct 30 01:43:45.916470 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:45.915105 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bb426f0-f37a-4660-819b-fa2bf8bb961c-kube-api-access-24p2b\" (UniqueName: \"kubernetes.io/projected/3bb426f0-f37a-4660-819b-fa2bf8bb961c-kube-api-access-24p2b\") pod \"3bb426f0-f37a-4660-819b-fa2bf8bb961c\" (UID: \"3bb426f0-f37a-4660-819b-fa2bf8bb961c\") " Oct 30 01:43:45.916470 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:45.916193 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bb426f0-f37a-4660-819b-fa2bf8bb961c-plugins-config-volume" pod "3bb426f0-f37a-4660-819b-fa2bf8bb961c" (UID: "3bb426f0-f37a-4660-819b-fa2bf8bb961c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:43:45.919538 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:45.918703 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb426f0-f37a-4660-819b-fa2bf8bb961c-kube-api-access-24p2b" pod "3bb426f0-f37a-4660-819b-fa2bf8bb961c" (UID: "3bb426f0-f37a-4660-819b-fa2bf8bb961c"). InnerVolumeSpecName "kube-api-access-24p2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:43:46.016775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:46.016721 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bb426f0-f37a-4660-819b-fa2bf8bb961c-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:46.016775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:46.016771 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24p2b\" (UniqueName: \"kubernetes.io/projected/3bb426f0-f37a-4660-819b-fa2bf8bb961c-kube-api-access-24p2b\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:46.507369 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:46.507334 6572 generic.go:356] "Generic (PLEG): container finished" podID="3bb426f0-f37a-4660-819b-fa2bf8bb961c" containerID="43f2f6294ee8f63aafb414ba2b48c77d00788a2b44dbaae7ebc9c9b2caa641ac" exitCode=0 Oct 30 01:43:46.507821 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:46.507384 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3fde1c-58cf94cd5b-nq9g2" Oct 30 01:43:46.507821 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:46.507420 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3fde1c-58cf94cd5b-nq9g2" event={"ID":"3bb426f0-f37a-4660-819b-fa2bf8bb961c","Type":"ContainerDied","Data":"43f2f6294ee8f63aafb414ba2b48c77d00788a2b44dbaae7ebc9c9b2caa641ac"} Oct 30 01:43:46.507821 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:46.507481 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3fde1c-58cf94cd5b-nq9g2" event={"ID":"3bb426f0-f37a-4660-819b-fa2bf8bb961c","Type":"ContainerDied","Data":"4849bcf2d87067d596a2b6b2d0899ae33415425c8dece1e2c3d11a4ba39b3baf"} Oct 30 01:43:46.507821 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:46.507506 6572 scope.go:122] "RemoveContainer" containerID="43f2f6294ee8f63aafb414ba2b48c77d00788a2b44dbaae7ebc9c9b2caa641ac" Oct 30 01:43:46.518858 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:46.518833 6572 scope.go:122] "RemoveContainer" containerID="43f2f6294ee8f63aafb414ba2b48c77d00788a2b44dbaae7ebc9c9b2caa641ac" Oct 30 01:43:46.519190 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:43:46.519171 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43f2f6294ee8f63aafb414ba2b48c77d00788a2b44dbaae7ebc9c9b2caa641ac\": container with ID starting with 43f2f6294ee8f63aafb414ba2b48c77d00788a2b44dbaae7ebc9c9b2caa641ac not found: ID does not exist" containerID="43f2f6294ee8f63aafb414ba2b48c77d00788a2b44dbaae7ebc9c9b2caa641ac" Oct 30 01:43:46.519243 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:46.519198 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43f2f6294ee8f63aafb414ba2b48c77d00788a2b44dbaae7ebc9c9b2caa641ac"} err="failed to get container status \"43f2f6294ee8f63aafb414ba2b48c77d00788a2b44dbaae7ebc9c9b2caa641ac\": rpc error: code = NotFound desc = could not find container \"43f2f6294ee8f63aafb414ba2b48c77d00788a2b44dbaae7ebc9c9b2caa641ac\": container with ID starting with 43f2f6294ee8f63aafb414ba2b48c77d00788a2b44dbaae7ebc9c9b2caa641ac not found: ID does not exist" Oct 30 01:43:46.543476 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:46.543440 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51763c04-1938-4d21-b5af-2f6529c64e6e" path="/var/lib/kubelet/pods/51763c04-1938-4d21-b5af-2f6529c64e6e/volumes" Oct 30 01:43:46.543903 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:46.543888 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53166fd9-5f8c-45c4-a69a-c919a1bf0201" path="/var/lib/kubelet/pods/53166fd9-5f8c-45c4-a69a-c919a1bf0201/volumes" Oct 30 01:43:46.567875 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:46.567824 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3fde1c-58cf94cd5b-nq9g2"] Oct 30 01:43:46.579254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:46.579208 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3fde1c-58cf94cd5b-nq9g2"] Oct 30 01:43:46.900939 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:46.900908 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:47.172806 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:47.172734 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" Oct 30 01:43:47.333192 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:47.333154 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7243362-b004-4393-8a15-8ea62c257b6d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7243362-b004-4393-8a15-8ea62c257b6d-credential-socket\") pod \"d7243362-b004-4393-8a15-8ea62c257b6d\" (UID: \"d7243362-b004-4393-8a15-8ea62c257b6d\") " Oct 30 01:43:47.333192 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:47.333197 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7243362-b004-4393-8a15-8ea62c257b6d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7243362-b004-4393-8a15-8ea62c257b6d-workload-certs\") pod \"d7243362-b004-4393-8a15-8ea62c257b6d\" (UID: \"d7243362-b004-4393-8a15-8ea62c257b6d\") " Oct 30 01:43:47.333492 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:47.333324 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7243362-b004-4393-8a15-8ea62c257b6d-istio-token\" (UniqueName: \"kubernetes.io/projected/d7243362-b004-4393-8a15-8ea62c257b6d-istio-token\") pod \"d7243362-b004-4393-8a15-8ea62c257b6d\" (UID: \"d7243362-b004-4393-8a15-8ea62c257b6d\") " Oct 30 01:43:47.333492 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:47.333357 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7243362-b004-4393-8a15-8ea62c257b6d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7243362-b004-4393-8a15-8ea62c257b6d-istio-envoy\") pod \"d7243362-b004-4393-8a15-8ea62c257b6d\" (UID: \"d7243362-b004-4393-8a15-8ea62c257b6d\") " Oct 30 01:43:47.333492 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:47.333385 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7243362-b004-4393-8a15-8ea62c257b6d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7243362-b004-4393-8a15-8ea62c257b6d-istiod-ca-cert\") pod \"d7243362-b004-4393-8a15-8ea62c257b6d\" (UID: \"d7243362-b004-4393-8a15-8ea62c257b6d\") " Oct 30 01:43:47.333492 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:47.333487 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7243362-b004-4393-8a15-8ea62c257b6d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7243362-b004-4393-8a15-8ea62c257b6d-istio-data\") pod \"d7243362-b004-4393-8a15-8ea62c257b6d\" (UID: \"d7243362-b004-4393-8a15-8ea62c257b6d\") " Oct 30 01:43:47.333713 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:47.333523 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7243362-b004-4393-8a15-8ea62c257b6d-kube-api-access-z46nz\" (UniqueName: \"kubernetes.io/projected/d7243362-b004-4393-8a15-8ea62c257b6d-kube-api-access-z46nz\") pod \"d7243362-b004-4393-8a15-8ea62c257b6d\" (UID: \"d7243362-b004-4393-8a15-8ea62c257b6d\") " Oct 30 01:43:47.333713 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:47.333531 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7243362-b004-4393-8a15-8ea62c257b6d-credential-socket" pod "d7243362-b004-4393-8a15-8ea62c257b6d" (UID: "d7243362-b004-4393-8a15-8ea62c257b6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:47.333713 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:47.333552 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7243362-b004-4393-8a15-8ea62c257b6d-workload-certs" pod "d7243362-b004-4393-8a15-8ea62c257b6d" (UID: "d7243362-b004-4393-8a15-8ea62c257b6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:47.333713 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:47.333573 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7243362-b004-4393-8a15-8ea62c257b6d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7243362-b004-4393-8a15-8ea62c257b6d-istio-podinfo\") pod \"d7243362-b004-4393-8a15-8ea62c257b6d\" (UID: \"d7243362-b004-4393-8a15-8ea62c257b6d\") " Oct 30 01:43:47.333713 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:47.333636 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7243362-b004-4393-8a15-8ea62c257b6d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7243362-b004-4393-8a15-8ea62c257b6d-workload-socket\") pod \"d7243362-b004-4393-8a15-8ea62c257b6d\" (UID: \"d7243362-b004-4393-8a15-8ea62c257b6d\") " Oct 30 01:43:47.334044 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:47.333874 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7243362-b004-4393-8a15-8ea62c257b6d-istio-data" pod "d7243362-b004-4393-8a15-8ea62c257b6d" (UID: "d7243362-b004-4393-8a15-8ea62c257b6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:47.334044 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:47.333901 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7243362-b004-4393-8a15-8ea62c257b6d-istiod-ca-cert" pod "d7243362-b004-4393-8a15-8ea62c257b6d" (UID: "d7243362-b004-4393-8a15-8ea62c257b6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:43:47.334156 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:47.334088 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7243362-b004-4393-8a15-8ea62c257b6d-workload-socket" pod "d7243362-b004-4393-8a15-8ea62c257b6d" (UID: "d7243362-b004-4393-8a15-8ea62c257b6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:47.335802 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:47.334834 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7243362-b004-4393-8a15-8ea62c257b6d-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:47.335802 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:47.334869 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7243362-b004-4393-8a15-8ea62c257b6d-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:47.335802 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:47.334887 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7243362-b004-4393-8a15-8ea62c257b6d-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:47.335802 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:47.334904 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7243362-b004-4393-8a15-8ea62c257b6d-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:47.336130 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:47.334925 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7243362-b004-4393-8a15-8ea62c257b6d-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:47.347186 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:47.347131 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7243362-b004-4393-8a15-8ea62c257b6d-kube-api-access-z46nz" pod "d7243362-b004-4393-8a15-8ea62c257b6d" (UID: "d7243362-b004-4393-8a15-8ea62c257b6d"). InnerVolumeSpecName "kube-api-access-z46nz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:43:47.347186 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:47.347139 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7243362-b004-4393-8a15-8ea62c257b6d-istio-envoy" pod "d7243362-b004-4393-8a15-8ea62c257b6d" (UID: "d7243362-b004-4393-8a15-8ea62c257b6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:47.347522 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:47.347494 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7243362-b004-4393-8a15-8ea62c257b6d-istio-podinfo" pod "d7243362-b004-4393-8a15-8ea62c257b6d" (UID: "d7243362-b004-4393-8a15-8ea62c257b6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:43:47.347587 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:47.347520 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7243362-b004-4393-8a15-8ea62c257b6d-istio-token" pod "d7243362-b004-4393-8a15-8ea62c257b6d" (UID: "d7243362-b004-4393-8a15-8ea62c257b6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:43:47.437516 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:47.437476 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7243362-b004-4393-8a15-8ea62c257b6d-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:47.437516 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:47.437510 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7243362-b004-4393-8a15-8ea62c257b6d-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:47.437516 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:47.437522 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z46nz\" (UniqueName: \"kubernetes.io/projected/d7243362-b004-4393-8a15-8ea62c257b6d-kube-api-access-z46nz\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:47.437878 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:47.437533 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7243362-b004-4393-8a15-8ea62c257b6d-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:47.518342 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:47.518293 6572 generic.go:356] "Generic (PLEG): container finished" podID="d7243362-b004-4393-8a15-8ea62c257b6d" containerID="9f775cffb5e8cb2436d7c7f18b3586d3a5d4ce1fc4e66ebed075b12a97a6b65c" exitCode=0 Oct 30 01:43:47.518871 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:47.518354 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" event={"ID":"d7243362-b004-4393-8a15-8ea62c257b6d","Type":"ContainerDied","Data":"9f775cffb5e8cb2436d7c7f18b3586d3a5d4ce1fc4e66ebed075b12a97a6b65c"} Oct 30 01:43:47.518871 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:47.518374 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" Oct 30 01:43:47.518871 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:47.518456 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n" event={"ID":"d7243362-b004-4393-8a15-8ea62c257b6d","Type":"ContainerDied","Data":"4568434182b4f0f0b184def829c99824f102ca54acf30a3b2861897404e00234"} Oct 30 01:43:47.518871 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:47.518486 6572 scope.go:122] "RemoveContainer" containerID="9f775cffb5e8cb2436d7c7f18b3586d3a5d4ce1fc4e66ebed075b12a97a6b65c" Oct 30 01:43:47.532268 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:47.532244 6572 scope.go:122] "RemoveContainer" containerID="9f775cffb5e8cb2436d7c7f18b3586d3a5d4ce1fc4e66ebed075b12a97a6b65c" Oct 30 01:43:47.532626 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:43:47.532603 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f775cffb5e8cb2436d7c7f18b3586d3a5d4ce1fc4e66ebed075b12a97a6b65c\": container with ID starting with 9f775cffb5e8cb2436d7c7f18b3586d3a5d4ce1fc4e66ebed075b12a97a6b65c not found: ID does not exist" containerID="9f775cffb5e8cb2436d7c7f18b3586d3a5d4ce1fc4e66ebed075b12a97a6b65c" Oct 30 01:43:47.532713 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:47.532639 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f775cffb5e8cb2436d7c7f18b3586d3a5d4ce1fc4e66ebed075b12a97a6b65c"} err="failed to get container status \"9f775cffb5e8cb2436d7c7f18b3586d3a5d4ce1fc4e66ebed075b12a97a6b65c\": rpc error: code = NotFound desc = could not find container \"9f775cffb5e8cb2436d7c7f18b3586d3a5d4ce1fc4e66ebed075b12a97a6b65c\": container with ID starting with 9f775cffb5e8cb2436d7c7f18b3586d3a5d4ce1fc4e66ebed075b12a97a6b65c not found: ID does not exist" Oct 30 01:43:47.574220 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:47.574182 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n"] Oct 30 01:43:47.583906 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:47.583865 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-dzz2n"] Oct 30 01:43:48.543407 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:48.543366 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bb426f0-f37a-4660-819b-fa2bf8bb961c" path="/var/lib/kubelet/pods/3bb426f0-f37a-4660-819b-fa2bf8bb961c/volumes" Oct 30 01:43:48.543908 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:48.543891 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7243362-b004-4393-8a15-8ea62c257b6d" path="/var/lib/kubelet/pods/d7243362-b004-4393-8a15-8ea62c257b6d/volumes" Oct 30 01:43:48.676570 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:48.676529 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:50.428143 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:50.428106 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:51.527189 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:51.527150 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:51.789783 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:51.789647 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb"] Oct 30 01:43:51.789956 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:51.789929 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" podUID="05ffea96-f1dc-407d-ad3e-df11499478ce" containerName="istio-proxy" containerID="cri-o://c33e6852a52071dc259863931dd9455f22fafa846a304082da471671bc0d167f" gracePeriod=30 Oct 30 01:43:51.799410 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:51.799372 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:52.022567 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:52.022517 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:52.249334 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:52.249287 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:52.520288 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:52.520202 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:52.754162 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:52.754122 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:52.985311 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:52.985273 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:54.330668 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.330624 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c22e6b-c78dd"] Oct 30 01:43:54.331454 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.331430 6572 cpu_manager.go:414] "containerMap: removing container" podUID="3bb426f0-f37a-4660-819b-fa2bf8bb961c" containerName="payload-processing" Oct 30 01:43:54.331572 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.331456 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bb426f0-f37a-4660-819b-fa2bf8bb961c" containerName="payload-processing" Oct 30 01:43:54.331572 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.331478 6572 cpu_manager.go:414] "containerMap: removing container" podUID="d7243362-b004-4393-8a15-8ea62c257b6d" containerName="istio-proxy" Oct 30 01:43:54.331572 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.331486 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7243362-b004-4393-8a15-8ea62c257b6d" containerName="istio-proxy" Oct 30 01:43:54.331572 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.331497 6572 cpu_manager.go:414] "containerMap: removing container" podUID="6b50de5b-2782-45a9-8b90-44e3508013f8" containerName="istio-proxy" Oct 30 01:43:54.331572 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.331504 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b50de5b-2782-45a9-8b90-44e3508013f8" containerName="istio-proxy" Oct 30 01:43:54.331572 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.331555 6572 cpu_manager.go:414] "containerMap: removing container" podUID="53166fd9-5f8c-45c4-a69a-c919a1bf0201" containerName="payload-pre-processing" Oct 30 01:43:54.331572 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.331562 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53166fd9-5f8c-45c4-a69a-c919a1bf0201" containerName="payload-pre-processing" Oct 30 01:43:54.331974 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.331580 6572 cpu_manager.go:414] "containerMap: removing container" podUID="a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c" containerName="revoke-keys" Oct 30 01:43:54.331974 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.331587 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c" containerName="revoke-keys" Oct 30 01:43:54.331974 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.331613 6572 cpu_manager.go:414] "containerMap: removing container" podUID="51763c04-1938-4d21-b5af-2f6529c64e6e" containerName="revoke-keys" Oct 30 01:43:54.331974 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.331620 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51763c04-1938-4d21-b5af-2f6529c64e6e" containerName="revoke-keys" Oct 30 01:43:54.331974 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.331789 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="51763c04-1938-4d21-b5af-2f6529c64e6e" containerName="revoke-keys" Oct 30 01:43:54.331974 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.331806 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bb426f0-f37a-4660-819b-fa2bf8bb961c" containerName="payload-processing" Oct 30 01:43:54.331974 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.331818 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7243362-b004-4393-8a15-8ea62c257b6d" containerName="istio-proxy" Oct 30 01:43:54.331974 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.331825 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b50de5b-2782-45a9-8b90-44e3508013f8" containerName="istio-proxy" Oct 30 01:43:54.331974 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.331842 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="53166fd9-5f8c-45c4-a69a-c919a1bf0201" containerName="payload-pre-processing" Oct 30 01:43:54.331974 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.331852 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7a0f4cd-9a7d-4108-aeb3-a5ba4ced935c" containerName="revoke-keys" Oct 30 01:43:54.337127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.337101 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c22e6b-c78dd" Oct 30 01:43:54.354474 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.354443 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c22e6b-c78dd"] Oct 30 01:43:54.361733 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.361643 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:43:54.361733 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.361652 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:43:54.362830 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.362805 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:43:54.362830 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.362811 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:43:54.382553 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.382515 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39155e3d-3a29-478e-8b36-5981ef046b0a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c22e6b-c78dd\" (UID: \"39155e3d-3a29-478e-8b36-5981ef046b0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c22e6b-c78dd" Oct 30 01:43:54.382694 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.382666 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39155e3d-3a29-478e-8b36-5981ef046b0a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c22e6b-c78dd\" (UID: \"39155e3d-3a29-478e-8b36-5981ef046b0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c22e6b-c78dd" Oct 30 01:43:54.383093 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.382806 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rns7\" (UniqueName: \"kubernetes.io/projected/39155e3d-3a29-478e-8b36-5981ef046b0a-kube-api-access-5rns7\") pod \"maas-api-revoke-keys-e2e-praxis-a-c22e6b-c78dd\" (UID: \"39155e3d-3a29-478e-8b36-5981ef046b0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c22e6b-c78dd" Oct 30 01:43:54.483791 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.483723 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rns7\" (UniqueName: \"kubernetes.io/projected/39155e3d-3a29-478e-8b36-5981ef046b0a-kube-api-access-5rns7\") pod \"maas-api-revoke-keys-e2e-praxis-a-c22e6b-c78dd\" (UID: \"39155e3d-3a29-478e-8b36-5981ef046b0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c22e6b-c78dd" Oct 30 01:43:54.483990 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.483868 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39155e3d-3a29-478e-8b36-5981ef046b0a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c22e6b-c78dd\" (UID: \"39155e3d-3a29-478e-8b36-5981ef046b0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c22e6b-c78dd" Oct 30 01:43:54.483990 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.483955 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39155e3d-3a29-478e-8b36-5981ef046b0a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c22e6b-c78dd\" (UID: \"39155e3d-3a29-478e-8b36-5981ef046b0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c22e6b-c78dd" Oct 30 01:43:54.484625 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.484604 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39155e3d-3a29-478e-8b36-5981ef046b0a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c22e6b-c78dd\" (UID: \"39155e3d-3a29-478e-8b36-5981ef046b0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c22e6b-c78dd" Oct 30 01:43:54.486474 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.486435 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39155e3d-3a29-478e-8b36-5981ef046b0a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c22e6b-c78dd\" (UID: \"39155e3d-3a29-478e-8b36-5981ef046b0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c22e6b-c78dd" Oct 30 01:43:54.502879 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.502842 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rns7\" (UniqueName: \"kubernetes.io/projected/39155e3d-3a29-478e-8b36-5981ef046b0a-kube-api-access-5rns7\") pod \"maas-api-revoke-keys-e2e-praxis-a-c22e6b-c78dd\" (UID: \"39155e3d-3a29-478e-8b36-5981ef046b0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c22e6b-c78dd" Oct 30 01:43:54.665160 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.665130 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:43:54.672694 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.672656 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c22e6b-c78dd" Oct 30 01:43:54.872701 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.872659 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:54.982266 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:54.982235 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c22e6b-c78dd"] Oct 30 01:43:54.982561 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:43:54.982537 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39155e3d_3a29_478e_8b36_5981ef046b0a.slice/crio-c7e05fc438bdbe7cacdbccd5daed691f66b990f208df2bdb6b9f97fbd61e6a57 WatchSource:0}: Error finding container c7e05fc438bdbe7cacdbccd5daed691f66b990f208df2bdb6b9f97fbd61e6a57: Status 404 returned error can't find the container with id c7e05fc438bdbe7cacdbccd5daed691f66b990f208df2bdb6b9f97fbd61e6a57 Oct 30 01:43:55.588035 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:55.587938 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c22e6b-c78dd" event={"ID":"39155e3d-3a29-478e-8b36-5981ef046b0a","Type":"ContainerStarted","Data":"e01ed8c8a0d4065d9ead85624fa784ef7f202bd22fd246efce25d6146c75216d"} Oct 30 01:43:55.588035 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:55.588001 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c22e6b-c78dd" event={"ID":"39155e3d-3a29-478e-8b36-5981ef046b0a","Type":"ContainerStarted","Data":"c7e05fc438bdbe7cacdbccd5daed691f66b990f208df2bdb6b9f97fbd61e6a57"} Oct 30 01:43:55.630533 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:55.630472 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c22e6b-c78dd" podStartSLOduration=1.630454763 podStartE2EDuration="1.630454763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:43:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:43:55.627639445 +0000 UTC m=+1561.204064847" watchObservedRunningTime="2026-10-30 01:43:55.630454763 +0000 UTC m=+1561.206880164" Oct 30 01:43:56.332256 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:56.332217 6572 scope.go:122] "RemoveContainer" containerID="061d6970c7b3241e6f5e4f7b40e92d03503d6299024fac1f41c55c73bd432ae4" Oct 30 01:43:56.343225 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:56.343196 6572 scope.go:122] "RemoveContainer" containerID="4463e3cde0d8502569c31d9039b3efa194350696c70e7e19f27e1e4ac07b690c" Oct 30 01:43:56.357249 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:56.357225 6572 scope.go:122] "RemoveContainer" containerID="fc1daa2108128a3dd14abde3c5ba71bd5c9d118e2ce116c823381e8b8c8400f7" Oct 30 01:43:56.597181 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:56.597092 6572 generic.go:356] "Generic (PLEG): container finished" podID="39155e3d-3a29-478e-8b36-5981ef046b0a" containerID="e01ed8c8a0d4065d9ead85624fa784ef7f202bd22fd246efce25d6146c75216d" exitCode=0 Oct 30 01:43:56.597181 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:56.597145 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c22e6b-c78dd" event={"ID":"39155e3d-3a29-478e-8b36-5981ef046b0a","Type":"ContainerDied","Data":"e01ed8c8a0d4065d9ead85624fa784ef7f202bd22fd246efce25d6146c75216d"} Oct 30 01:43:56.825121 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:56.825084 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:57.071595 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.071572 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:43:57.223323 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.223291 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05ffea96-f1dc-407d-ad3e-df11499478ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05ffea96-f1dc-407d-ad3e-df11499478ce-istio-podinfo\") pod \"05ffea96-f1dc-407d-ad3e-df11499478ce\" (UID: \"05ffea96-f1dc-407d-ad3e-df11499478ce\") " Oct 30 01:43:57.223516 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.223368 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05ffea96-f1dc-407d-ad3e-df11499478ce-istio-token\" (UniqueName: \"kubernetes.io/projected/05ffea96-f1dc-407d-ad3e-df11499478ce-istio-token\") pod \"05ffea96-f1dc-407d-ad3e-df11499478ce\" (UID: \"05ffea96-f1dc-407d-ad3e-df11499478ce\") " Oct 30 01:43:57.223516 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.223404 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ffea96-f1dc-407d-ad3e-df11499478ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05ffea96-f1dc-407d-ad3e-df11499478ce-istio-data\") pod \"05ffea96-f1dc-407d-ad3e-df11499478ce\" (UID: \"05ffea96-f1dc-407d-ad3e-df11499478ce\") " Oct 30 01:43:57.223516 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.223486 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05ffea96-f1dc-407d-ad3e-df11499478ce-kube-api-access-847q6\" (UniqueName: \"kubernetes.io/projected/05ffea96-f1dc-407d-ad3e-df11499478ce-kube-api-access-847q6\") pod \"05ffea96-f1dc-407d-ad3e-df11499478ce\" (UID: \"05ffea96-f1dc-407d-ad3e-df11499478ce\") " Oct 30 01:43:57.223700 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.223526 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ffea96-f1dc-407d-ad3e-df11499478ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05ffea96-f1dc-407d-ad3e-df11499478ce-istio-envoy\") pod \"05ffea96-f1dc-407d-ad3e-df11499478ce\" (UID: \"05ffea96-f1dc-407d-ad3e-df11499478ce\") " Oct 30 01:43:57.223700 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.223607 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ffea96-f1dc-407d-ad3e-df11499478ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ffea96-f1dc-407d-ad3e-df11499478ce-workload-socket\") pod \"05ffea96-f1dc-407d-ad3e-df11499478ce\" (UID: \"05ffea96-f1dc-407d-ad3e-df11499478ce\") " Oct 30 01:43:57.223700 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.223665 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05ffea96-f1dc-407d-ad3e-df11499478ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05ffea96-f1dc-407d-ad3e-df11499478ce-istiod-ca-cert\") pod \"05ffea96-f1dc-407d-ad3e-df11499478ce\" (UID: \"05ffea96-f1dc-407d-ad3e-df11499478ce\") " Oct 30 01:43:57.223920 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.223773 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ffea96-f1dc-407d-ad3e-df11499478ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ffea96-f1dc-407d-ad3e-df11499478ce-credential-socket\") pod \"05ffea96-f1dc-407d-ad3e-df11499478ce\" (UID: \"05ffea96-f1dc-407d-ad3e-df11499478ce\") " Oct 30 01:43:57.223920 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.223822 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ffea96-f1dc-407d-ad3e-df11499478ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05ffea96-f1dc-407d-ad3e-df11499478ce-workload-certs\") pod \"05ffea96-f1dc-407d-ad3e-df11499478ce\" (UID: \"05ffea96-f1dc-407d-ad3e-df11499478ce\") " Oct 30 01:43:57.223920 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.223849 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ffea96-f1dc-407d-ad3e-df11499478ce-istio-data" pod "05ffea96-f1dc-407d-ad3e-df11499478ce" (UID: "05ffea96-f1dc-407d-ad3e-df11499478ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:57.224431 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.224194 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ffea96-f1dc-407d-ad3e-df11499478ce-credential-socket" pod "05ffea96-f1dc-407d-ad3e-df11499478ce" (UID: "05ffea96-f1dc-407d-ad3e-df11499478ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:57.224431 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.224307 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05ffea96-f1dc-407d-ad3e-df11499478ce-istiod-ca-cert" pod "05ffea96-f1dc-407d-ad3e-df11499478ce" (UID: "05ffea96-f1dc-407d-ad3e-df11499478ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:43:57.224431 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.224403 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ffea96-f1dc-407d-ad3e-df11499478ce-workload-socket" pod "05ffea96-f1dc-407d-ad3e-df11499478ce" (UID: "05ffea96-f1dc-407d-ad3e-df11499478ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:57.224688 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.224620 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05ffea96-f1dc-407d-ad3e-df11499478ce-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:57.224688 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.224642 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ffea96-f1dc-407d-ad3e-df11499478ce-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:57.224688 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.224658 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05ffea96-f1dc-407d-ad3e-df11499478ce-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:57.224688 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.224672 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ffea96-f1dc-407d-ad3e-df11499478ce-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:57.225109 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.225079 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ffea96-f1dc-407d-ad3e-df11499478ce-workload-certs" pod "05ffea96-f1dc-407d-ad3e-df11499478ce" (UID: "05ffea96-f1dc-407d-ad3e-df11499478ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:57.225793 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.225742 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05ffea96-f1dc-407d-ad3e-df11499478ce-istio-podinfo" pod "05ffea96-f1dc-407d-ad3e-df11499478ce" (UID: "05ffea96-f1dc-407d-ad3e-df11499478ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:43:57.226115 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.226096 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05ffea96-f1dc-407d-ad3e-df11499478ce-istio-token" pod "05ffea96-f1dc-407d-ad3e-df11499478ce" (UID: "05ffea96-f1dc-407d-ad3e-df11499478ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:43:57.226300 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.226279 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05ffea96-f1dc-407d-ad3e-df11499478ce-kube-api-access-847q6" pod "05ffea96-f1dc-407d-ad3e-df11499478ce" (UID: "05ffea96-f1dc-407d-ad3e-df11499478ce"). InnerVolumeSpecName "kube-api-access-847q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:43:57.226656 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.226635 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ffea96-f1dc-407d-ad3e-df11499478ce-istio-envoy" pod "05ffea96-f1dc-407d-ad3e-df11499478ce" (UID: "05ffea96-f1dc-407d-ad3e-df11499478ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:57.250431 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.250403 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c9d174-ndk9n"] Oct 30 01:43:57.251011 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.250997 6572 cpu_manager.go:414] "containerMap: removing container" podUID="05ffea96-f1dc-407d-ad3e-df11499478ce" containerName="istio-proxy" Oct 30 01:43:57.251076 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.251012 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05ffea96-f1dc-407d-ad3e-df11499478ce" containerName="istio-proxy" Oct 30 01:43:57.251127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.251117 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="05ffea96-f1dc-407d-ad3e-df11499478ce" containerName="istio-proxy" Oct 30 01:43:57.254811 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.254794 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c9d174-ndk9n" Oct 30 01:43:57.277543 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.277513 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c9d174-ndk9n"] Oct 30 01:43:57.326303 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.326262 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f535096-d7ec-4bc0-bd5f-208b97d396da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c9d174-ndk9n\" (UID: \"8f535096-d7ec-4bc0-bd5f-208b97d396da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c9d174-ndk9n" Oct 30 01:43:57.326563 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.326379 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcdgd\" (UniqueName: \"kubernetes.io/projected/8f535096-d7ec-4bc0-bd5f-208b97d396da-kube-api-access-vcdgd\") pod \"maas-api-revoke-keys-e2e-praxis-b-c9d174-ndk9n\" (UID: \"8f535096-d7ec-4bc0-bd5f-208b97d396da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c9d174-ndk9n" Oct 30 01:43:57.326563 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.326468 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f535096-d7ec-4bc0-bd5f-208b97d396da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c9d174-ndk9n\" (UID: \"8f535096-d7ec-4bc0-bd5f-208b97d396da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c9d174-ndk9n" Oct 30 01:43:57.326664 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.326599 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05ffea96-f1dc-407d-ad3e-df11499478ce-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:57.326664 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.326611 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05ffea96-f1dc-407d-ad3e-df11499478ce-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:57.326664 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.326620 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05ffea96-f1dc-407d-ad3e-df11499478ce-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:57.326664 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.326629 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-847q6\" (UniqueName: \"kubernetes.io/projected/05ffea96-f1dc-407d-ad3e-df11499478ce-kube-api-access-847q6\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:57.326664 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.326638 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05ffea96-f1dc-407d-ad3e-df11499478ce-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:57.427443 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.427407 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f535096-d7ec-4bc0-bd5f-208b97d396da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c9d174-ndk9n\" (UID: \"8f535096-d7ec-4bc0-bd5f-208b97d396da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c9d174-ndk9n" Oct 30 01:43:57.427621 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.427461 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcdgd\" (UniqueName: \"kubernetes.io/projected/8f535096-d7ec-4bc0-bd5f-208b97d396da-kube-api-access-vcdgd\") pod \"maas-api-revoke-keys-e2e-praxis-b-c9d174-ndk9n\" (UID: \"8f535096-d7ec-4bc0-bd5f-208b97d396da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c9d174-ndk9n" Oct 30 01:43:57.427621 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.427492 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f535096-d7ec-4bc0-bd5f-208b97d396da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c9d174-ndk9n\" (UID: \"8f535096-d7ec-4bc0-bd5f-208b97d396da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c9d174-ndk9n" Oct 30 01:43:57.428221 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.428201 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f535096-d7ec-4bc0-bd5f-208b97d396da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c9d174-ndk9n\" (UID: \"8f535096-d7ec-4bc0-bd5f-208b97d396da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c9d174-ndk9n" Oct 30 01:43:57.429906 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.429878 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f535096-d7ec-4bc0-bd5f-208b97d396da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c9d174-ndk9n\" (UID: \"8f535096-d7ec-4bc0-bd5f-208b97d396da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c9d174-ndk9n" Oct 30 01:43:57.447310 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.447283 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcdgd\" (UniqueName: \"kubernetes.io/projected/8f535096-d7ec-4bc0-bd5f-208b97d396da-kube-api-access-vcdgd\") pod \"maas-api-revoke-keys-e2e-praxis-b-c9d174-ndk9n\" (UID: \"8f535096-d7ec-4bc0-bd5f-208b97d396da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c9d174-ndk9n" Oct 30 01:43:57.568702 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.568619 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c9d174-ndk9n" Oct 30 01:43:57.608689 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.608654 6572 generic.go:356] "Generic (PLEG): container finished" podID="05ffea96-f1dc-407d-ad3e-df11499478ce" containerID="c33e6852a52071dc259863931dd9455f22fafa846a304082da471671bc0d167f" exitCode=0 Oct 30 01:43:57.609211 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.608763 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" Oct 30 01:43:57.609211 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.608742 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" event={"ID":"05ffea96-f1dc-407d-ad3e-df11499478ce","Type":"ContainerDied","Data":"c33e6852a52071dc259863931dd9455f22fafa846a304082da471671bc0d167f"} Oct 30 01:43:57.609211 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.608907 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb" event={"ID":"05ffea96-f1dc-407d-ad3e-df11499478ce","Type":"ContainerDied","Data":"69a7ffa663f7caed4d64eabff9c01305d2cf33b6a54c25db5b80498b809fb02e"} Oct 30 01:43:57.609211 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.608931 6572 scope.go:122] "RemoveContainer" containerID="c33e6852a52071dc259863931dd9455f22fafa846a304082da471671bc0d167f" Oct 30 01:43:57.627614 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.627586 6572 scope.go:122] "RemoveContainer" containerID="4edac2164303c8213d07e4a4d7a3e68dd8017177ee7776fd842bdc85561ff29d" Oct 30 01:43:57.647926 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.647897 6572 scope.go:122] "RemoveContainer" containerID="c33e6852a52071dc259863931dd9455f22fafa846a304082da471671bc0d167f" Oct 30 01:43:57.652108 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:43:57.652073 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c33e6852a52071dc259863931dd9455f22fafa846a304082da471671bc0d167f\": container with ID starting with c33e6852a52071dc259863931dd9455f22fafa846a304082da471671bc0d167f not found: ID does not exist" containerID="c33e6852a52071dc259863931dd9455f22fafa846a304082da471671bc0d167f" Oct 30 01:43:57.652239 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.652118 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c33e6852a52071dc259863931dd9455f22fafa846a304082da471671bc0d167f"} err="failed to get container status \"c33e6852a52071dc259863931dd9455f22fafa846a304082da471671bc0d167f\": rpc error: code = NotFound desc = could not find container \"c33e6852a52071dc259863931dd9455f22fafa846a304082da471671bc0d167f\": container with ID starting with c33e6852a52071dc259863931dd9455f22fafa846a304082da471671bc0d167f not found: ID does not exist" Oct 30 01:43:57.652239 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.652150 6572 scope.go:122] "RemoveContainer" containerID="4edac2164303c8213d07e4a4d7a3e68dd8017177ee7776fd842bdc85561ff29d" Oct 30 01:43:57.652610 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:43:57.652585 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4edac2164303c8213d07e4a4d7a3e68dd8017177ee7776fd842bdc85561ff29d\": container with ID starting with 4edac2164303c8213d07e4a4d7a3e68dd8017177ee7776fd842bdc85561ff29d not found: ID does not exist" containerID="4edac2164303c8213d07e4a4d7a3e68dd8017177ee7776fd842bdc85561ff29d" Oct 30 01:43:57.652681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.652619 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4edac2164303c8213d07e4a4d7a3e68dd8017177ee7776fd842bdc85561ff29d"} err="failed to get container status \"4edac2164303c8213d07e4a4d7a3e68dd8017177ee7776fd842bdc85561ff29d\": rpc error: code = NotFound desc = could not find container \"4edac2164303c8213d07e4a4d7a3e68dd8017177ee7776fd842bdc85561ff29d\": container with ID starting with 4edac2164303c8213d07e4a4d7a3e68dd8017177ee7776fd842bdc85561ff29d not found: ID does not exist" Oct 30 01:43:57.670856 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.670819 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb"] Oct 30 01:43:57.673226 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.673201 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-3fde1c-openshift-default-5fd69bd765-76qvb"] Oct 30 01:43:57.766507 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.766483 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c22e6b-c78dd" Oct 30 01:43:57.833305 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.833270 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39155e3d-3a29-478e-8b36-5981ef046b0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39155e3d-3a29-478e-8b36-5981ef046b0a-maas-api-openshift-ca\") pod \"39155e3d-3a29-478e-8b36-5981ef046b0a\" (UID: \"39155e3d-3a29-478e-8b36-5981ef046b0a\") " Oct 30 01:43:57.833486 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.833411 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39155e3d-3a29-478e-8b36-5981ef046b0a-kube-api-access-5rns7\" (UniqueName: \"kubernetes.io/projected/39155e3d-3a29-478e-8b36-5981ef046b0a-kube-api-access-5rns7\") pod \"39155e3d-3a29-478e-8b36-5981ef046b0a\" (UID: \"39155e3d-3a29-478e-8b36-5981ef046b0a\") " Oct 30 01:43:57.833564 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.833525 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39155e3d-3a29-478e-8b36-5981ef046b0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39155e3d-3a29-478e-8b36-5981ef046b0a-maas-api-xks-ca\") pod \"39155e3d-3a29-478e-8b36-5981ef046b0a\" (UID: \"39155e3d-3a29-478e-8b36-5981ef046b0a\") " Oct 30 01:43:57.834704 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.834675 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39155e3d-3a29-478e-8b36-5981ef046b0a-maas-api-openshift-ca" pod "39155e3d-3a29-478e-8b36-5981ef046b0a" (UID: "39155e3d-3a29-478e-8b36-5981ef046b0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:43:57.836253 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.836220 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39155e3d-3a29-478e-8b36-5981ef046b0a-maas-api-xks-ca" pod "39155e3d-3a29-478e-8b36-5981ef046b0a" (UID: "39155e3d-3a29-478e-8b36-5981ef046b0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:43:57.836969 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.836948 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39155e3d-3a29-478e-8b36-5981ef046b0a-kube-api-access-5rns7" pod "39155e3d-3a29-478e-8b36-5981ef046b0a" (UID: "39155e3d-3a29-478e-8b36-5981ef046b0a"). InnerVolumeSpecName "kube-api-access-5rns7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:43:57.883923 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.883891 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c9d174-ndk9n"] Oct 30 01:43:57.885739 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:43:57.885713 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f535096_d7ec_4bc0_bd5f_208b97d396da.slice/crio-03fa4703604b7df4c534d49d2699b00f9b9ee320fe0ead9f99a1df8d48f671fe WatchSource:0}: Error finding container 03fa4703604b7df4c534d49d2699b00f9b9ee320fe0ead9f99a1df8d48f671fe: Status 404 returned error can't find the container with id 03fa4703604b7df4c534d49d2699b00f9b9ee320fe0ead9f99a1df8d48f671fe Oct 30 01:43:57.935126 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.935092 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39155e3d-3a29-478e-8b36-5981ef046b0a-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:57.935126 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.935126 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rns7\" (UniqueName: \"kubernetes.io/projected/39155e3d-3a29-478e-8b36-5981ef046b0a-kube-api-access-5rns7\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:57.935274 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:57.935141 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39155e3d-3a29-478e-8b36-5981ef046b0a-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:43:58.184140 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:58.184106 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea0fc4-4qpjg"] Oct 30 01:43:58.184674 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:58.184661 6572 cpu_manager.go:414] "containerMap: removing container" podUID="39155e3d-3a29-478e-8b36-5981ef046b0a" containerName="revoke-keys" Oct 30 01:43:58.184721 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:58.184675 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39155e3d-3a29-478e-8b36-5981ef046b0a" containerName="revoke-keys" Oct 30 01:43:58.184721 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:58.184713 6572 cpu_manager.go:414] "containerMap: removing container" podUID="05ffea96-f1dc-407d-ad3e-df11499478ce" containerName="istio-proxy" Oct 30 01:43:58.184823 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:58.184721 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05ffea96-f1dc-407d-ad3e-df11499478ce" containerName="istio-proxy" Oct 30 01:43:58.184864 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:58.184857 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="39155e3d-3a29-478e-8b36-5981ef046b0a" containerName="revoke-keys" Oct 30 01:43:58.184907 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:58.184866 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="05ffea96-f1dc-407d-ad3e-df11499478ce" containerName="istio-proxy" Oct 30 01:43:58.189649 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:58.189625 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea0fc4-4qpjg" Oct 30 01:43:58.214704 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:58.214681 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea0fc4-4qpjg"] Oct 30 01:43:58.340190 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:58.340156 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8452l\" (UniqueName: \"kubernetes.io/projected/c07642d5-28e5-433c-91ee-b78c8077e419-kube-api-access-8452l\") pod \"maas-api-revoke-keys-e2e-ipp-b-ea0fc4-4qpjg\" (UID: \"c07642d5-28e5-433c-91ee-b78c8077e419\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea0fc4-4qpjg" Oct 30 01:43:58.340391 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:58.340301 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07642d5-28e5-433c-91ee-b78c8077e419-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ea0fc4-4qpjg\" (UID: \"c07642d5-28e5-433c-91ee-b78c8077e419\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea0fc4-4qpjg" Oct 30 01:43:58.340391 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:58.340341 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07642d5-28e5-433c-91ee-b78c8077e419-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ea0fc4-4qpjg\" (UID: \"c07642d5-28e5-433c-91ee-b78c8077e419\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea0fc4-4qpjg" Oct 30 01:43:58.442171 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:58.442138 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07642d5-28e5-433c-91ee-b78c8077e419-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ea0fc4-4qpjg\" (UID: \"c07642d5-28e5-433c-91ee-b78c8077e419\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea0fc4-4qpjg" Oct 30 01:43:58.442306 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:58.442194 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07642d5-28e5-433c-91ee-b78c8077e419-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ea0fc4-4qpjg\" (UID: \"c07642d5-28e5-433c-91ee-b78c8077e419\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea0fc4-4qpjg" Oct 30 01:43:58.442364 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:58.442329 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8452l\" (UniqueName: \"kubernetes.io/projected/c07642d5-28e5-433c-91ee-b78c8077e419-kube-api-access-8452l\") pod \"maas-api-revoke-keys-e2e-ipp-b-ea0fc4-4qpjg\" (UID: \"c07642d5-28e5-433c-91ee-b78c8077e419\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea0fc4-4qpjg" Oct 30 01:43:58.442826 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:58.442797 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07642d5-28e5-433c-91ee-b78c8077e419-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ea0fc4-4qpjg\" (UID: \"c07642d5-28e5-433c-91ee-b78c8077e419\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea0fc4-4qpjg" Oct 30 01:43:58.444712 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:58.444692 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07642d5-28e5-433c-91ee-b78c8077e419-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ea0fc4-4qpjg\" (UID: \"c07642d5-28e5-433c-91ee-b78c8077e419\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea0fc4-4qpjg" Oct 30 01:43:58.461187 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:58.461160 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8452l\" (UniqueName: \"kubernetes.io/projected/c07642d5-28e5-433c-91ee-b78c8077e419-kube-api-access-8452l\") pod \"maas-api-revoke-keys-e2e-ipp-b-ea0fc4-4qpjg\" (UID: \"c07642d5-28e5-433c-91ee-b78c8077e419\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea0fc4-4qpjg" Oct 30 01:43:58.504385 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:58.504366 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea0fc4-4qpjg" Oct 30 01:43:58.543617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:58.543582 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05ffea96-f1dc-407d-ad3e-df11499478ce" path="/var/lib/kubelet/pods/05ffea96-f1dc-407d-ad3e-df11499478ce/volumes" Oct 30 01:43:58.619980 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:58.619946 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c22e6b-c78dd" event={"ID":"39155e3d-3a29-478e-8b36-5981ef046b0a","Type":"ContainerDied","Data":"c7e05fc438bdbe7cacdbccd5daed691f66b990f208df2bdb6b9f97fbd61e6a57"} Oct 30 01:43:58.619980 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:58.619978 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c22e6b-c78dd" Oct 30 01:43:58.620460 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:58.620002 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7e05fc438bdbe7cacdbccd5daed691f66b990f208df2bdb6b9f97fbd61e6a57" Oct 30 01:43:58.623674 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:58.623649 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c9d174-ndk9n" event={"ID":"8f535096-d7ec-4bc0-bd5f-208b97d396da","Type":"ContainerStarted","Data":"5f4abeaf206b5f2738cbe38e41ef24f2dae0682a08e380ecc80a0f293ccc225e"} Oct 30 01:43:58.623828 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:58.623689 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c9d174-ndk9n" event={"ID":"8f535096-d7ec-4bc0-bd5f-208b97d396da","Type":"ContainerStarted","Data":"03fa4703604b7df4c534d49d2699b00f9b9ee320fe0ead9f99a1df8d48f671fe"} Oct 30 01:43:58.682776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:58.682716 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c9d174-ndk9n" podStartSLOduration=1.6827012670000001 podStartE2EDuration="1.682701267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:43:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:43:58.675798122 +0000 UTC m=+1564.252223525" watchObservedRunningTime="2026-10-30 01:43:58.682701267 +0000 UTC m=+1564.259126676" Oct 30 01:43:58.819242 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:58.819214 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea0fc4-4qpjg"] Oct 30 01:43:58.820124 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:43:58.820096 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc07642d5_28e5_433c_91ee_b78c8077e419.slice/crio-e3427ce43e4efe0571f1537b9cf019501237f0c7adde0ec27bd3992fb7d9ea5e WatchSource:0}: Error finding container e3427ce43e4efe0571f1537b9cf019501237f0c7adde0ec27bd3992fb7d9ea5e: Status 404 returned error can't find the container with id e3427ce43e4efe0571f1537b9cf019501237f0c7adde0ec27bd3992fb7d9ea5e Oct 30 01:43:58.862800 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:58.862773 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:59.632717 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:59.632678 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea0fc4-4qpjg" event={"ID":"c07642d5-28e5-433c-91ee-b78c8077e419","Type":"ContainerStarted","Data":"d11af34c1dc087ec9ffe66ef4ea007a7f6b06791e2806e54a6a55fcfc2155eeb"} Oct 30 01:43:59.633148 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:59.632792 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea0fc4-4qpjg" event={"ID":"c07642d5-28e5-433c-91ee-b78c8077e419","Type":"ContainerStarted","Data":"e3427ce43e4efe0571f1537b9cf019501237f0c7adde0ec27bd3992fb7d9ea5e"} Oct 30 01:43:59.634445 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:59.634414 6572 generic.go:356] "Generic (PLEG): container finished" podID="8f535096-d7ec-4bc0-bd5f-208b97d396da" containerID="5f4abeaf206b5f2738cbe38e41ef24f2dae0682a08e380ecc80a0f293ccc225e" exitCode=0 Oct 30 01:43:59.634557 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:59.634492 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c9d174-ndk9n" event={"ID":"8f535096-d7ec-4bc0-bd5f-208b97d396da","Type":"ContainerDied","Data":"5f4abeaf206b5f2738cbe38e41ef24f2dae0682a08e380ecc80a0f293ccc225e"} Oct 30 01:43:59.671854 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:43:59.671795 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea0fc4-4qpjg" podStartSLOduration=1.671780118 podStartE2EDuration="1.671780118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:43:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:43:59.669462358 +0000 UTC m=+1565.245887760" watchObservedRunningTime="2026-10-30 01:43:59.671780118 +0000 UTC m=+1565.248205520" Oct 30 01:44:00.127703 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:00.127665 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:00.359947 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:00.359906 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:00.643533 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:00.643499 6572 generic.go:356] "Generic (PLEG): container finished" podID="c07642d5-28e5-433c-91ee-b78c8077e419" containerID="d11af34c1dc087ec9ffe66ef4ea007a7f6b06791e2806e54a6a55fcfc2155eeb" exitCode=0 Oct 30 01:44:00.643955 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:00.643591 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea0fc4-4qpjg" event={"ID":"c07642d5-28e5-433c-91ee-b78c8077e419","Type":"ContainerDied","Data":"d11af34c1dc087ec9ffe66ef4ea007a7f6b06791e2806e54a6a55fcfc2155eeb"} Oct 30 01:44:00.827253 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:00.827225 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c9d174-ndk9n" Oct 30 01:44:00.833199 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:00.833173 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:00.979208 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:00.979175 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f535096-d7ec-4bc0-bd5f-208b97d396da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f535096-d7ec-4bc0-bd5f-208b97d396da-maas-api-xks-ca\") pod \"8f535096-d7ec-4bc0-bd5f-208b97d396da\" (UID: \"8f535096-d7ec-4bc0-bd5f-208b97d396da\") " Oct 30 01:44:00.979427 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:00.979233 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f535096-d7ec-4bc0-bd5f-208b97d396da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f535096-d7ec-4bc0-bd5f-208b97d396da-maas-api-openshift-ca\") pod \"8f535096-d7ec-4bc0-bd5f-208b97d396da\" (UID: \"8f535096-d7ec-4bc0-bd5f-208b97d396da\") " Oct 30 01:44:00.979427 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:00.979321 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f535096-d7ec-4bc0-bd5f-208b97d396da-kube-api-access-vcdgd\" (UniqueName: \"kubernetes.io/projected/8f535096-d7ec-4bc0-bd5f-208b97d396da-kube-api-access-vcdgd\") pod \"8f535096-d7ec-4bc0-bd5f-208b97d396da\" (UID: \"8f535096-d7ec-4bc0-bd5f-208b97d396da\") " Oct 30 01:44:00.979599 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:00.979571 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f535096-d7ec-4bc0-bd5f-208b97d396da-maas-api-openshift-ca" pod "8f535096-d7ec-4bc0-bd5f-208b97d396da" (UID: "8f535096-d7ec-4bc0-bd5f-208b97d396da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:44:00.979843 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:00.979819 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f535096-d7ec-4bc0-bd5f-208b97d396da-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:00.981400 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:00.981379 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f535096-d7ec-4bc0-bd5f-208b97d396da-maas-api-xks-ca" pod "8f535096-d7ec-4bc0-bd5f-208b97d396da" (UID: "8f535096-d7ec-4bc0-bd5f-208b97d396da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:44:00.981533 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:00.981514 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f535096-d7ec-4bc0-bd5f-208b97d396da-kube-api-access-vcdgd" pod "8f535096-d7ec-4bc0-bd5f-208b97d396da" (UID: "8f535096-d7ec-4bc0-bd5f-208b97d396da"). InnerVolumeSpecName "kube-api-access-vcdgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:01.081095 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:01.081063 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f535096-d7ec-4bc0-bd5f-208b97d396da-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:01.081095 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:01.081089 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcdgd\" (UniqueName: \"kubernetes.io/projected/8f535096-d7ec-4bc0-bd5f-208b97d396da-kube-api-access-vcdgd\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:01.653265 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:01.653222 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c9d174-ndk9n" event={"ID":"8f535096-d7ec-4bc0-bd5f-208b97d396da","Type":"ContainerDied","Data":"03fa4703604b7df4c534d49d2699b00f9b9ee320fe0ead9f99a1df8d48f671fe"} Oct 30 01:44:01.653265 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:01.653248 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c9d174-ndk9n" Oct 30 01:44:01.653846 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:01.653283 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03fa4703604b7df4c534d49d2699b00f9b9ee320fe0ead9f99a1df8d48f671fe" Oct 30 01:44:01.809736 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:01.809707 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea0fc4-4qpjg" Oct 30 01:44:01.992886 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:01.992851 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c07642d5-28e5-433c-91ee-b78c8077e419-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07642d5-28e5-433c-91ee-b78c8077e419-maas-api-xks-ca\") pod \"c07642d5-28e5-433c-91ee-b78c8077e419\" (UID: \"c07642d5-28e5-433c-91ee-b78c8077e419\") " Oct 30 01:44:01.993069 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:01.992968 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c07642d5-28e5-433c-91ee-b78c8077e419-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07642d5-28e5-433c-91ee-b78c8077e419-maas-api-openshift-ca\") pod \"c07642d5-28e5-433c-91ee-b78c8077e419\" (UID: \"c07642d5-28e5-433c-91ee-b78c8077e419\") " Oct 30 01:44:01.993171 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:01.993156 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c07642d5-28e5-433c-91ee-b78c8077e419-kube-api-access-8452l\" (UniqueName: \"kubernetes.io/projected/c07642d5-28e5-433c-91ee-b78c8077e419-kube-api-access-8452l\") pod \"c07642d5-28e5-433c-91ee-b78c8077e419\" (UID: \"c07642d5-28e5-433c-91ee-b78c8077e419\") " Oct 30 01:44:01.993388 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:01.993359 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c07642d5-28e5-433c-91ee-b78c8077e419-maas-api-openshift-ca" pod "c07642d5-28e5-433c-91ee-b78c8077e419" (UID: "c07642d5-28e5-433c-91ee-b78c8077e419"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:44:01.993682 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:01.993667 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07642d5-28e5-433c-91ee-b78c8077e419-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:01.994981 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:01.994964 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c07642d5-28e5-433c-91ee-b78c8077e419-maas-api-xks-ca" pod "c07642d5-28e5-433c-91ee-b78c8077e419" (UID: "c07642d5-28e5-433c-91ee-b78c8077e419"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:44:01.995133 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:01.995118 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c07642d5-28e5-433c-91ee-b78c8077e419-kube-api-access-8452l" pod "c07642d5-28e5-433c-91ee-b78c8077e419" (UID: "c07642d5-28e5-433c-91ee-b78c8077e419"). InnerVolumeSpecName "kube-api-access-8452l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:02.095354 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:02.095314 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8452l\" (UniqueName: \"kubernetes.io/projected/c07642d5-28e5-433c-91ee-b78c8077e419-kube-api-access-8452l\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:02.095354 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:02.095350 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07642d5-28e5-433c-91ee-b78c8077e419-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:02.663134 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:02.663104 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea0fc4-4qpjg" Oct 30 01:44:02.663676 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:02.663132 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea0fc4-4qpjg" event={"ID":"c07642d5-28e5-433c-91ee-b78c8077e419","Type":"ContainerDied","Data":"e3427ce43e4efe0571f1537b9cf019501237f0c7adde0ec27bd3992fb7d9ea5e"} Oct 30 01:44:02.663676 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:02.663178 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3427ce43e4efe0571f1537b9cf019501237f0c7adde0ec27bd3992fb7d9ea5e" Oct 30 01:44:02.751639 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:02.751602 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:04.775424 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:04.775391 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c22e6b-6486d6dc9b-gkwvs"] Oct 30 01:44:04.793114 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:04.793085 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c22e6b-7cd96f6bdc-p9l2f"] Oct 30 01:44:05.034645 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:05.034552 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:05.247345 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:05.247313 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:05.547892 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:05.547858 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:07.913643 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:07.913607 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:08.127890 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:08.127855 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:08.730471 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:08.730436 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:09.776333 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:09.776291 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-c22e6b-6486d6dc9b-gkwvs" podUID="0d1710d2-4436-4f5b-bfb9-82bd8210f7b3" containerName="payload-processing" containerID="cri-o://22a40e88b3d0cf06e5201ae0ffe987ffb1b120dd4917e9d16134148d9c6046d4" gracePeriod=25 Oct 30 01:44:09.793627 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:09.793591 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c22e6b-7cd96f6bdc-p9l2f" podUID="fac5d647-98c6-4ac2-97ba-c66b93d7673e" containerName="payload-pre-processing" containerID="cri-o://41f3da1b4d7fd19e1d2d60fef826cca2e824c57b7b33ed52c044058f59689dba" gracePeriod=25 Oct 30 01:44:10.089575 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:10.089547 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c22e6b-6486d6dc9b-gkwvs" Oct 30 01:44:10.105692 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:10.105665 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d1710d2-4436-4f5b-bfb9-82bd8210f7b3-kube-api-access-6tq97\" (UniqueName: \"kubernetes.io/projected/0d1710d2-4436-4f5b-bfb9-82bd8210f7b3-kube-api-access-6tq97\") pod \"0d1710d2-4436-4f5b-bfb9-82bd8210f7b3\" (UID: \"0d1710d2-4436-4f5b-bfb9-82bd8210f7b3\") " Oct 30 01:44:10.105886 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:10.105874 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d1710d2-4436-4f5b-bfb9-82bd8210f7b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d1710d2-4436-4f5b-bfb9-82bd8210f7b3-plugins-config-volume\") pod \"0d1710d2-4436-4f5b-bfb9-82bd8210f7b3\" (UID: \"0d1710d2-4436-4f5b-bfb9-82bd8210f7b3\") " Oct 30 01:44:10.106293 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:10.106268 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d1710d2-4436-4f5b-bfb9-82bd8210f7b3-plugins-config-volume" pod "0d1710d2-4436-4f5b-bfb9-82bd8210f7b3" (UID: "0d1710d2-4436-4f5b-bfb9-82bd8210f7b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:44:10.107798 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:10.107776 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d1710d2-4436-4f5b-bfb9-82bd8210f7b3-kube-api-access-6tq97" pod "0d1710d2-4436-4f5b-bfb9-82bd8210f7b3" (UID: "0d1710d2-4436-4f5b-bfb9-82bd8210f7b3"). InnerVolumeSpecName "kube-api-access-6tq97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:10.115919 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:10.115900 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c22e6b-7cd96f6bdc-p9l2f" Oct 30 01:44:10.147277 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:10.147235 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:10.207837 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:10.207803 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fac5d647-98c6-4ac2-97ba-c66b93d7673e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fac5d647-98c6-4ac2-97ba-c66b93d7673e-plugins-config-volume\") pod \"fac5d647-98c6-4ac2-97ba-c66b93d7673e\" (UID: \"fac5d647-98c6-4ac2-97ba-c66b93d7673e\") " Oct 30 01:44:10.207837 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:10.207852 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fac5d647-98c6-4ac2-97ba-c66b93d7673e-kube-api-access-q94sn\" (UniqueName: \"kubernetes.io/projected/fac5d647-98c6-4ac2-97ba-c66b93d7673e-kube-api-access-q94sn\") pod \"fac5d647-98c6-4ac2-97ba-c66b93d7673e\" (UID: \"fac5d647-98c6-4ac2-97ba-c66b93d7673e\") " Oct 30 01:44:10.208194 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:10.208168 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fac5d647-98c6-4ac2-97ba-c66b93d7673e-plugins-config-volume" pod "fac5d647-98c6-4ac2-97ba-c66b93d7673e" (UID: "fac5d647-98c6-4ac2-97ba-c66b93d7673e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:44:10.208286 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:10.208218 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tq97\" (UniqueName: \"kubernetes.io/projected/0d1710d2-4436-4f5b-bfb9-82bd8210f7b3-kube-api-access-6tq97\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:10.208286 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:10.208234 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d1710d2-4436-4f5b-bfb9-82bd8210f7b3-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:10.210034 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:10.210003 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fac5d647-98c6-4ac2-97ba-c66b93d7673e-kube-api-access-q94sn" pod "fac5d647-98c6-4ac2-97ba-c66b93d7673e" (UID: "fac5d647-98c6-4ac2-97ba-c66b93d7673e"). InnerVolumeSpecName "kube-api-access-q94sn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:10.309343 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:10.309267 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fac5d647-98c6-4ac2-97ba-c66b93d7673e-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:10.309343 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:10.309296 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q94sn\" (UniqueName: \"kubernetes.io/projected/fac5d647-98c6-4ac2-97ba-c66b93d7673e-kube-api-access-q94sn\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:10.747636 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:10.747605 6572 generic.go:356] "Generic (PLEG): container finished" podID="0d1710d2-4436-4f5b-bfb9-82bd8210f7b3" containerID="22a40e88b3d0cf06e5201ae0ffe987ffb1b120dd4917e9d16134148d9c6046d4" exitCode=0 Oct 30 01:44:10.747861 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:10.747646 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c22e6b-6486d6dc9b-gkwvs" Oct 30 01:44:10.747861 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:10.747686 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c22e6b-6486d6dc9b-gkwvs" event={"ID":"0d1710d2-4436-4f5b-bfb9-82bd8210f7b3","Type":"ContainerDied","Data":"22a40e88b3d0cf06e5201ae0ffe987ffb1b120dd4917e9d16134148d9c6046d4"} Oct 30 01:44:10.747861 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:10.747792 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c22e6b-6486d6dc9b-gkwvs" event={"ID":"0d1710d2-4436-4f5b-bfb9-82bd8210f7b3","Type":"ContainerDied","Data":"a8c458f701b46bb8373aa8b7572a506fd805904dadd8092e2488719a2e930cf2"} Oct 30 01:44:10.747861 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:10.747833 6572 scope.go:122] "RemoveContainer" containerID="22a40e88b3d0cf06e5201ae0ffe987ffb1b120dd4917e9d16134148d9c6046d4" Oct 30 01:44:10.749300 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:10.749276 6572 generic.go:356] "Generic (PLEG): container finished" podID="fac5d647-98c6-4ac2-97ba-c66b93d7673e" containerID="41f3da1b4d7fd19e1d2d60fef826cca2e824c57b7b33ed52c044058f59689dba" exitCode=0 Oct 30 01:44:10.749394 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:10.749332 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c22e6b-7cd96f6bdc-p9l2f" Oct 30 01:44:10.749394 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:10.749359 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c22e6b-7cd96f6bdc-p9l2f" event={"ID":"fac5d647-98c6-4ac2-97ba-c66b93d7673e","Type":"ContainerDied","Data":"41f3da1b4d7fd19e1d2d60fef826cca2e824c57b7b33ed52c044058f59689dba"} Oct 30 01:44:10.749520 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:10.749397 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c22e6b-7cd96f6bdc-p9l2f" event={"ID":"fac5d647-98c6-4ac2-97ba-c66b93d7673e","Type":"ContainerDied","Data":"c2ef06b0f6be875e32b2814c4236e00972b09dce2b70406f923d2b86aedcf82b"} Oct 30 01:44:10.761196 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:10.761160 6572 scope.go:122] "RemoveContainer" containerID="22a40e88b3d0cf06e5201ae0ffe987ffb1b120dd4917e9d16134148d9c6046d4" Oct 30 01:44:10.761539 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:44:10.761516 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22a40e88b3d0cf06e5201ae0ffe987ffb1b120dd4917e9d16134148d9c6046d4\": container with ID starting with 22a40e88b3d0cf06e5201ae0ffe987ffb1b120dd4917e9d16134148d9c6046d4 not found: ID does not exist" containerID="22a40e88b3d0cf06e5201ae0ffe987ffb1b120dd4917e9d16134148d9c6046d4" Oct 30 01:44:10.761643 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:10.761545 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22a40e88b3d0cf06e5201ae0ffe987ffb1b120dd4917e9d16134148d9c6046d4"} err="failed to get container status \"22a40e88b3d0cf06e5201ae0ffe987ffb1b120dd4917e9d16134148d9c6046d4\": rpc error: code = NotFound desc = could not find container \"22a40e88b3d0cf06e5201ae0ffe987ffb1b120dd4917e9d16134148d9c6046d4\": container with ID starting with 22a40e88b3d0cf06e5201ae0ffe987ffb1b120dd4917e9d16134148d9c6046d4 not found: ID does not exist" Oct 30 01:44:10.761643 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:10.761563 6572 scope.go:122] "RemoveContainer" containerID="41f3da1b4d7fd19e1d2d60fef826cca2e824c57b7b33ed52c044058f59689dba" Oct 30 01:44:10.771133 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:10.771118 6572 scope.go:122] "RemoveContainer" containerID="41f3da1b4d7fd19e1d2d60fef826cca2e824c57b7b33ed52c044058f59689dba" Oct 30 01:44:10.771416 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:44:10.771398 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41f3da1b4d7fd19e1d2d60fef826cca2e824c57b7b33ed52c044058f59689dba\": container with ID starting with 41f3da1b4d7fd19e1d2d60fef826cca2e824c57b7b33ed52c044058f59689dba not found: ID does not exist" containerID="41f3da1b4d7fd19e1d2d60fef826cca2e824c57b7b33ed52c044058f59689dba" Oct 30 01:44:10.771466 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:10.771425 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f3da1b4d7fd19e1d2d60fef826cca2e824c57b7b33ed52c044058f59689dba"} err="failed to get container status \"41f3da1b4d7fd19e1d2d60fef826cca2e824c57b7b33ed52c044058f59689dba\": rpc error: code = NotFound desc = could not find container \"41f3da1b4d7fd19e1d2d60fef826cca2e824c57b7b33ed52c044058f59689dba\": container with ID starting with 41f3da1b4d7fd19e1d2d60fef826cca2e824c57b7b33ed52c044058f59689dba not found: ID does not exist" Oct 30 01:44:10.798471 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:10.798448 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c22e6b-7cd96f6bdc-p9l2f"] Oct 30 01:44:10.809830 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:10.809809 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c22e6b-7cd96f6bdc-p9l2f"] Oct 30 01:44:10.841230 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:10.841198 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c22e6b-6486d6dc9b-gkwvs"] Oct 30 01:44:10.848047 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:10.848019 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c22e6b-6486d6dc9b-gkwvs"] Oct 30 01:44:10.985546 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:10.985514 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:11.059546 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:11.059514 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c22e6b-c78dd"] Oct 30 01:44:11.064617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:11.064584 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c22e6b-c78dd"] Oct 30 01:44:11.191245 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:11.191210 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:11.735315 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:11.735281 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:12.543224 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:12.543189 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d1710d2-4436-4f5b-bfb9-82bd8210f7b3" path="/var/lib/kubelet/pods/0d1710d2-4436-4f5b-bfb9-82bd8210f7b3/volumes" Oct 30 01:44:12.543660 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:12.543611 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39155e3d-3a29-478e-8b36-5981ef046b0a" path="/var/lib/kubelet/pods/39155e3d-3a29-478e-8b36-5981ef046b0a/volumes" Oct 30 01:44:12.544004 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:12.543989 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fac5d647-98c6-4ac2-97ba-c66b93d7673e" path="/var/lib/kubelet/pods/fac5d647-98c6-4ac2-97ba-c66b93d7673e/volumes" Oct 30 01:44:13.220776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:13.220730 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:14.691552 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:44:14.691514 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:44:15.116273 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:15.116229 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:15.552782 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:15.552712 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7"] Oct 30 01:44:15.553035 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:15.553010 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" podUID="eced80d4-0bf9-4f4e-a25e-0448b3a2f70b" containerName="istio-proxy" containerID="cri-o://646812fd6e7a54a86ed8e647687f2b7f90232210e08da4ce53a5cd1afb084e21" gracePeriod=30 Oct 30 01:44:15.575168 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:15.575124 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:15.828004 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:15.827907 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:16.152367 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:16.152299 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f01cbc-zwfv9"] Oct 30 01:44:16.152877 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:16.152863 6572 cpu_manager.go:414] "containerMap: removing container" podUID="c07642d5-28e5-433c-91ee-b78c8077e419" containerName="revoke-keys" Oct 30 01:44:16.152920 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:16.152878 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c07642d5-28e5-433c-91ee-b78c8077e419" containerName="revoke-keys" Oct 30 01:44:16.152960 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:16.152910 6572 cpu_manager.go:414] "containerMap: removing container" podUID="0d1710d2-4436-4f5b-bfb9-82bd8210f7b3" containerName="payload-processing" Oct 30 01:44:16.152960 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:16.152940 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d1710d2-4436-4f5b-bfb9-82bd8210f7b3" containerName="payload-processing" Oct 30 01:44:16.152960 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:16.152952 6572 cpu_manager.go:414] "containerMap: removing container" podUID="8f535096-d7ec-4bc0-bd5f-208b97d396da" containerName="revoke-keys" Oct 30 01:44:16.152960 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:16.152957 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f535096-d7ec-4bc0-bd5f-208b97d396da" containerName="revoke-keys" Oct 30 01:44:16.153087 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:16.152969 6572 cpu_manager.go:414] "containerMap: removing container" podUID="fac5d647-98c6-4ac2-97ba-c66b93d7673e" containerName="payload-pre-processing" Oct 30 01:44:16.153087 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:16.152976 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fac5d647-98c6-4ac2-97ba-c66b93d7673e" containerName="payload-pre-processing" Oct 30 01:44:16.153087 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:16.153073 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="c07642d5-28e5-433c-91ee-b78c8077e419" containerName="revoke-keys" Oct 30 01:44:16.153087 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:16.153087 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d1710d2-4436-4f5b-bfb9-82bd8210f7b3" containerName="payload-processing" Oct 30 01:44:16.153209 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:16.153114 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f535096-d7ec-4bc0-bd5f-208b97d396da" containerName="revoke-keys" Oct 30 01:44:16.153209 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:16.153123 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="fac5d647-98c6-4ac2-97ba-c66b93d7673e" containerName="payload-pre-processing" Oct 30 01:44:16.156577 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:16.156561 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f01cbc-zwfv9" Oct 30 01:44:16.175698 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:16.175673 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:44:16.179430 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:16.179407 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f01cbc-zwfv9"] Oct 30 01:44:16.185388 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:16.185368 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:44:16.193301 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:16.193272 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdhsc\" (UniqueName: \"kubernetes.io/projected/391f6157-4713-4a87-b2b3-5eea33d13d09-kube-api-access-xdhsc\") pod \"maas-api-revoke-keys-e2e-ipp-b-f01cbc-zwfv9\" (UID: \"391f6157-4713-4a87-b2b3-5eea33d13d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f01cbc-zwfv9" Oct 30 01:44:16.193463 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:16.193443 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/391f6157-4713-4a87-b2b3-5eea33d13d09-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f01cbc-zwfv9\" (UID: \"391f6157-4713-4a87-b2b3-5eea33d13d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f01cbc-zwfv9" Oct 30 01:44:16.193556 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:16.193536 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/391f6157-4713-4a87-b2b3-5eea33d13d09-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f01cbc-zwfv9\" (UID: \"391f6157-4713-4a87-b2b3-5eea33d13d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f01cbc-zwfv9" Oct 30 01:44:16.238700 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:16.238671 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:44:16.238939 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:16.238923 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:44:16.294689 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:16.294659 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/391f6157-4713-4a87-b2b3-5eea33d13d09-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f01cbc-zwfv9\" (UID: \"391f6157-4713-4a87-b2b3-5eea33d13d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f01cbc-zwfv9" Oct 30 01:44:16.294856 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:16.294722 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/391f6157-4713-4a87-b2b3-5eea33d13d09-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f01cbc-zwfv9\" (UID: \"391f6157-4713-4a87-b2b3-5eea33d13d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f01cbc-zwfv9" Oct 30 01:44:16.294856 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:16.294837 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdhsc\" (UniqueName: \"kubernetes.io/projected/391f6157-4713-4a87-b2b3-5eea33d13d09-kube-api-access-xdhsc\") pod \"maas-api-revoke-keys-e2e-ipp-b-f01cbc-zwfv9\" (UID: \"391f6157-4713-4a87-b2b3-5eea33d13d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f01cbc-zwfv9" Oct 30 01:44:16.295412 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:16.295388 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/391f6157-4713-4a87-b2b3-5eea33d13d09-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f01cbc-zwfv9\" (UID: \"391f6157-4713-4a87-b2b3-5eea33d13d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f01cbc-zwfv9" Oct 30 01:44:16.296976 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:16.296957 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/391f6157-4713-4a87-b2b3-5eea33d13d09-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f01cbc-zwfv9\" (UID: \"391f6157-4713-4a87-b2b3-5eea33d13d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f01cbc-zwfv9" Oct 30 01:44:16.313666 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:16.313642 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdhsc\" (UniqueName: \"kubernetes.io/projected/391f6157-4713-4a87-b2b3-5eea33d13d09-kube-api-access-xdhsc\") pod \"maas-api-revoke-keys-e2e-ipp-b-f01cbc-zwfv9\" (UID: \"391f6157-4713-4a87-b2b3-5eea33d13d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f01cbc-zwfv9" Oct 30 01:44:16.469645 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:16.469606 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f01cbc-zwfv9" Oct 30 01:44:16.770385 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:16.770356 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f01cbc-zwfv9"] Oct 30 01:44:16.771865 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:44:16.771823 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod391f6157_4713_4a87_b2b3_5eea33d13d09.slice/crio-79987dbf1bf674003b6270158fe7aebdddbefc97ac64fe5bcddcd831cd4bb7e1 WatchSource:0}: Error finding container 79987dbf1bf674003b6270158fe7aebdddbefc97ac64fe5bcddcd831cd4bb7e1: Status 404 returned error can't find the container with id 79987dbf1bf674003b6270158fe7aebdddbefc97ac64fe5bcddcd831cd4bb7e1 Oct 30 01:44:16.833016 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:16.832980 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f01cbc-zwfv9" event={"ID":"391f6157-4713-4a87-b2b3-5eea33d13d09","Type":"ContainerStarted","Data":"79987dbf1bf674003b6270158fe7aebdddbefc97ac64fe5bcddcd831cd4bb7e1"} Oct 30 01:44:16.920969 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:16.920930 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:17.718633 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:17.718595 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:17.801846 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:17.801810 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c9d174-ndk9n"] Oct 30 01:44:17.804823 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:17.804799 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c9d174-ndk9n"] Oct 30 01:44:17.845721 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:17.845688 6572 generic.go:356] "Generic (PLEG): container finished" podID="391f6157-4713-4a87-b2b3-5eea33d13d09" containerID="af87b55d416f5f2d80311daac615ce2a4a1825340993bbc995fdfdcf76f9103b" exitCode=0 Oct 30 01:44:17.846323 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:17.845935 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f01cbc-zwfv9" event={"ID":"391f6157-4713-4a87-b2b3-5eea33d13d09","Type":"ContainerDied","Data":"af87b55d416f5f2d80311daac615ce2a4a1825340993bbc995fdfdcf76f9103b"} Oct 30 01:44:17.978254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:17.978174 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:18.184325 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:18.184293 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:18.409067 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:18.408980 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:18.542829 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:18.542799 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f535096-d7ec-4bc0-bd5f-208b97d396da" path="/var/lib/kubelet/pods/8f535096-d7ec-4bc0-bd5f-208b97d396da/volumes" Oct 30 01:44:19.001242 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:19.001212 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f01cbc-zwfv9" Oct 30 01:44:19.137643 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:19.137567 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/391f6157-4713-4a87-b2b3-5eea33d13d09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/391f6157-4713-4a87-b2b3-5eea33d13d09-maas-api-xks-ca\") pod \"391f6157-4713-4a87-b2b3-5eea33d13d09\" (UID: \"391f6157-4713-4a87-b2b3-5eea33d13d09\") " Oct 30 01:44:19.137643 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:19.137635 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/391f6157-4713-4a87-b2b3-5eea33d13d09-kube-api-access-xdhsc\" (UniqueName: \"kubernetes.io/projected/391f6157-4713-4a87-b2b3-5eea33d13d09-kube-api-access-xdhsc\") pod \"391f6157-4713-4a87-b2b3-5eea33d13d09\" (UID: \"391f6157-4713-4a87-b2b3-5eea33d13d09\") " Oct 30 01:44:19.137938 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:19.137709 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/391f6157-4713-4a87-b2b3-5eea33d13d09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/391f6157-4713-4a87-b2b3-5eea33d13d09-maas-api-openshift-ca\") pod \"391f6157-4713-4a87-b2b3-5eea33d13d09\" (UID: \"391f6157-4713-4a87-b2b3-5eea33d13d09\") " Oct 30 01:44:19.138136 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:19.138110 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/391f6157-4713-4a87-b2b3-5eea33d13d09-maas-api-openshift-ca" pod "391f6157-4713-4a87-b2b3-5eea33d13d09" (UID: "391f6157-4713-4a87-b2b3-5eea33d13d09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:44:19.139813 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:19.139784 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/391f6157-4713-4a87-b2b3-5eea33d13d09-maas-api-xks-ca" pod "391f6157-4713-4a87-b2b3-5eea33d13d09" (UID: "391f6157-4713-4a87-b2b3-5eea33d13d09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:44:19.139813 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:19.139795 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/391f6157-4713-4a87-b2b3-5eea33d13d09-kube-api-access-xdhsc" pod "391f6157-4713-4a87-b2b3-5eea33d13d09" (UID: "391f6157-4713-4a87-b2b3-5eea33d13d09"). InnerVolumeSpecName "kube-api-access-xdhsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:19.238762 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:19.238722 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/391f6157-4713-4a87-b2b3-5eea33d13d09-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:19.238918 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:19.238786 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/391f6157-4713-4a87-b2b3-5eea33d13d09-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:19.238918 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:19.238802 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdhsc\" (UniqueName: \"kubernetes.io/projected/391f6157-4713-4a87-b2b3-5eea33d13d09-kube-api-access-xdhsc\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:19.608843 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:19.608807 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:19.821498 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:19.821469 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:19.861478 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:19.861395 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f01cbc-zwfv9" event={"ID":"391f6157-4713-4a87-b2b3-5eea33d13d09","Type":"ContainerDied","Data":"79987dbf1bf674003b6270158fe7aebdddbefc97ac64fe5bcddcd831cd4bb7e1"} Oct 30 01:44:19.861478 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:19.861452 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79987dbf1bf674003b6270158fe7aebdddbefc97ac64fe5bcddcd831cd4bb7e1" Oct 30 01:44:19.861478 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:19.861411 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f01cbc-zwfv9" Oct 30 01:44:20.036779 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.034214 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:20.104559 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.104528 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld"] Oct 30 01:44:20.104883 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.104853 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" podUID="37079845-e0dc-4eef-844e-eb7f15ae8cb3" containerName="istio-proxy" containerID="cri-o://48c63fa656a13f2e80a255e1f267a12b3f31e982b429a760bb7d79b776894c8d" gracePeriod=30 Oct 30 01:44:20.283096 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.283062 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:20.524714 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.524673 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-be8a7d-2ngwm"] Oct 30 01:44:20.525309 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.525292 6572 cpu_manager.go:414] "containerMap: removing container" podUID="391f6157-4713-4a87-b2b3-5eea33d13d09" containerName="revoke-keys" Oct 30 01:44:20.525309 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.525309 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="391f6157-4713-4a87-b2b3-5eea33d13d09" containerName="revoke-keys" Oct 30 01:44:20.525450 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.525434 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="391f6157-4713-4a87-b2b3-5eea33d13d09" containerName="revoke-keys" Oct 30 01:44:20.529111 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.529093 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-be8a7d-2ngwm" Oct 30 01:44:20.545618 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.545594 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:44:20.545881 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.545862 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:44:20.546946 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.546928 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:44:20.551068 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.551047 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-be8a7d-2ngwm"] Oct 30 01:44:20.556665 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.556643 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:44:20.655001 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.654969 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05d5eccf-17ad-4a3e-a6fd-c9fb4c136721-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-be8a7d-2ngwm\" (UID: \"05d5eccf-17ad-4a3e-a6fd-c9fb4c136721\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-be8a7d-2ngwm" Oct 30 01:44:20.655145 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.655006 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6rtp\" (UniqueName: \"kubernetes.io/projected/05d5eccf-17ad-4a3e-a6fd-c9fb4c136721-kube-api-access-f6rtp\") pod \"maas-api-revoke-keys-e2e-praxis-b-be8a7d-2ngwm\" (UID: \"05d5eccf-17ad-4a3e-a6fd-c9fb4c136721\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-be8a7d-2ngwm" Oct 30 01:44:20.655262 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.655240 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05d5eccf-17ad-4a3e-a6fd-c9fb4c136721-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-be8a7d-2ngwm\" (UID: \"05d5eccf-17ad-4a3e-a6fd-c9fb4c136721\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-be8a7d-2ngwm" Oct 30 01:44:20.757016 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.756974 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05d5eccf-17ad-4a3e-a6fd-c9fb4c136721-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-be8a7d-2ngwm\" (UID: \"05d5eccf-17ad-4a3e-a6fd-c9fb4c136721\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-be8a7d-2ngwm" Oct 30 01:44:20.757233 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.757212 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05d5eccf-17ad-4a3e-a6fd-c9fb4c136721-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-be8a7d-2ngwm\" (UID: \"05d5eccf-17ad-4a3e-a6fd-c9fb4c136721\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-be8a7d-2ngwm" Oct 30 01:44:20.757341 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.757251 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6rtp\" (UniqueName: \"kubernetes.io/projected/05d5eccf-17ad-4a3e-a6fd-c9fb4c136721-kube-api-access-f6rtp\") pod \"maas-api-revoke-keys-e2e-praxis-b-be8a7d-2ngwm\" (UID: \"05d5eccf-17ad-4a3e-a6fd-c9fb4c136721\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-be8a7d-2ngwm" Oct 30 01:44:20.757847 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.757818 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05d5eccf-17ad-4a3e-a6fd-c9fb4c136721-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-be8a7d-2ngwm\" (UID: \"05d5eccf-17ad-4a3e-a6fd-c9fb4c136721\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-be8a7d-2ngwm" Oct 30 01:44:20.760012 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.759985 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05d5eccf-17ad-4a3e-a6fd-c9fb4c136721-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-be8a7d-2ngwm\" (UID: \"05d5eccf-17ad-4a3e-a6fd-c9fb4c136721\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-be8a7d-2ngwm" Oct 30 01:44:20.788661 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.788599 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6rtp\" (UniqueName: \"kubernetes.io/projected/05d5eccf-17ad-4a3e-a6fd-c9fb4c136721-kube-api-access-f6rtp\") pod \"maas-api-revoke-keys-e2e-praxis-b-be8a7d-2ngwm\" (UID: \"05d5eccf-17ad-4a3e-a6fd-c9fb4c136721\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-be8a7d-2ngwm" Oct 30 01:44:20.831195 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.831171 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:44:20.842991 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.842968 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-be8a7d-2ngwm" Oct 30 01:44:20.872179 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.872135 6572 generic.go:356] "Generic (PLEG): container finished" podID="eced80d4-0bf9-4f4e-a25e-0448b3a2f70b" containerID="646812fd6e7a54a86ed8e647687f2b7f90232210e08da4ce53a5cd1afb084e21" exitCode=0 Oct 30 01:44:20.872292 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.872251 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" Oct 30 01:44:20.872292 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.872257 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" event={"ID":"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b","Type":"ContainerDied","Data":"646812fd6e7a54a86ed8e647687f2b7f90232210e08da4ce53a5cd1afb084e21"} Oct 30 01:44:20.872376 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.872314 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7" event={"ID":"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b","Type":"ContainerDied","Data":"69af19c67252dd2733113e4d762d7625b8e4b1168c726a7e22c170e15e86de7f"} Oct 30 01:44:20.872376 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.872338 6572 scope.go:122] "RemoveContainer" containerID="646812fd6e7a54a86ed8e647687f2b7f90232210e08da4ce53a5cd1afb084e21" Oct 30 01:44:20.886041 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.885910 6572 scope.go:122] "RemoveContainer" containerID="6e137d476a869f00b916844a51918cba2c9b8f33aaccdbcba7b970bea4ff74c3" Oct 30 01:44:20.899101 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.899082 6572 scope.go:122] "RemoveContainer" containerID="646812fd6e7a54a86ed8e647687f2b7f90232210e08da4ce53a5cd1afb084e21" Oct 30 01:44:20.899535 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:44:20.899511 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"646812fd6e7a54a86ed8e647687f2b7f90232210e08da4ce53a5cd1afb084e21\": container with ID starting with 646812fd6e7a54a86ed8e647687f2b7f90232210e08da4ce53a5cd1afb084e21 not found: ID does not exist" containerID="646812fd6e7a54a86ed8e647687f2b7f90232210e08da4ce53a5cd1afb084e21" Oct 30 01:44:20.899637 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.899545 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"646812fd6e7a54a86ed8e647687f2b7f90232210e08da4ce53a5cd1afb084e21"} err="failed to get container status \"646812fd6e7a54a86ed8e647687f2b7f90232210e08da4ce53a5cd1afb084e21\": rpc error: code = NotFound desc = could not find container \"646812fd6e7a54a86ed8e647687f2b7f90232210e08da4ce53a5cd1afb084e21\": container with ID starting with 646812fd6e7a54a86ed8e647687f2b7f90232210e08da4ce53a5cd1afb084e21 not found: ID does not exist" Oct 30 01:44:20.899637 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.899566 6572 scope.go:122] "RemoveContainer" containerID="6e137d476a869f00b916844a51918cba2c9b8f33aaccdbcba7b970bea4ff74c3" Oct 30 01:44:20.900134 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:44:20.900108 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e137d476a869f00b916844a51918cba2c9b8f33aaccdbcba7b970bea4ff74c3\": container with ID starting with 6e137d476a869f00b916844a51918cba2c9b8f33aaccdbcba7b970bea4ff74c3 not found: ID does not exist" containerID="6e137d476a869f00b916844a51918cba2c9b8f33aaccdbcba7b970bea4ff74c3" Oct 30 01:44:20.900249 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.900143 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e137d476a869f00b916844a51918cba2c9b8f33aaccdbcba7b970bea4ff74c3"} err="failed to get container status \"6e137d476a869f00b916844a51918cba2c9b8f33aaccdbcba7b970bea4ff74c3\": rpc error: code = NotFound desc = could not find container \"6e137d476a869f00b916844a51918cba2c9b8f33aaccdbcba7b970bea4ff74c3\": container with ID starting with 6e137d476a869f00b916844a51918cba2c9b8f33aaccdbcba7b970bea4ff74c3 not found: ID does not exist" Oct 30 01:44:20.959482 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.959451 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-istio-podinfo\") pod \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\" (UID: \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\") " Oct 30 01:44:20.959625 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.959507 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-istiod-ca-cert\") pod \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\" (UID: \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\") " Oct 30 01:44:20.959706 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.959636 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-workload-certs\") pod \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\" (UID: \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\") " Oct 30 01:44:20.959706 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.959673 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-istio-envoy\") pod \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\" (UID: \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\") " Oct 30 01:44:20.959845 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.959708 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-credential-socket\") pod \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\" (UID: \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\") " Oct 30 01:44:20.959845 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.959764 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-istio-data\") pod \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\" (UID: \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\") " Oct 30 01:44:20.959997 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.959907 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-workload-socket\") pod \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\" (UID: \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\") " Oct 30 01:44:20.959997 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.959964 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-kube-api-access-9fp6t\" (UniqueName: \"kubernetes.io/projected/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-kube-api-access-9fp6t\") pod \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\" (UID: \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\") " Oct 30 01:44:20.959997 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.959965 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-workload-certs" pod "eced80d4-0bf9-4f4e-a25e-0448b3a2f70b" (UID: "eced80d4-0bf9-4f4e-a25e-0448b3a2f70b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:20.960180 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.960002 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-istio-token\" (UniqueName: \"kubernetes.io/projected/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-istio-token\") pod \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\" (UID: \"eced80d4-0bf9-4f4e-a25e-0448b3a2f70b\") " Oct 30 01:44:20.960343 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.960322 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-credential-socket" pod "eced80d4-0bf9-4f4e-a25e-0448b3a2f70b" (UID: "eced80d4-0bf9-4f4e-a25e-0448b3a2f70b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:20.960478 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.960456 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-istio-data" pod "eced80d4-0bf9-4f4e-a25e-0448b3a2f70b" (UID: "eced80d4-0bf9-4f4e-a25e-0448b3a2f70b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:20.960535 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.960517 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:20.960600 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.960537 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:20.960711 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.960695 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-istiod-ca-cert" pod "eced80d4-0bf9-4f4e-a25e-0448b3a2f70b" (UID: "eced80d4-0bf9-4f4e-a25e-0448b3a2f70b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:44:20.960881 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.960858 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-workload-socket" pod "eced80d4-0bf9-4f4e-a25e-0448b3a2f70b" (UID: "eced80d4-0bf9-4f4e-a25e-0448b3a2f70b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:20.962239 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.962211 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-istio-podinfo" pod "eced80d4-0bf9-4f4e-a25e-0448b3a2f70b" (UID: "eced80d4-0bf9-4f4e-a25e-0448b3a2f70b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:44:20.962483 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.962463 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-istio-envoy" pod "eced80d4-0bf9-4f4e-a25e-0448b3a2f70b" (UID: "eced80d4-0bf9-4f4e-a25e-0448b3a2f70b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:20.962735 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.962718 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-istio-token" pod "eced80d4-0bf9-4f4e-a25e-0448b3a2f70b" (UID: "eced80d4-0bf9-4f4e-a25e-0448b3a2f70b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:20.963013 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:20.962989 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-kube-api-access-9fp6t" pod "eced80d4-0bf9-4f4e-a25e-0448b3a2f70b" (UID: "eced80d4-0bf9-4f4e-a25e-0448b3a2f70b"). InnerVolumeSpecName "kube-api-access-9fp6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:21.061427 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:21.061340 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:21.061427 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:21.061365 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:21.061427 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:21.061376 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:21.061427 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:21.061387 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fp6t\" (UniqueName: \"kubernetes.io/projected/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-kube-api-access-9fp6t\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:21.061427 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:21.061399 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:21.061427 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:21.061411 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:21.061427 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:21.061420 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:21.146687 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:21.146662 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-be8a7d-2ngwm"] Oct 30 01:44:21.147145 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:44:21.147118 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05d5eccf_17ad_4a3e_a6fd_c9fb4c136721.slice/crio-8abe3b05e42a707207a43ea7a7f820f14e6a25c0f021564bfac21b10c44c951a WatchSource:0}: Error finding container 8abe3b05e42a707207a43ea7a7f820f14e6a25c0f021564bfac21b10c44c951a: Status 404 returned error can't find the container with id 8abe3b05e42a707207a43ea7a7f820f14e6a25c0f021564bfac21b10c44c951a Oct 30 01:44:21.225999 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:21.225960 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7"] Oct 30 01:44:21.245767 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:21.230231 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-6b3dbd-openshift-default-59bfc5df-54hw7"] Oct 30 01:44:21.883050 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:21.882959 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-be8a7d-2ngwm" event={"ID":"05d5eccf-17ad-4a3e-a6fd-c9fb4c136721","Type":"ContainerStarted","Data":"ec732daa0b1aa2efa1b32ec1597c5839de3c67dc2955245967b828c062cac3f3"} Oct 30 01:44:21.883235 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:21.883053 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-be8a7d-2ngwm" event={"ID":"05d5eccf-17ad-4a3e-a6fd-c9fb4c136721","Type":"ContainerStarted","Data":"8abe3b05e42a707207a43ea7a7f820f14e6a25c0f021564bfac21b10c44c951a"} Oct 30 01:44:21.933554 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:21.933498 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-be8a7d-2ngwm" podStartSLOduration=1.9334843350000002 podStartE2EDuration="1.933484335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:44:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:44:21.932528373 +0000 UTC m=+1587.508953769" watchObservedRunningTime="2026-10-30 01:44:21.933484335 +0000 UTC m=+1587.509909755" Oct 30 01:44:22.243168 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:22.243133 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:22.543167 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:22.543083 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eced80d4-0bf9-4f4e-a25e-0448b3a2f70b" path="/var/lib/kubelet/pods/eced80d4-0bf9-4f4e-a25e-0448b3a2f70b/volumes" Oct 30 01:44:22.892213 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:22.892128 6572 generic.go:356] "Generic (PLEG): container finished" podID="05d5eccf-17ad-4a3e-a6fd-c9fb4c136721" containerID="ec732daa0b1aa2efa1b32ec1597c5839de3c67dc2955245967b828c062cac3f3" exitCode=0 Oct 30 01:44:22.892364 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:22.892210 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-be8a7d-2ngwm" event={"ID":"05d5eccf-17ad-4a3e-a6fd-c9fb4c136721","Type":"ContainerDied","Data":"ec732daa0b1aa2efa1b32ec1597c5839de3c67dc2955245967b828c062cac3f3"} Oct 30 01:44:24.065361 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:24.065308 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-be8a7d-2ngwm" Oct 30 01:44:24.112531 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:24.112497 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:24.198633 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:24.198605 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05d5eccf-17ad-4a3e-a6fd-c9fb4c136721-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05d5eccf-17ad-4a3e-a6fd-c9fb4c136721-maas-api-xks-ca\") pod \"05d5eccf-17ad-4a3e-a6fd-c9fb4c136721\" (UID: \"05d5eccf-17ad-4a3e-a6fd-c9fb4c136721\") " Oct 30 01:44:24.198869 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:24.198691 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05d5eccf-17ad-4a3e-a6fd-c9fb4c136721-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05d5eccf-17ad-4a3e-a6fd-c9fb4c136721-maas-api-openshift-ca\") pod \"05d5eccf-17ad-4a3e-a6fd-c9fb4c136721\" (UID: \"05d5eccf-17ad-4a3e-a6fd-c9fb4c136721\") " Oct 30 01:44:24.198869 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:24.198826 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05d5eccf-17ad-4a3e-a6fd-c9fb4c136721-kube-api-access-f6rtp\" (UniqueName: \"kubernetes.io/projected/05d5eccf-17ad-4a3e-a6fd-c9fb4c136721-kube-api-access-f6rtp\") pod \"05d5eccf-17ad-4a3e-a6fd-c9fb4c136721\" (UID: \"05d5eccf-17ad-4a3e-a6fd-c9fb4c136721\") " Oct 30 01:44:24.199114 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:24.199076 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05d5eccf-17ad-4a3e-a6fd-c9fb4c136721-maas-api-openshift-ca" pod "05d5eccf-17ad-4a3e-a6fd-c9fb4c136721" (UID: "05d5eccf-17ad-4a3e-a6fd-c9fb4c136721"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:44:24.200964 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:24.200941 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05d5eccf-17ad-4a3e-a6fd-c9fb4c136721-maas-api-xks-ca" pod "05d5eccf-17ad-4a3e-a6fd-c9fb4c136721" (UID: "05d5eccf-17ad-4a3e-a6fd-c9fb4c136721"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:44:24.201043 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:24.201012 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05d5eccf-17ad-4a3e-a6fd-c9fb4c136721-kube-api-access-f6rtp" pod "05d5eccf-17ad-4a3e-a6fd-c9fb4c136721" (UID: "05d5eccf-17ad-4a3e-a6fd-c9fb4c136721"). InnerVolumeSpecName "kube-api-access-f6rtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:24.300116 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:24.300086 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6rtp\" (UniqueName: \"kubernetes.io/projected/05d5eccf-17ad-4a3e-a6fd-c9fb4c136721-kube-api-access-f6rtp\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:24.300116 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:24.300113 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05d5eccf-17ad-4a3e-a6fd-c9fb4c136721-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:24.300323 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:24.300127 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05d5eccf-17ad-4a3e-a6fd-c9fb4c136721-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:24.601506 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:24.601467 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:24.849352 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:24.849268 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:24.895870 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:24.895842 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2"] Oct 30 01:44:24.896059 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:24.896035 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" podUID="98cd042c-e590-4f84-8acc-220357883bef" containerName="istio-proxy" containerID="cri-o://7b7e97c2fb8ce21981618ce287f92a3a95843b1fc3c880d33c56ba354061375e" gracePeriod=30 Oct 30 01:44:24.909281 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:24.909260 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-be8a7d-2ngwm" Oct 30 01:44:24.909425 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:24.909297 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-be8a7d-2ngwm" event={"ID":"05d5eccf-17ad-4a3e-a6fd-c9fb4c136721","Type":"ContainerDied","Data":"8abe3b05e42a707207a43ea7a7f820f14e6a25c0f021564bfac21b10c44c951a"} Oct 30 01:44:24.909425 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:24.909358 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8abe3b05e42a707207a43ea7a7f820f14e6a25c0f021564bfac21b10c44c951a" Oct 30 01:44:25.075654 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.075619 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:25.377898 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.377842 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" Oct 30 01:44:25.515637 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.515595 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37079845-e0dc-4eef-844e-eb7f15ae8cb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37079845-e0dc-4eef-844e-eb7f15ae8cb3-istiod-ca-cert\") pod \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\" (UID: \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\") " Oct 30 01:44:25.515637 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.515636 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37079845-e0dc-4eef-844e-eb7f15ae8cb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37079845-e0dc-4eef-844e-eb7f15ae8cb3-istio-data\") pod \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\" (UID: \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\") " Oct 30 01:44:25.515941 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.515675 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37079845-e0dc-4eef-844e-eb7f15ae8cb3-istio-token\" (UniqueName: \"kubernetes.io/projected/37079845-e0dc-4eef-844e-eb7f15ae8cb3-istio-token\") pod \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\" (UID: \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\") " Oct 30 01:44:25.515941 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.515824 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37079845-e0dc-4eef-844e-eb7f15ae8cb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37079845-e0dc-4eef-844e-eb7f15ae8cb3-credential-socket\") pod \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\" (UID: \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\") " Oct 30 01:44:25.515941 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.515885 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37079845-e0dc-4eef-844e-eb7f15ae8cb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37079845-e0dc-4eef-844e-eb7f15ae8cb3-workload-certs\") pod \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\" (UID: \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\") " Oct 30 01:44:25.515941 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.515920 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37079845-e0dc-4eef-844e-eb7f15ae8cb3-kube-api-access-dmdsm\" (UniqueName: \"kubernetes.io/projected/37079845-e0dc-4eef-844e-eb7f15ae8cb3-kube-api-access-dmdsm\") pod \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\" (UID: \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\") " Oct 30 01:44:25.516164 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.515992 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37079845-e0dc-4eef-844e-eb7f15ae8cb3-istio-data" pod "37079845-e0dc-4eef-844e-eb7f15ae8cb3" (UID: "37079845-e0dc-4eef-844e-eb7f15ae8cb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:25.516164 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.516016 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37079845-e0dc-4eef-844e-eb7f15ae8cb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37079845-e0dc-4eef-844e-eb7f15ae8cb3-workload-socket\") pod \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\" (UID: \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\") " Oct 30 01:44:25.516164 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.516110 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37079845-e0dc-4eef-844e-eb7f15ae8cb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37079845-e0dc-4eef-844e-eb7f15ae8cb3-istio-envoy\") pod \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\" (UID: \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\") " Oct 30 01:44:25.516164 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.516124 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37079845-e0dc-4eef-844e-eb7f15ae8cb3-workload-certs" pod "37079845-e0dc-4eef-844e-eb7f15ae8cb3" (UID: "37079845-e0dc-4eef-844e-eb7f15ae8cb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:25.516164 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.516142 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37079845-e0dc-4eef-844e-eb7f15ae8cb3-credential-socket" pod "37079845-e0dc-4eef-844e-eb7f15ae8cb3" (UID: "37079845-e0dc-4eef-844e-eb7f15ae8cb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:25.516441 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.516039 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37079845-e0dc-4eef-844e-eb7f15ae8cb3-istiod-ca-cert" pod "37079845-e0dc-4eef-844e-eb7f15ae8cb3" (UID: "37079845-e0dc-4eef-844e-eb7f15ae8cb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:44:25.516441 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.516165 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37079845-e0dc-4eef-844e-eb7f15ae8cb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37079845-e0dc-4eef-844e-eb7f15ae8cb3-istio-podinfo\") pod \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\" (UID: \"37079845-e0dc-4eef-844e-eb7f15ae8cb3\") " Oct 30 01:44:25.516441 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.516342 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37079845-e0dc-4eef-844e-eb7f15ae8cb3-workload-socket" pod "37079845-e0dc-4eef-844e-eb7f15ae8cb3" (UID: "37079845-e0dc-4eef-844e-eb7f15ae8cb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:25.516780 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.516738 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37079845-e0dc-4eef-844e-eb7f15ae8cb3-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:25.516882 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.516787 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37079845-e0dc-4eef-844e-eb7f15ae8cb3-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:25.516882 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.516800 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37079845-e0dc-4eef-844e-eb7f15ae8cb3-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:25.516882 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.516814 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37079845-e0dc-4eef-844e-eb7f15ae8cb3-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:25.516882 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.516830 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37079845-e0dc-4eef-844e-eb7f15ae8cb3-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:25.518324 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.518303 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37079845-e0dc-4eef-844e-eb7f15ae8cb3-istio-podinfo" pod "37079845-e0dc-4eef-844e-eb7f15ae8cb3" (UID: "37079845-e0dc-4eef-844e-eb7f15ae8cb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:44:25.518424 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.518327 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37079845-e0dc-4eef-844e-eb7f15ae8cb3-kube-api-access-dmdsm" pod "37079845-e0dc-4eef-844e-eb7f15ae8cb3" (UID: "37079845-e0dc-4eef-844e-eb7f15ae8cb3"). InnerVolumeSpecName "kube-api-access-dmdsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:25.518593 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.518571 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37079845-e0dc-4eef-844e-eb7f15ae8cb3-istio-envoy" pod "37079845-e0dc-4eef-844e-eb7f15ae8cb3" (UID: "37079845-e0dc-4eef-844e-eb7f15ae8cb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:25.518593 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.518581 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37079845-e0dc-4eef-844e-eb7f15ae8cb3-istio-token" pod "37079845-e0dc-4eef-844e-eb7f15ae8cb3" (UID: "37079845-e0dc-4eef-844e-eb7f15ae8cb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:25.617779 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.617721 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37079845-e0dc-4eef-844e-eb7f15ae8cb3-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:25.617938 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.617786 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmdsm\" (UniqueName: \"kubernetes.io/projected/37079845-e0dc-4eef-844e-eb7f15ae8cb3-kube-api-access-dmdsm\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:25.617938 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.617800 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37079845-e0dc-4eef-844e-eb7f15ae8cb3-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:25.617938 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.617809 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37079845-e0dc-4eef-844e-eb7f15ae8cb3-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:25.921117 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.921078 6572 generic.go:356] "Generic (PLEG): container finished" podID="37079845-e0dc-4eef-844e-eb7f15ae8cb3" containerID="48c63fa656a13f2e80a255e1f267a12b3f31e982b429a760bb7d79b776894c8d" exitCode=0 Oct 30 01:44:25.921318 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.921144 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" Oct 30 01:44:25.921318 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.921166 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" event={"ID":"37079845-e0dc-4eef-844e-eb7f15ae8cb3","Type":"ContainerDied","Data":"48c63fa656a13f2e80a255e1f267a12b3f31e982b429a760bb7d79b776894c8d"} Oct 30 01:44:25.921318 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.921267 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld" event={"ID":"37079845-e0dc-4eef-844e-eb7f15ae8cb3","Type":"ContainerDied","Data":"8b77b2c939a0be216d074754e52da2903c97ee4c3145d45895914b361671dd51"} Oct 30 01:44:25.921318 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.921298 6572 scope.go:122] "RemoveContainer" containerID="48c63fa656a13f2e80a255e1f267a12b3f31e982b429a760bb7d79b776894c8d" Oct 30 01:44:25.934981 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.934962 6572 scope.go:122] "RemoveContainer" containerID="48c63fa656a13f2e80a255e1f267a12b3f31e982b429a760bb7d79b776894c8d" Oct 30 01:44:25.935484 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:44:25.935412 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48c63fa656a13f2e80a255e1f267a12b3f31e982b429a760bb7d79b776894c8d\": container with ID starting with 48c63fa656a13f2e80a255e1f267a12b3f31e982b429a760bb7d79b776894c8d not found: ID does not exist" containerID="48c63fa656a13f2e80a255e1f267a12b3f31e982b429a760bb7d79b776894c8d" Oct 30 01:44:25.935484 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.935449 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48c63fa656a13f2e80a255e1f267a12b3f31e982b429a760bb7d79b776894c8d"} err="failed to get container status \"48c63fa656a13f2e80a255e1f267a12b3f31e982b429a760bb7d79b776894c8d\": rpc error: code = NotFound desc = could not find container \"48c63fa656a13f2e80a255e1f267a12b3f31e982b429a760bb7d79b776894c8d\": container with ID starting with 48c63fa656a13f2e80a255e1f267a12b3f31e982b429a760bb7d79b776894c8d not found: ID does not exist" Oct 30 01:44:25.982692 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.982657 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld"] Oct 30 01:44:25.993826 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:25.993790 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rkqld"] Oct 30 01:44:26.004500 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:26.004470 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:26.542524 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:26.542492 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37079845-e0dc-4eef-844e-eb7f15ae8cb3" path="/var/lib/kubelet/pods/37079845-e0dc-4eef-844e-eb7f15ae8cb3/volumes" Oct 30 01:44:27.495452 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:27.495412 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:27.729509 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:27.729473 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:27.942651 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:27.942618 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:28.214231 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:28.214149 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:28.285067 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:28.285036 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f01cbc-zwfv9"] Oct 30 01:44:28.295548 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:28.295523 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f01cbc-zwfv9"] Oct 30 01:44:28.432254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:28.432220 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:28.542625 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:28.542551 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="391f6157-4713-4a87-b2b3-5eea33d13d09" path="/var/lib/kubelet/pods/391f6157-4713-4a87-b2b3-5eea33d13d09/volumes" Oct 30 01:44:29.145907 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:29.145877 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:29.603145 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:29.603114 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-493521fb-zttmx"] Oct 30 01:44:29.603650 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:29.603629 6572 cpu_manager.go:414] "containerMap: removing container" podUID="eced80d4-0bf9-4f4e-a25e-0448b3a2f70b" containerName="istio-proxy" Oct 30 01:44:29.603650 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:29.603649 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eced80d4-0bf9-4f4e-a25e-0448b3a2f70b" containerName="istio-proxy" Oct 30 01:44:29.603834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:29.603699 6572 cpu_manager.go:414] "containerMap: removing container" podUID="37079845-e0dc-4eef-844e-eb7f15ae8cb3" containerName="istio-proxy" Oct 30 01:44:29.603834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:29.603708 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37079845-e0dc-4eef-844e-eb7f15ae8cb3" containerName="istio-proxy" Oct 30 01:44:29.603834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:29.603728 6572 cpu_manager.go:414] "containerMap: removing container" podUID="05d5eccf-17ad-4a3e-a6fd-c9fb4c136721" containerName="revoke-keys" Oct 30 01:44:29.603834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:29.603735 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05d5eccf-17ad-4a3e-a6fd-c9fb4c136721" containerName="revoke-keys" Oct 30 01:44:29.603976 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:29.603877 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="37079845-e0dc-4eef-844e-eb7f15ae8cb3" containerName="istio-proxy" Oct 30 01:44:29.603976 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:29.603889 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="05d5eccf-17ad-4a3e-a6fd-c9fb4c136721" containerName="revoke-keys" Oct 30 01:44:29.603976 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:29.603896 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="eced80d4-0bf9-4f4e-a25e-0448b3a2f70b" containerName="istio-proxy" Oct 30 01:44:29.603976 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:29.603905 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="eced80d4-0bf9-4f4e-a25e-0448b3a2f70b" containerName="istio-proxy" Oct 30 01:44:29.608716 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:29.608699 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493521fb-zttmx" Oct 30 01:44:29.624845 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:29.624820 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:44:29.636324 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:29.636302 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:44:29.640795 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:29.636589 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:44:29.640795 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:29.636590 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:44:29.640795 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:29.640135 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-493521fb-zttmx"] Oct 30 01:44:29.770674 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:29.770639 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2588b110-cd82-4f31-a36b-4f2fc12876f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-493521fb-zttmx\" (UID: \"2588b110-cd82-4f31-a36b-4f2fc12876f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493521fb-zttmx" Oct 30 01:44:29.770876 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:29.770709 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2588b110-cd82-4f31-a36b-4f2fc12876f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-493521fb-zttmx\" (UID: \"2588b110-cd82-4f31-a36b-4f2fc12876f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493521fb-zttmx" Oct 30 01:44:29.770876 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:29.770838 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcstb\" (UniqueName: \"kubernetes.io/projected/2588b110-cd82-4f31-a36b-4f2fc12876f5-kube-api-access-lcstb\") pod \"maas-api-revoke-sub-493521fb-zttmx\" (UID: \"2588b110-cd82-4f31-a36b-4f2fc12876f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493521fb-zttmx" Oct 30 01:44:29.872317 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:29.872244 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2588b110-cd82-4f31-a36b-4f2fc12876f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-493521fb-zttmx\" (UID: \"2588b110-cd82-4f31-a36b-4f2fc12876f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493521fb-zttmx" Oct 30 01:44:29.872317 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:29.872282 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcstb\" (UniqueName: \"kubernetes.io/projected/2588b110-cd82-4f31-a36b-4f2fc12876f5-kube-api-access-lcstb\") pod \"maas-api-revoke-sub-493521fb-zttmx\" (UID: \"2588b110-cd82-4f31-a36b-4f2fc12876f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493521fb-zttmx" Oct 30 01:44:29.872530 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:29.872371 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2588b110-cd82-4f31-a36b-4f2fc12876f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-493521fb-zttmx\" (UID: \"2588b110-cd82-4f31-a36b-4f2fc12876f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493521fb-zttmx" Oct 30 01:44:29.872911 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:29.872888 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2588b110-cd82-4f31-a36b-4f2fc12876f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-493521fb-zttmx\" (UID: \"2588b110-cd82-4f31-a36b-4f2fc12876f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493521fb-zttmx" Oct 30 01:44:29.874698 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:29.874675 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2588b110-cd82-4f31-a36b-4f2fc12876f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-493521fb-zttmx\" (UID: \"2588b110-cd82-4f31-a36b-4f2fc12876f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493521fb-zttmx" Oct 30 01:44:29.890942 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:29.890916 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcstb\" (UniqueName: \"kubernetes.io/projected/2588b110-cd82-4f31-a36b-4f2fc12876f5-kube-api-access-lcstb\") pod \"maas-api-revoke-sub-493521fb-zttmx\" (UID: \"2588b110-cd82-4f31-a36b-4f2fc12876f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493521fb-zttmx" Oct 30 01:44:29.922258 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:29.921711 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493521fb-zttmx" Oct 30 01:44:30.088878 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.088740 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:30.133784 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.133672 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-648f7c96c4-8jp52"] Oct 30 01:44:30.133955 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.133926 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-648f7c96c4-8jp52" podUID="864805b2-2538-4ca4-89c8-f4b5e756c0a4" containerName="payload-processing" containerID="cri-o://324018c10e751618c08e963be7cfdc08b9bcb62428cec9788b2ee8cdb9f2727a" gracePeriod=30 Oct 30 01:44:30.154829 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.154802 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-9fc8b47c5-wxtb4"] Oct 30 01:44:30.155242 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.155036 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-9fc8b47c5-wxtb4" podUID="69aaaf90-0480-4fe7-bf6b-313ddb1f0feb" containerName="payload-pre-processing" containerID="cri-o://9f222c3a7682b4762b8eab2603a68ef36d034a686910c1c553550932de42babc" gracePeriod=30 Oct 30 01:44:30.243201 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.243177 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-493521fb-zttmx"] Oct 30 01:44:30.259572 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:44:30.259549 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2588b110_cd82_4f31_a36b_4f2fc12876f5.slice/crio-52c6555f27c42a698ad7090504d11e5a334cdd181a7f121c03f3b311a9d1cb1c WatchSource:0}: Error finding container 52c6555f27c42a698ad7090504d11e5a334cdd181a7f121c03f3b311a9d1cb1c: Status 404 returned error can't find the container with id 52c6555f27c42a698ad7090504d11e5a334cdd181a7f121c03f3b311a9d1cb1c Oct 30 01:44:30.275566 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.275537 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:44:30.363508 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.363419 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:30.378248 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.378225 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98cd042c-e590-4f84-8acc-220357883bef-istio-token\" (UniqueName: \"kubernetes.io/projected/98cd042c-e590-4f84-8acc-220357883bef-istio-token\") pod \"98cd042c-e590-4f84-8acc-220357883bef\" (UID: \"98cd042c-e590-4f84-8acc-220357883bef\") " Oct 30 01:44:30.378376 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.378259 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98cd042c-e590-4f84-8acc-220357883bef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98cd042c-e590-4f84-8acc-220357883bef-istiod-ca-cert\") pod \"98cd042c-e590-4f84-8acc-220357883bef\" (UID: \"98cd042c-e590-4f84-8acc-220357883bef\") " Oct 30 01:44:30.378376 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.378288 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98cd042c-e590-4f84-8acc-220357883bef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98cd042c-e590-4f84-8acc-220357883bef-istio-podinfo\") pod \"98cd042c-e590-4f84-8acc-220357883bef\" (UID: \"98cd042c-e590-4f84-8acc-220357883bef\") " Oct 30 01:44:30.378376 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.378308 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98cd042c-e590-4f84-8acc-220357883bef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cd042c-e590-4f84-8acc-220357883bef-credential-socket\") pod \"98cd042c-e590-4f84-8acc-220357883bef\" (UID: \"98cd042c-e590-4f84-8acc-220357883bef\") " Oct 30 01:44:30.378532 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.378383 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98cd042c-e590-4f84-8acc-220357883bef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98cd042c-e590-4f84-8acc-220357883bef-istio-envoy\") pod \"98cd042c-e590-4f84-8acc-220357883bef\" (UID: \"98cd042c-e590-4f84-8acc-220357883bef\") " Oct 30 01:44:30.378532 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.378421 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98cd042c-e590-4f84-8acc-220357883bef-kube-api-access-zgqgf\" (UniqueName: \"kubernetes.io/projected/98cd042c-e590-4f84-8acc-220357883bef-kube-api-access-zgqgf\") pod \"98cd042c-e590-4f84-8acc-220357883bef\" (UID: \"98cd042c-e590-4f84-8acc-220357883bef\") " Oct 30 01:44:30.378532 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.378447 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98cd042c-e590-4f84-8acc-220357883bef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cd042c-e590-4f84-8acc-220357883bef-workload-socket\") pod \"98cd042c-e590-4f84-8acc-220357883bef\" (UID: \"98cd042c-e590-4f84-8acc-220357883bef\") " Oct 30 01:44:30.378532 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.378480 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98cd042c-e590-4f84-8acc-220357883bef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98cd042c-e590-4f84-8acc-220357883bef-istio-data\") pod \"98cd042c-e590-4f84-8acc-220357883bef\" (UID: \"98cd042c-e590-4f84-8acc-220357883bef\") " Oct 30 01:44:30.378777 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.378648 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98cd042c-e590-4f84-8acc-220357883bef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98cd042c-e590-4f84-8acc-220357883bef-workload-certs\") pod \"98cd042c-e590-4f84-8acc-220357883bef\" (UID: \"98cd042c-e590-4f84-8acc-220357883bef\") " Oct 30 01:44:30.378777 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.378691 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98cd042c-e590-4f84-8acc-220357883bef-istiod-ca-cert" pod "98cd042c-e590-4f84-8acc-220357883bef" (UID: "98cd042c-e590-4f84-8acc-220357883bef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:44:30.379409 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.378964 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98cd042c-e590-4f84-8acc-220357883bef-workload-socket" pod "98cd042c-e590-4f84-8acc-220357883bef" (UID: "98cd042c-e590-4f84-8acc-220357883bef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:30.379409 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.379091 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98cd042c-e590-4f84-8acc-220357883bef-istio-data" pod "98cd042c-e590-4f84-8acc-220357883bef" (UID: "98cd042c-e590-4f84-8acc-220357883bef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:30.379409 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.379306 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98cd042c-e590-4f84-8acc-220357883bef-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:30.379409 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.379334 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cd042c-e590-4f84-8acc-220357883bef-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:30.379409 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.379349 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98cd042c-e590-4f84-8acc-220357883bef-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:30.379409 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.379306 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98cd042c-e590-4f84-8acc-220357883bef-credential-socket" pod "98cd042c-e590-4f84-8acc-220357883bef" (UID: "98cd042c-e590-4f84-8acc-220357883bef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:30.379811 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.379448 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98cd042c-e590-4f84-8acc-220357883bef-workload-certs" pod "98cd042c-e590-4f84-8acc-220357883bef" (UID: "98cd042c-e590-4f84-8acc-220357883bef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:30.381931 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.381893 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98cd042c-e590-4f84-8acc-220357883bef-istio-envoy" pod "98cd042c-e590-4f84-8acc-220357883bef" (UID: "98cd042c-e590-4f84-8acc-220357883bef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:30.381931 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.381927 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98cd042c-e590-4f84-8acc-220357883bef-istio-token" pod "98cd042c-e590-4f84-8acc-220357883bef" (UID: "98cd042c-e590-4f84-8acc-220357883bef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:30.382094 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.381952 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98cd042c-e590-4f84-8acc-220357883bef-kube-api-access-zgqgf" pod "98cd042c-e590-4f84-8acc-220357883bef" (UID: "98cd042c-e590-4f84-8acc-220357883bef"). InnerVolumeSpecName "kube-api-access-zgqgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:30.383356 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.383303 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98cd042c-e590-4f84-8acc-220357883bef-istio-podinfo" pod "98cd042c-e590-4f84-8acc-220357883bef" (UID: "98cd042c-e590-4f84-8acc-220357883bef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:44:30.456408 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.456374 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-648f7c96c4-8jp52" Oct 30 01:44:30.458922 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.458899 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-9fc8b47c5-wxtb4" Oct 30 01:44:30.481044 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.480949 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98cd042c-e590-4f84-8acc-220357883bef-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:30.481044 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.480984 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cd042c-e590-4f84-8acc-220357883bef-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:30.481044 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.481000 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98cd042c-e590-4f84-8acc-220357883bef-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:30.481410 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.481378 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgqgf\" (UniqueName: \"kubernetes.io/projected/98cd042c-e590-4f84-8acc-220357883bef-kube-api-access-zgqgf\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:30.481410 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.481405 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98cd042c-e590-4f84-8acc-220357883bef-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:30.481605 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.481421 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98cd042c-e590-4f84-8acc-220357883bef-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:30.582779 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.581360 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:30.582779 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.581706 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/864805b2-2538-4ca4-89c8-f4b5e756c0a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/864805b2-2538-4ca4-89c8-f4b5e756c0a4-plugins-config-volume\") pod \"864805b2-2538-4ca4-89c8-f4b5e756c0a4\" (UID: \"864805b2-2538-4ca4-89c8-f4b5e756c0a4\") " Oct 30 01:44:30.582779 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.581870 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69aaaf90-0480-4fe7-bf6b-313ddb1f0feb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69aaaf90-0480-4fe7-bf6b-313ddb1f0feb-plugins-config-volume\") pod \"69aaaf90-0480-4fe7-bf6b-313ddb1f0feb\" (UID: \"69aaaf90-0480-4fe7-bf6b-313ddb1f0feb\") " Oct 30 01:44:30.582779 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.581970 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69aaaf90-0480-4fe7-bf6b-313ddb1f0feb-kube-api-access-s8kmt\" (UniqueName: \"kubernetes.io/projected/69aaaf90-0480-4fe7-bf6b-313ddb1f0feb-kube-api-access-s8kmt\") pod \"69aaaf90-0480-4fe7-bf6b-313ddb1f0feb\" (UID: \"69aaaf90-0480-4fe7-bf6b-313ddb1f0feb\") " Oct 30 01:44:30.582779 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.582039 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/864805b2-2538-4ca4-89c8-f4b5e756c0a4-kube-api-access-w976n\" (UniqueName: \"kubernetes.io/projected/864805b2-2538-4ca4-89c8-f4b5e756c0a4-kube-api-access-w976n\") pod \"864805b2-2538-4ca4-89c8-f4b5e756c0a4\" (UID: \"864805b2-2538-4ca4-89c8-f4b5e756c0a4\") " Oct 30 01:44:30.583467 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.583153 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69aaaf90-0480-4fe7-bf6b-313ddb1f0feb-plugins-config-volume" pod "69aaaf90-0480-4fe7-bf6b-313ddb1f0feb" (UID: "69aaaf90-0480-4fe7-bf6b-313ddb1f0feb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:44:30.583848 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.583543 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/864805b2-2538-4ca4-89c8-f4b5e756c0a4-plugins-config-volume" pod "864805b2-2538-4ca4-89c8-f4b5e756c0a4" (UID: "864805b2-2538-4ca4-89c8-f4b5e756c0a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:44:30.585120 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.585095 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/864805b2-2538-4ca4-89c8-f4b5e756c0a4-kube-api-access-w976n" pod "864805b2-2538-4ca4-89c8-f4b5e756c0a4" (UID: "864805b2-2538-4ca4-89c8-f4b5e756c0a4"). InnerVolumeSpecName "kube-api-access-w976n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:30.585683 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.585663 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69aaaf90-0480-4fe7-bf6b-313ddb1f0feb-kube-api-access-s8kmt" pod "69aaaf90-0480-4fe7-bf6b-313ddb1f0feb" (UID: "69aaaf90-0480-4fe7-bf6b-313ddb1f0feb"). InnerVolumeSpecName "kube-api-access-s8kmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:30.684324 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.684291 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/864805b2-2538-4ca4-89c8-f4b5e756c0a4-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:30.684324 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.684325 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69aaaf90-0480-4fe7-bf6b-313ddb1f0feb-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:30.684561 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.684340 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8kmt\" (UniqueName: \"kubernetes.io/projected/69aaaf90-0480-4fe7-bf6b-313ddb1f0feb-kube-api-access-s8kmt\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:30.684561 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.684486 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w976n\" (UniqueName: \"kubernetes.io/projected/864805b2-2538-4ca4-89c8-f4b5e756c0a4-kube-api-access-w976n\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:30.834062 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.834028 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:30.918025 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.917996 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea0fc4-4qpjg"] Oct 30 01:44:30.921460 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.921439 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ea0fc4-4qpjg"] Oct 30 01:44:30.988109 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.988021 6572 generic.go:356] "Generic (PLEG): container finished" podID="98cd042c-e590-4f84-8acc-220357883bef" containerID="7b7e97c2fb8ce21981618ce287f92a3a95843b1fc3c880d33c56ba354061375e" exitCode=0 Oct 30 01:44:30.988109 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.988078 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" Oct 30 01:44:30.988109 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.988098 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" event={"ID":"98cd042c-e590-4f84-8acc-220357883bef","Type":"ContainerDied","Data":"7b7e97c2fb8ce21981618ce287f92a3a95843b1fc3c880d33c56ba354061375e"} Oct 30 01:44:30.988413 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.988152 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2" event={"ID":"98cd042c-e590-4f84-8acc-220357883bef","Type":"ContainerDied","Data":"cfdc96ec140ef5c34fb7b4f3ff7f5aecd59c4e0b6d9dc0a8573bd9f96460df92"} Oct 30 01:44:30.988413 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.988176 6572 scope.go:122] "RemoveContainer" containerID="7b7e97c2fb8ce21981618ce287f92a3a95843b1fc3c880d33c56ba354061375e" Oct 30 01:44:30.989945 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.989919 6572 generic.go:356] "Generic (PLEG): container finished" podID="69aaaf90-0480-4fe7-bf6b-313ddb1f0feb" containerID="9f222c3a7682b4762b8eab2603a68ef36d034a686910c1c553550932de42babc" exitCode=0 Oct 30 01:44:30.990088 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.989958 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-9fc8b47c5-wxtb4" Oct 30 01:44:30.990088 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.989987 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-9fc8b47c5-wxtb4" event={"ID":"69aaaf90-0480-4fe7-bf6b-313ddb1f0feb","Type":"ContainerDied","Data":"9f222c3a7682b4762b8eab2603a68ef36d034a686910c1c553550932de42babc"} Oct 30 01:44:30.990088 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.990046 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-9fc8b47c5-wxtb4" event={"ID":"69aaaf90-0480-4fe7-bf6b-313ddb1f0feb","Type":"ContainerDied","Data":"77043ebf4b468cb68004e79251207cce7e1949341d59c9fb3beadcb553876ba7"} Oct 30 01:44:30.991687 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.991661 6572 generic.go:356] "Generic (PLEG): container finished" podID="864805b2-2538-4ca4-89c8-f4b5e756c0a4" containerID="324018c10e751618c08e963be7cfdc08b9bcb62428cec9788b2ee8cdb9f2727a" exitCode=0 Oct 30 01:44:30.991841 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.991706 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-648f7c96c4-8jp52" Oct 30 01:44:30.991841 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.991757 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-648f7c96c4-8jp52" event={"ID":"864805b2-2538-4ca4-89c8-f4b5e756c0a4","Type":"ContainerDied","Data":"324018c10e751618c08e963be7cfdc08b9bcb62428cec9788b2ee8cdb9f2727a"} Oct 30 01:44:30.991841 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.991806 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-648f7c96c4-8jp52" event={"ID":"864805b2-2538-4ca4-89c8-f4b5e756c0a4","Type":"ContainerDied","Data":"4daca96da2a4a53019676f58e32878c4d974057bf892ae6ec6d2b2e8c247972a"} Oct 30 01:44:30.993823 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.993792 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493521fb-zttmx" event={"ID":"2588b110-cd82-4f31-a36b-4f2fc12876f5","Type":"ContainerStarted","Data":"221c8472b869a0eb6e9d39f2b10058bcd3af51fa14fd59a5a81540eacb8a155f"} Oct 30 01:44:30.993925 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:30.993840 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493521fb-zttmx" event={"ID":"2588b110-cd82-4f31-a36b-4f2fc12876f5","Type":"ContainerStarted","Data":"52c6555f27c42a698ad7090504d11e5a334cdd181a7f121c03f3b311a9d1cb1c"} Oct 30 01:44:31.000697 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:31.000682 6572 scope.go:122] "RemoveContainer" containerID="1ab66c1173fb183401523059288f77755e92e96d4c5f3f58813475a22a26c788" Oct 30 01:44:31.012547 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:31.012529 6572 scope.go:122] "RemoveContainer" containerID="7b7e97c2fb8ce21981618ce287f92a3a95843b1fc3c880d33c56ba354061375e" Oct 30 01:44:31.012941 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:44:31.012915 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b7e97c2fb8ce21981618ce287f92a3a95843b1fc3c880d33c56ba354061375e\": container with ID starting with 7b7e97c2fb8ce21981618ce287f92a3a95843b1fc3c880d33c56ba354061375e not found: ID does not exist" containerID="7b7e97c2fb8ce21981618ce287f92a3a95843b1fc3c880d33c56ba354061375e" Oct 30 01:44:31.013015 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:31.012954 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b7e97c2fb8ce21981618ce287f92a3a95843b1fc3c880d33c56ba354061375e"} err="failed to get container status \"7b7e97c2fb8ce21981618ce287f92a3a95843b1fc3c880d33c56ba354061375e\": rpc error: code = NotFound desc = could not find container \"7b7e97c2fb8ce21981618ce287f92a3a95843b1fc3c880d33c56ba354061375e\": container with ID starting with 7b7e97c2fb8ce21981618ce287f92a3a95843b1fc3c880d33c56ba354061375e not found: ID does not exist" Oct 30 01:44:31.013015 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:31.012978 6572 scope.go:122] "RemoveContainer" containerID="1ab66c1173fb183401523059288f77755e92e96d4c5f3f58813475a22a26c788" Oct 30 01:44:31.013266 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:44:31.013244 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ab66c1173fb183401523059288f77755e92e96d4c5f3f58813475a22a26c788\": container with ID starting with 1ab66c1173fb183401523059288f77755e92e96d4c5f3f58813475a22a26c788 not found: ID does not exist" containerID="1ab66c1173fb183401523059288f77755e92e96d4c5f3f58813475a22a26c788" Oct 30 01:44:31.013315 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:31.013278 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ab66c1173fb183401523059288f77755e92e96d4c5f3f58813475a22a26c788"} err="failed to get container status \"1ab66c1173fb183401523059288f77755e92e96d4c5f3f58813475a22a26c788\": rpc error: code = NotFound desc = could not find container \"1ab66c1173fb183401523059288f77755e92e96d4c5f3f58813475a22a26c788\": container with ID starting with 1ab66c1173fb183401523059288f77755e92e96d4c5f3f58813475a22a26c788 not found: ID does not exist" Oct 30 01:44:31.013315 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:31.013299 6572 scope.go:122] "RemoveContainer" containerID="9f222c3a7682b4762b8eab2603a68ef36d034a686910c1c553550932de42babc" Oct 30 01:44:31.023599 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:31.023580 6572 scope.go:122] "RemoveContainer" containerID="9f222c3a7682b4762b8eab2603a68ef36d034a686910c1c553550932de42babc" Oct 30 01:44:31.023889 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:44:31.023859 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f222c3a7682b4762b8eab2603a68ef36d034a686910c1c553550932de42babc\": container with ID starting with 9f222c3a7682b4762b8eab2603a68ef36d034a686910c1c553550932de42babc not found: ID does not exist" containerID="9f222c3a7682b4762b8eab2603a68ef36d034a686910c1c553550932de42babc" Oct 30 01:44:31.023967 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:31.023887 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f222c3a7682b4762b8eab2603a68ef36d034a686910c1c553550932de42babc"} err="failed to get container status \"9f222c3a7682b4762b8eab2603a68ef36d034a686910c1c553550932de42babc\": rpc error: code = NotFound desc = could not find container \"9f222c3a7682b4762b8eab2603a68ef36d034a686910c1c553550932de42babc\": container with ID starting with 9f222c3a7682b4762b8eab2603a68ef36d034a686910c1c553550932de42babc not found: ID does not exist" Oct 30 01:44:31.023967 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:31.023906 6572 scope.go:122] "RemoveContainer" containerID="324018c10e751618c08e963be7cfdc08b9bcb62428cec9788b2ee8cdb9f2727a" Oct 30 01:44:31.033889 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:31.033868 6572 scope.go:122] "RemoveContainer" containerID="324018c10e751618c08e963be7cfdc08b9bcb62428cec9788b2ee8cdb9f2727a" Oct 30 01:44:31.034175 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:44:31.034153 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"324018c10e751618c08e963be7cfdc08b9bcb62428cec9788b2ee8cdb9f2727a\": container with ID starting with 324018c10e751618c08e963be7cfdc08b9bcb62428cec9788b2ee8cdb9f2727a not found: ID does not exist" containerID="324018c10e751618c08e963be7cfdc08b9bcb62428cec9788b2ee8cdb9f2727a" Oct 30 01:44:31.034224 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:31.034184 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"324018c10e751618c08e963be7cfdc08b9bcb62428cec9788b2ee8cdb9f2727a"} err="failed to get container status \"324018c10e751618c08e963be7cfdc08b9bcb62428cec9788b2ee8cdb9f2727a\": rpc error: code = NotFound desc = could not find container \"324018c10e751618c08e963be7cfdc08b9bcb62428cec9788b2ee8cdb9f2727a\": container with ID starting with 324018c10e751618c08e963be7cfdc08b9bcb62428cec9788b2ee8cdb9f2727a not found: ID does not exist" Oct 30 01:44:31.050507 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:31.050457 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493521fb-zttmx" podStartSLOduration=2.050440063 podStartE2EDuration="2.050440063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:44:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:44:31.048100511 +0000 UTC m=+1596.624525913" watchObservedRunningTime="2026-10-30 01:44:31.050440063 +0000 UTC m=+1596.626865465" Oct 30 01:44:31.060368 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:31.060343 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:31.087223 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:31.087189 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2"] Oct 30 01:44:31.092468 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:31.092450 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-c9d174-openshift-default-7c4bb5678f-9d6s2"] Oct 30 01:44:31.125188 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:31.125167 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-9fc8b47c5-wxtb4"] Oct 30 01:44:31.130543 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:31.130522 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ea0fc4-9fc8b47c5-wxtb4"] Oct 30 01:44:31.164034 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:31.164011 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-648f7c96c4-8jp52"] Oct 30 01:44:31.168785 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:31.168764 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-648f7c96c4-8jp52"] Oct 30 01:44:31.539381 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:31.539346 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:32.004632 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:32.004592 6572 generic.go:356] "Generic (PLEG): container finished" podID="2588b110-cd82-4f31-a36b-4f2fc12876f5" containerID="221c8472b869a0eb6e9d39f2b10058bcd3af51fa14fd59a5a81540eacb8a155f" exitCode=0 Oct 30 01:44:32.004874 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:32.004673 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493521fb-zttmx" event={"ID":"2588b110-cd82-4f31-a36b-4f2fc12876f5","Type":"ContainerDied","Data":"221c8472b869a0eb6e9d39f2b10058bcd3af51fa14fd59a5a81540eacb8a155f"} Oct 30 01:44:32.542497 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:32.542467 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69aaaf90-0480-4fe7-bf6b-313ddb1f0feb" path="/var/lib/kubelet/pods/69aaaf90-0480-4fe7-bf6b-313ddb1f0feb/volumes" Oct 30 01:44:32.542917 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:32.542902 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="864805b2-2538-4ca4-89c8-f4b5e756c0a4" path="/var/lib/kubelet/pods/864805b2-2538-4ca4-89c8-f4b5e756c0a4/volumes" Oct 30 01:44:32.543332 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:32.543317 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98cd042c-e590-4f84-8acc-220357883bef" path="/var/lib/kubelet/pods/98cd042c-e590-4f84-8acc-220357883bef/volumes" Oct 30 01:44:32.544032 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:32.544017 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c07642d5-28e5-433c-91ee-b78c8077e419" path="/var/lib/kubelet/pods/c07642d5-28e5-433c-91ee-b78c8077e419/volumes" Oct 30 01:44:32.613653 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:32.613616 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:32.859370 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:32.859273 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:33.095691 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:33.095663 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:33.204871 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:33.204846 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493521fb-zttmx" Oct 30 01:44:33.213268 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:33.213246 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2588b110-cd82-4f31-a36b-4f2fc12876f5-kube-api-access-lcstb\" (UniqueName: \"kubernetes.io/projected/2588b110-cd82-4f31-a36b-4f2fc12876f5-kube-api-access-lcstb\") pod \"2588b110-cd82-4f31-a36b-4f2fc12876f5\" (UID: \"2588b110-cd82-4f31-a36b-4f2fc12876f5\") " Oct 30 01:44:33.213389 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:33.213373 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2588b110-cd82-4f31-a36b-4f2fc12876f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2588b110-cd82-4f31-a36b-4f2fc12876f5-maas-api-openshift-ca\") pod \"2588b110-cd82-4f31-a36b-4f2fc12876f5\" (UID: \"2588b110-cd82-4f31-a36b-4f2fc12876f5\") " Oct 30 01:44:33.213434 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:33.213421 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2588b110-cd82-4f31-a36b-4f2fc12876f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2588b110-cd82-4f31-a36b-4f2fc12876f5-maas-api-xks-ca\") pod \"2588b110-cd82-4f31-a36b-4f2fc12876f5\" (UID: \"2588b110-cd82-4f31-a36b-4f2fc12876f5\") " Oct 30 01:44:33.213711 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:33.213688 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2588b110-cd82-4f31-a36b-4f2fc12876f5-maas-api-openshift-ca" pod "2588b110-cd82-4f31-a36b-4f2fc12876f5" (UID: "2588b110-cd82-4f31-a36b-4f2fc12876f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:44:33.213888 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:33.213871 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2588b110-cd82-4f31-a36b-4f2fc12876f5-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:33.215277 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:33.215257 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2588b110-cd82-4f31-a36b-4f2fc12876f5-kube-api-access-lcstb" pod "2588b110-cd82-4f31-a36b-4f2fc12876f5" (UID: "2588b110-cd82-4f31-a36b-4f2fc12876f5"). InnerVolumeSpecName "kube-api-access-lcstb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:33.215406 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:33.215387 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2588b110-cd82-4f31-a36b-4f2fc12876f5-maas-api-xks-ca" pod "2588b110-cd82-4f31-a36b-4f2fc12876f5" (UID: "2588b110-cd82-4f31-a36b-4f2fc12876f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:44:33.314571 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:33.314543 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2588b110-cd82-4f31-a36b-4f2fc12876f5-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:33.314571 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:33.314567 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcstb\" (UniqueName: \"kubernetes.io/projected/2588b110-cd82-4f31-a36b-4f2fc12876f5-kube-api-access-lcstb\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:34.028420 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:34.028381 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493521fb-zttmx" event={"ID":"2588b110-cd82-4f31-a36b-4f2fc12876f5","Type":"ContainerDied","Data":"52c6555f27c42a698ad7090504d11e5a334cdd181a7f121c03f3b311a9d1cb1c"} Oct 30 01:44:34.028420 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:34.028420 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493521fb-zttmx" Oct 30 01:44:34.028909 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:34.028439 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52c6555f27c42a698ad7090504d11e5a334cdd181a7f121c03f3b311a9d1cb1c" Oct 30 01:44:34.554600 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:34.554568 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:35.160935 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:35.160901 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:35.222284 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:35.222254 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-be8a7d-2ngwm"] Oct 30 01:44:35.233670 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:35.233638 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-be8a7d-2ngwm"] Oct 30 01:44:35.369015 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:35.368983 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:35.620664 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:35.620576 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:35.812570 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:35.812532 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859"] Oct 30 01:44:35.812871 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:35.812839 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" podUID="bfd2951a-980f-4a1a-bb49-20e1d9fce44d" containerName="istio-proxy" containerID="cri-o://cae2e61e4343e7b48804ce1812c41a9ec81782033838c25e75712fdd6963f4dc" gracePeriod=30 Oct 30 01:44:35.858807 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:35.858776 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:36.073943 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:36.073912 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:36.542145 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:36.542113 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05d5eccf-17ad-4a3e-a6fd-c9fb4c136721" path="/var/lib/kubelet/pods/05d5eccf-17ad-4a3e-a6fd-c9fb4c136721/volumes" Oct 30 01:44:37.520877 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:37.520848 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:38.960803 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:38.960734 6572 patch_prober.go:29] interesting pod/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:44:38.961283 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:38.960848 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" podUID="bfd2951a-980f-4a1a-bb49-20e1d9fce44d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:44:39.082420 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:39.082372 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:39.330375 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:39.330280 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:39.560917 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:39.560888 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:41.086045 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.085966 6572 generic.go:356] "Generic (PLEG): container finished" podID="bfd2951a-980f-4a1a-bb49-20e1d9fce44d" containerID="cae2e61e4343e7b48804ce1812c41a9ec81782033838c25e75712fdd6963f4dc" exitCode=0 Oct 30 01:44:41.086421 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.086044 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" event={"ID":"bfd2951a-980f-4a1a-bb49-20e1d9fce44d","Type":"ContainerDied","Data":"cae2e61e4343e7b48804ce1812c41a9ec81782033838c25e75712fdd6963f4dc"} Oct 30 01:44:41.086421 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.086137 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" event={"ID":"bfd2951a-980f-4a1a-bb49-20e1d9fce44d","Type":"ContainerDied","Data":"34035b8b52676aaaff038657218eecb4b2883da100059f8bfd2bc5a6e60f127d"} Oct 30 01:44:41.086421 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.086157 6572 scope.go:122] "RemoveContainer" containerID="cabbabac34140d1126a01d4d0a722cd69a31736a775ed126cbfff1e32c6c74d2" Oct 30 01:44:41.086421 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.086166 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34035b8b52676aaaff038657218eecb4b2883da100059f8bfd2bc5a6e60f127d" Oct 30 01:44:41.089918 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.089898 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:44:41.104859 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.104830 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:41.145386 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.145353 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-workload-socket\") pod \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\" (UID: \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\") " Oct 30 01:44:41.145539 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.145448 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-istiod-ca-cert\") pod \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\" (UID: \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\") " Oct 30 01:44:41.145539 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.145476 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-workload-certs\") pod \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\" (UID: \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\") " Oct 30 01:44:41.145539 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.145495 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-credential-socket\") pod \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\" (UID: \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\") " Oct 30 01:44:41.145697 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.145586 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-istio-podinfo\") pod \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\" (UID: \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\") " Oct 30 01:44:41.145697 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.145621 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-istio-data\") pod \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\" (UID: \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\") " Oct 30 01:44:41.145697 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.145655 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-workload-socket" pod "bfd2951a-980f-4a1a-bb49-20e1d9fce44d" (UID: "bfd2951a-980f-4a1a-bb49-20e1d9fce44d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:41.145884 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.145789 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-workload-certs" pod "bfd2951a-980f-4a1a-bb49-20e1d9fce44d" (UID: "bfd2951a-980f-4a1a-bb49-20e1d9fce44d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:41.145884 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.145669 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-istio-envoy\") pod \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\" (UID: \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\") " Oct 30 01:44:41.145884 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.145862 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-istiod-ca-cert" pod "bfd2951a-980f-4a1a-bb49-20e1d9fce44d" (UID: "bfd2951a-980f-4a1a-bb49-20e1d9fce44d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:44:41.146063 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.145968 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-credential-socket" pod "bfd2951a-980f-4a1a-bb49-20e1d9fce44d" (UID: "bfd2951a-980f-4a1a-bb49-20e1d9fce44d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:41.146063 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.145980 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-kube-api-access-zmzg2\" (UniqueName: \"kubernetes.io/projected/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-kube-api-access-zmzg2\") pod \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\" (UID: \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\") " Oct 30 01:44:41.146063 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.145981 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-istio-data" pod "bfd2951a-980f-4a1a-bb49-20e1d9fce44d" (UID: "bfd2951a-980f-4a1a-bb49-20e1d9fce44d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:41.146063 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.146024 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-istio-token\" (UniqueName: \"kubernetes.io/projected/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-istio-token\") pod \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\" (UID: \"bfd2951a-980f-4a1a-bb49-20e1d9fce44d\") " Oct 30 01:44:41.146449 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.146428 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:41.146536 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.146455 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:41.146536 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.146469 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:41.146536 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.146478 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:41.146536 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.146486 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:41.147797 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.147775 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-istio-envoy" pod "bfd2951a-980f-4a1a-bb49-20e1d9fce44d" (UID: "bfd2951a-980f-4a1a-bb49-20e1d9fce44d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:41.147880 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.147829 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-istio-podinfo" pod "bfd2951a-980f-4a1a-bb49-20e1d9fce44d" (UID: "bfd2951a-980f-4a1a-bb49-20e1d9fce44d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:44:41.148315 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.148282 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-istio-token" pod "bfd2951a-980f-4a1a-bb49-20e1d9fce44d" (UID: "bfd2951a-980f-4a1a-bb49-20e1d9fce44d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:41.148405 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.148348 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-kube-api-access-zmzg2" pod "bfd2951a-980f-4a1a-bb49-20e1d9fce44d" (UID: "bfd2951a-980f-4a1a-bb49-20e1d9fce44d"). InnerVolumeSpecName "kube-api-access-zmzg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:41.247912 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.247872 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:41.248357 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.248330 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:41.248357 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.248355 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmzg2\" (UniqueName: \"kubernetes.io/projected/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-kube-api-access-zmzg2\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:41.248553 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.248366 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfd2951a-980f-4a1a-bb49-20e1d9fce44d-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:41.441784 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.441716 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl"] Oct 30 01:44:41.442432 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.442412 6572 cpu_manager.go:414] "containerMap: removing container" podUID="2588b110-cd82-4f31-a36b-4f2fc12876f5" containerName="revoke-keys" Oct 30 01:44:41.442432 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.442430 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2588b110-cd82-4f31-a36b-4f2fc12876f5" containerName="revoke-keys" Oct 30 01:44:41.442601 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.442451 6572 cpu_manager.go:414] "containerMap: removing container" podUID="98cd042c-e590-4f84-8acc-220357883bef" containerName="istio-proxy" Oct 30 01:44:41.442601 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.442457 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98cd042c-e590-4f84-8acc-220357883bef" containerName="istio-proxy" Oct 30 01:44:41.442601 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.442470 6572 cpu_manager.go:414] "containerMap: removing container" podUID="bfd2951a-980f-4a1a-bb49-20e1d9fce44d" containerName="istio-proxy" Oct 30 01:44:41.442601 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.442477 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfd2951a-980f-4a1a-bb49-20e1d9fce44d" containerName="istio-proxy" Oct 30 01:44:41.442601 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.442490 6572 cpu_manager.go:414] "containerMap: removing container" podUID="98cd042c-e590-4f84-8acc-220357883bef" containerName="istio-proxy" Oct 30 01:44:41.442601 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.442497 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98cd042c-e590-4f84-8acc-220357883bef" containerName="istio-proxy" Oct 30 01:44:41.442601 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.442518 6572 cpu_manager.go:414] "containerMap: removing container" podUID="bfd2951a-980f-4a1a-bb49-20e1d9fce44d" containerName="istio-proxy" Oct 30 01:44:41.442601 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.442525 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfd2951a-980f-4a1a-bb49-20e1d9fce44d" containerName="istio-proxy" Oct 30 01:44:41.442601 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.442537 6572 cpu_manager.go:414] "containerMap: removing container" podUID="864805b2-2538-4ca4-89c8-f4b5e756c0a4" containerName="payload-processing" Oct 30 01:44:41.442601 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.442541 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="864805b2-2538-4ca4-89c8-f4b5e756c0a4" containerName="payload-processing" Oct 30 01:44:41.442601 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.442548 6572 cpu_manager.go:414] "containerMap: removing container" podUID="69aaaf90-0480-4fe7-bf6b-313ddb1f0feb" containerName="payload-pre-processing" Oct 30 01:44:41.442601 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.442554 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69aaaf90-0480-4fe7-bf6b-313ddb1f0feb" containerName="payload-pre-processing" Oct 30 01:44:41.442601 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.442600 6572 cpu_manager.go:414] "containerMap: removing container" podUID="eced80d4-0bf9-4f4e-a25e-0448b3a2f70b" containerName="istio-proxy" Oct 30 01:44:41.442601 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.442605 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eced80d4-0bf9-4f4e-a25e-0448b3a2f70b" containerName="istio-proxy" Oct 30 01:44:41.443092 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.442702 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="98cd042c-e590-4f84-8acc-220357883bef" containerName="istio-proxy" Oct 30 01:44:41.443092 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.442711 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfd2951a-980f-4a1a-bb49-20e1d9fce44d" containerName="istio-proxy" Oct 30 01:44:41.443092 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.442718 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfd2951a-980f-4a1a-bb49-20e1d9fce44d" containerName="istio-proxy" Oct 30 01:44:41.443092 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.442730 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="864805b2-2538-4ca4-89c8-f4b5e756c0a4" containerName="payload-processing" Oct 30 01:44:41.443092 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.442738 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="69aaaf90-0480-4fe7-bf6b-313ddb1f0feb" containerName="payload-pre-processing" Oct 30 01:44:41.443092 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.442771 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="2588b110-cd82-4f31-a36b-4f2fc12876f5" containerName="revoke-keys" Oct 30 01:44:41.448023 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.447997 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:44:41.470036 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.469978 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-f197b1-openshift-default-dockercfg-2bwnq\"" Oct 30 01:44:41.480014 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.479963 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl"] Oct 30 01:44:41.552610 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.552575 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c1aa377-ab2c-4653-a630-3d4eef49526d-istio-envoy\") pod \"e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl\" (UID: \"2c1aa377-ab2c-4653-a630-3d4eef49526d\") " pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:44:41.552829 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.552628 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1aa377-ab2c-4653-a630-3d4eef49526d-workload-socket\") pod \"e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl\" (UID: \"2c1aa377-ab2c-4653-a630-3d4eef49526d\") " pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:44:41.552829 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.552706 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c1aa377-ab2c-4653-a630-3d4eef49526d-istio-data\") pod \"e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl\" (UID: \"2c1aa377-ab2c-4653-a630-3d4eef49526d\") " pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:44:41.552829 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.552807 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c1aa377-ab2c-4653-a630-3d4eef49526d-istio-podinfo\") pod \"e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl\" (UID: \"2c1aa377-ab2c-4653-a630-3d4eef49526d\") " pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:44:41.552951 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.552839 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n58v\" (UniqueName: \"kubernetes.io/projected/2c1aa377-ab2c-4653-a630-3d4eef49526d-kube-api-access-9n58v\") pod \"e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl\" (UID: \"2c1aa377-ab2c-4653-a630-3d4eef49526d\") " pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:44:41.552989 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.552950 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c1aa377-ab2c-4653-a630-3d4eef49526d-workload-certs\") pod \"e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl\" (UID: \"2c1aa377-ab2c-4653-a630-3d4eef49526d\") " pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:44:41.552989 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.552978 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c1aa377-ab2c-4653-a630-3d4eef49526d-istiod-ca-cert\") pod \"e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl\" (UID: \"2c1aa377-ab2c-4653-a630-3d4eef49526d\") " pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:44:41.553057 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.553035 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1aa377-ab2c-4653-a630-3d4eef49526d-credential-socket\") pod \"e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl\" (UID: \"2c1aa377-ab2c-4653-a630-3d4eef49526d\") " pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:44:41.553057 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.553053 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c1aa377-ab2c-4653-a630-3d4eef49526d-istio-token\") pod \"e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl\" (UID: \"2c1aa377-ab2c-4653-a630-3d4eef49526d\") " pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:44:41.666943 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.666905 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c1aa377-ab2c-4653-a630-3d4eef49526d-istio-envoy\") pod \"e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl\" (UID: \"2c1aa377-ab2c-4653-a630-3d4eef49526d\") " pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:44:41.668087 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.668062 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1aa377-ab2c-4653-a630-3d4eef49526d-workload-socket\") pod \"e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl\" (UID: \"2c1aa377-ab2c-4653-a630-3d4eef49526d\") " pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:44:41.668271 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.668256 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c1aa377-ab2c-4653-a630-3d4eef49526d-istio-data\") pod \"e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl\" (UID: \"2c1aa377-ab2c-4653-a630-3d4eef49526d\") " pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:44:41.668397 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.668384 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c1aa377-ab2c-4653-a630-3d4eef49526d-istio-podinfo\") pod \"e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl\" (UID: \"2c1aa377-ab2c-4653-a630-3d4eef49526d\") " pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:44:41.668520 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.668504 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n58v\" (UniqueName: \"kubernetes.io/projected/2c1aa377-ab2c-4653-a630-3d4eef49526d-kube-api-access-9n58v\") pod \"e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl\" (UID: \"2c1aa377-ab2c-4653-a630-3d4eef49526d\") " pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:44:41.668707 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.667924 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:41.668881 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.668830 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c1aa377-ab2c-4653-a630-3d4eef49526d-workload-certs\") pod \"e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl\" (UID: \"2c1aa377-ab2c-4653-a630-3d4eef49526d\") " pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:44:41.669022 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.669007 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c1aa377-ab2c-4653-a630-3d4eef49526d-istiod-ca-cert\") pod \"e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl\" (UID: \"2c1aa377-ab2c-4653-a630-3d4eef49526d\") " pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:44:41.669137 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.669114 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1aa377-ab2c-4653-a630-3d4eef49526d-credential-socket\") pod \"e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl\" (UID: \"2c1aa377-ab2c-4653-a630-3d4eef49526d\") " pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:44:41.669260 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.669244 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c1aa377-ab2c-4653-a630-3d4eef49526d-istio-token\") pod \"e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl\" (UID: \"2c1aa377-ab2c-4653-a630-3d4eef49526d\") " pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:44:41.675788 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.669460 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c1aa377-ab2c-4653-a630-3d4eef49526d-workload-certs\") pod \"e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl\" (UID: \"2c1aa377-ab2c-4653-a630-3d4eef49526d\") " pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:44:41.675788 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.669067 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c1aa377-ab2c-4653-a630-3d4eef49526d-istio-data\") pod \"e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl\" (UID: \"2c1aa377-ab2c-4653-a630-3d4eef49526d\") " pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:44:41.675788 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.669261 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1aa377-ab2c-4653-a630-3d4eef49526d-workload-socket\") pod \"e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl\" (UID: \"2c1aa377-ab2c-4653-a630-3d4eef49526d\") " pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:44:41.675788 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.669884 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c1aa377-ab2c-4653-a630-3d4eef49526d-istio-envoy\") pod \"e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl\" (UID: \"2c1aa377-ab2c-4653-a630-3d4eef49526d\") " pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:44:41.675788 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.674178 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1aa377-ab2c-4653-a630-3d4eef49526d-credential-socket\") pod \"e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl\" (UID: \"2c1aa377-ab2c-4653-a630-3d4eef49526d\") " pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:44:41.675788 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.675229 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c1aa377-ab2c-4653-a630-3d4eef49526d-istiod-ca-cert\") pod \"e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl\" (UID: \"2c1aa377-ab2c-4653-a630-3d4eef49526d\") " pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:44:41.676531 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.676509 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c1aa377-ab2c-4653-a630-3d4eef49526d-istio-podinfo\") pod \"e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl\" (UID: \"2c1aa377-ab2c-4653-a630-3d4eef49526d\") " pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:44:41.693201 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.693138 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n58v\" (UniqueName: \"kubernetes.io/projected/2c1aa377-ab2c-4653-a630-3d4eef49526d-kube-api-access-9n58v\") pod \"e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl\" (UID: \"2c1aa377-ab2c-4653-a630-3d4eef49526d\") " pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:44:41.694152 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.694132 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c1aa377-ab2c-4653-a630-3d4eef49526d-istio-token\") pod \"e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl\" (UID: \"2c1aa377-ab2c-4653-a630-3d4eef49526d\") " pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:44:41.762487 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.762453 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:44:41.883192 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:41.883162 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:42.091801 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:42.091769 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl"] Oct 30 01:44:42.095062 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:44:42.095019 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c1aa377_ab2c_4653_a630_3d4eef49526d.slice/crio-2aabc151dcdab27d124e5748d10ca31228a03cbec38b93914f0353c878dfe7f1 WatchSource:0}: Error finding container 2aabc151dcdab27d124e5748d10ca31228a03cbec38b93914f0353c878dfe7f1: Status 404 returned error can't find the container with id 2aabc151dcdab27d124e5748d10ca31228a03cbec38b93914f0353c878dfe7f1 Oct 30 01:44:42.097798 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:42.097763 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:44:42.097907 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:42.097838 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:44:42.097907 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:42.097880 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:44:42.098397 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:42.098380 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859" Oct 30 01:44:42.180374 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:42.180342 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859"] Oct 30 01:44:42.185302 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:42.185275 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f01cbc-openshift-default-579586d54c-6n859"] Oct 30 01:44:42.543092 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:42.543051 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfd2951a-980f-4a1a-bb49-20e1d9fce44d" path="/var/lib/kubelet/pods/bfd2951a-980f-4a1a-bb49-20e1d9fce44d/volumes" Oct 30 01:44:42.924463 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:42.924433 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-952d1786-6wx7g"] Oct 30 01:44:42.925069 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:42.925054 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="98cd042c-e590-4f84-8acc-220357883bef" containerName="istio-proxy" Oct 30 01:44:42.928903 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:42.928887 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-952d1786-6wx7g" Oct 30 01:44:42.948290 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:42.948260 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-952d1786-6wx7g"] Oct 30 01:44:42.957073 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:42.957028 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:44:42.957248 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:42.957114 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:44:42.958622 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:42.958601 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:44:42.959780 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:42.958888 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:44:43.091368 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:43.091331 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98e0bfb4-2b67-4c67-91ff-72a484a8e201-maas-api-xks-ca\") pod \"maas-api-revoke-sub-952d1786-6wx7g\" (UID: \"98e0bfb4-2b67-4c67-91ff-72a484a8e201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-952d1786-6wx7g" Oct 30 01:44:43.091576 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:43.091389 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggc4p\" (UniqueName: \"kubernetes.io/projected/98e0bfb4-2b67-4c67-91ff-72a484a8e201-kube-api-access-ggc4p\") pod \"maas-api-revoke-sub-952d1786-6wx7g\" (UID: \"98e0bfb4-2b67-4c67-91ff-72a484a8e201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-952d1786-6wx7g" Oct 30 01:44:43.091576 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:43.091458 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98e0bfb4-2b67-4c67-91ff-72a484a8e201-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-952d1786-6wx7g\" (UID: \"98e0bfb4-2b67-4c67-91ff-72a484a8e201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-952d1786-6wx7g" Oct 30 01:44:43.108372 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:43.108329 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" event={"ID":"2c1aa377-ab2c-4653-a630-3d4eef49526d","Type":"ContainerStarted","Data":"14fa9a4fa5bce69078f1ee86094df3c5f5a8a02dd48631a01adacb880f7bfdf8"} Oct 30 01:44:43.108894 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:43.108398 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" event={"ID":"2c1aa377-ab2c-4653-a630-3d4eef49526d","Type":"ContainerStarted","Data":"2aabc151dcdab27d124e5748d10ca31228a03cbec38b93914f0353c878dfe7f1"} Oct 30 01:44:43.159475 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:43.159412 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" podStartSLOduration=2.159392989 podStartE2EDuration="2.159392989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:44:43.158846401 +0000 UTC m=+1608.735271805" watchObservedRunningTime="2026-10-30 01:44:43.159392989 +0000 UTC m=+1608.735818392" Oct 30 01:44:43.192436 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:43.192345 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggc4p\" (UniqueName: \"kubernetes.io/projected/98e0bfb4-2b67-4c67-91ff-72a484a8e201-kube-api-access-ggc4p\") pod \"maas-api-revoke-sub-952d1786-6wx7g\" (UID: \"98e0bfb4-2b67-4c67-91ff-72a484a8e201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-952d1786-6wx7g" Oct 30 01:44:43.192436 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:43.192404 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98e0bfb4-2b67-4c67-91ff-72a484a8e201-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-952d1786-6wx7g\" (UID: \"98e0bfb4-2b67-4c67-91ff-72a484a8e201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-952d1786-6wx7g" Oct 30 01:44:43.192627 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:43.192595 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98e0bfb4-2b67-4c67-91ff-72a484a8e201-maas-api-xks-ca\") pod \"maas-api-revoke-sub-952d1786-6wx7g\" (UID: \"98e0bfb4-2b67-4c67-91ff-72a484a8e201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-952d1786-6wx7g" Oct 30 01:44:43.193131 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:43.193109 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98e0bfb4-2b67-4c67-91ff-72a484a8e201-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-952d1786-6wx7g\" (UID: \"98e0bfb4-2b67-4c67-91ff-72a484a8e201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-952d1786-6wx7g" Oct 30 01:44:43.195000 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:43.194980 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98e0bfb4-2b67-4c67-91ff-72a484a8e201-maas-api-xks-ca\") pod \"maas-api-revoke-sub-952d1786-6wx7g\" (UID: \"98e0bfb4-2b67-4c67-91ff-72a484a8e201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-952d1786-6wx7g" Oct 30 01:44:43.212225 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:43.212194 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggc4p\" (UniqueName: \"kubernetes.io/projected/98e0bfb4-2b67-4c67-91ff-72a484a8e201-kube-api-access-ggc4p\") pod \"maas-api-revoke-sub-952d1786-6wx7g\" (UID: \"98e0bfb4-2b67-4c67-91ff-72a484a8e201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-952d1786-6wx7g" Oct 30 01:44:43.242718 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:43.242689 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-952d1786-6wx7g" Oct 30 01:44:43.267092 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:43.267053 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:43.569420 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:43.569379 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-952d1786-6wx7g"] Oct 30 01:44:43.578397 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:44:43.578359 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e0bfb4_2b67_4c67_91ff_72a484a8e201.slice/crio-515ceecad72ecf3b717acb99e150603b24f6bdd928b26212298ff52859e559ec WatchSource:0}: Error finding container 515ceecad72ecf3b717acb99e150603b24f6bdd928b26212298ff52859e559ec: Status 404 returned error can't find the container with id 515ceecad72ecf3b717acb99e150603b24f6bdd928b26212298ff52859e559ec Oct 30 01:44:43.763655 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:43.763620 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:44:43.769665 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:43.769634 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:44:43.941163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:43.941121 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3612ab3f-p4hql"] Oct 30 01:44:43.945295 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:43.945269 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3612ab3f-p4hql" Oct 30 01:44:43.973232 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:43.973192 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3612ab3f-p4hql"] Oct 30 01:44:44.109181 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:44.109082 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/326e903b-0283-438b-b9c6-f463b7f685ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3612ab3f-p4hql\" (UID: \"326e903b-0283-438b-b9c6-f463b7f685ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3612ab3f-p4hql" Oct 30 01:44:44.109595 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:44.109210 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/326e903b-0283-438b-b9c6-f463b7f685ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3612ab3f-p4hql\" (UID: \"326e903b-0283-438b-b9c6-f463b7f685ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3612ab3f-p4hql" Oct 30 01:44:44.109595 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:44.109318 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pd7f\" (UniqueName: \"kubernetes.io/projected/326e903b-0283-438b-b9c6-f463b7f685ae-kube-api-access-4pd7f\") pod \"maas-api-revoke-sub-3612ab3f-p4hql\" (UID: \"326e903b-0283-438b-b9c6-f463b7f685ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3612ab3f-p4hql" Oct 30 01:44:44.114612 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:44.114572 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:44.117917 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:44.117883 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-952d1786-6wx7g" event={"ID":"98e0bfb4-2b67-4c67-91ff-72a484a8e201","Type":"ContainerStarted","Data":"26f0ed90857f308c581a77c72be6a788fd8df45b92d96c9fa6935ecf51a207bd"} Oct 30 01:44:44.118088 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:44.117945 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-952d1786-6wx7g" event={"ID":"98e0bfb4-2b67-4c67-91ff-72a484a8e201","Type":"ContainerStarted","Data":"515ceecad72ecf3b717acb99e150603b24f6bdd928b26212298ff52859e559ec"} Oct 30 01:44:44.118359 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:44.118342 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:44:44.119356 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:44.119335 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:44:44.159194 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:44.159140 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-952d1786-6wx7g" podStartSLOduration=2.159119672 podStartE2EDuration="2.159119672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:44:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:44:44.158294325 +0000 UTC m=+1609.734728354" watchObservedRunningTime="2026-10-30 01:44:44.159119672 +0000 UTC m=+1609.735545075" Oct 30 01:44:44.211036 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:44.210998 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pd7f\" (UniqueName: \"kubernetes.io/projected/326e903b-0283-438b-b9c6-f463b7f685ae-kube-api-access-4pd7f\") pod \"maas-api-revoke-sub-3612ab3f-p4hql\" (UID: \"326e903b-0283-438b-b9c6-f463b7f685ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3612ab3f-p4hql" Oct 30 01:44:44.211211 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:44.211188 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/326e903b-0283-438b-b9c6-f463b7f685ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3612ab3f-p4hql\" (UID: \"326e903b-0283-438b-b9c6-f463b7f685ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3612ab3f-p4hql" Oct 30 01:44:44.211274 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:44.211212 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/326e903b-0283-438b-b9c6-f463b7f685ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3612ab3f-p4hql\" (UID: \"326e903b-0283-438b-b9c6-f463b7f685ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3612ab3f-p4hql" Oct 30 01:44:44.212155 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:44.212120 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/326e903b-0283-438b-b9c6-f463b7f685ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3612ab3f-p4hql\" (UID: \"326e903b-0283-438b-b9c6-f463b7f685ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3612ab3f-p4hql" Oct 30 01:44:44.214153 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:44.214124 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/326e903b-0283-438b-b9c6-f463b7f685ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3612ab3f-p4hql\" (UID: \"326e903b-0283-438b-b9c6-f463b7f685ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3612ab3f-p4hql" Oct 30 01:44:44.234173 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:44.234143 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pd7f\" (UniqueName: \"kubernetes.io/projected/326e903b-0283-438b-b9c6-f463b7f685ae-kube-api-access-4pd7f\") pod \"maas-api-revoke-sub-3612ab3f-p4hql\" (UID: \"326e903b-0283-438b-b9c6-f463b7f685ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3612ab3f-p4hql" Oct 30 01:44:44.260412 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:44.260382 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3612ab3f-p4hql" Oct 30 01:44:44.344577 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:44.344541 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:44.568010 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:44.567969 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:44.579359 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:44:44.579328 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod326e903b_0283_438b_b9c6_f463b7f685ae.slice/crio-ba0589fb584ee51f2f9900e34ea87b215d0bc7dc06d82e885658c55c5f4a6cf3 WatchSource:0}: Error finding container ba0589fb584ee51f2f9900e34ea87b215d0bc7dc06d82e885658c55c5f4a6cf3: Status 404 returned error can't find the container with id ba0589fb584ee51f2f9900e34ea87b215d0bc7dc06d82e885658c55c5f4a6cf3 Oct 30 01:44:44.580524 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:44.580498 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3612ab3f-p4hql"] Oct 30 01:44:44.690200 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:44:44.690157 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:44:45.127812 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:45.127709 6572 generic.go:356] "Generic (PLEG): container finished" podID="98e0bfb4-2b67-4c67-91ff-72a484a8e201" containerID="26f0ed90857f308c581a77c72be6a788fd8df45b92d96c9fa6935ecf51a207bd" exitCode=0 Oct 30 01:44:45.127812 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:45.127789 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-952d1786-6wx7g" event={"ID":"98e0bfb4-2b67-4c67-91ff-72a484a8e201","Type":"ContainerDied","Data":"26f0ed90857f308c581a77c72be6a788fd8df45b92d96c9fa6935ecf51a207bd"} Oct 30 01:44:45.130017 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:45.129986 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3612ab3f-p4hql" event={"ID":"326e903b-0283-438b-b9c6-f463b7f685ae","Type":"ContainerStarted","Data":"c13db6414e2222b842933236b47de9a4cf6b2716d4caf57b7bfcd5c6c9c606bd"} Oct 30 01:44:45.130185 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:45.130034 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3612ab3f-p4hql" event={"ID":"326e903b-0283-438b-b9c6-f463b7f685ae","Type":"ContainerStarted","Data":"ba0589fb584ee51f2f9900e34ea87b215d0bc7dc06d82e885658c55c5f4a6cf3"} Oct 30 01:44:45.210537 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:45.210484 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3612ab3f-p4hql" podStartSLOduration=2.210468439 podStartE2EDuration="2.210468439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:44:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:44:45.20886802 +0000 UTC m=+1610.785293423" watchObservedRunningTime="2026-10-30 01:44:45.210468439 +0000 UTC m=+1610.786893841" Oct 30 01:44:45.335301 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:45.335267 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:45.794698 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:45.794612 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:46.018234 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:46.018190 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:46.140248 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:46.140160 6572 generic.go:356] "Generic (PLEG): container finished" podID="326e903b-0283-438b-b9c6-f463b7f685ae" containerID="c13db6414e2222b842933236b47de9a4cf6b2716d4caf57b7bfcd5c6c9c606bd" exitCode=0 Oct 30 01:44:46.140708 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:46.140250 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3612ab3f-p4hql" event={"ID":"326e903b-0283-438b-b9c6-f463b7f685ae","Type":"ContainerDied","Data":"c13db6414e2222b842933236b47de9a4cf6b2716d4caf57b7bfcd5c6c9c606bd"} Oct 30 01:44:46.295396 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:46.295366 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-952d1786-6wx7g" Oct 30 01:44:46.345992 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:46.345950 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98e0bfb4-2b67-4c67-91ff-72a484a8e201-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98e0bfb4-2b67-4c67-91ff-72a484a8e201-maas-api-openshift-ca\") pod \"98e0bfb4-2b67-4c67-91ff-72a484a8e201\" (UID: \"98e0bfb4-2b67-4c67-91ff-72a484a8e201\") " Oct 30 01:44:46.346172 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:46.346125 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98e0bfb4-2b67-4c67-91ff-72a484a8e201-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98e0bfb4-2b67-4c67-91ff-72a484a8e201-maas-api-xks-ca\") pod \"98e0bfb4-2b67-4c67-91ff-72a484a8e201\" (UID: \"98e0bfb4-2b67-4c67-91ff-72a484a8e201\") " Oct 30 01:44:46.346172 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:46.346163 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98e0bfb4-2b67-4c67-91ff-72a484a8e201-kube-api-access-ggc4p\" (UniqueName: \"kubernetes.io/projected/98e0bfb4-2b67-4c67-91ff-72a484a8e201-kube-api-access-ggc4p\") pod \"98e0bfb4-2b67-4c67-91ff-72a484a8e201\" (UID: \"98e0bfb4-2b67-4c67-91ff-72a484a8e201\") " Oct 30 01:44:46.346430 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:46.346399 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98e0bfb4-2b67-4c67-91ff-72a484a8e201-maas-api-openshift-ca" pod "98e0bfb4-2b67-4c67-91ff-72a484a8e201" (UID: "98e0bfb4-2b67-4c67-91ff-72a484a8e201"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:44:46.346669 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:46.346632 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98e0bfb4-2b67-4c67-91ff-72a484a8e201-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:46.349351 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:46.349318 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98e0bfb4-2b67-4c67-91ff-72a484a8e201-maas-api-xks-ca" pod "98e0bfb4-2b67-4c67-91ff-72a484a8e201" (UID: "98e0bfb4-2b67-4c67-91ff-72a484a8e201"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:44:46.349509 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:46.349346 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98e0bfb4-2b67-4c67-91ff-72a484a8e201-kube-api-access-ggc4p" pod "98e0bfb4-2b67-4c67-91ff-72a484a8e201" (UID: "98e0bfb4-2b67-4c67-91ff-72a484a8e201"). InnerVolumeSpecName "kube-api-access-ggc4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:46.448469 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:46.448422 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98e0bfb4-2b67-4c67-91ff-72a484a8e201-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:46.448469 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:46.448459 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggc4p\" (UniqueName: \"kubernetes.io/projected/98e0bfb4-2b67-4c67-91ff-72a484a8e201-kube-api-access-ggc4p\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:46.771836 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:46.771711 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55e61c16-9hgtn"] Oct 30 01:44:46.772299 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:46.772283 6572 cpu_manager.go:414] "containerMap: removing container" podUID="98e0bfb4-2b67-4c67-91ff-72a484a8e201" containerName="revoke-keys" Oct 30 01:44:46.772337 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:46.772300 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98e0bfb4-2b67-4c67-91ff-72a484a8e201" containerName="revoke-keys" Oct 30 01:44:46.772428 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:46.772419 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="98e0bfb4-2b67-4c67-91ff-72a484a8e201" containerName="revoke-keys" Oct 30 01:44:46.775820 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:46.775800 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e61c16-9hgtn" Oct 30 01:44:46.798682 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:46.798642 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55e61c16-9hgtn"] Oct 30 01:44:46.852387 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:46.852351 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55e61c16-9hgtn\" (UID: \"bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e61c16-9hgtn" Oct 30 01:44:46.852589 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:46.852490 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55e61c16-9hgtn\" (UID: \"bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e61c16-9hgtn" Oct 30 01:44:46.852641 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:46.852597 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgqwm\" (UniqueName: \"kubernetes.io/projected/bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4-kube-api-access-wgqwm\") pod \"maas-api-revoke-sub-55e61c16-9hgtn\" (UID: \"bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e61c16-9hgtn" Oct 30 01:44:46.953954 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:46.953914 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55e61c16-9hgtn\" (UID: \"bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e61c16-9hgtn" Oct 30 01:44:46.954178 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:46.953987 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55e61c16-9hgtn\" (UID: \"bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e61c16-9hgtn" Oct 30 01:44:46.954178 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:46.954044 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgqwm\" (UniqueName: \"kubernetes.io/projected/bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4-kube-api-access-wgqwm\") pod \"maas-api-revoke-sub-55e61c16-9hgtn\" (UID: \"bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e61c16-9hgtn" Oct 30 01:44:46.954684 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:46.954662 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55e61c16-9hgtn\" (UID: \"bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e61c16-9hgtn" Oct 30 01:44:46.956468 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:46.956447 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55e61c16-9hgtn\" (UID: \"bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e61c16-9hgtn" Oct 30 01:44:46.973369 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:46.973338 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgqwm\" (UniqueName: \"kubernetes.io/projected/bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4-kube-api-access-wgqwm\") pod \"maas-api-revoke-sub-55e61c16-9hgtn\" (UID: \"bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e61c16-9hgtn" Oct 30 01:44:47.090106 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:47.090017 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e61c16-9hgtn" Oct 30 01:44:47.190314 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:47.190281 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-952d1786-6wx7g" Oct 30 01:44:47.190691 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:47.190301 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-952d1786-6wx7g" event={"ID":"98e0bfb4-2b67-4c67-91ff-72a484a8e201","Type":"ContainerDied","Data":"515ceecad72ecf3b717acb99e150603b24f6bdd928b26212298ff52859e559ec"} Oct 30 01:44:47.190691 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:47.190367 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="515ceecad72ecf3b717acb99e150603b24f6bdd928b26212298ff52859e559ec" Oct 30 01:44:47.362205 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:47.362170 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:47.375294 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:47.375263 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3612ab3f-p4hql" Oct 30 01:44:47.424717 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:47.424690 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55e61c16-9hgtn"] Oct 30 01:44:47.426233 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:44:47.426202 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbff214ad_ad9e_4e94_ba24_e43ce7dfa8b4.slice/crio-8c5c6ed9d274bbe7bbf4d7c01443df1212a86fc1b09b1946359a6000e9651ee0 WatchSource:0}: Error finding container 8c5c6ed9d274bbe7bbf4d7c01443df1212a86fc1b09b1946359a6000e9651ee0: Status 404 returned error can't find the container with id 8c5c6ed9d274bbe7bbf4d7c01443df1212a86fc1b09b1946359a6000e9651ee0 Oct 30 01:44:47.491783 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:47.490711 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/326e903b-0283-438b-b9c6-f463b7f685ae-kube-api-access-4pd7f\" (UniqueName: \"kubernetes.io/projected/326e903b-0283-438b-b9c6-f463b7f685ae-kube-api-access-4pd7f\") pod \"326e903b-0283-438b-b9c6-f463b7f685ae\" (UID: \"326e903b-0283-438b-b9c6-f463b7f685ae\") " Oct 30 01:44:47.491783 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:47.490782 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/326e903b-0283-438b-b9c6-f463b7f685ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/326e903b-0283-438b-b9c6-f463b7f685ae-maas-api-xks-ca\") pod \"326e903b-0283-438b-b9c6-f463b7f685ae\" (UID: \"326e903b-0283-438b-b9c6-f463b7f685ae\") " Oct 30 01:44:47.491783 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:47.490978 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/326e903b-0283-438b-b9c6-f463b7f685ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/326e903b-0283-438b-b9c6-f463b7f685ae-maas-api-openshift-ca\") pod \"326e903b-0283-438b-b9c6-f463b7f685ae\" (UID: \"326e903b-0283-438b-b9c6-f463b7f685ae\") " Oct 30 01:44:47.491783 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:47.491780 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/326e903b-0283-438b-b9c6-f463b7f685ae-maas-api-openshift-ca" pod "326e903b-0283-438b-b9c6-f463b7f685ae" (UID: "326e903b-0283-438b-b9c6-f463b7f685ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:44:47.495544 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:47.495508 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/326e903b-0283-438b-b9c6-f463b7f685ae-kube-api-access-4pd7f" pod "326e903b-0283-438b-b9c6-f463b7f685ae" (UID: "326e903b-0283-438b-b9c6-f463b7f685ae"). InnerVolumeSpecName "kube-api-access-4pd7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:47.500400 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:47.500356 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/326e903b-0283-438b-b9c6-f463b7f685ae-maas-api-xks-ca" pod "326e903b-0283-438b-b9c6-f463b7f685ae" (UID: "326e903b-0283-438b-b9c6-f463b7f685ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:44:47.592770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:47.592716 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pd7f\" (UniqueName: \"kubernetes.io/projected/326e903b-0283-438b-b9c6-f463b7f685ae-kube-api-access-4pd7f\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:47.592770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:47.592773 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/326e903b-0283-438b-b9c6-f463b7f685ae-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:47.593030 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:47.592791 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/326e903b-0283-438b-b9c6-f463b7f685ae-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:47.627825 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:47.627790 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:47.634976 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:47.634944 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f197b1-6dc895f6f6-qqnvt"] Oct 30 01:44:47.635569 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:47.635550 6572 cpu_manager.go:414] "containerMap: removing container" podUID="326e903b-0283-438b-b9c6-f463b7f685ae" containerName="revoke-keys" Oct 30 01:44:47.635688 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:47.635571 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="326e903b-0283-438b-b9c6-f463b7f685ae" containerName="revoke-keys" Oct 30 01:44:47.635806 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:47.635790 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="326e903b-0283-438b-b9c6-f463b7f685ae" containerName="revoke-keys" Oct 30 01:44:47.640893 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:47.640868 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f197b1-6dc895f6f6-qqnvt" Oct 30 01:44:47.659155 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:47.659120 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-f197b1\"" Oct 30 01:44:47.659387 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:47.659361 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-f197b1-dockercfg-h2wtb\"" Oct 30 01:44:47.662634 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:47.662608 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f197b1-6dc895f6f6-qqnvt"] Oct 30 01:44:47.694119 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:47.694073 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzzs6\" (UniqueName: \"kubernetes.io/projected/8136562a-9ad1-41c5-9139-677268c5da39-kube-api-access-zzzs6\") pod \"payload-processing-e2e-praxis-a-f197b1-6dc895f6f6-qqnvt\" (UID: \"8136562a-9ad1-41c5-9139-677268c5da39\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f197b1-6dc895f6f6-qqnvt" Oct 30 01:44:47.694119 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:47.694123 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8136562a-9ad1-41c5-9139-677268c5da39-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f197b1-6dc895f6f6-qqnvt\" (UID: \"8136562a-9ad1-41c5-9139-677268c5da39\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f197b1-6dc895f6f6-qqnvt" Oct 30 01:44:47.805824 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:47.795458 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzzs6\" (UniqueName: \"kubernetes.io/projected/8136562a-9ad1-41c5-9139-677268c5da39-kube-api-access-zzzs6\") pod \"payload-processing-e2e-praxis-a-f197b1-6dc895f6f6-qqnvt\" (UID: \"8136562a-9ad1-41c5-9139-677268c5da39\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f197b1-6dc895f6f6-qqnvt" Oct 30 01:44:47.805824 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:47.795522 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8136562a-9ad1-41c5-9139-677268c5da39-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f197b1-6dc895f6f6-qqnvt\" (UID: \"8136562a-9ad1-41c5-9139-677268c5da39\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f197b1-6dc895f6f6-qqnvt" Oct 30 01:44:47.805824 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:47.796660 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8136562a-9ad1-41c5-9139-677268c5da39-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f197b1-6dc895f6f6-qqnvt\" (UID: \"8136562a-9ad1-41c5-9139-677268c5da39\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f197b1-6dc895f6f6-qqnvt" Oct 30 01:44:47.848294 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:47.848205 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:47.849388 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:47.849348 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzzs6\" (UniqueName: \"kubernetes.io/projected/8136562a-9ad1-41c5-9139-677268c5da39-kube-api-access-zzzs6\") pod \"payload-processing-e2e-praxis-a-f197b1-6dc895f6f6-qqnvt\" (UID: \"8136562a-9ad1-41c5-9139-677268c5da39\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f197b1-6dc895f6f6-qqnvt" Oct 30 01:44:47.956467 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:47.956425 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f197b1-6dc895f6f6-qqnvt" Oct 30 01:44:48.072363 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:48.072326 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:48.199546 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:48.199510 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3612ab3f-p4hql" event={"ID":"326e903b-0283-438b-b9c6-f463b7f685ae","Type":"ContainerDied","Data":"ba0589fb584ee51f2f9900e34ea87b215d0bc7dc06d82e885658c55c5f4a6cf3"} Oct 30 01:44:48.200016 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:48.199569 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba0589fb584ee51f2f9900e34ea87b215d0bc7dc06d82e885658c55c5f4a6cf3" Oct 30 01:44:48.200016 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:48.199517 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3612ab3f-p4hql" Oct 30 01:44:48.201575 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:48.201547 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e61c16-9hgtn" event={"ID":"bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4","Type":"ContainerStarted","Data":"3d1ba0386b1328db6b9d3583deb6e3c70fe2b6ceee978f6a58e50ee0a54dbf20"} Oct 30 01:44:48.201680 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:48.201603 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e61c16-9hgtn" event={"ID":"bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4","Type":"ContainerStarted","Data":"8c5c6ed9d274bbe7bbf4d7c01443df1212a86fc1b09b1946359a6000e9651ee0"} Oct 30 01:44:48.270292 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:48.270219 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e61c16-9hgtn" podStartSLOduration=2.2701956 podStartE2EDuration="2.2701956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:44:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:44:48.266110799 +0000 UTC m=+1613.842536203" watchObservedRunningTime="2026-10-30 01:44:48.2701956 +0000 UTC m=+1613.846621004" Oct 30 01:44:48.276871 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:48.276738 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f197b1-6dc895f6f6-qqnvt"] Oct 30 01:44:48.283097 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:44:48.283060 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8136562a_9ad1_41c5_9139_677268c5da39.slice/crio-b8f7748c4f92b577dd4c85f99a88a7fe3ac506d21313a78e7d048cbd93a5cb01 WatchSource:0}: Error finding container b8f7748c4f92b577dd4c85f99a88a7fe3ac506d21313a78e7d048cbd93a5cb01: Status 404 returned error can't find the container with id b8f7748c4f92b577dd4c85f99a88a7fe3ac506d21313a78e7d048cbd93a5cb01 Oct 30 01:44:49.213505 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:49.213464 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f197b1-6dc895f6f6-qqnvt" event={"ID":"8136562a-9ad1-41c5-9139-677268c5da39","Type":"ContainerStarted","Data":"aab9a4a191b245c7dc3acae84840cfb44e08df0702c286e92480a8c243a3cb90"} Oct 30 01:44:49.214043 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:49.213534 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-f197b1-6dc895f6f6-qqnvt" Oct 30 01:44:49.214043 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:49.213559 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f197b1-6dc895f6f6-qqnvt" event={"ID":"8136562a-9ad1-41c5-9139-677268c5da39","Type":"ContainerStarted","Data":"b8f7748c4f92b577dd4c85f99a88a7fe3ac506d21313a78e7d048cbd93a5cb01"} Oct 30 01:44:49.215364 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:49.215339 6572 generic.go:356] "Generic (PLEG): container finished" podID="bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4" containerID="3d1ba0386b1328db6b9d3583deb6e3c70fe2b6ceee978f6a58e50ee0a54dbf20" exitCode=0 Oct 30 01:44:49.215488 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:49.215415 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e61c16-9hgtn" event={"ID":"bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4","Type":"ContainerDied","Data":"3d1ba0386b1328db6b9d3583deb6e3c70fe2b6ceee978f6a58e50ee0a54dbf20"} Oct 30 01:44:49.262497 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:49.262421 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-f197b1-6dc895f6f6-qqnvt" podStartSLOduration=2.262397055 podStartE2EDuration="2.262397055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:44:49.25152148 +0000 UTC m=+1614.827946918" watchObservedRunningTime="2026-10-30 01:44:49.262397055 +0000 UTC m=+1614.838822458" Oct 30 01:44:49.304250 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:49.304208 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:50.361829 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:50.361801 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e61c16-9hgtn" Oct 30 01:44:50.428865 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:50.428829 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4-maas-api-openshift-ca\") pod \"bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4\" (UID: \"bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4\") " Oct 30 01:44:50.429043 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:50.428898 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4-kube-api-access-wgqwm\" (UniqueName: \"kubernetes.io/projected/bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4-kube-api-access-wgqwm\") pod \"bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4\" (UID: \"bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4\") " Oct 30 01:44:50.429043 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:50.428960 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4-maas-api-xks-ca\") pod \"bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4\" (UID: \"bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4\") " Oct 30 01:44:50.429299 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:50.429274 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4-maas-api-openshift-ca" pod "bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4" (UID: "bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:44:50.429497 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:50.429480 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:50.431200 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:50.431164 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4-kube-api-access-wgqwm" pod "bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4" (UID: "bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4"). InnerVolumeSpecName "kube-api-access-wgqwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:50.431556 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:50.431511 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4-maas-api-xks-ca" pod "bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4" (UID: "bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:44:50.530526 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:50.530437 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgqwm\" (UniqueName: \"kubernetes.io/projected/bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4-kube-api-access-wgqwm\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:50.530526 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:50.530472 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:51.016138 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:51.016094 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:51.232554 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:51.232521 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e61c16-9hgtn" Oct 30 01:44:51.232724 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:51.232560 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e61c16-9hgtn" event={"ID":"bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4","Type":"ContainerDied","Data":"8c5c6ed9d274bbe7bbf4d7c01443df1212a86fc1b09b1946359a6000e9651ee0"} Oct 30 01:44:51.232724 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:51.232609 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c5c6ed9d274bbe7bbf4d7c01443df1212a86fc1b09b1946359a6000e9651ee0" Oct 30 01:44:52.872978 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:52.872934 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:53.477831 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:53.477799 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c045c797-2fpp8"] Oct 30 01:44:53.478379 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:53.478365 6572 cpu_manager.go:414] "containerMap: removing container" podUID="bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4" containerName="revoke-keys" Oct 30 01:44:53.478435 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:53.478380 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4" containerName="revoke-keys" Oct 30 01:44:53.478518 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:53.478480 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4" containerName="revoke-keys" Oct 30 01:44:53.482088 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:53.482066 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c045c797-2fpp8" Oct 30 01:44:53.500738 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:53.500702 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c045c797-2fpp8"] Oct 30 01:44:53.507698 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:53.507664 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:44:53.507961 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:53.507845 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:44:53.507961 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:53.507916 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:44:53.508127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:53.508037 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:44:53.581143 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:53.581102 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9ss8\" (UniqueName: \"kubernetes.io/projected/ee4f2e25-28ea-41f9-9e33-001c0f482823-kube-api-access-b9ss8\") pod \"maas-api-revoke-sub-c045c797-2fpp8\" (UID: \"ee4f2e25-28ea-41f9-9e33-001c0f482823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c045c797-2fpp8" Oct 30 01:44:53.581333 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:53.581156 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee4f2e25-28ea-41f9-9e33-001c0f482823-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c045c797-2fpp8\" (UID: \"ee4f2e25-28ea-41f9-9e33-001c0f482823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c045c797-2fpp8" Oct 30 01:44:53.581333 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:53.581288 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee4f2e25-28ea-41f9-9e33-001c0f482823-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c045c797-2fpp8\" (UID: \"ee4f2e25-28ea-41f9-9e33-001c0f482823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c045c797-2fpp8" Oct 30 01:44:53.682472 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:53.682427 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9ss8\" (UniqueName: \"kubernetes.io/projected/ee4f2e25-28ea-41f9-9e33-001c0f482823-kube-api-access-b9ss8\") pod \"maas-api-revoke-sub-c045c797-2fpp8\" (UID: \"ee4f2e25-28ea-41f9-9e33-001c0f482823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c045c797-2fpp8" Oct 30 01:44:53.682659 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:53.682510 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee4f2e25-28ea-41f9-9e33-001c0f482823-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c045c797-2fpp8\" (UID: \"ee4f2e25-28ea-41f9-9e33-001c0f482823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c045c797-2fpp8" Oct 30 01:44:53.682659 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:53.682560 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee4f2e25-28ea-41f9-9e33-001c0f482823-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c045c797-2fpp8\" (UID: \"ee4f2e25-28ea-41f9-9e33-001c0f482823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c045c797-2fpp8" Oct 30 01:44:53.683252 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:53.683226 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee4f2e25-28ea-41f9-9e33-001c0f482823-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c045c797-2fpp8\" (UID: \"ee4f2e25-28ea-41f9-9e33-001c0f482823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c045c797-2fpp8" Oct 30 01:44:53.685034 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:53.685014 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee4f2e25-28ea-41f9-9e33-001c0f482823-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c045c797-2fpp8\" (UID: \"ee4f2e25-28ea-41f9-9e33-001c0f482823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c045c797-2fpp8" Oct 30 01:44:53.701818 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:53.701778 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9ss8\" (UniqueName: \"kubernetes.io/projected/ee4f2e25-28ea-41f9-9e33-001c0f482823-kube-api-access-b9ss8\") pod \"maas-api-revoke-sub-c045c797-2fpp8\" (UID: \"ee4f2e25-28ea-41f9-9e33-001c0f482823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c045c797-2fpp8" Oct 30 01:44:53.796647 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:53.796550 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c045c797-2fpp8" Oct 30 01:44:54.109188 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:54.109157 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c045c797-2fpp8"] Oct 30 01:44:54.111496 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:44:54.111467 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee4f2e25_28ea_41f9_9e33_001c0f482823.slice/crio-bce4cf281abb960795994773b3295519f31cd518cf94d1a0c409610e1038dd5d WatchSource:0}: Error finding container bce4cf281abb960795994773b3295519f31cd518cf94d1a0c409610e1038dd5d: Status 404 returned error can't find the container with id bce4cf281abb960795994773b3295519f31cd518cf94d1a0c409610e1038dd5d Oct 30 01:44:54.260267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:54.260213 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c045c797-2fpp8" event={"ID":"ee4f2e25-28ea-41f9-9e33-001c0f482823","Type":"ContainerStarted","Data":"bce4cf281abb960795994773b3295519f31cd518cf94d1a0c409610e1038dd5d"} Oct 30 01:44:55.067044 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:55.067007 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:55.269725 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:55.269629 6572 generic.go:356] "Generic (PLEG): container finished" podID="ee4f2e25-28ea-41f9-9e33-001c0f482823" containerID="85be00953de5b97ae327f898d5cb13716763ff1955ad5749d08ff7d20abef4c8" exitCode=0 Oct 30 01:44:55.269725 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:55.269698 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c045c797-2fpp8" event={"ID":"ee4f2e25-28ea-41f9-9e33-001c0f482823","Type":"ContainerDied","Data":"85be00953de5b97ae327f898d5cb13716763ff1955ad5749d08ff7d20abef4c8"} Oct 30 01:44:56.435757 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:56.435715 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c045c797-2fpp8" Oct 30 01:44:56.437880 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:56.437857 6572 scope.go:122] "RemoveContainer" containerID="f86982ce81697156fde84f84b59973662e196d6bf72eea730fcd43f3ea0765b1" Oct 30 01:44:56.618510 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:56.618412 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee4f2e25-28ea-41f9-9e33-001c0f482823-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee4f2e25-28ea-41f9-9e33-001c0f482823-maas-api-xks-ca\") pod \"ee4f2e25-28ea-41f9-9e33-001c0f482823\" (UID: \"ee4f2e25-28ea-41f9-9e33-001c0f482823\") " Oct 30 01:44:56.618694 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:56.618521 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee4f2e25-28ea-41f9-9e33-001c0f482823-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee4f2e25-28ea-41f9-9e33-001c0f482823-maas-api-openshift-ca\") pod \"ee4f2e25-28ea-41f9-9e33-001c0f482823\" (UID: \"ee4f2e25-28ea-41f9-9e33-001c0f482823\") " Oct 30 01:44:56.618694 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:56.618547 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee4f2e25-28ea-41f9-9e33-001c0f482823-kube-api-access-b9ss8\" (UniqueName: \"kubernetes.io/projected/ee4f2e25-28ea-41f9-9e33-001c0f482823-kube-api-access-b9ss8\") pod \"ee4f2e25-28ea-41f9-9e33-001c0f482823\" (UID: \"ee4f2e25-28ea-41f9-9e33-001c0f482823\") " Oct 30 01:44:56.618990 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:56.618966 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee4f2e25-28ea-41f9-9e33-001c0f482823-maas-api-openshift-ca" pod "ee4f2e25-28ea-41f9-9e33-001c0f482823" (UID: "ee4f2e25-28ea-41f9-9e33-001c0f482823"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:44:56.620712 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:56.620675 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee4f2e25-28ea-41f9-9e33-001c0f482823-kube-api-access-b9ss8" pod "ee4f2e25-28ea-41f9-9e33-001c0f482823" (UID: "ee4f2e25-28ea-41f9-9e33-001c0f482823"). InnerVolumeSpecName "kube-api-access-b9ss8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:56.620712 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:56.620692 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee4f2e25-28ea-41f9-9e33-001c0f482823-maas-api-xks-ca" pod "ee4f2e25-28ea-41f9-9e33-001c0f482823" (UID: "ee4f2e25-28ea-41f9-9e33-001c0f482823"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:44:56.720099 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:56.720055 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee4f2e25-28ea-41f9-9e33-001c0f482823-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:56.720099 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:56.720092 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee4f2e25-28ea-41f9-9e33-001c0f482823-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:56.720099 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:56.720102 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9ss8\" (UniqueName: \"kubernetes.io/projected/ee4f2e25-28ea-41f9-9e33-001c0f482823-kube-api-access-b9ss8\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:44:56.962366 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:56.962316 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:57.257605 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:57.257518 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b80122c5-lc25m"] Oct 30 01:44:57.258097 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:57.258083 6572 cpu_manager.go:414] "containerMap: removing container" podUID="ee4f2e25-28ea-41f9-9e33-001c0f482823" containerName="revoke-keys" Oct 30 01:44:57.258139 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:57.258099 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee4f2e25-28ea-41f9-9e33-001c0f482823" containerName="revoke-keys" Oct 30 01:44:57.258215 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:57.258206 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee4f2e25-28ea-41f9-9e33-001c0f482823" containerName="revoke-keys" Oct 30 01:44:57.263976 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:57.263957 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80122c5-lc25m" Oct 30 01:44:57.284360 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:57.284328 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b80122c5-lc25m"] Oct 30 01:44:57.288556 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:57.288534 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c045c797-2fpp8" Oct 30 01:44:57.288689 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:57.288568 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c045c797-2fpp8" event={"ID":"ee4f2e25-28ea-41f9-9e33-001c0f482823","Type":"ContainerDied","Data":"bce4cf281abb960795994773b3295519f31cd518cf94d1a0c409610e1038dd5d"} Oct 30 01:44:57.288689 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:57.288620 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bce4cf281abb960795994773b3295519f31cd518cf94d1a0c409610e1038dd5d" Oct 30 01:44:57.427662 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:57.427621 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t54s6\" (UniqueName: \"kubernetes.io/projected/c2810118-aba3-474f-aefb-ce3393aaef31-kube-api-access-t54s6\") pod \"maas-api-revoke-sub-b80122c5-lc25m\" (UID: \"c2810118-aba3-474f-aefb-ce3393aaef31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80122c5-lc25m" Oct 30 01:44:57.427892 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:57.427796 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2810118-aba3-474f-aefb-ce3393aaef31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b80122c5-lc25m\" (UID: \"c2810118-aba3-474f-aefb-ce3393aaef31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80122c5-lc25m" Oct 30 01:44:57.427892 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:57.427831 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2810118-aba3-474f-aefb-ce3393aaef31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b80122c5-lc25m\" (UID: \"c2810118-aba3-474f-aefb-ce3393aaef31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80122c5-lc25m" Oct 30 01:44:57.528686 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:57.528589 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2810118-aba3-474f-aefb-ce3393aaef31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b80122c5-lc25m\" (UID: \"c2810118-aba3-474f-aefb-ce3393aaef31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80122c5-lc25m" Oct 30 01:44:57.528686 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:57.528642 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2810118-aba3-474f-aefb-ce3393aaef31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b80122c5-lc25m\" (UID: \"c2810118-aba3-474f-aefb-ce3393aaef31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80122c5-lc25m" Oct 30 01:44:57.529201 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:57.528813 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t54s6\" (UniqueName: \"kubernetes.io/projected/c2810118-aba3-474f-aefb-ce3393aaef31-kube-api-access-t54s6\") pod \"maas-api-revoke-sub-b80122c5-lc25m\" (UID: \"c2810118-aba3-474f-aefb-ce3393aaef31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80122c5-lc25m" Oct 30 01:44:57.529456 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:57.529436 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2810118-aba3-474f-aefb-ce3393aaef31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b80122c5-lc25m\" (UID: \"c2810118-aba3-474f-aefb-ce3393aaef31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80122c5-lc25m" Oct 30 01:44:57.531022 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:57.531003 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2810118-aba3-474f-aefb-ce3393aaef31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b80122c5-lc25m\" (UID: \"c2810118-aba3-474f-aefb-ce3393aaef31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80122c5-lc25m" Oct 30 01:44:57.548486 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:57.548460 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t54s6\" (UniqueName: \"kubernetes.io/projected/c2810118-aba3-474f-aefb-ce3393aaef31-kube-api-access-t54s6\") pod \"maas-api-revoke-sub-b80122c5-lc25m\" (UID: \"c2810118-aba3-474f-aefb-ce3393aaef31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80122c5-lc25m" Oct 30 01:44:57.578076 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:57.578044 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80122c5-lc25m" Oct 30 01:44:57.876826 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:57.876793 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b80122c5-lc25m"] Oct 30 01:44:57.877984 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:44:57.877957 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2810118_aba3_474f_aefb_ce3393aaef31.slice/crio-d89dfba030e95c41f9e106177c39945d6506e1a55a9c12d4e0d08b2dfcce57c7 WatchSource:0}: Error finding container d89dfba030e95c41f9e106177c39945d6506e1a55a9c12d4e0d08b2dfcce57c7: Status 404 returned error can't find the container with id d89dfba030e95c41f9e106177c39945d6506e1a55a9c12d4e0d08b2dfcce57c7 Oct 30 01:44:58.071951 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:58.071913 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1050396f-r58zb"] Oct 30 01:44:58.077916 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:58.077885 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1050396f-r58zb"] Oct 30 01:44:58.297417 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:58.297376 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80122c5-lc25m" event={"ID":"c2810118-aba3-474f-aefb-ce3393aaef31","Type":"ContainerStarted","Data":"f3d1435ebaada96efd09dfebabeefeb5d8b744f063cfa6198693124942c05fb9"} Oct 30 01:44:58.297583 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:58.297473 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80122c5-lc25m" event={"ID":"c2810118-aba3-474f-aefb-ce3393aaef31","Type":"ContainerStarted","Data":"d89dfba030e95c41f9e106177c39945d6506e1a55a9c12d4e0d08b2dfcce57c7"} Oct 30 01:44:58.347888 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:58.347830 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80122c5-lc25m" podStartSLOduration=1.347814557 podStartE2EDuration="1.347814557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:44:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:44:58.345655151 +0000 UTC m=+1623.922080553" watchObservedRunningTime="2026-10-30 01:44:58.347814557 +0000 UTC m=+1623.924239959" Oct 30 01:44:58.543934 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:58.543859 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46ee12a0-1673-4ba6-b16d-2be18884f588" path="/var/lib/kubelet/pods/46ee12a0-1673-4ba6-b16d-2be18884f588/volumes" Oct 30 01:44:58.730854 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:58.730815 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:59.168957 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:59.168916 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9315ff4-f46th"] Oct 30 01:44:59.173258 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:59.173232 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9315ff4-f46th" Oct 30 01:44:59.193845 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:59.193812 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9315ff4-f46th"] Oct 30 01:44:59.307627 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:59.307590 6572 generic.go:356] "Generic (PLEG): container finished" podID="c2810118-aba3-474f-aefb-ce3393aaef31" containerID="f3d1435ebaada96efd09dfebabeefeb5d8b744f063cfa6198693124942c05fb9" exitCode=0 Oct 30 01:44:59.307904 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:59.307674 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80122c5-lc25m" event={"ID":"c2810118-aba3-474f-aefb-ce3393aaef31","Type":"ContainerDied","Data":"f3d1435ebaada96efd09dfebabeefeb5d8b744f063cfa6198693124942c05fb9"} Oct 30 01:44:59.350523 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:59.350489 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5mk2\" (UniqueName: \"kubernetes.io/projected/7057658c-88a0-4bce-bb3f-765903a4b7fa-kube-api-access-s5mk2\") pod \"maas-api-revoke-sub-b9315ff4-f46th\" (UID: \"7057658c-88a0-4bce-bb3f-765903a4b7fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9315ff4-f46th" Oct 30 01:44:59.350796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:59.350554 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7057658c-88a0-4bce-bb3f-765903a4b7fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9315ff4-f46th\" (UID: \"7057658c-88a0-4bce-bb3f-765903a4b7fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9315ff4-f46th" Oct 30 01:44:59.350796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:59.350648 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7057658c-88a0-4bce-bb3f-765903a4b7fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9315ff4-f46th\" (UID: \"7057658c-88a0-4bce-bb3f-765903a4b7fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9315ff4-f46th" Oct 30 01:44:59.451761 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:59.451643 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5mk2\" (UniqueName: \"kubernetes.io/projected/7057658c-88a0-4bce-bb3f-765903a4b7fa-kube-api-access-s5mk2\") pod \"maas-api-revoke-sub-b9315ff4-f46th\" (UID: \"7057658c-88a0-4bce-bb3f-765903a4b7fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9315ff4-f46th" Oct 30 01:44:59.452051 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:59.451738 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7057658c-88a0-4bce-bb3f-765903a4b7fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9315ff4-f46th\" (UID: \"7057658c-88a0-4bce-bb3f-765903a4b7fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9315ff4-f46th" Oct 30 01:44:59.452051 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:59.451836 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7057658c-88a0-4bce-bb3f-765903a4b7fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9315ff4-f46th\" (UID: \"7057658c-88a0-4bce-bb3f-765903a4b7fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9315ff4-f46th" Oct 30 01:44:59.452510 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:59.452482 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7057658c-88a0-4bce-bb3f-765903a4b7fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9315ff4-f46th\" (UID: \"7057658c-88a0-4bce-bb3f-765903a4b7fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9315ff4-f46th" Oct 30 01:44:59.454204 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:59.454181 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7057658c-88a0-4bce-bb3f-765903a4b7fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9315ff4-f46th\" (UID: \"7057658c-88a0-4bce-bb3f-765903a4b7fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9315ff4-f46th" Oct 30 01:44:59.471445 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:59.471412 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5mk2\" (UniqueName: \"kubernetes.io/projected/7057658c-88a0-4bce-bb3f-765903a4b7fa-kube-api-access-s5mk2\") pod \"maas-api-revoke-sub-b9315ff4-f46th\" (UID: \"7057658c-88a0-4bce-bb3f-765903a4b7fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9315ff4-f46th" Oct 30 01:44:59.487436 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:59.487402 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9315ff4-f46th" Oct 30 01:44:59.793718 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:44:59.793686 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9315ff4-f46th"] Oct 30 01:44:59.795092 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:44:59.795066 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7057658c_88a0_4bce_bb3f_765903a4b7fa.slice/crio-015f435a9aa898c2d755f4ab1a6d6f2d41c914b4d2aa438914ea605fe52d3872 WatchSource:0}: Error finding container 015f435a9aa898c2d755f4ab1a6d6f2d41c914b4d2aa438914ea605fe52d3872: Status 404 returned error can't find the container with id 015f435a9aa898c2d755f4ab1a6d6f2d41c914b4d2aa438914ea605fe52d3872 Oct 30 01:45:00.175938 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.175899 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888745-bffjq"] Oct 30 01:45:00.180146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.180111 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888745-68pqj"] Oct 30 01:45:00.180337 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.180233 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888745-bffjq" Oct 30 01:45:00.184360 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.184330 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888745-68pqj" Oct 30 01:45:00.187905 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.187880 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888745-w7rlv"] Oct 30 01:45:00.192116 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.192092 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888745-w7rlv" Oct 30 01:45:00.215012 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.214892 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888745-bffjq"] Oct 30 01:45:00.219201 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.219173 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-58632b-29888745-gqzpm"] Oct 30 01:45:00.223512 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.223484 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888745-68pqj"] Oct 30 01:45:00.223683 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.223602 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-58632b-29888745-gqzpm" Oct 30 01:45:00.230623 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.226242 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-f197b1-6dc895f6f6-qqnvt" Oct 30 01:45:00.236979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.236907 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888745-w7rlv"] Oct 30 01:45:00.265910 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.265871 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-58632b-29888745-gqzpm"] Oct 30 01:45:00.321404 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.321364 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9315ff4-f46th" event={"ID":"7057658c-88a0-4bce-bb3f-765903a4b7fa","Type":"ContainerStarted","Data":"8bb2193200f14fddc22587bcc06c0c4a48ee78dd4ea90a1b682069c6700f4a68"} Oct 30 01:45:00.321809 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.321681 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9315ff4-f46th" event={"ID":"7057658c-88a0-4bce-bb3f-765903a4b7fa","Type":"ContainerStarted","Data":"015f435a9aa898c2d755f4ab1a6d6f2d41c914b4d2aa438914ea605fe52d3872"} Oct 30 01:45:00.362858 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.362118 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c76a323c-f168-42f8-933b-b28c333dcac4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-eda881-29888745-68pqj\" (UID: \"c76a323c-f168-42f8-933b-b28c333dcac4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888745-68pqj" Oct 30 01:45:00.362858 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.362291 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4f29\" (UniqueName: \"kubernetes.io/projected/fae65009-54c2-4407-90c3-bc753280c957-kube-api-access-r4f29\") pod \"maas-api-key-cleanup-e2e-praxis-b-58632b-29888745-gqzpm\" (UID: \"fae65009-54c2-4407-90c3-bc753280c957\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-58632b-29888745-gqzpm" Oct 30 01:45:00.362858 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.362486 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fae65009-54c2-4407-90c3-bc753280c957-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-58632b-29888745-gqzpm\" (UID: \"fae65009-54c2-4407-90c3-bc753280c957\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-58632b-29888745-gqzpm" Oct 30 01:45:00.362858 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.362547 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db1407a-ef40-4f6a-8d81-5e3a1e502c33-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-803472-29888745-bffjq\" (UID: \"5db1407a-ef40-4f6a-8d81-5e3a1e502c33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888745-bffjq" Oct 30 01:45:00.362858 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.362599 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lclgl\" (UniqueName: \"kubernetes.io/projected/c76a323c-f168-42f8-933b-b28c333dcac4-kube-api-access-lclgl\") pod \"maas-api-key-cleanup-e2e-worker-w2-eda881-29888745-68pqj\" (UID: \"c76a323c-f168-42f8-933b-b28c333dcac4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888745-68pqj" Oct 30 01:45:00.362858 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.362661 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scbpt\" (UniqueName: \"kubernetes.io/projected/1aa4b366-acc4-44d6-9208-3884e0349bb4-kube-api-access-scbpt\") pod \"maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888745-w7rlv\" (UID: \"1aa4b366-acc4-44d6-9208-3884e0349bb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888745-w7rlv" Oct 30 01:45:00.362858 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.362796 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa4b366-acc4-44d6-9208-3884e0349bb4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888745-w7rlv\" (UID: \"1aa4b366-acc4-44d6-9208-3884e0349bb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888745-w7rlv" Oct 30 01:45:00.372570 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.366293 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf82j\" (UniqueName: \"kubernetes.io/projected/5db1407a-ef40-4f6a-8d81-5e3a1e502c33-kube-api-access-tf82j\") pod \"maas-api-key-cleanup-e2e-worker-w4-803472-29888745-bffjq\" (UID: \"5db1407a-ef40-4f6a-8d81-5e3a1e502c33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888745-bffjq" Oct 30 01:45:00.433890 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.433674 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9315ff4-f46th" podStartSLOduration=1.433649651 podStartE2EDuration="1.433649651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:44:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:00.432988622 +0000 UTC m=+1626.009414025" watchObservedRunningTime="2026-10-30 01:45:00.433649651 +0000 UTC m=+1626.010075053" Oct 30 01:45:00.467514 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.467488 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4f29\" (UniqueName: \"kubernetes.io/projected/fae65009-54c2-4407-90c3-bc753280c957-kube-api-access-r4f29\") pod \"maas-api-key-cleanup-e2e-praxis-b-58632b-29888745-gqzpm\" (UID: \"fae65009-54c2-4407-90c3-bc753280c957\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-58632b-29888745-gqzpm" Oct 30 01:45:00.467647 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.467527 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fae65009-54c2-4407-90c3-bc753280c957-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-58632b-29888745-gqzpm\" (UID: \"fae65009-54c2-4407-90c3-bc753280c957\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-58632b-29888745-gqzpm" Oct 30 01:45:00.467647 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.467563 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db1407a-ef40-4f6a-8d81-5e3a1e502c33-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-803472-29888745-bffjq\" (UID: \"5db1407a-ef40-4f6a-8d81-5e3a1e502c33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888745-bffjq" Oct 30 01:45:00.467647 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.467591 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lclgl\" (UniqueName: \"kubernetes.io/projected/c76a323c-f168-42f8-933b-b28c333dcac4-kube-api-access-lclgl\") pod \"maas-api-key-cleanup-e2e-worker-w2-eda881-29888745-68pqj\" (UID: \"c76a323c-f168-42f8-933b-b28c333dcac4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888745-68pqj" Oct 30 01:45:00.467647 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.467632 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scbpt\" (UniqueName: \"kubernetes.io/projected/1aa4b366-acc4-44d6-9208-3884e0349bb4-kube-api-access-scbpt\") pod \"maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888745-w7rlv\" (UID: \"1aa4b366-acc4-44d6-9208-3884e0349bb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888745-w7rlv" Oct 30 01:45:00.467896 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.467701 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa4b366-acc4-44d6-9208-3884e0349bb4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888745-w7rlv\" (UID: \"1aa4b366-acc4-44d6-9208-3884e0349bb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888745-w7rlv" Oct 30 01:45:00.467896 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.467788 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf82j\" (UniqueName: \"kubernetes.io/projected/5db1407a-ef40-4f6a-8d81-5e3a1e502c33-kube-api-access-tf82j\") pod \"maas-api-key-cleanup-e2e-worker-w4-803472-29888745-bffjq\" (UID: \"5db1407a-ef40-4f6a-8d81-5e3a1e502c33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888745-bffjq" Oct 30 01:45:00.468140 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.468114 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c76a323c-f168-42f8-933b-b28c333dcac4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-eda881-29888745-68pqj\" (UID: \"c76a323c-f168-42f8-933b-b28c333dcac4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888745-68pqj" Oct 30 01:45:00.468450 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.468427 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fae65009-54c2-4407-90c3-bc753280c957-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-58632b-29888745-gqzpm\" (UID: \"fae65009-54c2-4407-90c3-bc753280c957\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-58632b-29888745-gqzpm" Oct 30 01:45:00.468520 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.468430 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db1407a-ef40-4f6a-8d81-5e3a1e502c33-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-803472-29888745-bffjq\" (UID: \"5db1407a-ef40-4f6a-8d81-5e3a1e502c33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888745-bffjq" Oct 30 01:45:00.468520 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.468471 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa4b366-acc4-44d6-9208-3884e0349bb4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888745-w7rlv\" (UID: \"1aa4b366-acc4-44d6-9208-3884e0349bb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888745-w7rlv" Oct 30 01:45:00.468778 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.468729 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c76a323c-f168-42f8-933b-b28c333dcac4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-eda881-29888745-68pqj\" (UID: \"c76a323c-f168-42f8-933b-b28c333dcac4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888745-68pqj" Oct 30 01:45:00.489997 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.489966 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4f29\" (UniqueName: \"kubernetes.io/projected/fae65009-54c2-4407-90c3-bc753280c957-kube-api-access-r4f29\") pod \"maas-api-key-cleanup-e2e-praxis-b-58632b-29888745-gqzpm\" (UID: \"fae65009-54c2-4407-90c3-bc753280c957\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-58632b-29888745-gqzpm" Oct 30 01:45:00.492005 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.491978 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf82j\" (UniqueName: \"kubernetes.io/projected/5db1407a-ef40-4f6a-8d81-5e3a1e502c33-kube-api-access-tf82j\") pod \"maas-api-key-cleanup-e2e-worker-w4-803472-29888745-bffjq\" (UID: \"5db1407a-ef40-4f6a-8d81-5e3a1e502c33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888745-bffjq" Oct 30 01:45:00.493443 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.493411 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lclgl\" (UniqueName: \"kubernetes.io/projected/c76a323c-f168-42f8-933b-b28c333dcac4-kube-api-access-lclgl\") pod \"maas-api-key-cleanup-e2e-worker-w2-eda881-29888745-68pqj\" (UID: \"c76a323c-f168-42f8-933b-b28c333dcac4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888745-68pqj" Oct 30 01:45:00.493556 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.493508 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scbpt\" (UniqueName: \"kubernetes.io/projected/1aa4b366-acc4-44d6-9208-3884e0349bb4-kube-api-access-scbpt\") pod \"maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888745-w7rlv\" (UID: \"1aa4b366-acc4-44d6-9208-3884e0349bb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888745-w7rlv" Oct 30 01:45:00.495128 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.495111 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80122c5-lc25m" Oct 30 01:45:00.497415 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.497395 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888745-bffjq" Oct 30 01:45:00.505612 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.505584 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888745-68pqj" Oct 30 01:45:00.517510 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.517483 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888745-w7rlv" Oct 30 01:45:00.536453 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.536406 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:00.556702 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.556044 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-58632b-29888745-gqzpm" Oct 30 01:45:00.571640 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.570107 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2810118-aba3-474f-aefb-ce3393aaef31-kube-api-access-t54s6\" (UniqueName: \"kubernetes.io/projected/c2810118-aba3-474f-aefb-ce3393aaef31-kube-api-access-t54s6\") pod \"c2810118-aba3-474f-aefb-ce3393aaef31\" (UID: \"c2810118-aba3-474f-aefb-ce3393aaef31\") " Oct 30 01:45:00.571640 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.570321 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2810118-aba3-474f-aefb-ce3393aaef31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2810118-aba3-474f-aefb-ce3393aaef31-maas-api-xks-ca\") pod \"c2810118-aba3-474f-aefb-ce3393aaef31\" (UID: \"c2810118-aba3-474f-aefb-ce3393aaef31\") " Oct 30 01:45:00.571640 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.570431 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2810118-aba3-474f-aefb-ce3393aaef31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2810118-aba3-474f-aefb-ce3393aaef31-maas-api-openshift-ca\") pod \"c2810118-aba3-474f-aefb-ce3393aaef31\" (UID: \"c2810118-aba3-474f-aefb-ce3393aaef31\") " Oct 30 01:45:00.571640 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.571221 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2810118-aba3-474f-aefb-ce3393aaef31-maas-api-openshift-ca" pod "c2810118-aba3-474f-aefb-ce3393aaef31" (UID: "c2810118-aba3-474f-aefb-ce3393aaef31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:45:00.574334 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.573689 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2810118-aba3-474f-aefb-ce3393aaef31-maas-api-xks-ca" pod "c2810118-aba3-474f-aefb-ce3393aaef31" (UID: "c2810118-aba3-474f-aefb-ce3393aaef31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:45:00.577823 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.577229 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2810118-aba3-474f-aefb-ce3393aaef31-kube-api-access-t54s6" pod "c2810118-aba3-474f-aefb-ce3393aaef31" (UID: "c2810118-aba3-474f-aefb-ce3393aaef31"). InnerVolumeSpecName "kube-api-access-t54s6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:45:00.672396 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.672351 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2810118-aba3-474f-aefb-ce3393aaef31-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:00.672396 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.672396 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2810118-aba3-474f-aefb-ce3393aaef31-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:00.672675 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.672411 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t54s6\" (UniqueName: \"kubernetes.io/projected/c2810118-aba3-474f-aefb-ce3393aaef31-kube-api-access-t54s6\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:00.758414 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.758378 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:00.853886 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.853859 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888745-bffjq"] Oct 30 01:45:00.854595 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:45:00.854574 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5db1407a_ef40_4f6a_8d81_5e3a1e502c33.slice/crio-b1a80b6d839f5548254df01fd96536f06176ad8e05e08cb5cf5e94cebfa144fc WatchSource:0}: Error finding container b1a80b6d839f5548254df01fd96536f06176ad8e05e08cb5cf5e94cebfa144fc: Status 404 returned error can't find the container with id b1a80b6d839f5548254df01fd96536f06176ad8e05e08cb5cf5e94cebfa144fc Oct 30 01:45:00.883148 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.883116 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888745-68pqj"] Oct 30 01:45:00.895153 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:45:00.895128 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc76a323c_f168_42f8_933b_b28c333dcac4.slice/crio-52950fd440ee03bf19e276c4019535bce3ef40c2c6e29f90bafb28800f569197 WatchSource:0}: Error finding container 52950fd440ee03bf19e276c4019535bce3ef40c2c6e29f90bafb28800f569197: Status 404 returned error can't find the container with id 52950fd440ee03bf19e276c4019535bce3ef40c2c6e29f90bafb28800f569197 Oct 30 01:45:00.941324 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.941294 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888745-w7rlv"] Oct 30 01:45:00.942884 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:45:00.942859 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae65009_54c2_4407_90c3_bc753280c957.slice/crio-6deedc086f2204f3366d096787c77b53f5bcd24d98a37d574525e5e05b7e0812 WatchSource:0}: Error finding container 6deedc086f2204f3366d096787c77b53f5bcd24d98a37d574525e5e05b7e0812: Status 404 returned error can't find the container with id 6deedc086f2204f3366d096787c77b53f5bcd24d98a37d574525e5e05b7e0812 Oct 30 01:45:00.943785 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:45:00.943708 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa4b366_acc4_44d6_9208_3884e0349bb4.slice/crio-6fe173c848de0fa8726fb2fd86bf33045c02de808b544bcfc473294e51d94426 WatchSource:0}: Error finding container 6fe173c848de0fa8726fb2fd86bf33045c02de808b544bcfc473294e51d94426: Status 404 returned error can't find the container with id 6fe173c848de0fa8726fb2fd86bf33045c02de808b544bcfc473294e51d94426 Oct 30 01:45:00.944920 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.944853 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-58632b-29888745-gqzpm"] Oct 30 01:45:00.987897 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:00.987864 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:01.330485 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:01.330452 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-58632b-29888745-gqzpm" event={"ID":"fae65009-54c2-4407-90c3-bc753280c957","Type":"ContainerStarted","Data":"e0132205c0a78c8e9f33d759f2a9ea474bbb39b20bc0dc5843c96f19f7432947"} Oct 30 01:45:01.330608 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:01.330509 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-58632b-29888745-gqzpm" event={"ID":"fae65009-54c2-4407-90c3-bc753280c957","Type":"ContainerStarted","Data":"6deedc086f2204f3366d096787c77b53f5bcd24d98a37d574525e5e05b7e0812"} Oct 30 01:45:01.331853 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:01.331834 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80122c5-lc25m" Oct 30 01:45:01.331973 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:01.331865 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80122c5-lc25m" event={"ID":"c2810118-aba3-474f-aefb-ce3393aaef31","Type":"ContainerDied","Data":"d89dfba030e95c41f9e106177c39945d6506e1a55a9c12d4e0d08b2dfcce57c7"} Oct 30 01:45:01.331973 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:01.331903 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d89dfba030e95c41f9e106177c39945d6506e1a55a9c12d4e0d08b2dfcce57c7" Oct 30 01:45:01.333611 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:01.333583 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888745-68pqj" event={"ID":"c76a323c-f168-42f8-933b-b28c333dcac4","Type":"ContainerStarted","Data":"b3267d240c455f1f441cccadd0299ed5ca4447d6e42b3b98e195be20810ddde0"} Oct 30 01:45:01.333756 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:01.333642 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888745-68pqj" event={"ID":"c76a323c-f168-42f8-933b-b28c333dcac4","Type":"ContainerStarted","Data":"52950fd440ee03bf19e276c4019535bce3ef40c2c6e29f90bafb28800f569197"} Oct 30 01:45:01.335428 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:01.335400 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888745-bffjq" event={"ID":"5db1407a-ef40-4f6a-8d81-5e3a1e502c33","Type":"ContainerStarted","Data":"f46369ca0ea055de109d1f660c8e2d75e600923cb28ce7761fcffe4a800d8cc9"} Oct 30 01:45:01.335594 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:01.335453 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888745-bffjq" event={"ID":"5db1407a-ef40-4f6a-8d81-5e3a1e502c33","Type":"ContainerStarted","Data":"b1a80b6d839f5548254df01fd96536f06176ad8e05e08cb5cf5e94cebfa144fc"} Oct 30 01:45:01.337177 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:01.337147 6572 generic.go:356] "Generic (PLEG): container finished" podID="7057658c-88a0-4bce-bb3f-765903a4b7fa" containerID="8bb2193200f14fddc22587bcc06c0c4a48ee78dd4ea90a1b682069c6700f4a68" exitCode=0 Oct 30 01:45:01.337299 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:01.337268 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9315ff4-f46th" event={"ID":"7057658c-88a0-4bce-bb3f-765903a4b7fa","Type":"ContainerDied","Data":"8bb2193200f14fddc22587bcc06c0c4a48ee78dd4ea90a1b682069c6700f4a68"} Oct 30 01:45:01.339001 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:01.338974 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888745-w7rlv" event={"ID":"1aa4b366-acc4-44d6-9208-3884e0349bb4","Type":"ContainerStarted","Data":"7539c50ed172816f8f16f3c1bc62f3f1e0b30781b1885f50b34621dfaafcfc9b"} Oct 30 01:45:01.339103 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:01.339010 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888745-w7rlv" event={"ID":"1aa4b366-acc4-44d6-9208-3884e0349bb4","Type":"ContainerStarted","Data":"6fe173c848de0fa8726fb2fd86bf33045c02de808b544bcfc473294e51d94426"} Oct 30 01:45:01.382735 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:01.382675 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-58632b-29888745-gqzpm" podStartSLOduration=1.382657486 podStartE2EDuration="1.382657486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:01.380530561 +0000 UTC m=+1626.956955963" watchObservedRunningTime="2026-10-30 01:45:01.382657486 +0000 UTC m=+1626.959082889" Oct 30 01:45:01.421329 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:01.421272 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888745-w7rlv" podStartSLOduration=1.421256617 podStartE2EDuration="1.421256617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:01.419052038 +0000 UTC m=+1626.995477420" watchObservedRunningTime="2026-10-30 01:45:01.421256617 +0000 UTC m=+1626.997682020" Oct 30 01:45:01.457539 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:01.457475 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888745-68pqj" podStartSLOduration=1.457458909 podStartE2EDuration="1.457458909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:01.456427238 +0000 UTC m=+1627.032852619" watchObservedRunningTime="2026-10-30 01:45:01.457458909 +0000 UTC m=+1627.033884312" Oct 30 01:45:01.497093 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:01.496990 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888745-bffjq" podStartSLOduration=1.496975474 podStartE2EDuration="1.496975474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:01.493509972 +0000 UTC m=+1627.069935385" watchObservedRunningTime="2026-10-30 01:45:01.496975474 +0000 UTC m=+1627.073400907" Oct 30 01:45:02.347740 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:02.347697 6572 generic.go:356] "Generic (PLEG): container finished" podID="1aa4b366-acc4-44d6-9208-3884e0349bb4" containerID="7539c50ed172816f8f16f3c1bc62f3f1e0b30781b1885f50b34621dfaafcfc9b" exitCode=60 Oct 30 01:45:02.348231 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:02.347795 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888745-w7rlv" event={"ID":"1aa4b366-acc4-44d6-9208-3884e0349bb4","Type":"ContainerDied","Data":"7539c50ed172816f8f16f3c1bc62f3f1e0b30781b1885f50b34621dfaafcfc9b"} Oct 30 01:45:02.348231 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:02.348163 6572 scope.go:122] "RemoveContainer" containerID="7539c50ed172816f8f16f3c1bc62f3f1e0b30781b1885f50b34621dfaafcfc9b" Oct 30 01:45:02.350087 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:02.349626 6572 generic.go:356] "Generic (PLEG): container finished" podID="fae65009-54c2-4407-90c3-bc753280c957" containerID="e0132205c0a78c8e9f33d759f2a9ea474bbb39b20bc0dc5843c96f19f7432947" exitCode=60 Oct 30 01:45:02.350087 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:02.349657 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-58632b-29888745-gqzpm" event={"ID":"fae65009-54c2-4407-90c3-bc753280c957","Type":"ContainerDied","Data":"e0132205c0a78c8e9f33d759f2a9ea474bbb39b20bc0dc5843c96f19f7432947"} Oct 30 01:45:02.350087 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:02.349954 6572 scope.go:122] "RemoveContainer" containerID="e0132205c0a78c8e9f33d759f2a9ea474bbb39b20bc0dc5843c96f19f7432947" Oct 30 01:45:02.351630 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:02.351608 6572 generic.go:356] "Generic (PLEG): container finished" podID="c76a323c-f168-42f8-933b-b28c333dcac4" containerID="b3267d240c455f1f441cccadd0299ed5ca4447d6e42b3b98e195be20810ddde0" exitCode=60 Oct 30 01:45:02.351833 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:02.351675 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888745-68pqj" event={"ID":"c76a323c-f168-42f8-933b-b28c333dcac4","Type":"ContainerDied","Data":"b3267d240c455f1f441cccadd0299ed5ca4447d6e42b3b98e195be20810ddde0"} Oct 30 01:45:02.351906 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:02.351885 6572 scope.go:122] "RemoveContainer" containerID="b3267d240c455f1f441cccadd0299ed5ca4447d6e42b3b98e195be20810ddde0" Oct 30 01:45:02.353613 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:02.353593 6572 generic.go:356] "Generic (PLEG): container finished" podID="5db1407a-ef40-4f6a-8d81-5e3a1e502c33" containerID="f46369ca0ea055de109d1f660c8e2d75e600923cb28ce7761fcffe4a800d8cc9" exitCode=60 Oct 30 01:45:02.353692 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:02.353671 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888745-bffjq" event={"ID":"5db1407a-ef40-4f6a-8d81-5e3a1e502c33","Type":"ContainerDied","Data":"f46369ca0ea055de109d1f660c8e2d75e600923cb28ce7761fcffe4a800d8cc9"} Oct 30 01:45:02.353831 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:02.353816 6572 scope.go:122] "RemoveContainer" containerID="f46369ca0ea055de109d1f660c8e2d75e600923cb28ce7761fcffe4a800d8cc9" Oct 30 01:45:02.561165 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:02.561135 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9315ff4-f46th" Oct 30 01:45:02.565819 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:02.565791 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:02.704859 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:02.704817 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7057658c-88a0-4bce-bb3f-765903a4b7fa-kube-api-access-s5mk2\" (UniqueName: \"kubernetes.io/projected/7057658c-88a0-4bce-bb3f-765903a4b7fa-kube-api-access-s5mk2\") pod \"7057658c-88a0-4bce-bb3f-765903a4b7fa\" (UID: \"7057658c-88a0-4bce-bb3f-765903a4b7fa\") " Oct 30 01:45:02.704859 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:02.704865 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7057658c-88a0-4bce-bb3f-765903a4b7fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7057658c-88a0-4bce-bb3f-765903a4b7fa-maas-api-openshift-ca\") pod \"7057658c-88a0-4bce-bb3f-765903a4b7fa\" (UID: \"7057658c-88a0-4bce-bb3f-765903a4b7fa\") " Oct 30 01:45:02.705140 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:02.705053 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7057658c-88a0-4bce-bb3f-765903a4b7fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7057658c-88a0-4bce-bb3f-765903a4b7fa-maas-api-xks-ca\") pod \"7057658c-88a0-4bce-bb3f-765903a4b7fa\" (UID: \"7057658c-88a0-4bce-bb3f-765903a4b7fa\") " Oct 30 01:45:02.705261 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:02.705236 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7057658c-88a0-4bce-bb3f-765903a4b7fa-maas-api-openshift-ca" pod "7057658c-88a0-4bce-bb3f-765903a4b7fa" (UID: "7057658c-88a0-4bce-bb3f-765903a4b7fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:45:02.705529 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:02.705507 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7057658c-88a0-4bce-bb3f-765903a4b7fa-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:02.707865 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:02.707838 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7057658c-88a0-4bce-bb3f-765903a4b7fa-kube-api-access-s5mk2" pod "7057658c-88a0-4bce-bb3f-765903a4b7fa" (UID: "7057658c-88a0-4bce-bb3f-765903a4b7fa"). InnerVolumeSpecName "kube-api-access-s5mk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:45:02.708002 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:02.707834 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7057658c-88a0-4bce-bb3f-765903a4b7fa-maas-api-xks-ca" pod "7057658c-88a0-4bce-bb3f-765903a4b7fa" (UID: "7057658c-88a0-4bce-bb3f-765903a4b7fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:45:02.806519 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:02.806474 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7057658c-88a0-4bce-bb3f-765903a4b7fa-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:02.806519 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:02.806509 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5mk2\" (UniqueName: \"kubernetes.io/projected/7057658c-88a0-4bce-bb3f-765903a4b7fa-kube-api-access-s5mk2\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:03.364216 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:03.364172 6572 generic.go:356] "Generic (PLEG): container finished" podID="fae65009-54c2-4407-90c3-bc753280c957" containerID="30746ff47e4e3103295a8237e334d61b170ed02afad49dbbfaf8cc6a0ce79c6a" exitCode=60 Oct 30 01:45:03.364666 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:03.364254 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-58632b-29888745-gqzpm" event={"ID":"fae65009-54c2-4407-90c3-bc753280c957","Type":"ContainerDied","Data":"30746ff47e4e3103295a8237e334d61b170ed02afad49dbbfaf8cc6a0ce79c6a"} Oct 30 01:45:03.364666 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:03.364317 6572 scope.go:122] "RemoveContainer" containerID="e0132205c0a78c8e9f33d759f2a9ea474bbb39b20bc0dc5843c96f19f7432947" Oct 30 01:45:03.364666 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:03.364606 6572 scope.go:122] "RemoveContainer" containerID="30746ff47e4e3103295a8237e334d61b170ed02afad49dbbfaf8cc6a0ce79c6a" Oct 30 01:45:03.364987 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:45:03.364961 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-58632b-29888745-gqzpm_odh-ai-gateway-infra(fae65009-54c2-4407-90c3-bc753280c957)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-58632b-29888745-gqzpm" podUID="fae65009-54c2-4407-90c3-bc753280c957" Oct 30 01:45:03.366642 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:03.366616 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888745-68pqj" event={"ID":"c76a323c-f168-42f8-933b-b28c333dcac4","Type":"ContainerStarted","Data":"ddfea1e2e7b86c33fd304e13f1f045cba9c72e98f4a4ec79d36bc9383a3729f4"} Oct 30 01:45:03.368667 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:03.368634 6572 generic.go:356] "Generic (PLEG): container finished" podID="5db1407a-ef40-4f6a-8d81-5e3a1e502c33" containerID="fecfc2e745a2463acb5122304e25698098c0fb6e0d0ee4c4fef162a51c2666ab" exitCode=60 Oct 30 01:45:03.368808 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:03.368702 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888745-bffjq" event={"ID":"5db1407a-ef40-4f6a-8d81-5e3a1e502c33","Type":"ContainerDied","Data":"fecfc2e745a2463acb5122304e25698098c0fb6e0d0ee4c4fef162a51c2666ab"} Oct 30 01:45:03.369014 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:03.368992 6572 scope.go:122] "RemoveContainer" containerID="fecfc2e745a2463acb5122304e25698098c0fb6e0d0ee4c4fef162a51c2666ab" Oct 30 01:45:03.369300 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:45:03.369273 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-803472-29888745-bffjq_odh-ai-gateway-infra(5db1407a-ef40-4f6a-8d81-5e3a1e502c33)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888745-bffjq" podUID="5db1407a-ef40-4f6a-8d81-5e3a1e502c33" Oct 30 01:45:03.370304 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:03.370275 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9315ff4-f46th" Oct 30 01:45:03.370400 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:03.370314 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9315ff4-f46th" event={"ID":"7057658c-88a0-4bce-bb3f-765903a4b7fa","Type":"ContainerDied","Data":"015f435a9aa898c2d755f4ab1a6d6f2d41c914b4d2aa438914ea605fe52d3872"} Oct 30 01:45:03.370400 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:03.370347 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="015f435a9aa898c2d755f4ab1a6d6f2d41c914b4d2aa438914ea605fe52d3872" Oct 30 01:45:03.372421 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:03.372394 6572 generic.go:356] "Generic (PLEG): container finished" podID="1aa4b366-acc4-44d6-9208-3884e0349bb4" containerID="dc2ada3d8ac1882d699a05b3093feb5523f8d7a9fcaa6f4092238e8f4cbbf05d" exitCode=60 Oct 30 01:45:03.372512 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:03.372472 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888745-w7rlv" event={"ID":"1aa4b366-acc4-44d6-9208-3884e0349bb4","Type":"ContainerDied","Data":"dc2ada3d8ac1882d699a05b3093feb5523f8d7a9fcaa6f4092238e8f4cbbf05d"} Oct 30 01:45:03.372695 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:03.372679 6572 scope.go:122] "RemoveContainer" containerID="dc2ada3d8ac1882d699a05b3093feb5523f8d7a9fcaa6f4092238e8f4cbbf05d" Oct 30 01:45:03.372958 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:45:03.372939 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888745-w7rlv_odh-ai-gateway-infra(1aa4b366-acc4-44d6-9208-3884e0349bb4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888745-w7rlv" podUID="1aa4b366-acc4-44d6-9208-3884e0349bb4" Oct 30 01:45:03.393207 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:03.393179 6572 scope.go:122] "RemoveContainer" containerID="f46369ca0ea055de109d1f660c8e2d75e600923cb28ce7761fcffe4a800d8cc9" Oct 30 01:45:03.409919 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:03.409894 6572 scope.go:122] "RemoveContainer" containerID="7539c50ed172816f8f16f3c1bc62f3f1e0b30781b1885f50b34621dfaafcfc9b" Oct 30 01:45:04.339054 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:04.339021 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:04.382153 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:04.382125 6572 scope.go:122] "RemoveContainer" containerID="dc2ada3d8ac1882d699a05b3093feb5523f8d7a9fcaa6f4092238e8f4cbbf05d" Oct 30 01:45:04.382540 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:45:04.382436 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888745-w7rlv_odh-ai-gateway-infra(1aa4b366-acc4-44d6-9208-3884e0349bb4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888745-w7rlv" podUID="1aa4b366-acc4-44d6-9208-3884e0349bb4" Oct 30 01:45:04.383800 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:04.383778 6572 scope.go:122] "RemoveContainer" containerID="30746ff47e4e3103295a8237e334d61b170ed02afad49dbbfaf8cc6a0ce79c6a" Oct 30 01:45:04.384081 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:45:04.384054 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-58632b-29888745-gqzpm_odh-ai-gateway-infra(fae65009-54c2-4407-90c3-bc753280c957)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-58632b-29888745-gqzpm" podUID="fae65009-54c2-4407-90c3-bc753280c957" Oct 30 01:45:04.385622 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:04.385597 6572 generic.go:356] "Generic (PLEG): container finished" podID="c76a323c-f168-42f8-933b-b28c333dcac4" containerID="ddfea1e2e7b86c33fd304e13f1f045cba9c72e98f4a4ec79d36bc9383a3729f4" exitCode=60 Oct 30 01:45:04.385717 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:04.385666 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888745-68pqj" event={"ID":"c76a323c-f168-42f8-933b-b28c333dcac4","Type":"ContainerDied","Data":"ddfea1e2e7b86c33fd304e13f1f045cba9c72e98f4a4ec79d36bc9383a3729f4"} Oct 30 01:45:04.385717 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:04.385713 6572 scope.go:122] "RemoveContainer" containerID="b3267d240c455f1f441cccadd0299ed5ca4447d6e42b3b98e195be20810ddde0" Oct 30 01:45:04.385967 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:04.385919 6572 scope.go:122] "RemoveContainer" containerID="ddfea1e2e7b86c33fd304e13f1f045cba9c72e98f4a4ec79d36bc9383a3729f4" Oct 30 01:45:04.386225 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:45:04.386204 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-eda881-29888745-68pqj_odh-ai-gateway-infra(c76a323c-f168-42f8-933b-b28c333dcac4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888745-68pqj" podUID="c76a323c-f168-42f8-933b-b28c333dcac4" Oct 30 01:45:04.387681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:04.387665 6572 scope.go:122] "RemoveContainer" containerID="fecfc2e745a2463acb5122304e25698098c0fb6e0d0ee4c4fef162a51c2666ab" Oct 30 01:45:04.387905 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:45:04.387889 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-803472-29888745-bffjq_odh-ai-gateway-infra(5db1407a-ef40-4f6a-8d81-5e3a1e502c33)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888745-bffjq" podUID="5db1407a-ef40-4f6a-8d81-5e3a1e502c33" Oct 30 01:45:05.127271 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:05.127231 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:05.398821 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:05.398708 6572 scope.go:122] "RemoveContainer" containerID="ddfea1e2e7b86c33fd304e13f1f045cba9c72e98f4a4ec79d36bc9383a3729f4" Oct 30 01:45:05.399284 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:45:05.399114 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-eda881-29888745-68pqj_odh-ai-gateway-infra(c76a323c-f168-42f8-933b-b28c333dcac4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888745-68pqj" podUID="c76a323c-f168-42f8-933b-b28c333dcac4" Oct 30 01:45:06.077144 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:06.077112 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:06.554900 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:06.554866 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5fa3e48-djt6k"] Oct 30 01:45:06.555439 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:06.555426 6572 cpu_manager.go:414] "containerMap: removing container" podUID="7057658c-88a0-4bce-bb3f-765903a4b7fa" containerName="revoke-keys" Oct 30 01:45:06.555499 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:06.555440 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7057658c-88a0-4bce-bb3f-765903a4b7fa" containerName="revoke-keys" Oct 30 01:45:06.555499 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:06.555449 6572 cpu_manager.go:414] "containerMap: removing container" podUID="c2810118-aba3-474f-aefb-ce3393aaef31" containerName="revoke-keys" Oct 30 01:45:06.555499 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:06.555455 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2810118-aba3-474f-aefb-ce3393aaef31" containerName="revoke-keys" Oct 30 01:45:06.555598 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:06.555559 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="7057658c-88a0-4bce-bb3f-765903a4b7fa" containerName="revoke-keys" Oct 30 01:45:06.555598 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:06.555567 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2810118-aba3-474f-aefb-ce3393aaef31" containerName="revoke-keys" Oct 30 01:45:06.560330 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:06.560313 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fa3e48-djt6k" Oct 30 01:45:06.579254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:06.579220 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5fa3e48-djt6k"] Oct 30 01:45:06.585820 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:06.585796 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:45:06.651276 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:06.651235 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b64f76b0-d2d3-48ea-a283-b901c5ac4b67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5fa3e48-djt6k\" (UID: \"b64f76b0-d2d3-48ea-a283-b901c5ac4b67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fa3e48-djt6k" Oct 30 01:45:06.651471 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:06.651409 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b64f76b0-d2d3-48ea-a283-b901c5ac4b67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5fa3e48-djt6k\" (UID: \"b64f76b0-d2d3-48ea-a283-b901c5ac4b67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fa3e48-djt6k" Oct 30 01:45:06.651546 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:06.651471 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf9qn\" (UniqueName: \"kubernetes.io/projected/b64f76b0-d2d3-48ea-a283-b901c5ac4b67-kube-api-access-rf9qn\") pod \"maas-api-revoke-sub-e5fa3e48-djt6k\" (UID: \"b64f76b0-d2d3-48ea-a283-b901c5ac4b67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fa3e48-djt6k" Oct 30 01:45:06.752704 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:06.752658 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b64f76b0-d2d3-48ea-a283-b901c5ac4b67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5fa3e48-djt6k\" (UID: \"b64f76b0-d2d3-48ea-a283-b901c5ac4b67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fa3e48-djt6k" Oct 30 01:45:06.752912 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:06.752872 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b64f76b0-d2d3-48ea-a283-b901c5ac4b67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5fa3e48-djt6k\" (UID: \"b64f76b0-d2d3-48ea-a283-b901c5ac4b67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fa3e48-djt6k" Oct 30 01:45:06.752979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:06.752945 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf9qn\" (UniqueName: \"kubernetes.io/projected/b64f76b0-d2d3-48ea-a283-b901c5ac4b67-kube-api-access-rf9qn\") pod \"maas-api-revoke-sub-e5fa3e48-djt6k\" (UID: \"b64f76b0-d2d3-48ea-a283-b901c5ac4b67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fa3e48-djt6k" Oct 30 01:45:06.753525 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:06.753502 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b64f76b0-d2d3-48ea-a283-b901c5ac4b67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5fa3e48-djt6k\" (UID: \"b64f76b0-d2d3-48ea-a283-b901c5ac4b67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fa3e48-djt6k" Oct 30 01:45:06.755262 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:06.755239 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b64f76b0-d2d3-48ea-a283-b901c5ac4b67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5fa3e48-djt6k\" (UID: \"b64f76b0-d2d3-48ea-a283-b901c5ac4b67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fa3e48-djt6k" Oct 30 01:45:06.772514 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:06.772480 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf9qn\" (UniqueName: \"kubernetes.io/projected/b64f76b0-d2d3-48ea-a283-b901c5ac4b67-kube-api-access-rf9qn\") pod \"maas-api-revoke-sub-e5fa3e48-djt6k\" (UID: \"b64f76b0-d2d3-48ea-a283-b901c5ac4b67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fa3e48-djt6k" Oct 30 01:45:06.874174 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:06.874088 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fa3e48-djt6k" Oct 30 01:45:07.177610 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:07.177580 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5fa3e48-djt6k"] Oct 30 01:45:07.178420 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:45:07.178390 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb64f76b0_d2d3_48ea_a283_b901c5ac4b67.slice/crio-4a1cbf47ae6ccd29397c9181cba7f2a2e37260f92c00c1cef2648f98ba8c5429 WatchSource:0}: Error finding container 4a1cbf47ae6ccd29397c9181cba7f2a2e37260f92c00c1cef2648f98ba8c5429: Status 404 returned error can't find the container with id 4a1cbf47ae6ccd29397c9181cba7f2a2e37260f92c00c1cef2648f98ba8c5429 Oct 30 01:45:07.417632 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:07.417597 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fa3e48-djt6k" event={"ID":"b64f76b0-d2d3-48ea-a283-b901c5ac4b67","Type":"ContainerStarted","Data":"4a1cbf47ae6ccd29397c9181cba7f2a2e37260f92c00c1cef2648f98ba8c5429"} Oct 30 01:45:07.813927 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:07.813653 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:08.427220 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:08.427188 6572 generic.go:356] "Generic (PLEG): container finished" podID="b64f76b0-d2d3-48ea-a283-b901c5ac4b67" containerID="b84175e6d38b2c997cfb4448761b1ef455d3df28876af3d5d48d1908c6e075d8" exitCode=0 Oct 30 01:45:08.427419 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:08.427248 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fa3e48-djt6k" event={"ID":"b64f76b0-d2d3-48ea-a283-b901c5ac4b67","Type":"ContainerDied","Data":"b84175e6d38b2c997cfb4448761b1ef455d3df28876af3d5d48d1908c6e075d8"} Oct 30 01:45:09.570473 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:09.570436 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:09.618386 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:09.618358 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fa3e48-djt6k" Oct 30 01:45:09.686992 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:09.686904 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b64f76b0-d2d3-48ea-a283-b901c5ac4b67-kube-api-access-rf9qn\" (UniqueName: \"kubernetes.io/projected/b64f76b0-d2d3-48ea-a283-b901c5ac4b67-kube-api-access-rf9qn\") pod \"b64f76b0-d2d3-48ea-a283-b901c5ac4b67\" (UID: \"b64f76b0-d2d3-48ea-a283-b901c5ac4b67\") " Oct 30 01:45:09.686992 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:09.686976 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b64f76b0-d2d3-48ea-a283-b901c5ac4b67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b64f76b0-d2d3-48ea-a283-b901c5ac4b67-maas-api-xks-ca\") pod \"b64f76b0-d2d3-48ea-a283-b901c5ac4b67\" (UID: \"b64f76b0-d2d3-48ea-a283-b901c5ac4b67\") " Oct 30 01:45:09.687225 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:09.687049 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b64f76b0-d2d3-48ea-a283-b901c5ac4b67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b64f76b0-d2d3-48ea-a283-b901c5ac4b67-maas-api-openshift-ca\") pod \"b64f76b0-d2d3-48ea-a283-b901c5ac4b67\" (UID: \"b64f76b0-d2d3-48ea-a283-b901c5ac4b67\") " Oct 30 01:45:09.687497 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:09.687460 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b64f76b0-d2d3-48ea-a283-b901c5ac4b67-maas-api-openshift-ca" pod "b64f76b0-d2d3-48ea-a283-b901c5ac4b67" (UID: "b64f76b0-d2d3-48ea-a283-b901c5ac4b67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:45:09.687665 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:09.687641 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b64f76b0-d2d3-48ea-a283-b901c5ac4b67-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:09.689233 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:09.689211 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b64f76b0-d2d3-48ea-a283-b901c5ac4b67-maas-api-xks-ca" pod "b64f76b0-d2d3-48ea-a283-b901c5ac4b67" (UID: "b64f76b0-d2d3-48ea-a283-b901c5ac4b67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:45:09.689327 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:09.689235 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b64f76b0-d2d3-48ea-a283-b901c5ac4b67-kube-api-access-rf9qn" pod "b64f76b0-d2d3-48ea-a283-b901c5ac4b67" (UID: "b64f76b0-d2d3-48ea-a283-b901c5ac4b67"). InnerVolumeSpecName "kube-api-access-rf9qn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:45:09.788992 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:09.788956 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf9qn\" (UniqueName: \"kubernetes.io/projected/b64f76b0-d2d3-48ea-a283-b901c5ac4b67-kube-api-access-rf9qn\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:09.788992 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:09.788987 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b64f76b0-d2d3-48ea-a283-b901c5ac4b67-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:10.445461 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:10.445424 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fa3e48-djt6k" Oct 30 01:45:10.445654 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:10.445463 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fa3e48-djt6k" event={"ID":"b64f76b0-d2d3-48ea-a283-b901c5ac4b67","Type":"ContainerDied","Data":"4a1cbf47ae6ccd29397c9181cba7f2a2e37260f92c00c1cef2648f98ba8c5429"} Oct 30 01:45:10.445654 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:10.445536 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a1cbf47ae6ccd29397c9181cba7f2a2e37260f92c00c1cef2648f98ba8c5429" Oct 30 01:45:11.325762 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:11.325715 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:13.044191 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.044153 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w"] Oct 30 01:45:13.044838 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.044819 6572 cpu_manager.go:414] "containerMap: removing container" podUID="b64f76b0-d2d3-48ea-a283-b901c5ac4b67" containerName="revoke-keys" Oct 30 01:45:13.044838 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.044837 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b64f76b0-d2d3-48ea-a283-b901c5ac4b67" containerName="revoke-keys" Oct 30 01:45:13.044986 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.044936 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="b64f76b0-d2d3-48ea-a283-b901c5ac4b67" containerName="revoke-keys" Oct 30 01:45:13.049315 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.049289 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" Oct 30 01:45:13.117556 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.117515 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:13.130781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.130279 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a74259-b0af-4c30-b41a-75835331b838-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w\" (UID: \"99a74259-b0af-4c30-b41a-75835331b838\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" Oct 30 01:45:13.130781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.130474 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgqms\" (UniqueName: \"kubernetes.io/projected/99a74259-b0af-4c30-b41a-75835331b838-kube-api-access-lgqms\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w\" (UID: \"99a74259-b0af-4c30-b41a-75835331b838\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" Oct 30 01:45:13.130781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.130574 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99a74259-b0af-4c30-b41a-75835331b838-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w\" (UID: \"99a74259-b0af-4c30-b41a-75835331b838\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" Oct 30 01:45:13.130781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.130624 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99a74259-b0af-4c30-b41a-75835331b838-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w\" (UID: \"99a74259-b0af-4c30-b41a-75835331b838\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" Oct 30 01:45:13.130781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.130655 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99a74259-b0af-4c30-b41a-75835331b838-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w\" (UID: \"99a74259-b0af-4c30-b41a-75835331b838\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" Oct 30 01:45:13.130781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.130698 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99a74259-b0af-4c30-b41a-75835331b838-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w\" (UID: \"99a74259-b0af-4c30-b41a-75835331b838\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" Oct 30 01:45:13.130781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.130729 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99a74259-b0af-4c30-b41a-75835331b838-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w\" (UID: \"99a74259-b0af-4c30-b41a-75835331b838\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" Oct 30 01:45:13.137265 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.130794 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a74259-b0af-4c30-b41a-75835331b838-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w\" (UID: \"99a74259-b0af-4c30-b41a-75835331b838\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" Oct 30 01:45:13.137265 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.130825 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99a74259-b0af-4c30-b41a-75835331b838-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w\" (UID: \"99a74259-b0af-4c30-b41a-75835331b838\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" Oct 30 01:45:13.137265 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.134496 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w"] Oct 30 01:45:13.232385 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.232342 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgqms\" (UniqueName: \"kubernetes.io/projected/99a74259-b0af-4c30-b41a-75835331b838-kube-api-access-lgqms\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w\" (UID: \"99a74259-b0af-4c30-b41a-75835331b838\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" Oct 30 01:45:13.232568 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.232437 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99a74259-b0af-4c30-b41a-75835331b838-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w\" (UID: \"99a74259-b0af-4c30-b41a-75835331b838\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" Oct 30 01:45:13.232568 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.232480 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99a74259-b0af-4c30-b41a-75835331b838-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w\" (UID: \"99a74259-b0af-4c30-b41a-75835331b838\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" Oct 30 01:45:13.232568 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.232508 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99a74259-b0af-4c30-b41a-75835331b838-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w\" (UID: \"99a74259-b0af-4c30-b41a-75835331b838\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" Oct 30 01:45:13.232568 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.232547 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99a74259-b0af-4c30-b41a-75835331b838-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w\" (UID: \"99a74259-b0af-4c30-b41a-75835331b838\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" Oct 30 01:45:13.232834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.232575 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99a74259-b0af-4c30-b41a-75835331b838-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w\" (UID: \"99a74259-b0af-4c30-b41a-75835331b838\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" Oct 30 01:45:13.232834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.232612 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a74259-b0af-4c30-b41a-75835331b838-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w\" (UID: \"99a74259-b0af-4c30-b41a-75835331b838\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" Oct 30 01:45:13.232834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.232636 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99a74259-b0af-4c30-b41a-75835331b838-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w\" (UID: \"99a74259-b0af-4c30-b41a-75835331b838\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" Oct 30 01:45:13.232834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.232692 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a74259-b0af-4c30-b41a-75835331b838-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w\" (UID: \"99a74259-b0af-4c30-b41a-75835331b838\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" Oct 30 01:45:13.233037 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.233005 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99a74259-b0af-4c30-b41a-75835331b838-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w\" (UID: \"99a74259-b0af-4c30-b41a-75835331b838\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" Oct 30 01:45:13.233378 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.233332 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99a74259-b0af-4c30-b41a-75835331b838-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w\" (UID: \"99a74259-b0af-4c30-b41a-75835331b838\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" Oct 30 01:45:13.233378 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.233347 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a74259-b0af-4c30-b41a-75835331b838-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w\" (UID: \"99a74259-b0af-4c30-b41a-75835331b838\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" Oct 30 01:45:13.233565 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.233421 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99a74259-b0af-4c30-b41a-75835331b838-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w\" (UID: \"99a74259-b0af-4c30-b41a-75835331b838\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" Oct 30 01:45:13.233907 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.233883 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a74259-b0af-4c30-b41a-75835331b838-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w\" (UID: \"99a74259-b0af-4c30-b41a-75835331b838\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" Oct 30 01:45:13.235792 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.235735 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99a74259-b0af-4c30-b41a-75835331b838-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w\" (UID: \"99a74259-b0af-4c30-b41a-75835331b838\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" Oct 30 01:45:13.235988 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.235965 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99a74259-b0af-4c30-b41a-75835331b838-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w\" (UID: \"99a74259-b0af-4c30-b41a-75835331b838\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" Oct 30 01:45:13.256356 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.256322 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99a74259-b0af-4c30-b41a-75835331b838-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w\" (UID: \"99a74259-b0af-4c30-b41a-75835331b838\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" Oct 30 01:45:13.256497 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.256452 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgqms\" (UniqueName: \"kubernetes.io/projected/99a74259-b0af-4c30-b41a-75835331b838-kube-api-access-lgqms\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w\" (UID: \"99a74259-b0af-4c30-b41a-75835331b838\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" Oct 30 01:45:13.380549 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.380468 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" Oct 30 01:45:13.705645 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.705601 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w"] Oct 30 01:45:13.706111 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:45:13.706090 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a74259_b0af_4c30_b41a_75835331b838.slice/crio-323df7d26b4372961b78499a49ae8c47e9364cbedc44028d7b6d9aa9e82b43f4 WatchSource:0}: Error finding container 323df7d26b4372961b78499a49ae8c47e9364cbedc44028d7b6d9aa9e82b43f4: Status 404 returned error can't find the container with id 323df7d26b4372961b78499a49ae8c47e9364cbedc44028d7b6d9aa9e82b43f4 Oct 30 01:45:13.709117 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.709073 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:45:13.709248 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.709168 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:45:13.709248 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:13.709216 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:45:14.496841 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:14.496799 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" event={"ID":"99a74259-b0af-4c30-b41a-75835331b838","Type":"ContainerStarted","Data":"9b1720bdfd7678e36efbccbf9cea31fb49cd331ea9d68d469e23965e80dcd39e"} Oct 30 01:45:14.497228 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:14.496872 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" event={"ID":"99a74259-b0af-4c30-b41a-75835331b838","Type":"ContainerStarted","Data":"323df7d26b4372961b78499a49ae8c47e9364cbedc44028d7b6d9aa9e82b43f4"} Oct 30 01:45:14.548824 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:14.541862 6572 scope.go:122] "RemoveContainer" containerID="fecfc2e745a2463acb5122304e25698098c0fb6e0d0ee4c4fef162a51c2666ab" Oct 30 01:45:14.584819 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:14.583512 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" podStartSLOduration=1.58349535 podStartE2EDuration="1.58349535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:14.580032058 +0000 UTC m=+1640.156457461" watchObservedRunningTime="2026-10-30 01:45:14.58349535 +0000 UTC m=+1640.159920751" Oct 30 01:45:14.695544 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:45:14.695500 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:45:15.386770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:15.385959 6572 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 01:45:15.386770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:15.386017 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" podUID="99a74259-b0af-4c30-b41a-75835331b838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 01:45:15.386770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:15.386561 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" Oct 30 01:45:15.594774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:15.594089 6572 scope.go:122] "RemoveContainer" containerID="dc2ada3d8ac1882d699a05b3093feb5523f8d7a9fcaa6f4092238e8f4cbbf05d" Oct 30 01:45:15.693499 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:15.692399 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:15.823667 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:15.823631 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888745-bffjq" event={"ID":"5db1407a-ef40-4f6a-8d81-5e3a1e502c33","Type":"ContainerStarted","Data":"8a736438320335af804929480d323162fe19f0a1d4d12283b5f7727049eca17c"} Oct 30 01:45:16.381356 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:16.381318 6572 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 01:45:16.381535 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:16.381371 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" podUID="99a74259-b0af-4c30-b41a-75835331b838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 01:45:16.838482 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:16.838449 6572 generic.go:356] "Generic (PLEG): container finished" podID="5db1407a-ef40-4f6a-8d81-5e3a1e502c33" containerID="8a736438320335af804929480d323162fe19f0a1d4d12283b5f7727049eca17c" exitCode=60 Oct 30 01:45:16.838978 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:16.838524 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888745-bffjq" event={"ID":"5db1407a-ef40-4f6a-8d81-5e3a1e502c33","Type":"ContainerDied","Data":"8a736438320335af804929480d323162fe19f0a1d4d12283b5f7727049eca17c"} Oct 30 01:45:16.838978 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:16.838591 6572 scope.go:122] "RemoveContainer" containerID="fecfc2e745a2463acb5122304e25698098c0fb6e0d0ee4c4fef162a51c2666ab" Oct 30 01:45:16.839113 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:16.839040 6572 scope.go:122] "RemoveContainer" containerID="8a736438320335af804929480d323162fe19f0a1d4d12283b5f7727049eca17c" Oct 30 01:45:16.839429 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:45:16.839404 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-803472-29888745-bffjq_odh-ai-gateway-infra(5db1407a-ef40-4f6a-8d81-5e3a1e502c33)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888745-bffjq" podUID="5db1407a-ef40-4f6a-8d81-5e3a1e502c33" Oct 30 01:45:16.841366 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:16.841343 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888745-w7rlv" event={"ID":"1aa4b366-acc4-44d6-9208-3884e0349bb4","Type":"ContainerStarted","Data":"504b53793919a5ef57a436dadcf46cdd2dd5d1de0dbce82d5d09fbb43727b108"} Oct 30 01:45:16.876500 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:16.876468 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888745-bffjq"] Oct 30 01:45:17.222771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:17.206942 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-513ef70e-zwvrc"] Oct 30 01:45:17.241150 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:17.241118 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-513ef70e-zwvrc" Oct 30 01:45:17.250565 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:17.250529 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-513ef70e-zwvrc"] Oct 30 01:45:17.254836 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:17.254807 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0169ab93-f50e-4660-b20c-72ccde5254c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-513ef70e-zwvrc\" (UID: \"0169ab93-f50e-4660-b20c-72ccde5254c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-513ef70e-zwvrc" Oct 30 01:45:17.255093 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:17.255077 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0169ab93-f50e-4660-b20c-72ccde5254c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-513ef70e-zwvrc\" (UID: \"0169ab93-f50e-4660-b20c-72ccde5254c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-513ef70e-zwvrc" Oct 30 01:45:17.255216 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:17.255205 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-452p6\" (UniqueName: \"kubernetes.io/projected/0169ab93-f50e-4660-b20c-72ccde5254c9-kube-api-access-452p6\") pod \"maas-api-revoke-sub-513ef70e-zwvrc\" (UID: \"0169ab93-f50e-4660-b20c-72ccde5254c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-513ef70e-zwvrc" Oct 30 01:45:17.258431 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:17.258358 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:45:17.356180 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:17.356139 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-452p6\" (UniqueName: \"kubernetes.io/projected/0169ab93-f50e-4660-b20c-72ccde5254c9-kube-api-access-452p6\") pod \"maas-api-revoke-sub-513ef70e-zwvrc\" (UID: \"0169ab93-f50e-4660-b20c-72ccde5254c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-513ef70e-zwvrc" Oct 30 01:45:17.356369 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:17.356257 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0169ab93-f50e-4660-b20c-72ccde5254c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-513ef70e-zwvrc\" (UID: \"0169ab93-f50e-4660-b20c-72ccde5254c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-513ef70e-zwvrc" Oct 30 01:45:17.356420 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:17.356389 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0169ab93-f50e-4660-b20c-72ccde5254c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-513ef70e-zwvrc\" (UID: \"0169ab93-f50e-4660-b20c-72ccde5254c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-513ef70e-zwvrc" Oct 30 01:45:17.357133 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:17.357106 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0169ab93-f50e-4660-b20c-72ccde5254c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-513ef70e-zwvrc\" (UID: \"0169ab93-f50e-4660-b20c-72ccde5254c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-513ef70e-zwvrc" Oct 30 01:45:17.358725 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:17.358699 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0169ab93-f50e-4660-b20c-72ccde5254c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-513ef70e-zwvrc\" (UID: \"0169ab93-f50e-4660-b20c-72ccde5254c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-513ef70e-zwvrc" Oct 30 01:45:17.381010 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:17.380982 6572 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 01:45:17.381121 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:17.381037 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" podUID="99a74259-b0af-4c30-b41a-75835331b838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 01:45:17.381121 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:17.381109 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-452p6\" (UniqueName: \"kubernetes.io/projected/0169ab93-f50e-4660-b20c-72ccde5254c9-kube-api-access-452p6\") pod \"maas-api-revoke-sub-513ef70e-zwvrc\" (UID: \"0169ab93-f50e-4660-b20c-72ccde5254c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-513ef70e-zwvrc" Oct 30 01:45:17.535830 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:17.535581 6572 scope.go:122] "RemoveContainer" containerID="30746ff47e4e3103295a8237e334d61b170ed02afad49dbbfaf8cc6a0ce79c6a" Oct 30 01:45:17.536024 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:17.535978 6572 scope.go:122] "RemoveContainer" containerID="ddfea1e2e7b86c33fd304e13f1f045cba9c72e98f4a4ec79d36bc9383a3729f4" Oct 30 01:45:17.565516 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:17.565485 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-513ef70e-zwvrc" Oct 30 01:45:17.855201 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:17.855156 6572 generic.go:356] "Generic (PLEG): container finished" podID="1aa4b366-acc4-44d6-9208-3884e0349bb4" containerID="504b53793919a5ef57a436dadcf46cdd2dd5d1de0dbce82d5d09fbb43727b108" exitCode=60 Oct 30 01:45:17.855800 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:17.855245 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888745-w7rlv" event={"ID":"1aa4b366-acc4-44d6-9208-3884e0349bb4","Type":"ContainerDied","Data":"504b53793919a5ef57a436dadcf46cdd2dd5d1de0dbce82d5d09fbb43727b108"} Oct 30 01:45:17.855800 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:17.855344 6572 scope.go:122] "RemoveContainer" containerID="dc2ada3d8ac1882d699a05b3093feb5523f8d7a9fcaa6f4092238e8f4cbbf05d" Oct 30 01:45:17.855980 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:17.855804 6572 scope.go:122] "RemoveContainer" containerID="504b53793919a5ef57a436dadcf46cdd2dd5d1de0dbce82d5d09fbb43727b108" Oct 30 01:45:17.856155 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:45:17.856133 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888745-w7rlv_odh-ai-gateway-infra(1aa4b366-acc4-44d6-9208-3884e0349bb4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888745-w7rlv" podUID="1aa4b366-acc4-44d6-9208-3884e0349bb4" Oct 30 01:45:17.954896 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:17.947184 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-513ef70e-zwvrc"] Oct 30 01:45:17.954896 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:17.954312 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888745-w7rlv"] Oct 30 01:45:17.956137 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:45:17.956101 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0169ab93_f50e_4660_b20c_72ccde5254c9.slice/crio-53a18b16fd0548a3a51daf973b86b149d9d1625bacaef542c247178e52c74771 WatchSource:0}: Error finding container 53a18b16fd0548a3a51daf973b86b149d9d1625bacaef542c247178e52c74771: Status 404 returned error can't find the container with id 53a18b16fd0548a3a51daf973b86b149d9d1625bacaef542c247178e52c74771 Oct 30 01:45:17.990881 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:17.990565 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:18.254436 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.254404 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:18.256939 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.256919 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888745-bffjq" Oct 30 01:45:18.372804 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.372671 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5db1407a-ef40-4f6a-8d81-5e3a1e502c33-kube-api-access-tf82j\" (UniqueName: \"kubernetes.io/projected/5db1407a-ef40-4f6a-8d81-5e3a1e502c33-kube-api-access-tf82j\") pod \"5db1407a-ef40-4f6a-8d81-5e3a1e502c33\" (UID: \"5db1407a-ef40-4f6a-8d81-5e3a1e502c33\") " Oct 30 01:45:18.372959 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.372866 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5db1407a-ef40-4f6a-8d81-5e3a1e502c33-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db1407a-ef40-4f6a-8d81-5e3a1e502c33-maas-api-service-ca\") pod \"5db1407a-ef40-4f6a-8d81-5e3a1e502c33\" (UID: \"5db1407a-ef40-4f6a-8d81-5e3a1e502c33\") " Oct 30 01:45:18.373273 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.373247 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5db1407a-ef40-4f6a-8d81-5e3a1e502c33-maas-api-service-ca" pod "5db1407a-ef40-4f6a-8d81-5e3a1e502c33" (UID: "5db1407a-ef40-4f6a-8d81-5e3a1e502c33"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:45:18.373380 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.373352 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db1407a-ef40-4f6a-8d81-5e3a1e502c33-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:18.381513 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.381485 6572 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 01:45:18.381673 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.381539 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" podUID="99a74259-b0af-4c30-b41a-75835331b838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 01:45:18.430890 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.421192 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db1407a-ef40-4f6a-8d81-5e3a1e502c33-kube-api-access-tf82j" pod "5db1407a-ef40-4f6a-8d81-5e3a1e502c33" (UID: "5db1407a-ef40-4f6a-8d81-5e3a1e502c33"). InnerVolumeSpecName "kube-api-access-tf82j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:45:18.475170 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.475126 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf82j\" (UniqueName: \"kubernetes.io/projected/5db1407a-ef40-4f6a-8d81-5e3a1e502c33-kube-api-access-tf82j\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:18.680445 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.680410 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5559857-wv4vq"] Oct 30 01:45:18.681013 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.680996 6572 cpu_manager.go:414] "containerMap: removing container" podUID="5db1407a-ef40-4f6a-8d81-5e3a1e502c33" containerName="cleanup" Oct 30 01:45:18.681013 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.681013 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5db1407a-ef40-4f6a-8d81-5e3a1e502c33" containerName="cleanup" Oct 30 01:45:18.681237 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.681043 6572 cpu_manager.go:414] "containerMap: removing container" podUID="5db1407a-ef40-4f6a-8d81-5e3a1e502c33" containerName="cleanup" Oct 30 01:45:18.681237 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.681049 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5db1407a-ef40-4f6a-8d81-5e3a1e502c33" containerName="cleanup" Oct 30 01:45:18.681237 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.681181 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="5db1407a-ef40-4f6a-8d81-5e3a1e502c33" containerName="cleanup" Oct 30 01:45:18.681237 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.681195 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="5db1407a-ef40-4f6a-8d81-5e3a1e502c33" containerName="cleanup" Oct 30 01:45:18.681237 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.681202 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="5db1407a-ef40-4f6a-8d81-5e3a1e502c33" containerName="cleanup" Oct 30 01:45:18.684838 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.684819 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5559857-wv4vq" Oct 30 01:45:18.703706 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.703671 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5559857-wv4vq"] Oct 30 01:45:18.779778 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.779707 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq696\" (UniqueName: \"kubernetes.io/projected/bd21026f-4167-49df-ace8-dbc4d9060fc0-kube-api-access-qq696\") pod \"maas-api-revoke-sub-a5559857-wv4vq\" (UID: \"bd21026f-4167-49df-ace8-dbc4d9060fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5559857-wv4vq" Oct 30 01:45:18.779931 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.779914 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd21026f-4167-49df-ace8-dbc4d9060fc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5559857-wv4vq\" (UID: \"bd21026f-4167-49df-ace8-dbc4d9060fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5559857-wv4vq" Oct 30 01:45:18.779996 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.779973 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd21026f-4167-49df-ace8-dbc4d9060fc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5559857-wv4vq\" (UID: \"bd21026f-4167-49df-ace8-dbc4d9060fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5559857-wv4vq" Oct 30 01:45:18.864257 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.864224 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-513ef70e-zwvrc" event={"ID":"0169ab93-f50e-4660-b20c-72ccde5254c9","Type":"ContainerStarted","Data":"334781df94aba36cadabc85b3b842584da92a13cbb19d1c43552617a031d8b69"} Oct 30 01:45:18.864722 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.864281 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-513ef70e-zwvrc" event={"ID":"0169ab93-f50e-4660-b20c-72ccde5254c9","Type":"ContainerStarted","Data":"53a18b16fd0548a3a51daf973b86b149d9d1625bacaef542c247178e52c74771"} Oct 30 01:45:18.866593 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.866559 6572 generic.go:356] "Generic (PLEG): container finished" podID="fae65009-54c2-4407-90c3-bc753280c957" containerID="6cbad56da63a4b7c9a8ab98ae78495f40d5c9d6ca7633ec6f74fa82e1942abb5" exitCode=60 Oct 30 01:45:18.866736 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.866586 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-58632b-29888745-gqzpm" event={"ID":"fae65009-54c2-4407-90c3-bc753280c957","Type":"ContainerDied","Data":"6cbad56da63a4b7c9a8ab98ae78495f40d5c9d6ca7633ec6f74fa82e1942abb5"} Oct 30 01:45:18.866736 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.866664 6572 scope.go:122] "RemoveContainer" containerID="30746ff47e4e3103295a8237e334d61b170ed02afad49dbbfaf8cc6a0ce79c6a" Oct 30 01:45:18.866994 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.866976 6572 scope.go:122] "RemoveContainer" containerID="6cbad56da63a4b7c9a8ab98ae78495f40d5c9d6ca7633ec6f74fa82e1942abb5" Oct 30 01:45:18.867323 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:45:18.867296 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-58632b-29888745-gqzpm_odh-ai-gateway-infra(fae65009-54c2-4407-90c3-bc753280c957)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-58632b-29888745-gqzpm" podUID="fae65009-54c2-4407-90c3-bc753280c957" Oct 30 01:45:18.869786 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.869428 6572 generic.go:356] "Generic (PLEG): container finished" podID="c76a323c-f168-42f8-933b-b28c333dcac4" containerID="fb9f7ba102030ae4adf69b66ff453203404ee69ef83058ccc344f031fc88dcf1" exitCode=60 Oct 30 01:45:18.869786 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.869479 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888745-68pqj" event={"ID":"c76a323c-f168-42f8-933b-b28c333dcac4","Type":"ContainerDied","Data":"fb9f7ba102030ae4adf69b66ff453203404ee69ef83058ccc344f031fc88dcf1"} Oct 30 01:45:18.869786 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.869710 6572 scope.go:122] "RemoveContainer" containerID="fb9f7ba102030ae4adf69b66ff453203404ee69ef83058ccc344f031fc88dcf1" Oct 30 01:45:18.870030 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:45:18.870008 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-eda881-29888745-68pqj_odh-ai-gateway-infra(c76a323c-f168-42f8-933b-b28c333dcac4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888745-68pqj" podUID="c76a323c-f168-42f8-933b-b28c333dcac4" Oct 30 01:45:18.871034 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.871018 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888745-bffjq" Oct 30 01:45:18.871126 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.871075 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888745-bffjq" event={"ID":"5db1407a-ef40-4f6a-8d81-5e3a1e502c33","Type":"ContainerDied","Data":"b1a80b6d839f5548254df01fd96536f06176ad8e05e08cb5cf5e94cebfa144fc"} Oct 30 01:45:18.880855 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.880819 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq696\" (UniqueName: \"kubernetes.io/projected/bd21026f-4167-49df-ace8-dbc4d9060fc0-kube-api-access-qq696\") pod \"maas-api-revoke-sub-a5559857-wv4vq\" (UID: \"bd21026f-4167-49df-ace8-dbc4d9060fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5559857-wv4vq" Oct 30 01:45:18.880998 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.880919 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd21026f-4167-49df-ace8-dbc4d9060fc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5559857-wv4vq\" (UID: \"bd21026f-4167-49df-ace8-dbc4d9060fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5559857-wv4vq" Oct 30 01:45:18.880998 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.880987 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd21026f-4167-49df-ace8-dbc4d9060fc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5559857-wv4vq\" (UID: \"bd21026f-4167-49df-ace8-dbc4d9060fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5559857-wv4vq" Oct 30 01:45:18.881643 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.881619 6572 scope.go:122] "RemoveContainer" containerID="ddfea1e2e7b86c33fd304e13f1f045cba9c72e98f4a4ec79d36bc9383a3729f4" Oct 30 01:45:18.881715 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.881673 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd21026f-4167-49df-ace8-dbc4d9060fc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5559857-wv4vq\" (UID: \"bd21026f-4167-49df-ace8-dbc4d9060fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5559857-wv4vq" Oct 30 01:45:18.883586 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.883561 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd21026f-4167-49df-ace8-dbc4d9060fc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5559857-wv4vq\" (UID: \"bd21026f-4167-49df-ace8-dbc4d9060fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5559857-wv4vq" Oct 30 01:45:18.896989 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.896969 6572 scope.go:122] "RemoveContainer" containerID="8a736438320335af804929480d323162fe19f0a1d4d12283b5f7727049eca17c" Oct 30 01:45:18.899630 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.899610 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq696\" (UniqueName: \"kubernetes.io/projected/bd21026f-4167-49df-ace8-dbc4d9060fc0-kube-api-access-qq696\") pod \"maas-api-revoke-sub-a5559857-wv4vq\" (UID: \"bd21026f-4167-49df-ace8-dbc4d9060fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5559857-wv4vq" Oct 30 01:45:18.929771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:18.928910 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-513ef70e-zwvrc" podStartSLOduration=1.928887065 podStartE2EDuration="1.928887065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:18.904455705 +0000 UTC m=+1644.480881110" watchObservedRunningTime="2026-10-30 01:45:18.928887065 +0000 UTC m=+1644.505312468" Oct 30 01:45:19.043050 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.042940 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5559857-wv4vq" Oct 30 01:45:19.078015 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.077805 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888745-bffjq"] Oct 30 01:45:19.097582 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.097543 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888745-bffjq"] Oct 30 01:45:19.114149 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.113061 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888745-w7rlv" Oct 30 01:45:19.167667 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.167610 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p"] Oct 30 01:45:19.176775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.168695 6572 cpu_manager.go:414] "containerMap: removing container" podUID="1aa4b366-acc4-44d6-9208-3884e0349bb4" containerName="cleanup" Oct 30 01:45:19.183697 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.179938 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aa4b366-acc4-44d6-9208-3884e0349bb4" containerName="cleanup" Oct 30 01:45:19.183697 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.180057 6572 cpu_manager.go:414] "containerMap: removing container" podUID="5db1407a-ef40-4f6a-8d81-5e3a1e502c33" containerName="cleanup" Oct 30 01:45:19.183697 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.180068 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5db1407a-ef40-4f6a-8d81-5e3a1e502c33" containerName="cleanup" Oct 30 01:45:19.183697 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.180084 6572 cpu_manager.go:414] "containerMap: removing container" podUID="1aa4b366-acc4-44d6-9208-3884e0349bb4" containerName="cleanup" Oct 30 01:45:19.183697 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.180090 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aa4b366-acc4-44d6-9208-3884e0349bb4" containerName="cleanup" Oct 30 01:45:19.183697 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.180272 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aa4b366-acc4-44d6-9208-3884e0349bb4" containerName="cleanup" Oct 30 01:45:19.183697 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.180291 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aa4b366-acc4-44d6-9208-3884e0349bb4" containerName="cleanup" Oct 30 01:45:19.183697 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.180308 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aa4b366-acc4-44d6-9208-3884e0349bb4" containerName="cleanup" Oct 30 01:45:19.184936 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.184914 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aa4b366-acc4-44d6-9208-3884e0349bb4-kube-api-access-scbpt\" (UniqueName: \"kubernetes.io/projected/1aa4b366-acc4-44d6-9208-3884e0349bb4-kube-api-access-scbpt\") pod \"1aa4b366-acc4-44d6-9208-3884e0349bb4\" (UID: \"1aa4b366-acc4-44d6-9208-3884e0349bb4\") " Oct 30 01:45:19.185111 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.185096 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aa4b366-acc4-44d6-9208-3884e0349bb4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa4b366-acc4-44d6-9208-3884e0349bb4-maas-api-service-ca\") pod \"1aa4b366-acc4-44d6-9208-3884e0349bb4\" (UID: \"1aa4b366-acc4-44d6-9208-3884e0349bb4\") " Oct 30 01:45:19.185411 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.185394 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt"] Oct 30 01:45:19.187531 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.187503 6572 cpu_manager.go:414] "containerMap: removing container" podUID="1aa4b366-acc4-44d6-9208-3884e0349bb4" containerName="cleanup" Oct 30 01:45:19.187653 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.187639 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aa4b366-acc4-44d6-9208-3884e0349bb4" containerName="cleanup" Oct 30 01:45:19.191582 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.191527 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aa4b366-acc4-44d6-9208-3884e0349bb4-maas-api-service-ca" pod "1aa4b366-acc4-44d6-9208-3884e0349bb4" (UID: "1aa4b366-acc4-44d6-9208-3884e0349bb4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:45:19.192573 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.192544 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" Oct 30 01:45:19.194914 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.194893 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p"] Oct 30 01:45:19.195133 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.195116 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" Oct 30 01:45:19.253826 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.253783 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt"] Oct 30 01:45:19.269832 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.269802 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-eda881-openshift-default-dockercfg-jwbsd\"" Oct 30 01:45:19.270596 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.270567 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aa4b366-acc4-44d6-9208-3884e0349bb4-kube-api-access-scbpt" pod "1aa4b366-acc4-44d6-9208-3884e0349bb4" (UID: "1aa4b366-acc4-44d6-9208-3884e0349bb4"). InnerVolumeSpecName "kube-api-access-scbpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:45:19.292020 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.291988 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-9227p\" (UID: \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" Oct 30 01:45:19.292137 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.292038 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt\" (UID: \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" Oct 30 01:45:19.292137 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.292065 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-9227p\" (UID: \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" Oct 30 01:45:19.292137 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.292122 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt\" (UID: \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" Oct 30 01:45:19.292254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.292149 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt\" (UID: \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" Oct 30 01:45:19.292254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.292182 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt\" (UID: \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" Oct 30 01:45:19.292254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.292215 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-9227p\" (UID: \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" Oct 30 01:45:19.292354 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.292265 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt\" (UID: \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" Oct 30 01:45:19.292354 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.292345 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-9227p\" (UID: \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" Oct 30 01:45:19.292414 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.292370 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt\" (UID: \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" Oct 30 01:45:19.292414 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.292396 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt\" (UID: \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" Oct 30 01:45:19.292478 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.292429 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-9227p\" (UID: \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" Oct 30 01:45:19.292478 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.292467 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmrqr\" (UniqueName: \"kubernetes.io/projected/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-kube-api-access-wmrqr\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-9227p\" (UID: \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" Oct 30 01:45:19.292543 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.292505 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzndw\" (UniqueName: \"kubernetes.io/projected/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-kube-api-access-pzndw\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt\" (UID: \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" Oct 30 01:45:19.292543 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.292528 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt\" (UID: \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" Oct 30 01:45:19.292602 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.292561 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-9227p\" (UID: \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" Oct 30 01:45:19.292602 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.292590 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-9227p\" (UID: \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" Oct 30 01:45:19.292673 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.292655 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-9227p\" (UID: \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" Oct 30 01:45:19.292932 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.292915 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scbpt\" (UniqueName: \"kubernetes.io/projected/1aa4b366-acc4-44d6-9208-3884e0349bb4-kube-api-access-scbpt\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:19.292987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.292935 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa4b366-acc4-44d6-9208-3884e0349bb4-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:19.381151 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.381118 6572 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 01:45:19.381330 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.381173 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" podUID="99a74259-b0af-4c30-b41a-75835331b838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 01:45:19.394343 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.394308 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzndw\" (UniqueName: \"kubernetes.io/projected/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-kube-api-access-pzndw\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt\" (UID: \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" Oct 30 01:45:19.394544 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.394529 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt\" (UID: \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" Oct 30 01:45:19.394690 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.394677 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-9227p\" (UID: \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" Oct 30 01:45:19.394817 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.394802 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-9227p\" (UID: \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" Oct 30 01:45:19.394960 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.394947 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-9227p\" (UID: \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" Oct 30 01:45:19.395074 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.395053 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-9227p\" (UID: \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" Oct 30 01:45:19.395223 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.395104 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-9227p\" (UID: \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" Oct 30 01:45:19.395356 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.395342 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt\" (UID: \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" Oct 30 01:45:19.395681 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.395669 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-9227p\" (UID: \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" Oct 30 01:45:19.395821 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.395809 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt\" (UID: \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" Oct 30 01:45:19.396107 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.396096 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt\" (UID: \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" Oct 30 01:45:19.397107 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.397086 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt\" (UID: \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" Oct 30 01:45:19.397229 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.397217 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-9227p\" (UID: \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" Oct 30 01:45:19.397904 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.395132 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-9227p\" (UID: \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" Oct 30 01:45:19.397904 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.397308 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt\" (UID: \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" Oct 30 01:45:19.397904 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.395727 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-9227p\" (UID: \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" Oct 30 01:45:19.397904 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.395955 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-9227p\" (UID: \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" Oct 30 01:45:19.397904 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.397427 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-9227p\" (UID: \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" Oct 30 01:45:19.397904 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.397456 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt\" (UID: \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" Oct 30 01:45:19.397904 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.397486 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt\" (UID: \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" Oct 30 01:45:19.397904 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.397496 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt\" (UID: \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" Oct 30 01:45:19.397904 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.397522 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-9227p\" (UID: \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" Oct 30 01:45:19.397904 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.397566 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmrqr\" (UniqueName: \"kubernetes.io/projected/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-kube-api-access-wmrqr\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-9227p\" (UID: \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" Oct 30 01:45:19.397904 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.397718 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-9227p\" (UID: \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" Oct 30 01:45:19.397904 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.397734 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt\" (UID: \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" Oct 30 01:45:19.397904 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.397735 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-9227p\" (UID: \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" Oct 30 01:45:19.397904 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.395627 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt\" (UID: \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" Oct 30 01:45:19.397904 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.396056 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt\" (UID: \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" Oct 30 01:45:19.398537 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.397970 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt\" (UID: \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" Oct 30 01:45:19.398537 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.398127 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt\" (UID: \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" Oct 30 01:45:19.399422 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.399403 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-9227p\" (UID: \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" Oct 30 01:45:19.399964 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.399942 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt\" (UID: \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" Oct 30 01:45:19.444514 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.444456 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5559857-wv4vq"] Oct 30 01:45:19.445450 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.445427 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzndw\" (UniqueName: \"kubernetes.io/projected/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-kube-api-access-pzndw\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt\" (UID: \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" Oct 30 01:45:19.446805 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.446779 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-9227p\" (UID: \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" Oct 30 01:45:19.447143 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.447116 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmrqr\" (UniqueName: \"kubernetes.io/projected/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-kube-api-access-wmrqr\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-9227p\" (UID: \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" Oct 30 01:45:19.447285 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.447260 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt\" (UID: \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" Oct 30 01:45:19.588107 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.588070 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" Oct 30 01:45:19.595649 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.595623 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" Oct 30 01:45:19.897028 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.896368 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5559857-wv4vq" event={"ID":"bd21026f-4167-49df-ace8-dbc4d9060fc0","Type":"ContainerStarted","Data":"b438b6300ca0023bed380fc8bf30bc742a5d159fa1b9f7ec144aaea4943456f9"} Oct 30 01:45:19.897028 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.896427 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5559857-wv4vq" event={"ID":"bd21026f-4167-49df-ace8-dbc4d9060fc0","Type":"ContainerStarted","Data":"c19463bc5b24781f7549627118c7ae6662239fe1a2224e021a21255b8468a623"} Oct 30 01:45:19.898963 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.898937 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888745-w7rlv" event={"ID":"1aa4b366-acc4-44d6-9208-3884e0349bb4","Type":"ContainerDied","Data":"6fe173c848de0fa8726fb2fd86bf33045c02de808b544bcfc473294e51d94426"} Oct 30 01:45:19.898963 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.898955 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888745-w7rlv" Oct 30 01:45:19.899131 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.898984 6572 scope.go:122] "RemoveContainer" containerID="504b53793919a5ef57a436dadcf46cdd2dd5d1de0dbce82d5d09fbb43727b108" Oct 30 01:45:19.905716 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.905688 6572 generic.go:356] "Generic (PLEG): container finished" podID="0169ab93-f50e-4660-b20c-72ccde5254c9" containerID="334781df94aba36cadabc85b3b842584da92a13cbb19d1c43552617a031d8b69" exitCode=0 Oct 30 01:45:19.930837 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.905784 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-513ef70e-zwvrc" event={"ID":"0169ab93-f50e-4660-b20c-72ccde5254c9","Type":"ContainerDied","Data":"334781df94aba36cadabc85b3b842584da92a13cbb19d1c43552617a031d8b69"} Oct 30 01:45:19.970988 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:19.970797 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5559857-wv4vq" podStartSLOduration=1.97078084 podStartE2EDuration="1.97078084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:19.969650607 +0000 UTC m=+1645.546076010" watchObservedRunningTime="2026-10-30 01:45:19.97078084 +0000 UTC m=+1645.547206239" Oct 30 01:45:20.018696 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.017966 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888745-68pqj"] Oct 30 01:45:20.051518 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:45:20.051464 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe01e89d_ffd9_40bc_8720_eee65f4a42eb.slice/crio-0102f470f963359de5cb964ca9a04c4f66eccaacf528ad8010119deb35b1ecfa WatchSource:0}: Error finding container 0102f470f963359de5cb964ca9a04c4f66eccaacf528ad8010119deb35b1ecfa: Status 404 returned error can't find the container with id 0102f470f963359de5cb964ca9a04c4f66eccaacf528ad8010119deb35b1ecfa Oct 30 01:45:20.056264 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.055445 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:45:20.056264 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.055531 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:45:20.056264 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.055575 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:45:20.056264 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.056174 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt"] Oct 30 01:45:20.065636 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.065605 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p"] Oct 30 01:45:20.069544 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.069514 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-58632b-29888745-gqzpm"] Oct 30 01:45:20.086996 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.086103 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888745-w7rlv"] Oct 30 01:45:20.094026 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:45:20.093985 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdadf1cc_610e_4e7b_83cf_5a173f5779eb.slice/crio-6961b05897ecb4a9256f116d2e5f7a4a37a2bd7bf24ac05c3648ffef1a09f893 WatchSource:0}: Error finding container 6961b05897ecb4a9256f116d2e5f7a4a37a2bd7bf24ac05c3648ffef1a09f893: Status 404 returned error can't find the container with id 6961b05897ecb4a9256f116d2e5f7a4a37a2bd7bf24ac05c3648ffef1a09f893 Oct 30 01:45:20.099227 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.098632 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:45:20.099227 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.098723 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:45:20.099227 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.098788 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:45:20.106942 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.106882 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888745-w7rlv"] Oct 30 01:45:20.111004 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.110970 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cb08419-d8vwm"] Oct 30 01:45:20.121274 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.121160 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb08419-d8vwm" Oct 30 01:45:20.164823 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.152477 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cb08419-d8vwm"] Oct 30 01:45:20.218279 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.218151 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:20.229731 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.229359 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cb08419-d8vwm\" (UID: \"6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb08419-d8vwm" Oct 30 01:45:20.229731 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.229433 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cb08419-d8vwm\" (UID: \"6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb08419-d8vwm" Oct 30 01:45:20.229731 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.229532 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47j7x\" (UniqueName: \"kubernetes.io/projected/6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df-kube-api-access-47j7x\") pod \"maas-api-revoke-sub-3cb08419-d8vwm\" (UID: \"6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb08419-d8vwm" Oct 30 01:45:20.343266 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.330524 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47j7x\" (UniqueName: \"kubernetes.io/projected/6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df-kube-api-access-47j7x\") pod \"maas-api-revoke-sub-3cb08419-d8vwm\" (UID: \"6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb08419-d8vwm" Oct 30 01:45:20.343266 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.330736 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cb08419-d8vwm\" (UID: \"6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb08419-d8vwm" Oct 30 01:45:20.343266 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.330802 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cb08419-d8vwm\" (UID: \"6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb08419-d8vwm" Oct 30 01:45:20.343266 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.331616 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cb08419-d8vwm\" (UID: \"6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb08419-d8vwm" Oct 30 01:45:20.371687 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.371482 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cb08419-d8vwm\" (UID: \"6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb08419-d8vwm" Oct 30 01:45:20.375012 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.374933 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47j7x\" (UniqueName: \"kubernetes.io/projected/6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df-kube-api-access-47j7x\") pod \"maas-api-revoke-sub-3cb08419-d8vwm\" (UID: \"6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb08419-d8vwm" Oct 30 01:45:20.386924 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.385276 6572 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 01:45:20.386924 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.385327 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" podUID="99a74259-b0af-4c30-b41a-75835331b838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 01:45:20.505033 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.504500 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888745-68pqj" Oct 30 01:45:20.560444 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.560126 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb08419-d8vwm" Oct 30 01:45:20.600530 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.600472 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aa4b366-acc4-44d6-9208-3884e0349bb4" path="/var/lib/kubelet/pods/1aa4b366-acc4-44d6-9208-3884e0349bb4/volumes" Oct 30 01:45:20.601415 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.601390 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5db1407a-ef40-4f6a-8d81-5e3a1e502c33" path="/var/lib/kubelet/pods/5db1407a-ef40-4f6a-8d81-5e3a1e502c33/volumes" Oct 30 01:45:20.656778 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.643383 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c76a323c-f168-42f8-933b-b28c333dcac4-kube-api-access-lclgl\" (UniqueName: \"kubernetes.io/projected/c76a323c-f168-42f8-933b-b28c333dcac4-kube-api-access-lclgl\") pod \"c76a323c-f168-42f8-933b-b28c333dcac4\" (UID: \"c76a323c-f168-42f8-933b-b28c333dcac4\") " Oct 30 01:45:20.656778 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.643477 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c76a323c-f168-42f8-933b-b28c333dcac4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c76a323c-f168-42f8-933b-b28c333dcac4-maas-api-service-ca\") pod \"c76a323c-f168-42f8-933b-b28c333dcac4\" (UID: \"c76a323c-f168-42f8-933b-b28c333dcac4\") " Oct 30 01:45:20.669093 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.668035 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c76a323c-f168-42f8-933b-b28c333dcac4-maas-api-service-ca" pod "c76a323c-f168-42f8-933b-b28c333dcac4" (UID: "c76a323c-f168-42f8-933b-b28c333dcac4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:45:20.691457 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.691398 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c76a323c-f168-42f8-933b-b28c333dcac4-kube-api-access-lclgl" pod "c76a323c-f168-42f8-933b-b28c333dcac4" (UID: "c76a323c-f168-42f8-933b-b28c333dcac4"). InnerVolumeSpecName "kube-api-access-lclgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:45:20.737070 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.736681 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-58632b-29888745-gqzpm" Oct 30 01:45:20.749774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.748917 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lclgl\" (UniqueName: \"kubernetes.io/projected/c76a323c-f168-42f8-933b-b28c333dcac4-kube-api-access-lclgl\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:20.749774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.748957 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c76a323c-f168-42f8-933b-b28c333dcac4-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:20.967010 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.966928 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fae65009-54c2-4407-90c3-bc753280c957-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fae65009-54c2-4407-90c3-bc753280c957-maas-api-service-ca\") pod \"fae65009-54c2-4407-90c3-bc753280c957\" (UID: \"fae65009-54c2-4407-90c3-bc753280c957\") " Oct 30 01:45:20.970580 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.970539 6572 generic.go:356] "Generic (PLEG): container finished" podID="bd21026f-4167-49df-ace8-dbc4d9060fc0" containerID="b438b6300ca0023bed380fc8bf30bc742a5d159fa1b9f7ec144aaea4943456f9" exitCode=0 Oct 30 01:45:20.970734 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.970685 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5559857-wv4vq" event={"ID":"bd21026f-4167-49df-ace8-dbc4d9060fc0","Type":"ContainerDied","Data":"b438b6300ca0023bed380fc8bf30bc742a5d159fa1b9f7ec144aaea4943456f9"} Oct 30 01:45:20.974548 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.973637 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fae65009-54c2-4407-90c3-bc753280c957-kube-api-access-r4f29\" (UniqueName: \"kubernetes.io/projected/fae65009-54c2-4407-90c3-bc753280c957-kube-api-access-r4f29\") pod \"fae65009-54c2-4407-90c3-bc753280c957\" (UID: \"fae65009-54c2-4407-90c3-bc753280c957\") " Oct 30 01:45:20.975155 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:20.975117 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fae65009-54c2-4407-90c3-bc753280c957-maas-api-service-ca" pod "fae65009-54c2-4407-90c3-bc753280c957" (UID: "fae65009-54c2-4407-90c3-bc753280c957"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:45:21.008631 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:21.008583 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae65009-54c2-4407-90c3-bc753280c957-kube-api-access-r4f29" pod "fae65009-54c2-4407-90c3-bc753280c957" (UID: "fae65009-54c2-4407-90c3-bc753280c957"). InnerVolumeSpecName "kube-api-access-r4f29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:45:21.038322 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:21.026509 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" event={"ID":"fe01e89d-ffd9-40bc-8720-eee65f4a42eb","Type":"ContainerStarted","Data":"12996ff0bc936e2d0a017b435104cbe551e061b1ddb1c7b7b24d3ac1c0bd2e35"} Oct 30 01:45:21.038322 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:21.026580 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" event={"ID":"fe01e89d-ffd9-40bc-8720-eee65f4a42eb","Type":"ContainerStarted","Data":"0102f470f963359de5cb964ca9a04c4f66eccaacf528ad8010119deb35b1ecfa"} Oct 30 01:45:21.099782 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:21.090689 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-58632b-29888745-gqzpm" Oct 30 01:45:21.099782 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:21.090879 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-58632b-29888745-gqzpm" event={"ID":"fae65009-54c2-4407-90c3-bc753280c957","Type":"ContainerDied","Data":"6deedc086f2204f3366d096787c77b53f5bcd24d98a37d574525e5e05b7e0812"} Oct 30 01:45:21.099782 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:21.090954 6572 scope.go:122] "RemoveContainer" containerID="6cbad56da63a4b7c9a8ab98ae78495f40d5c9d6ca7633ec6f74fa82e1942abb5" Oct 30 01:45:21.103742 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:21.103708 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888745-68pqj" event={"ID":"c76a323c-f168-42f8-933b-b28c333dcac4","Type":"ContainerDied","Data":"52950fd440ee03bf19e276c4019535bce3ef40c2c6e29f90bafb28800f569197"} Oct 30 01:45:21.104014 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:21.103991 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888745-68pqj" Oct 30 01:45:21.137689 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:21.134853 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" podStartSLOduration=2.134826424 podStartE2EDuration="2.134826424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:21.133398134 +0000 UTC m=+1646.709823540" watchObservedRunningTime="2026-10-30 01:45:21.134826424 +0000 UTC m=+1646.711251828" Oct 30 01:45:21.153834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:21.151536 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" event={"ID":"fdadf1cc-610e-4e7b-83cf-5a173f5779eb","Type":"ContainerStarted","Data":"96b0395c0ebede04db1c43677d9faa01e3353c3d8a987b373b2ceeb928c533b5"} Oct 30 01:45:21.153834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:21.151614 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" event={"ID":"fdadf1cc-610e-4e7b-83cf-5a173f5779eb","Type":"ContainerStarted","Data":"6961b05897ecb4a9256f116d2e5f7a4a37a2bd7bf24ac05c3648ffef1a09f893"} Oct 30 01:45:21.153834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:21.152259 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4f29\" (UniqueName: \"kubernetes.io/projected/fae65009-54c2-4407-90c3-bc753280c957-kube-api-access-r4f29\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:21.153834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:21.152312 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fae65009-54c2-4407-90c3-bc753280c957-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:21.170528 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:21.169900 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cb08419-d8vwm"] Oct 30 01:45:21.172972 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:21.172941 6572 scope.go:122] "RemoveContainer" containerID="fb9f7ba102030ae4adf69b66ff453203404ee69ef83058ccc344f031fc88dcf1" Oct 30 01:45:21.277768 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:21.277685 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" podStartSLOduration=2.277661443 podStartE2EDuration="2.277661443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:21.240126341 +0000 UTC m=+1646.816551745" watchObservedRunningTime="2026-10-30 01:45:21.277661443 +0000 UTC m=+1646.854086846" Oct 30 01:45:21.279243 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:21.279207 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-58632b-29888745-gqzpm"] Oct 30 01:45:21.361834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:21.334718 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-58632b-29888745-gqzpm"] Oct 30 01:45:21.392771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:21.392369 6572 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 01:45:21.392771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:21.392424 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" podUID="99a74259-b0af-4c30-b41a-75835331b838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 01:45:21.488267 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:21.488188 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888745-68pqj"] Oct 30 01:45:21.489716 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:21.489683 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888745-68pqj"] Oct 30 01:45:21.571773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:21.571086 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-513ef70e-zwvrc" Oct 30 01:45:21.588465 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:21.588417 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" Oct 30 01:45:21.590422 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:21.590390 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Oct 30 01:45:21.590569 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:21.590439 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" podUID="fdadf1cc-610e-4e7b-83cf-5a173f5779eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Oct 30 01:45:21.595806 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:21.595779 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" Oct 30 01:45:21.598181 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:21.598152 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 30 01:45:21.598316 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:21.598203 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" podUID="fe01e89d-ffd9-40bc-8720-eee65f4a42eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 30 01:45:21.704971 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:21.704919 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0169ab93-f50e-4660-b20c-72ccde5254c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0169ab93-f50e-4660-b20c-72ccde5254c9-maas-api-xks-ca\") pod \"0169ab93-f50e-4660-b20c-72ccde5254c9\" (UID: \"0169ab93-f50e-4660-b20c-72ccde5254c9\") " Oct 30 01:45:21.705216 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:21.705068 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0169ab93-f50e-4660-b20c-72ccde5254c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0169ab93-f50e-4660-b20c-72ccde5254c9-maas-api-openshift-ca\") pod \"0169ab93-f50e-4660-b20c-72ccde5254c9\" (UID: \"0169ab93-f50e-4660-b20c-72ccde5254c9\") " Oct 30 01:45:21.705430 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:21.705407 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0169ab93-f50e-4660-b20c-72ccde5254c9-kube-api-access-452p6\" (UniqueName: \"kubernetes.io/projected/0169ab93-f50e-4660-b20c-72ccde5254c9-kube-api-access-452p6\") pod \"0169ab93-f50e-4660-b20c-72ccde5254c9\" (UID: \"0169ab93-f50e-4660-b20c-72ccde5254c9\") " Oct 30 01:45:21.732791 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:21.728240 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0169ab93-f50e-4660-b20c-72ccde5254c9-maas-api-openshift-ca" pod "0169ab93-f50e-4660-b20c-72ccde5254c9" (UID: "0169ab93-f50e-4660-b20c-72ccde5254c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:45:21.794810 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:21.793603 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0169ab93-f50e-4660-b20c-72ccde5254c9-kube-api-access-452p6" pod "0169ab93-f50e-4660-b20c-72ccde5254c9" (UID: "0169ab93-f50e-4660-b20c-72ccde5254c9"). InnerVolumeSpecName "kube-api-access-452p6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:45:21.804934 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:21.795117 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-452p6\" (UniqueName: \"kubernetes.io/projected/0169ab93-f50e-4660-b20c-72ccde5254c9-kube-api-access-452p6\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:21.804934 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:21.795151 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0169ab93-f50e-4660-b20c-72ccde5254c9-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:22.023424 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:22.023320 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:22.143782 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:22.142952 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0169ab93-f50e-4660-b20c-72ccde5254c9-maas-api-xks-ca" pod "0169ab93-f50e-4660-b20c-72ccde5254c9" (UID: "0169ab93-f50e-4660-b20c-72ccde5254c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:45:22.150782 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:22.148126 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0169ab93-f50e-4660-b20c-72ccde5254c9-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:22.520997 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:22.520962 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:22.618851 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:22.618812 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Oct 30 01:45:22.619106 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:22.619072 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" podUID="fdadf1cc-610e-4e7b-83cf-5a173f5779eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Oct 30 01:45:22.641772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:22.636495 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 30 01:45:22.641772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:22.636576 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" podUID="fe01e89d-ffd9-40bc-8720-eee65f4a42eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 30 01:45:22.704958 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:22.704833 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-513ef70e-zwvrc" Oct 30 01:45:22.729790 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:22.726366 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c76a323c-f168-42f8-933b-b28c333dcac4" path="/var/lib/kubelet/pods/c76a323c-f168-42f8-933b-b28c333dcac4/volumes" Oct 30 01:45:22.729790 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:22.727059 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fae65009-54c2-4407-90c3-bc753280c957" path="/var/lib/kubelet/pods/fae65009-54c2-4407-90c3-bc753280c957/volumes" Oct 30 01:45:22.729790 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:22.727739 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-513ef70e-zwvrc" event={"ID":"0169ab93-f50e-4660-b20c-72ccde5254c9","Type":"ContainerDied","Data":"53a18b16fd0548a3a51daf973b86b149d9d1625bacaef542c247178e52c74771"} Oct 30 01:45:22.729790 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:22.727818 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53a18b16fd0548a3a51daf973b86b149d9d1625bacaef542c247178e52c74771" Oct 30 01:45:22.729790 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:22.727899 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" Oct 30 01:45:22.731316 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:22.730527 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" Oct 30 01:45:22.768674 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:22.765293 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" Oct 30 01:45:22.788117 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:22.787210 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb08419-d8vwm" event={"ID":"6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df","Type":"ContainerStarted","Data":"c0183ebdcb58f614dbc508fbd8e3f9a2f058f52f940564f18f6c021cd04fef4a"} Oct 30 01:45:22.788117 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:22.787278 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb08419-d8vwm" event={"ID":"6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df","Type":"ContainerStarted","Data":"fe7e8c276c90a58d90c8bf42dad7ee6f207825669408135b3360cdf4310610df"} Oct 30 01:45:23.008476 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:23.007507 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb08419-d8vwm" podStartSLOduration=3.007488353 podStartE2EDuration="3.007488353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:22.927343896 +0000 UTC m=+1648.503769299" watchObservedRunningTime="2026-10-30 01:45:23.007488353 +0000 UTC m=+1648.583913755" Oct 30 01:45:23.278517 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:23.278481 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:23.595872 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:23.595786 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Oct 30 01:45:23.595872 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:23.595852 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" podUID="fdadf1cc-610e-4e7b-83cf-5a173f5779eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Oct 30 01:45:23.596948 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:23.596742 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 30 01:45:23.596948 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:23.596799 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" podUID="fe01e89d-ffd9-40bc-8720-eee65f4a42eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 30 01:45:23.700846 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:23.699109 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5559857-wv4vq" Oct 30 01:45:23.801615 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:23.801577 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd21026f-4167-49df-ace8-dbc4d9060fc0-kube-api-access-qq696\" (UniqueName: \"kubernetes.io/projected/bd21026f-4167-49df-ace8-dbc4d9060fc0-kube-api-access-qq696\") pod \"bd21026f-4167-49df-ace8-dbc4d9060fc0\" (UID: \"bd21026f-4167-49df-ace8-dbc4d9060fc0\") " Oct 30 01:45:23.803024 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:23.802994 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd21026f-4167-49df-ace8-dbc4d9060fc0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd21026f-4167-49df-ace8-dbc4d9060fc0-maas-api-openshift-ca\") pod \"bd21026f-4167-49df-ace8-dbc4d9060fc0\" (UID: \"bd21026f-4167-49df-ace8-dbc4d9060fc0\") " Oct 30 01:45:23.803903 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:23.803875 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd21026f-4167-49df-ace8-dbc4d9060fc0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd21026f-4167-49df-ace8-dbc4d9060fc0-maas-api-xks-ca\") pod \"bd21026f-4167-49df-ace8-dbc4d9060fc0\" (UID: \"bd21026f-4167-49df-ace8-dbc4d9060fc0\") " Oct 30 01:45:23.804370 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:23.804306 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd21026f-4167-49df-ace8-dbc4d9060fc0-maas-api-openshift-ca" pod "bd21026f-4167-49df-ace8-dbc4d9060fc0" (UID: "bd21026f-4167-49df-ace8-dbc4d9060fc0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:45:23.804834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:23.804621 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd21026f-4167-49df-ace8-dbc4d9060fc0-kube-api-access-qq696" pod "bd21026f-4167-49df-ace8-dbc4d9060fc0" (UID: "bd21026f-4167-49df-ace8-dbc4d9060fc0"). InnerVolumeSpecName "kube-api-access-qq696". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:45:23.811785 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:23.805587 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq696\" (UniqueName: \"kubernetes.io/projected/bd21026f-4167-49df-ace8-dbc4d9060fc0-kube-api-access-qq696\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:23.865827 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:23.852807 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd21026f-4167-49df-ace8-dbc4d9060fc0-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:23.865827 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:23.852115 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd21026f-4167-49df-ace8-dbc4d9060fc0-maas-api-xks-ca" pod "bd21026f-4167-49df-ace8-dbc4d9060fc0" (UID: "bd21026f-4167-49df-ace8-dbc4d9060fc0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:45:24.005138 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:23.947807 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:24.005138 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:23.957677 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd21026f-4167-49df-ace8-dbc4d9060fc0-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:24.081930 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:24.044055 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5559857-wv4vq" event={"ID":"bd21026f-4167-49df-ace8-dbc4d9060fc0","Type":"ContainerDied","Data":"c19463bc5b24781f7549627118c7ae6662239fe1a2224e021a21255b8468a623"} Oct 30 01:45:24.081930 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:24.044126 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c19463bc5b24781f7549627118c7ae6662239fe1a2224e021a21255b8468a623" Oct 30 01:45:24.081930 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:24.044213 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5559857-wv4vq" Oct 30 01:45:24.187357 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:24.175339 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:24.187357 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:24.177663 6572 generic.go:356] "Generic (PLEG): container finished" podID="6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df" containerID="c0183ebdcb58f614dbc508fbd8e3f9a2f058f52f940564f18f6c021cd04fef4a" exitCode=0 Oct 30 01:45:24.187357 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:24.178823 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb08419-d8vwm" event={"ID":"6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df","Type":"ContainerDied","Data":"c0183ebdcb58f614dbc508fbd8e3f9a2f058f52f940564f18f6c021cd04fef4a"} Oct 30 01:45:24.599643 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:24.599604 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 30 01:45:24.600211 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:24.599701 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" podUID="fe01e89d-ffd9-40bc-8720-eee65f4a42eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 30 01:45:24.600765 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:24.600715 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Oct 30 01:45:24.600871 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:24.600850 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" podUID="fdadf1cc-610e-4e7b-83cf-5a173f5779eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Oct 30 01:45:24.740816 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:24.740765 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:25.598431 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:25.598339 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Oct 30 01:45:25.598690 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:25.598658 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" podUID="fdadf1cc-610e-4e7b-83cf-5a173f5779eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Oct 30 01:45:25.600142 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:25.600117 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 30 01:45:25.600608 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:25.600586 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" podUID="fe01e89d-ffd9-40bc-8720-eee65f4a42eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 30 01:45:25.813772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:25.809927 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb08419-d8vwm" Oct 30 01:45:25.900062 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:25.896002 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df-kube-api-access-47j7x\" (UniqueName: \"kubernetes.io/projected/6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df-kube-api-access-47j7x\") pod \"6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df\" (UID: \"6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df\") " Oct 30 01:45:25.900062 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:25.896166 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df-maas-api-openshift-ca\") pod \"6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df\" (UID: \"6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df\") " Oct 30 01:45:25.900062 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:25.896225 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df-maas-api-xks-ca\") pod \"6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df\" (UID: \"6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df\") " Oct 30 01:45:25.900062 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:25.897635 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df-maas-api-openshift-ca" pod "6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df" (UID: "6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:45:25.965801 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:25.964545 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df-maas-api-xks-ca" pod "6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df" (UID: "6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:45:25.986911 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:25.986880 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df-kube-api-access-47j7x" pod "6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df" (UID: "6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df"). InnerVolumeSpecName "kube-api-access-47j7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:45:25.997536 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:25.997509 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47j7x\" (UniqueName: \"kubernetes.io/projected/6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df-kube-api-access-47j7x\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:25.997536 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:25.997534 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:25.997536 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:25.997545 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:26.263988 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:26.263951 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb08419-d8vwm" Oct 30 01:45:26.264186 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:26.264008 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb08419-d8vwm" event={"ID":"6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df","Type":"ContainerDied","Data":"fe7e8c276c90a58d90c8bf42dad7ee6f207825669408135b3360cdf4310610df"} Oct 30 01:45:26.264186 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:26.264085 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe7e8c276c90a58d90c8bf42dad7ee6f207825669408135b3360cdf4310610df" Oct 30 01:45:26.589418 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:26.589341 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Oct 30 01:45:26.589418 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:26.589393 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" podUID="fdadf1cc-610e-4e7b-83cf-5a173f5779eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Oct 30 01:45:26.601379 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:26.594825 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:26.601379 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:26.596024 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 30 01:45:26.601379 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:26.596069 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" podUID="fe01e89d-ffd9-40bc-8720-eee65f4a42eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 30 01:45:27.588669 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:27.588627 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Oct 30 01:45:27.588935 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:27.588707 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" podUID="fdadf1cc-610e-4e7b-83cf-5a173f5779eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Oct 30 01:45:27.597057 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:27.597022 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body= Oct 30 01:45:27.597226 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:27.597074 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" podUID="fe01e89d-ffd9-40bc-8720-eee65f4a42eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" Oct 30 01:45:27.697147 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:27.697098 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:27.912425 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:27.912392 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:28.526093 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:28.526052 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:28.666855 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:28.666821 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" Oct 30 01:45:28.667644 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:28.667519 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" Oct 30 01:45:28.668769 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:28.668723 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" Oct 30 01:45:29.071910 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:29.071878 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" Oct 30 01:45:29.072580 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:29.072560 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" Oct 30 01:45:29.073622 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:29.073604 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" Oct 30 01:45:30.159964 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.159922 6572 patch_prober.go:29] interesting pod/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:45:30.160446 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.160002 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" podUID="2c1aa377-ab2c-4653-a630-3d4eef49526d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:45:30.218121 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.215581 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:30.404174 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.404131 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:30.445614 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.445531 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-8f68ab-7dbc58dc79-zl8qv"] Oct 30 01:45:30.446116 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.446101 6572 cpu_manager.go:414] "containerMap: removing container" podUID="c76a323c-f168-42f8-933b-b28c333dcac4" containerName="cleanup" Oct 30 01:45:30.446191 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.446116 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c76a323c-f168-42f8-933b-b28c333dcac4" containerName="cleanup" Oct 30 01:45:30.446191 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.446129 6572 cpu_manager.go:414] "containerMap: removing container" podUID="fae65009-54c2-4407-90c3-bc753280c957" containerName="cleanup" Oct 30 01:45:30.446191 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.446134 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fae65009-54c2-4407-90c3-bc753280c957" containerName="cleanup" Oct 30 01:45:30.446191 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.446141 6572 cpu_manager.go:414] "containerMap: removing container" podUID="bd21026f-4167-49df-ace8-dbc4d9060fc0" containerName="revoke-keys" Oct 30 01:45:30.446191 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.446147 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd21026f-4167-49df-ace8-dbc4d9060fc0" containerName="revoke-keys" Oct 30 01:45:30.446191 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.446159 6572 cpu_manager.go:414] "containerMap: removing container" podUID="c76a323c-f168-42f8-933b-b28c333dcac4" containerName="cleanup" Oct 30 01:45:30.446191 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.446164 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c76a323c-f168-42f8-933b-b28c333dcac4" containerName="cleanup" Oct 30 01:45:30.446191 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.446174 6572 cpu_manager.go:414] "containerMap: removing container" podUID="6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df" containerName="revoke-keys" Oct 30 01:45:30.446191 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.446178 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df" containerName="revoke-keys" Oct 30 01:45:30.446191 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.446187 6572 cpu_manager.go:414] "containerMap: removing container" podUID="fae65009-54c2-4407-90c3-bc753280c957" containerName="cleanup" Oct 30 01:45:30.446191 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.446192 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fae65009-54c2-4407-90c3-bc753280c957" containerName="cleanup" Oct 30 01:45:30.446796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.446206 6572 cpu_manager.go:414] "containerMap: removing container" podUID="c76a323c-f168-42f8-933b-b28c333dcac4" containerName="cleanup" Oct 30 01:45:30.446796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.446210 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c76a323c-f168-42f8-933b-b28c333dcac4" containerName="cleanup" Oct 30 01:45:30.446796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.446216 6572 cpu_manager.go:414] "containerMap: removing container" podUID="0169ab93-f50e-4660-b20c-72ccde5254c9" containerName="revoke-keys" Oct 30 01:45:30.446796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.446220 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0169ab93-f50e-4660-b20c-72ccde5254c9" containerName="revoke-keys" Oct 30 01:45:30.446796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.446232 6572 cpu_manager.go:414] "containerMap: removing container" podUID="fae65009-54c2-4407-90c3-bc753280c957" containerName="cleanup" Oct 30 01:45:30.446796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.446236 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fae65009-54c2-4407-90c3-bc753280c957" containerName="cleanup" Oct 30 01:45:30.446796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.446311 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="c76a323c-f168-42f8-933b-b28c333dcac4" containerName="cleanup" Oct 30 01:45:30.446796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.446319 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="fae65009-54c2-4407-90c3-bc753280c957" containerName="cleanup" Oct 30 01:45:30.446796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.446325 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd21026f-4167-49df-ace8-dbc4d9060fc0" containerName="revoke-keys" Oct 30 01:45:30.446796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.446333 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="c76a323c-f168-42f8-933b-b28c333dcac4" containerName="cleanup" Oct 30 01:45:30.446796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.446340 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df" containerName="revoke-keys" Oct 30 01:45:30.446796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.446346 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="fae65009-54c2-4407-90c3-bc753280c957" containerName="cleanup" Oct 30 01:45:30.446796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.446354 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="0169ab93-f50e-4660-b20c-72ccde5254c9" containerName="revoke-keys" Oct 30 01:45:30.448946 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.448923 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8f68ab-7dbc58dc79-zl8qv" Oct 30 01:45:30.462685 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.462649 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-8f68ab\"" Oct 30 01:45:30.474227 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.474142 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-8f68ab-dockercfg-f76w7\"" Oct 30 01:45:30.478002 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.477971 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-8f68ab-7dbc58dc79-zl8qv"] Oct 30 01:45:30.521808 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.521771 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-8f68ab-57c584d79b-n88mc"] Oct 30 01:45:30.522447 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.522433 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="c76a323c-f168-42f8-933b-b28c333dcac4" containerName="cleanup" Oct 30 01:45:30.522499 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.522451 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="fae65009-54c2-4407-90c3-bc753280c957" containerName="cleanup" Oct 30 01:45:30.525026 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.525007 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-8f68ab-57c584d79b-n88mc" Oct 30 01:45:30.556222 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.554583 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-8f68ab-57c584d79b-n88mc"] Oct 30 01:45:30.606775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.589603 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkrsk\" (UniqueName: \"kubernetes.io/projected/0731d0f8-7e67-4fce-84ea-84441098c02d-kube-api-access-mkrsk\") pod \"payload-processing-e2e-praxis-b-8f68ab-57c584d79b-n88mc\" (UID: \"0731d0f8-7e67-4fce-84ea-84441098c02d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-8f68ab-57c584d79b-n88mc" Oct 30 01:45:30.606775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.589701 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26657c68-1439-4806-b039-13baea561144-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-8f68ab-7dbc58dc79-zl8qv\" (UID: \"26657c68-1439-4806-b039-13baea561144\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8f68ab-7dbc58dc79-zl8qv" Oct 30 01:45:30.606775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.590210 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0731d0f8-7e67-4fce-84ea-84441098c02d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-8f68ab-57c584d79b-n88mc\" (UID: \"0731d0f8-7e67-4fce-84ea-84441098c02d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-8f68ab-57c584d79b-n88mc" Oct 30 01:45:30.606775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.590281 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vhsn\" (UniqueName: \"kubernetes.io/projected/26657c68-1439-4806-b039-13baea561144-kube-api-access-2vhsn\") pod \"payload-pre-processing-e2e-praxis-b-8f68ab-7dbc58dc79-zl8qv\" (UID: \"26657c68-1439-4806-b039-13baea561144\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8f68ab-7dbc58dc79-zl8qv" Oct 30 01:45:30.606775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.590766 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:30.692225 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.692183 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0731d0f8-7e67-4fce-84ea-84441098c02d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-8f68ab-57c584d79b-n88mc\" (UID: \"0731d0f8-7e67-4fce-84ea-84441098c02d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-8f68ab-57c584d79b-n88mc" Oct 30 01:45:30.692426 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.692267 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vhsn\" (UniqueName: \"kubernetes.io/projected/26657c68-1439-4806-b039-13baea561144-kube-api-access-2vhsn\") pod \"payload-pre-processing-e2e-praxis-b-8f68ab-7dbc58dc79-zl8qv\" (UID: \"26657c68-1439-4806-b039-13baea561144\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8f68ab-7dbc58dc79-zl8qv" Oct 30 01:45:30.692535 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.692488 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkrsk\" (UniqueName: \"kubernetes.io/projected/0731d0f8-7e67-4fce-84ea-84441098c02d-kube-api-access-mkrsk\") pod \"payload-processing-e2e-praxis-b-8f68ab-57c584d79b-n88mc\" (UID: \"0731d0f8-7e67-4fce-84ea-84441098c02d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-8f68ab-57c584d79b-n88mc" Oct 30 01:45:30.692606 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.692555 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26657c68-1439-4806-b039-13baea561144-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-8f68ab-7dbc58dc79-zl8qv\" (UID: \"26657c68-1439-4806-b039-13baea561144\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8f68ab-7dbc58dc79-zl8qv" Oct 30 01:45:30.695307 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.695280 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26657c68-1439-4806-b039-13baea561144-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-8f68ab-7dbc58dc79-zl8qv\" (UID: \"26657c68-1439-4806-b039-13baea561144\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8f68ab-7dbc58dc79-zl8qv" Oct 30 01:45:30.695517 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.695492 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0731d0f8-7e67-4fce-84ea-84441098c02d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-8f68ab-57c584d79b-n88mc\" (UID: \"0731d0f8-7e67-4fce-84ea-84441098c02d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-8f68ab-57c584d79b-n88mc" Oct 30 01:45:30.713246 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.713180 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkrsk\" (UniqueName: \"kubernetes.io/projected/0731d0f8-7e67-4fce-84ea-84441098c02d-kube-api-access-mkrsk\") pod \"payload-processing-e2e-praxis-b-8f68ab-57c584d79b-n88mc\" (UID: \"0731d0f8-7e67-4fce-84ea-84441098c02d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-8f68ab-57c584d79b-n88mc" Oct 30 01:45:30.713246 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.713236 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vhsn\" (UniqueName: \"kubernetes.io/projected/26657c68-1439-4806-b039-13baea561144-kube-api-access-2vhsn\") pod \"payload-pre-processing-e2e-praxis-b-8f68ab-7dbc58dc79-zl8qv\" (UID: \"26657c68-1439-4806-b039-13baea561144\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8f68ab-7dbc58dc79-zl8qv" Oct 30 01:45:30.762949 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.762919 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8f68ab-7dbc58dc79-zl8qv" Oct 30 01:45:30.839165 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:30.838175 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-8f68ab-57c584d79b-n88mc" Oct 30 01:45:31.060370 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:31.060342 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-8f68ab-7dbc58dc79-zl8qv"] Oct 30 01:45:31.060515 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:45:31.060492 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26657c68_1439_4806_b039_13baea561144.slice/crio-3894c641bf534def77b58e2110715c978f3466f43172af6a728e90791edd96b9 WatchSource:0}: Error finding container 3894c641bf534def77b58e2110715c978f3466f43172af6a728e90791edd96b9: Status 404 returned error can't find the container with id 3894c641bf534def77b58e2110715c978f3466f43172af6a728e90791edd96b9 Oct 30 01:45:31.149165 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:31.149129 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-8f68ab-57c584d79b-n88mc"] Oct 30 01:45:31.149165 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:45:31.149149 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0731d0f8_7e67_4fce_84ea_84441098c02d.slice/crio-b63d79e1441c85134de219f1302dc9f1793cd43521a74781bf1fe94dc985af0d WatchSource:0}: Error finding container b63d79e1441c85134de219f1302dc9f1793cd43521a74781bf1fe94dc985af0d: Status 404 returned error can't find the container with id b63d79e1441c85134de219f1302dc9f1793cd43521a74781bf1fe94dc985af0d Oct 30 01:45:31.317643 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:31.317555 6572 generic.go:356] "Generic (PLEG): container finished" podID="2c1aa377-ab2c-4653-a630-3d4eef49526d" containerID="14fa9a4fa5bce69078f1ee86094df3c5f5a8a02dd48631a01adacb880f7bfdf8" exitCode=0 Oct 30 01:45:31.317643 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:31.317624 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" event={"ID":"2c1aa377-ab2c-4653-a630-3d4eef49526d","Type":"ContainerDied","Data":"14fa9a4fa5bce69078f1ee86094df3c5f5a8a02dd48631a01adacb880f7bfdf8"} Oct 30 01:45:31.318175 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:31.318054 6572 scope.go:122] "RemoveContainer" containerID="14fa9a4fa5bce69078f1ee86094df3c5f5a8a02dd48631a01adacb880f7bfdf8" Oct 30 01:45:31.319409 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:31.319385 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-8f68ab-57c584d79b-n88mc" event={"ID":"0731d0f8-7e67-4fce-84ea-84441098c02d","Type":"ContainerStarted","Data":"1a274491c04dce642c0d9007b015b44bdf0f0bf84120362fb9e8812706004cd3"} Oct 30 01:45:31.319527 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:31.319428 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-8f68ab-57c584d79b-n88mc" event={"ID":"0731d0f8-7e67-4fce-84ea-84441098c02d","Type":"ContainerStarted","Data":"b63d79e1441c85134de219f1302dc9f1793cd43521a74781bf1fe94dc985af0d"} Oct 30 01:45:31.319527 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:31.319478 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-8f68ab-57c584d79b-n88mc" Oct 30 01:45:31.319527 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:31.319503 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:45:31.319712 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:31.319570 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:45:31.319712 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:31.319606 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:45:31.321545 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:31.321525 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8f68ab-7dbc58dc79-zl8qv" event={"ID":"26657c68-1439-4806-b039-13baea561144","Type":"ContainerStarted","Data":"171f11a91479f0ddb23834903a3742cec4df8ffa6ce8e28b7101017e8ae15e5f"} Oct 30 01:45:31.321661 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:31.321558 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8f68ab-7dbc58dc79-zl8qv" event={"ID":"26657c68-1439-4806-b039-13baea561144","Type":"ContainerStarted","Data":"3894c641bf534def77b58e2110715c978f3466f43172af6a728e90791edd96b9"} Oct 30 01:45:31.321730 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:31.321670 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8f68ab-7dbc58dc79-zl8qv" Oct 30 01:45:31.406104 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:31.406053 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8f68ab-7dbc58dc79-zl8qv" podStartSLOduration=1.406039758 podStartE2EDuration="1.406039758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:31.400160035 +0000 UTC m=+1656.976585439" watchObservedRunningTime="2026-10-30 01:45:31.406039758 +0000 UTC m=+1656.982465180" Oct 30 01:45:31.442857 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:31.438260 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-8f68ab-57c584d79b-n88mc" podStartSLOduration=1.438241898 podStartE2EDuration="1.438241898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:31.437581227 +0000 UTC m=+1657.014006631" watchObservedRunningTime="2026-10-30 01:45:31.438241898 +0000 UTC m=+1657.014667300" Oct 30 01:45:32.063392 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:32.063340 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:32.340816 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:32.340696 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" event={"ID":"2c1aa377-ab2c-4653-a630-3d4eef49526d","Type":"ContainerStarted","Data":"72cc259568b451aac5ebe330f41c573ac8d2b1c09862bcf3a6962ec910780676"} Oct 30 01:45:32.763179 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:32.763143 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:45:32.763362 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:32.763343 6572 patch_prober.go:29] interesting pod/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Oct 30 01:45:32.763404 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:32.763388 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" podUID="2c1aa377-ab2c-4653-a630-3d4eef49526d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Oct 30 01:45:33.763811 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:33.763776 6572 patch_prober.go:29] interesting pod/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Oct 30 01:45:33.764286 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:33.763838 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" podUID="2c1aa377-ab2c-4653-a630-3d4eef49526d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Oct 30 01:45:33.800125 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:33.800074 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:33.984795 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:33.984764 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebfaa245-25g2p"] Oct 30 01:45:33.987979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:33.987958 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebfaa245-25g2p" Oct 30 01:45:34.029816 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.029118 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:45:34.029816 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.029190 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:45:34.029816 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.029303 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:45:34.029816 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.029476 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:45:34.037773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.036811 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebfaa245-25g2p"] Oct 30 01:45:34.155515 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.155474 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f2871c4-f508-4186-8e8c-1f0cfaae81fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebfaa245-25g2p\" (UID: \"8f2871c4-f508-4186-8e8c-1f0cfaae81fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebfaa245-25g2p" Oct 30 01:45:34.155680 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.155600 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f2871c4-f508-4186-8e8c-1f0cfaae81fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebfaa245-25g2p\" (UID: \"8f2871c4-f508-4186-8e8c-1f0cfaae81fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebfaa245-25g2p" Oct 30 01:45:34.155680 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.155645 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtcrl\" (UniqueName: \"kubernetes.io/projected/8f2871c4-f508-4186-8e8c-1f0cfaae81fb-kube-api-access-mtcrl\") pod \"maas-api-revoke-sub-ebfaa245-25g2p\" (UID: \"8f2871c4-f508-4186-8e8c-1f0cfaae81fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebfaa245-25g2p" Oct 30 01:45:34.157282 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.157258 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r"] Oct 30 01:45:34.160607 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.160586 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" Oct 30 01:45:34.185353 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.185325 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r"] Oct 30 01:45:34.257415 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.257385 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1174e25-63b1-4693-841f-72732c52e068-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r\" (UID: \"f1174e25-63b1-4693-841f-72732c52e068\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" Oct 30 01:45:34.257415 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.257417 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1174e25-63b1-4693-841f-72732c52e068-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r\" (UID: \"f1174e25-63b1-4693-841f-72732c52e068\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" Oct 30 01:45:34.257628 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.257445 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1174e25-63b1-4693-841f-72732c52e068-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r\" (UID: \"f1174e25-63b1-4693-841f-72732c52e068\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" Oct 30 01:45:34.257628 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.257517 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1174e25-63b1-4693-841f-72732c52e068-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r\" (UID: \"f1174e25-63b1-4693-841f-72732c52e068\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" Oct 30 01:45:34.257628 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.257600 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1174e25-63b1-4693-841f-72732c52e068-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r\" (UID: \"f1174e25-63b1-4693-841f-72732c52e068\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" Oct 30 01:45:34.257731 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.257654 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1174e25-63b1-4693-841f-72732c52e068-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r\" (UID: \"f1174e25-63b1-4693-841f-72732c52e068\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" Oct 30 01:45:34.257731 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.257693 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvf2j\" (UniqueName: \"kubernetes.io/projected/f1174e25-63b1-4693-841f-72732c52e068-kube-api-access-tvf2j\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r\" (UID: \"f1174e25-63b1-4693-841f-72732c52e068\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" Oct 30 01:45:34.257836 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.257812 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f2871c4-f508-4186-8e8c-1f0cfaae81fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebfaa245-25g2p\" (UID: \"8f2871c4-f508-4186-8e8c-1f0cfaae81fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebfaa245-25g2p" Oct 30 01:45:34.257875 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.257857 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1174e25-63b1-4693-841f-72732c52e068-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r\" (UID: \"f1174e25-63b1-4693-841f-72732c52e068\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" Oct 30 01:45:34.257915 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.257902 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f2871c4-f508-4186-8e8c-1f0cfaae81fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebfaa245-25g2p\" (UID: \"8f2871c4-f508-4186-8e8c-1f0cfaae81fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebfaa245-25g2p" Oct 30 01:45:34.257950 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.257927 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtcrl\" (UniqueName: \"kubernetes.io/projected/8f2871c4-f508-4186-8e8c-1f0cfaae81fb-kube-api-access-mtcrl\") pod \"maas-api-revoke-sub-ebfaa245-25g2p\" (UID: \"8f2871c4-f508-4186-8e8c-1f0cfaae81fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebfaa245-25g2p" Oct 30 01:45:34.258008 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.257987 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1174e25-63b1-4693-841f-72732c52e068-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r\" (UID: \"f1174e25-63b1-4693-841f-72732c52e068\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" Oct 30 01:45:34.258598 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.258576 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f2871c4-f508-4186-8e8c-1f0cfaae81fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebfaa245-25g2p\" (UID: \"8f2871c4-f508-4186-8e8c-1f0cfaae81fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebfaa245-25g2p" Oct 30 01:45:34.260224 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.260200 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f2871c4-f508-4186-8e8c-1f0cfaae81fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebfaa245-25g2p\" (UID: \"8f2871c4-f508-4186-8e8c-1f0cfaae81fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebfaa245-25g2p" Oct 30 01:45:34.276990 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.276969 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtcrl\" (UniqueName: \"kubernetes.io/projected/8f2871c4-f508-4186-8e8c-1f0cfaae81fb-kube-api-access-mtcrl\") pod \"maas-api-revoke-sub-ebfaa245-25g2p\" (UID: \"8f2871c4-f508-4186-8e8c-1f0cfaae81fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebfaa245-25g2p" Oct 30 01:45:34.301542 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.301489 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebfaa245-25g2p" Oct 30 01:45:34.359419 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.359383 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1174e25-63b1-4693-841f-72732c52e068-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r\" (UID: \"f1174e25-63b1-4693-841f-72732c52e068\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" Oct 30 01:45:34.359419 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.359421 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1174e25-63b1-4693-841f-72732c52e068-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r\" (UID: \"f1174e25-63b1-4693-841f-72732c52e068\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" Oct 30 01:45:34.359638 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.359456 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1174e25-63b1-4693-841f-72732c52e068-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r\" (UID: \"f1174e25-63b1-4693-841f-72732c52e068\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" Oct 30 01:45:34.359638 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.359483 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1174e25-63b1-4693-841f-72732c52e068-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r\" (UID: \"f1174e25-63b1-4693-841f-72732c52e068\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" Oct 30 01:45:34.359638 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.359574 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvf2j\" (UniqueName: \"kubernetes.io/projected/f1174e25-63b1-4693-841f-72732c52e068-kube-api-access-tvf2j\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r\" (UID: \"f1174e25-63b1-4693-841f-72732c52e068\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" Oct 30 01:45:34.359846 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.359727 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1174e25-63b1-4693-841f-72732c52e068-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r\" (UID: \"f1174e25-63b1-4693-841f-72732c52e068\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" Oct 30 01:45:34.359846 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.359795 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1174e25-63b1-4693-841f-72732c52e068-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r\" (UID: \"f1174e25-63b1-4693-841f-72732c52e068\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" Oct 30 01:45:34.359974 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.359911 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1174e25-63b1-4693-841f-72732c52e068-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r\" (UID: \"f1174e25-63b1-4693-841f-72732c52e068\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" Oct 30 01:45:34.359974 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.359941 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1174e25-63b1-4693-841f-72732c52e068-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r\" (UID: \"f1174e25-63b1-4693-841f-72732c52e068\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" Oct 30 01:45:34.360094 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.360030 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1174e25-63b1-4693-841f-72732c52e068-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r\" (UID: \"f1174e25-63b1-4693-841f-72732c52e068\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" Oct 30 01:45:34.360094 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.359795 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1174e25-63b1-4693-841f-72732c52e068-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r\" (UID: \"f1174e25-63b1-4693-841f-72732c52e068\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" Oct 30 01:45:34.360211 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.360169 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1174e25-63b1-4693-841f-72732c52e068-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r\" (UID: \"f1174e25-63b1-4693-841f-72732c52e068\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" Oct 30 01:45:34.360315 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.360291 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1174e25-63b1-4693-841f-72732c52e068-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r\" (UID: \"f1174e25-63b1-4693-841f-72732c52e068\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" Oct 30 01:45:34.360415 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.360393 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1174e25-63b1-4693-841f-72732c52e068-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r\" (UID: \"f1174e25-63b1-4693-841f-72732c52e068\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" Oct 30 01:45:34.362687 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.362664 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1174e25-63b1-4693-841f-72732c52e068-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r\" (UID: \"f1174e25-63b1-4693-841f-72732c52e068\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" Oct 30 01:45:34.363122 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.363102 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1174e25-63b1-4693-841f-72732c52e068-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r\" (UID: \"f1174e25-63b1-4693-841f-72732c52e068\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" Oct 30 01:45:34.387470 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.380791 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1174e25-63b1-4693-841f-72732c52e068-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r\" (UID: \"f1174e25-63b1-4693-841f-72732c52e068\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" Oct 30 01:45:34.387470 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.383873 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvf2j\" (UniqueName: \"kubernetes.io/projected/f1174e25-63b1-4693-841f-72732c52e068-kube-api-access-tvf2j\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r\" (UID: \"f1174e25-63b1-4693-841f-72732c52e068\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" Oct 30 01:45:34.473988 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.473262 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" Oct 30 01:45:34.637505 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.637476 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebfaa245-25g2p"] Oct 30 01:45:34.638476 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:45:34.638445 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f2871c4_f508_4186_8e8c_1f0cfaae81fb.slice/crio-2abfcfae5ffee9f795e0794028958b94baf949266fece73fb635a378a714dcca WatchSource:0}: Error finding container 2abfcfae5ffee9f795e0794028958b94baf949266fece73fb635a378a714dcca: Status 404 returned error can't find the container with id 2abfcfae5ffee9f795e0794028958b94baf949266fece73fb635a378a714dcca Oct 30 01:45:34.763365 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.763331 6572 patch_prober.go:29] interesting pod/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Oct 30 01:45:34.763529 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.763389 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" podUID="2c1aa377-ab2c-4653-a630-3d4eef49526d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Oct 30 01:45:34.797934 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.797903 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r"] Oct 30 01:45:34.830824 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:45:34.830392 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1174e25_63b1_4693_841f_72732c52e068.slice/crio-b94a4125174b3aa3071b720772980041fc077d7b5008c2d6b6f324817c08a47f WatchSource:0}: Error finding container b94a4125174b3aa3071b720772980041fc077d7b5008c2d6b6f324817c08a47f: Status 404 returned error can't find the container with id b94a4125174b3aa3071b720772980041fc077d7b5008c2d6b6f324817c08a47f Oct 30 01:45:34.834895 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.834860 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:45:34.835015 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.834936 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:45:34.835015 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:34.834969 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:45:35.371217 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.371182 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79"] Oct 30 01:45:35.375601 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.375565 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebfaa245-25g2p" event={"ID":"8f2871c4-f508-4186-8e8c-1f0cfaae81fb","Type":"ContainerStarted","Data":"4bcf8f538c6bed420f0f63babb43032e36bb6dc2bc8dab980d8169dfe5ef7466"} Oct 30 01:45:35.375822 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.375616 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebfaa245-25g2p" event={"ID":"8f2871c4-f508-4186-8e8c-1f0cfaae81fb","Type":"ContainerStarted","Data":"2abfcfae5ffee9f795e0794028958b94baf949266fece73fb635a378a714dcca"} Oct 30 01:45:35.375915 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.375859 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" Oct 30 01:45:35.376515 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.376489 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" event={"ID":"f1174e25-63b1-4693-841f-72732c52e068","Type":"ContainerStarted","Data":"9e7cb755c1b61d34c51778f691ffb09b0fb8116ea606ec53d6b96207b1df620a"} Oct 30 01:45:35.376636 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.376540 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" event={"ID":"f1174e25-63b1-4693-841f-72732c52e068","Type":"ContainerStarted","Data":"b94a4125174b3aa3071b720772980041fc077d7b5008c2d6b6f324817c08a47f"} Oct 30 01:45:35.490082 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.472840 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79"] Oct 30 01:45:35.490082 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.474177 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" Oct 30 01:45:35.490082 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.477646 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Oct 30 01:45:35.490082 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.477712 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" podUID="f1174e25-63b1-4693-841f-72732c52e068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Oct 30 01:45:35.490082 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.478860 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:35.490082 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.480913 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b24c1c0-b61b-43fc-84b0-994045057bc6-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79\" (UID: \"8b24c1c0-b61b-43fc-84b0-994045057bc6\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" Oct 30 01:45:35.490082 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.480996 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b24c1c0-b61b-43fc-84b0-994045057bc6-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79\" (UID: \"8b24c1c0-b61b-43fc-84b0-994045057bc6\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" Oct 30 01:45:35.490082 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.481028 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b24c1c0-b61b-43fc-84b0-994045057bc6-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79\" (UID: \"8b24c1c0-b61b-43fc-84b0-994045057bc6\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" Oct 30 01:45:35.490082 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.481080 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q9l9\" (UniqueName: \"kubernetes.io/projected/8b24c1c0-b61b-43fc-84b0-994045057bc6-kube-api-access-4q9l9\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79\" (UID: \"8b24c1c0-b61b-43fc-84b0-994045057bc6\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" Oct 30 01:45:35.490082 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.481232 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b24c1c0-b61b-43fc-84b0-994045057bc6-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79\" (UID: \"8b24c1c0-b61b-43fc-84b0-994045057bc6\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" Oct 30 01:45:35.490082 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.481279 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b24c1c0-b61b-43fc-84b0-994045057bc6-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79\" (UID: \"8b24c1c0-b61b-43fc-84b0-994045057bc6\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" Oct 30 01:45:35.490082 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.481373 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b24c1c0-b61b-43fc-84b0-994045057bc6-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79\" (UID: \"8b24c1c0-b61b-43fc-84b0-994045057bc6\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" Oct 30 01:45:35.490082 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.481435 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b24c1c0-b61b-43fc-84b0-994045057bc6-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79\" (UID: \"8b24c1c0-b61b-43fc-84b0-994045057bc6\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" Oct 30 01:45:35.490082 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.481659 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b24c1c0-b61b-43fc-84b0-994045057bc6-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79\" (UID: \"8b24c1c0-b61b-43fc-84b0-994045057bc6\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" Oct 30 01:45:35.496783 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.496699 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebfaa245-25g2p" podStartSLOduration=2.496680948 podStartE2EDuration="2.496680948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:35.469281739 +0000 UTC m=+1661.045707172" watchObservedRunningTime="2026-10-30 01:45:35.496680948 +0000 UTC m=+1661.073106351" Oct 30 01:45:35.540703 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.540633 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" podStartSLOduration=1.540614535 podStartE2EDuration="1.540614535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:35.539361809 +0000 UTC m=+1661.115787212" watchObservedRunningTime="2026-10-30 01:45:35.540614535 +0000 UTC m=+1661.117039937" Oct 30 01:45:35.582689 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.582642 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b24c1c0-b61b-43fc-84b0-994045057bc6-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79\" (UID: \"8b24c1c0-b61b-43fc-84b0-994045057bc6\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" Oct 30 01:45:35.582892 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.582721 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b24c1c0-b61b-43fc-84b0-994045057bc6-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79\" (UID: \"8b24c1c0-b61b-43fc-84b0-994045057bc6\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" Oct 30 01:45:35.582892 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.582850 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b24c1c0-b61b-43fc-84b0-994045057bc6-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79\" (UID: \"8b24c1c0-b61b-43fc-84b0-994045057bc6\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" Oct 30 01:45:35.583029 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.582905 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b24c1c0-b61b-43fc-84b0-994045057bc6-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79\" (UID: \"8b24c1c0-b61b-43fc-84b0-994045057bc6\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" Oct 30 01:45:35.583091 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.583059 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b24c1c0-b61b-43fc-84b0-994045057bc6-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79\" (UID: \"8b24c1c0-b61b-43fc-84b0-994045057bc6\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" Oct 30 01:45:35.583091 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.583072 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b24c1c0-b61b-43fc-84b0-994045057bc6-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79\" (UID: \"8b24c1c0-b61b-43fc-84b0-994045057bc6\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" Oct 30 01:45:35.583198 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.583140 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b24c1c0-b61b-43fc-84b0-994045057bc6-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79\" (UID: \"8b24c1c0-b61b-43fc-84b0-994045057bc6\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" Oct 30 01:45:35.583198 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.583188 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b24c1c0-b61b-43fc-84b0-994045057bc6-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79\" (UID: \"8b24c1c0-b61b-43fc-84b0-994045057bc6\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" Oct 30 01:45:35.583302 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.583215 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b24c1c0-b61b-43fc-84b0-994045057bc6-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79\" (UID: \"8b24c1c0-b61b-43fc-84b0-994045057bc6\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" Oct 30 01:45:35.583302 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.583290 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q9l9\" (UniqueName: \"kubernetes.io/projected/8b24c1c0-b61b-43fc-84b0-994045057bc6-kube-api-access-4q9l9\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79\" (UID: \"8b24c1c0-b61b-43fc-84b0-994045057bc6\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" Oct 30 01:45:35.583530 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.583510 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b24c1c0-b61b-43fc-84b0-994045057bc6-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79\" (UID: \"8b24c1c0-b61b-43fc-84b0-994045057bc6\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" Oct 30 01:45:35.583847 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.583826 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b24c1c0-b61b-43fc-84b0-994045057bc6-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79\" (UID: \"8b24c1c0-b61b-43fc-84b0-994045057bc6\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" Oct 30 01:45:35.583959 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.583900 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b24c1c0-b61b-43fc-84b0-994045057bc6-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79\" (UID: \"8b24c1c0-b61b-43fc-84b0-994045057bc6\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" Oct 30 01:45:35.584285 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.584263 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b24c1c0-b61b-43fc-84b0-994045057bc6-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79\" (UID: \"8b24c1c0-b61b-43fc-84b0-994045057bc6\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" Oct 30 01:45:35.586444 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.586416 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b24c1c0-b61b-43fc-84b0-994045057bc6-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79\" (UID: \"8b24c1c0-b61b-43fc-84b0-994045057bc6\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" Oct 30 01:45:35.586827 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.586805 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b24c1c0-b61b-43fc-84b0-994045057bc6-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79\" (UID: \"8b24c1c0-b61b-43fc-84b0-994045057bc6\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" Oct 30 01:45:35.603178 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.603140 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q9l9\" (UniqueName: \"kubernetes.io/projected/8b24c1c0-b61b-43fc-84b0-994045057bc6-kube-api-access-4q9l9\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79\" (UID: \"8b24c1c0-b61b-43fc-84b0-994045057bc6\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" Oct 30 01:45:35.603322 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.603227 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b24c1c0-b61b-43fc-84b0-994045057bc6-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79\" (UID: \"8b24c1c0-b61b-43fc-84b0-994045057bc6\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" Oct 30 01:45:35.697891 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.697852 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" Oct 30 01:45:35.763262 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.763185 6572 patch_prober.go:29] interesting pod/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Oct 30 01:45:35.763262 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:35.763246 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" podUID="2c1aa377-ab2c-4653-a630-3d4eef49526d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Oct 30 01:45:36.159420 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:36.159296 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79"] Oct 30 01:45:36.160393 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:36.160355 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:45:36.160499 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:36.160442 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:45:36.160499 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:36.160488 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:45:36.391634 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:36.391550 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" event={"ID":"8b24c1c0-b61b-43fc-84b0-994045057bc6","Type":"ContainerStarted","Data":"68d9b90790271e8bf890d5d8e4ceffe0bf9459fd275aeacd20de216ac22d9920"} Oct 30 01:45:36.391634 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:36.391608 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" event={"ID":"8b24c1c0-b61b-43fc-84b0-994045057bc6","Type":"ContainerStarted","Data":"87856ddba85bd3c72ef4c4697f262598d3c0a44843288f748ba30db71d5a0cab"} Oct 30 01:45:36.394218 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:36.394185 6572 generic.go:356] "Generic (PLEG): container finished" podID="8f2871c4-f508-4186-8e8c-1f0cfaae81fb" containerID="4bcf8f538c6bed420f0f63babb43032e36bb6dc2bc8dab980d8169dfe5ef7466" exitCode=0 Oct 30 01:45:36.394355 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:36.394220 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebfaa245-25g2p" event={"ID":"8f2871c4-f508-4186-8e8c-1f0cfaae81fb","Type":"ContainerDied","Data":"4bcf8f538c6bed420f0f63babb43032e36bb6dc2bc8dab980d8169dfe5ef7466"} Oct 30 01:45:36.448777 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:36.447253 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" podStartSLOduration=1.447230834 podStartE2EDuration="1.447230834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:36.444712023 +0000 UTC m=+1662.021137427" watchObservedRunningTime="2026-10-30 01:45:36.447230834 +0000 UTC m=+1662.023656237" Oct 30 01:45:36.474804 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:36.474504 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Oct 30 01:45:36.474804 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:36.474602 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" podUID="f1174e25-63b1-4693-841f-72732c52e068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Oct 30 01:45:36.698981 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:36.698933 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" Oct 30 01:45:36.765241 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:36.765159 6572 patch_prober.go:29] interesting pod/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Oct 30 01:45:36.765241 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:36.765224 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" podUID="2c1aa377-ab2c-4653-a630-3d4eef49526d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Oct 30 01:45:37.215768 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:37.215379 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:37.485825 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:37.484909 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Oct 30 01:45:37.485825 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:37.484971 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" podUID="f1174e25-63b1-4693-841f-72732c52e068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Oct 30 01:45:37.704777 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:37.704035 6572 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Oct 30 01:45:37.704777 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:37.704102 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" podUID="8b24c1c0-b61b-43fc-84b0-994045057bc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Oct 30 01:45:37.766829 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:37.766135 6572 patch_prober.go:29] interesting pod/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Oct 30 01:45:37.766829 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:37.766210 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" podUID="2c1aa377-ab2c-4653-a630-3d4eef49526d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Oct 30 01:45:38.018825 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:38.017081 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" podUID="fac3ca55-64af-42d9-90b1-0e821b893282" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:45:38.095775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:38.094950 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" podUID="317505ef-238c-458e-a74e-1198806b94d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:45:38.434771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:38.414234 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" podUID="0ee3f8e3-9b55-4ce5-afaf-86780a26391e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:45:38.461128 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:38.455809 6572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" podUID="317505ef-238c-458e-a74e-1198806b94d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:45:38.730528 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:38.516311 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Oct 30 01:45:38.730528 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:38.516387 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" podUID="f1174e25-63b1-4693-841f-72732c52e068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Oct 30 01:45:38.730528 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:38.715156 6572 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Oct 30 01:45:38.730528 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:38.715213 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" podUID="8b24c1c0-b61b-43fc-84b0-994045057bc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Oct 30 01:45:38.873035 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:38.734610 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebfaa245-25g2p" Oct 30 01:45:38.873035 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:38.806979 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" podUID="9a56da7a-9b92-40e2-b08b-43cf591b5221" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:45:38.873035 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:38.842016 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:45:38.873035 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:38.871121 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:45:38.883225 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:38.883141 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:45:39.009864 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:39.007143 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f2871c4-f508-4186-8e8c-1f0cfaae81fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f2871c4-f508-4186-8e8c-1f0cfaae81fb-maas-api-openshift-ca\") pod \"8f2871c4-f508-4186-8e8c-1f0cfaae81fb\" (UID: \"8f2871c4-f508-4186-8e8c-1f0cfaae81fb\") " Oct 30 01:45:39.009864 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:39.007264 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f2871c4-f508-4186-8e8c-1f0cfaae81fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f2871c4-f508-4186-8e8c-1f0cfaae81fb-maas-api-xks-ca\") pod \"8f2871c4-f508-4186-8e8c-1f0cfaae81fb\" (UID: \"8f2871c4-f508-4186-8e8c-1f0cfaae81fb\") " Oct 30 01:45:39.009864 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:39.007377 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f2871c4-f508-4186-8e8c-1f0cfaae81fb-kube-api-access-mtcrl\" (UniqueName: \"kubernetes.io/projected/8f2871c4-f508-4186-8e8c-1f0cfaae81fb-kube-api-access-mtcrl\") pod \"8f2871c4-f508-4186-8e8c-1f0cfaae81fb\" (UID: \"8f2871c4-f508-4186-8e8c-1f0cfaae81fb\") " Oct 30 01:45:39.152785 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:39.075844 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" podUID="fac3ca55-64af-42d9-90b1-0e821b893282" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:45:39.152785 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:39.079560 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" Oct 30 01:45:39.152785 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:39.092915 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f2871c4-f508-4186-8e8c-1f0cfaae81fb-kube-api-access-mtcrl" pod "8f2871c4-f508-4186-8e8c-1f0cfaae81fb" (UID: "8f2871c4-f508-4186-8e8c-1f0cfaae81fb"). InnerVolumeSpecName "kube-api-access-mtcrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:45:39.152785 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:39.099825 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" podUID="317505ef-238c-458e-a74e-1198806b94d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 01:45:39.152785 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:39.101193 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" Oct 30 01:45:39.152785 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:39.120123 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtcrl\" (UniqueName: \"kubernetes.io/projected/8f2871c4-f508-4186-8e8c-1f0cfaae81fb-kube-api-access-mtcrl\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:39.185011 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:39.184974 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f2871c4-f508-4186-8e8c-1f0cfaae81fb-maas-api-openshift-ca" pod "8f2871c4-f508-4186-8e8c-1f0cfaae81fb" (UID: "8f2871c4-f508-4186-8e8c-1f0cfaae81fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:45:39.270247 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:39.258256 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f2871c4-f508-4186-8e8c-1f0cfaae81fb-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:39.292777 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:39.277146 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f2871c4-f508-4186-8e8c-1f0cfaae81fb-maas-api-xks-ca" pod "8f2871c4-f508-4186-8e8c-1f0cfaae81fb" (UID: "8f2871c4-f508-4186-8e8c-1f0cfaae81fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:45:39.359772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:39.359632 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f2871c4-f508-4186-8e8c-1f0cfaae81fb-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:39.435220 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:39.435161 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" podUID="0ee3f8e3-9b55-4ce5-afaf-86780a26391e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:45:39.462266 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:39.462234 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" Oct 30 01:45:39.487420 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:39.487386 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Oct 30 01:45:39.487708 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:39.487681 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" podUID="f1174e25-63b1-4693-841f-72732c52e068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Oct 30 01:45:39.635314 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:39.635215 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:39.720397 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:39.706130 6572 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Oct 30 01:45:39.720397 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:39.706235 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" podUID="8b24c1c0-b61b-43fc-84b0-994045057bc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Oct 30 01:45:39.783968 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:39.783715 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" Oct 30 01:45:39.888485 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:39.888397 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" Oct 30 01:45:39.922772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:39.905901 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebfaa245-25g2p" event={"ID":"8f2871c4-f508-4186-8e8c-1f0cfaae81fb","Type":"ContainerDied","Data":"2abfcfae5ffee9f795e0794028958b94baf949266fece73fb635a378a714dcca"} Oct 30 01:45:39.922772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:39.905972 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2abfcfae5ffee9f795e0794028958b94baf949266fece73fb635a378a714dcca" Oct 30 01:45:39.922772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:39.906061 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebfaa245-25g2p" Oct 30 01:45:40.072787 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:40.070854 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" Oct 30 01:45:40.473949 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:40.473916 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Oct 30 01:45:40.474333 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:40.473992 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" podUID="f1174e25-63b1-4693-841f-72732c52e068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Oct 30 01:45:40.698495 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:40.698443 6572 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Oct 30 01:45:40.698723 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:40.698509 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" podUID="8b24c1c0-b61b-43fc-84b0-994045057bc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Oct 30 01:45:41.234836 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:41.234797 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:41.474498 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:41.474463 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Oct 30 01:45:41.475008 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:41.474519 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" podUID="f1174e25-63b1-4693-841f-72732c52e068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Oct 30 01:45:41.699228 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:41.699190 6572 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Oct 30 01:45:41.699441 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:41.699255 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" podUID="8b24c1c0-b61b-43fc-84b0-994045057bc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Oct 30 01:45:42.343851 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:42.343823 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8f68ab-7dbc58dc79-zl8qv" Oct 30 01:45:42.344031 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:42.343901 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-8f68ab-57c584d79b-n88mc" Oct 30 01:45:42.474191 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:42.474159 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body= Oct 30 01:45:42.474361 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:42.474213 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" podUID="f1174e25-63b1-4693-841f-72732c52e068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" Oct 30 01:45:42.698372 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:42.698336 6572 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Oct 30 01:45:42.698889 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:42.698397 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" podUID="8b24c1c0-b61b-43fc-84b0-994045057bc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Oct 30 01:45:42.930773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:42.917417 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:43.166330 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:43.166288 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:43.698515 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:43.698471 6572 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Oct 30 01:45:43.699095 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:43.698537 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" podUID="8b24c1c0-b61b-43fc-84b0-994045057bc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Oct 30 01:45:44.435186 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:44.435143 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7c26448-th5p8"] Oct 30 01:45:44.436217 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:44.436195 6572 cpu_manager.go:414] "containerMap: removing container" podUID="8f2871c4-f508-4186-8e8c-1f0cfaae81fb" containerName="revoke-keys" Oct 30 01:45:44.436293 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:44.436220 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f2871c4-f508-4186-8e8c-1f0cfaae81fb" containerName="revoke-keys" Oct 30 01:45:44.436394 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:44.436380 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f2871c4-f508-4186-8e8c-1f0cfaae81fb" containerName="revoke-keys" Oct 30 01:45:44.439628 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:44.439606 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c26448-th5p8" Oct 30 01:45:44.465299 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:44.465266 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:45:44.465456 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:44.465403 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:45:44.473802 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:44.473682 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:45:44.473938 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:44.473820 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:45:44.481538 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:44.481505 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7c26448-th5p8"] Oct 30 01:45:44.505851 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:44.505805 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8e92603-8f43-4815-bc95-1b88ab8dd06c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7c26448-th5p8\" (UID: \"c8e92603-8f43-4815-bc95-1b88ab8dd06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c26448-th5p8" Oct 30 01:45:44.506064 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:44.505908 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c49g7\" (UniqueName: \"kubernetes.io/projected/c8e92603-8f43-4815-bc95-1b88ab8dd06c-kube-api-access-c49g7\") pod \"maas-api-revoke-sub-c7c26448-th5p8\" (UID: \"c8e92603-8f43-4815-bc95-1b88ab8dd06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c26448-th5p8" Oct 30 01:45:44.506064 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:44.506055 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8e92603-8f43-4815-bc95-1b88ab8dd06c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7c26448-th5p8\" (UID: \"c8e92603-8f43-4815-bc95-1b88ab8dd06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c26448-th5p8" Oct 30 01:45:44.515926 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:44.515893 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:45:44.516197 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:44.516166 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" podUID="f1174e25-63b1-4693-841f-72732c52e068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:45:44.607456 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:44.607413 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8e92603-8f43-4815-bc95-1b88ab8dd06c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7c26448-th5p8\" (UID: \"c8e92603-8f43-4815-bc95-1b88ab8dd06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c26448-th5p8" Oct 30 01:45:44.607676 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:44.607490 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c49g7\" (UniqueName: \"kubernetes.io/projected/c8e92603-8f43-4815-bc95-1b88ab8dd06c-kube-api-access-c49g7\") pod \"maas-api-revoke-sub-c7c26448-th5p8\" (UID: \"c8e92603-8f43-4815-bc95-1b88ab8dd06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c26448-th5p8" Oct 30 01:45:44.607676 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:44.607623 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8e92603-8f43-4815-bc95-1b88ab8dd06c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7c26448-th5p8\" (UID: \"c8e92603-8f43-4815-bc95-1b88ab8dd06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c26448-th5p8" Oct 30 01:45:44.608519 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:44.608483 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8e92603-8f43-4815-bc95-1b88ab8dd06c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7c26448-th5p8\" (UID: \"c8e92603-8f43-4815-bc95-1b88ab8dd06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c26448-th5p8" Oct 30 01:45:44.622773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:44.610294 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8e92603-8f43-4815-bc95-1b88ab8dd06c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7c26448-th5p8\" (UID: \"c8e92603-8f43-4815-bc95-1b88ab8dd06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c26448-th5p8" Oct 30 01:45:44.631345 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:44.631321 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c49g7\" (UniqueName: \"kubernetes.io/projected/c8e92603-8f43-4815-bc95-1b88ab8dd06c-kube-api-access-c49g7\") pod \"maas-api-revoke-sub-c7c26448-th5p8\" (UID: \"c8e92603-8f43-4815-bc95-1b88ab8dd06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c26448-th5p8" Oct 30 01:45:44.695768 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:45:44.695670 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:45:44.699070 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:44.699042 6572 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Oct 30 01:45:44.699475 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:44.699096 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" podUID="8b24c1c0-b61b-43fc-84b0-994045057bc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Oct 30 01:45:44.776317 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:44.776286 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c26448-th5p8" Oct 30 01:45:44.899356 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:44.898212 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" Oct 30 01:45:44.900001 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:44.899846 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" Oct 30 01:45:44.937390 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:44.936175 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" Oct 30 01:45:45.105337 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:45.105309 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7c26448-th5p8"] Oct 30 01:45:45.106879 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:45:45.106829 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e92603_8f43_4815_bc95_1b88ab8dd06c.slice/crio-45546da61ae86c9cb8b6ac9b91268b743de230771a6c1ee073425564195ce3a7 WatchSource:0}: Error finding container 45546da61ae86c9cb8b6ac9b91268b743de230771a6c1ee073425564195ce3a7: Status 404 returned error can't find the container with id 45546da61ae86c9cb8b6ac9b91268b743de230771a6c1ee073425564195ce3a7 Oct 30 01:45:45.193015 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:45.192976 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:45.476656 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:45.476618 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0ae258-k6wjd"] Oct 30 01:45:45.480702 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:45.480681 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0ae258-k6wjd" Oct 30 01:45:45.497829 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:45.497780 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0ae258-k6wjd"] Oct 30 01:45:45.546801 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:45.546768 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/822d8556-80de-405f-a0cf-8ab09bfbca3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0ae258-k6wjd\" (UID: \"822d8556-80de-405f-a0cf-8ab09bfbca3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0ae258-k6wjd" Oct 30 01:45:45.546952 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:45.546910 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q77jv\" (UniqueName: \"kubernetes.io/projected/822d8556-80de-405f-a0cf-8ab09bfbca3e-kube-api-access-q77jv\") pod \"maas-api-revoke-keys-e2e-praxis-a-0ae258-k6wjd\" (UID: \"822d8556-80de-405f-a0cf-8ab09bfbca3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0ae258-k6wjd" Oct 30 01:45:45.547067 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:45.547051 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/822d8556-80de-405f-a0cf-8ab09bfbca3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0ae258-k6wjd\" (UID: \"822d8556-80de-405f-a0cf-8ab09bfbca3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0ae258-k6wjd" Oct 30 01:45:45.648252 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:45.648220 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/822d8556-80de-405f-a0cf-8ab09bfbca3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0ae258-k6wjd\" (UID: \"822d8556-80de-405f-a0cf-8ab09bfbca3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0ae258-k6wjd" Oct 30 01:45:45.648400 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:45.648308 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q77jv\" (UniqueName: \"kubernetes.io/projected/822d8556-80de-405f-a0cf-8ab09bfbca3e-kube-api-access-q77jv\") pod \"maas-api-revoke-keys-e2e-praxis-a-0ae258-k6wjd\" (UID: \"822d8556-80de-405f-a0cf-8ab09bfbca3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0ae258-k6wjd" Oct 30 01:45:45.648471 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:45.648420 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/822d8556-80de-405f-a0cf-8ab09bfbca3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0ae258-k6wjd\" (UID: \"822d8556-80de-405f-a0cf-8ab09bfbca3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0ae258-k6wjd" Oct 30 01:45:45.649177 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:45.649152 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/822d8556-80de-405f-a0cf-8ab09bfbca3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0ae258-k6wjd\" (UID: \"822d8556-80de-405f-a0cf-8ab09bfbca3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0ae258-k6wjd" Oct 30 01:45:45.650921 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:45.650891 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/822d8556-80de-405f-a0cf-8ab09bfbca3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0ae258-k6wjd\" (UID: \"822d8556-80de-405f-a0cf-8ab09bfbca3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0ae258-k6wjd" Oct 30 01:45:45.674837 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:45.674770 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q77jv\" (UniqueName: \"kubernetes.io/projected/822d8556-80de-405f-a0cf-8ab09bfbca3e-kube-api-access-q77jv\") pod \"maas-api-revoke-keys-e2e-praxis-a-0ae258-k6wjd\" (UID: \"822d8556-80de-405f-a0cf-8ab09bfbca3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0ae258-k6wjd" Oct 30 01:45:45.796023 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:45.795986 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0ae258-k6wjd" Oct 30 01:45:46.041212 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:46.041099 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c26448-th5p8" event={"ID":"c8e92603-8f43-4815-bc95-1b88ab8dd06c","Type":"ContainerStarted","Data":"84b96e42b94e8028f8ea8dec77368ba88ffb039b95f844aad402ad355905a608"} Oct 30 01:45:46.041212 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:46.041171 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c26448-th5p8" event={"ID":"c8e92603-8f43-4815-bc95-1b88ab8dd06c","Type":"ContainerStarted","Data":"45546da61ae86c9cb8b6ac9b91268b743de230771a6c1ee073425564195ce3a7"} Oct 30 01:45:46.082063 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:46.082012 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c26448-th5p8" podStartSLOduration=2.081998855 podStartE2EDuration="2.081998855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:46.079091332 +0000 UTC m=+1671.655516736" watchObservedRunningTime="2026-10-30 01:45:46.081998855 +0000 UTC m=+1671.658424256" Oct 30 01:45:46.116133 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:46.116106 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0ae258-k6wjd"] Oct 30 01:45:46.158847 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:45:46.158801 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod822d8556_80de_405f_a0cf_8ab09bfbca3e.slice/crio-56047540a48e9fae7a4f9294afc67eb7681c9417cd74373d7f372d4720ecdd1a WatchSource:0}: Error finding container 56047540a48e9fae7a4f9294afc67eb7681c9417cd74373d7f372d4720ecdd1a: Status 404 returned error can't find the container with id 56047540a48e9fae7a4f9294afc67eb7681c9417cd74373d7f372d4720ecdd1a Oct 30 01:45:46.676873 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:46.676709 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" Oct 30 01:45:46.677533 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:46.677507 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" Oct 30 01:45:46.678656 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:46.678621 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" Oct 30 01:45:47.051526 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:47.051445 6572 generic.go:356] "Generic (PLEG): container finished" podID="c8e92603-8f43-4815-bc95-1b88ab8dd06c" containerID="84b96e42b94e8028f8ea8dec77368ba88ffb039b95f844aad402ad355905a608" exitCode=0 Oct 30 01:45:47.051965 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:47.051533 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c26448-th5p8" event={"ID":"c8e92603-8f43-4815-bc95-1b88ab8dd06c","Type":"ContainerDied","Data":"84b96e42b94e8028f8ea8dec77368ba88ffb039b95f844aad402ad355905a608"} Oct 30 01:45:47.053383 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:47.053351 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0ae258-k6wjd" event={"ID":"822d8556-80de-405f-a0cf-8ab09bfbca3e","Type":"ContainerStarted","Data":"2c9e598321b8a25383fd3353dc908dc21cdeede678c0588f0b7e9d053b2f73cf"} Oct 30 01:45:47.053503 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:47.053403 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0ae258-k6wjd" event={"ID":"822d8556-80de-405f-a0cf-8ab09bfbca3e","Type":"ContainerStarted","Data":"56047540a48e9fae7a4f9294afc67eb7681c9417cd74373d7f372d4720ecdd1a"} Oct 30 01:45:47.137938 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:47.137884 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0ae258-k6wjd" podStartSLOduration=2.137869371 podStartE2EDuration="2.137869371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:47.133040958 +0000 UTC m=+1672.709466361" watchObservedRunningTime="2026-10-30 01:45:47.137869371 +0000 UTC m=+1672.714294750" Oct 30 01:45:47.389491 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:47.389455 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:48.064320 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:48.064287 6572 generic.go:356] "Generic (PLEG): container finished" podID="822d8556-80de-405f-a0cf-8ab09bfbca3e" containerID="2c9e598321b8a25383fd3353dc908dc21cdeede678c0588f0b7e9d053b2f73cf" exitCode=0 Oct 30 01:45:48.064774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:48.064378 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0ae258-k6wjd" event={"ID":"822d8556-80de-405f-a0cf-8ab09bfbca3e","Type":"ContainerDied","Data":"2c9e598321b8a25383fd3353dc908dc21cdeede678c0588f0b7e9d053b2f73cf"} Oct 30 01:45:48.227950 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:48.227925 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c26448-th5p8" Oct 30 01:45:48.385455 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:48.385373 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8e92603-8f43-4815-bc95-1b88ab8dd06c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8e92603-8f43-4815-bc95-1b88ab8dd06c-maas-api-openshift-ca\") pod \"c8e92603-8f43-4815-bc95-1b88ab8dd06c\" (UID: \"c8e92603-8f43-4815-bc95-1b88ab8dd06c\") " Oct 30 01:45:48.385455 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:48.385450 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8e92603-8f43-4815-bc95-1b88ab8dd06c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8e92603-8f43-4815-bc95-1b88ab8dd06c-maas-api-xks-ca\") pod \"c8e92603-8f43-4815-bc95-1b88ab8dd06c\" (UID: \"c8e92603-8f43-4815-bc95-1b88ab8dd06c\") " Oct 30 01:45:48.385708 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:48.385535 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8e92603-8f43-4815-bc95-1b88ab8dd06c-kube-api-access-c49g7\" (UniqueName: \"kubernetes.io/projected/c8e92603-8f43-4815-bc95-1b88ab8dd06c-kube-api-access-c49g7\") pod \"c8e92603-8f43-4815-bc95-1b88ab8dd06c\" (UID: \"c8e92603-8f43-4815-bc95-1b88ab8dd06c\") " Oct 30 01:45:48.385708 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:48.385694 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8e92603-8f43-4815-bc95-1b88ab8dd06c-maas-api-openshift-ca" pod "c8e92603-8f43-4815-bc95-1b88ab8dd06c" (UID: "c8e92603-8f43-4815-bc95-1b88ab8dd06c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:45:48.386107 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:48.386082 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8e92603-8f43-4815-bc95-1b88ab8dd06c-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:48.387597 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:48.387568 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8e92603-8f43-4815-bc95-1b88ab8dd06c-maas-api-xks-ca" pod "c8e92603-8f43-4815-bc95-1b88ab8dd06c" (UID: "c8e92603-8f43-4815-bc95-1b88ab8dd06c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:45:48.387680 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:48.387634 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8e92603-8f43-4815-bc95-1b88ab8dd06c-kube-api-access-c49g7" pod "c8e92603-8f43-4815-bc95-1b88ab8dd06c" (UID: "c8e92603-8f43-4815-bc95-1b88ab8dd06c"). InnerVolumeSpecName "kube-api-access-c49g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:45:48.486847 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:48.486813 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8e92603-8f43-4815-bc95-1b88ab8dd06c-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:48.486847 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:48.486841 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c49g7\" (UniqueName: \"kubernetes.io/projected/c8e92603-8f43-4815-bc95-1b88ab8dd06c-kube-api-access-c49g7\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:48.647775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:48.647663 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ca28f14-26xcn"] Oct 30 01:45:48.648390 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:48.648370 6572 cpu_manager.go:414] "containerMap: removing container" podUID="c8e92603-8f43-4815-bc95-1b88ab8dd06c" containerName="revoke-keys" Oct 30 01:45:48.648492 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:48.648393 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8e92603-8f43-4815-bc95-1b88ab8dd06c" containerName="revoke-keys" Oct 30 01:45:48.648573 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:48.648560 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8e92603-8f43-4815-bc95-1b88ab8dd06c" containerName="revoke-keys" Oct 30 01:45:48.651185 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:48.651163 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca28f14-26xcn" Oct 30 01:45:48.669613 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:48.669591 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ca28f14-26xcn"] Oct 30 01:45:48.791198 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:48.791167 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ce0dba-99e6-475e-92e5-01bc7a9514c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ca28f14-26xcn\" (UID: \"08ce0dba-99e6-475e-92e5-01bc7a9514c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca28f14-26xcn" Oct 30 01:45:48.791359 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:48.791278 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc4jr\" (UniqueName: \"kubernetes.io/projected/08ce0dba-99e6-475e-92e5-01bc7a9514c9-kube-api-access-dc4jr\") pod \"maas-api-revoke-sub-1ca28f14-26xcn\" (UID: \"08ce0dba-99e6-475e-92e5-01bc7a9514c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca28f14-26xcn" Oct 30 01:45:48.791359 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:48.791335 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ce0dba-99e6-475e-92e5-01bc7a9514c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ca28f14-26xcn\" (UID: \"08ce0dba-99e6-475e-92e5-01bc7a9514c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca28f14-26xcn" Oct 30 01:45:48.892359 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:48.892318 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ce0dba-99e6-475e-92e5-01bc7a9514c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ca28f14-26xcn\" (UID: \"08ce0dba-99e6-475e-92e5-01bc7a9514c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca28f14-26xcn" Oct 30 01:45:48.892543 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:48.892424 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc4jr\" (UniqueName: \"kubernetes.io/projected/08ce0dba-99e6-475e-92e5-01bc7a9514c9-kube-api-access-dc4jr\") pod \"maas-api-revoke-sub-1ca28f14-26xcn\" (UID: \"08ce0dba-99e6-475e-92e5-01bc7a9514c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca28f14-26xcn" Oct 30 01:45:48.892543 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:48.892478 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ce0dba-99e6-475e-92e5-01bc7a9514c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ca28f14-26xcn\" (UID: \"08ce0dba-99e6-475e-92e5-01bc7a9514c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca28f14-26xcn" Oct 30 01:45:48.893042 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:48.893018 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ce0dba-99e6-475e-92e5-01bc7a9514c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ca28f14-26xcn\" (UID: \"08ce0dba-99e6-475e-92e5-01bc7a9514c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca28f14-26xcn" Oct 30 01:45:48.894828 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:48.894809 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ce0dba-99e6-475e-92e5-01bc7a9514c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ca28f14-26xcn\" (UID: \"08ce0dba-99e6-475e-92e5-01bc7a9514c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca28f14-26xcn" Oct 30 01:45:48.910969 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:48.910945 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc4jr\" (UniqueName: \"kubernetes.io/projected/08ce0dba-99e6-475e-92e5-01bc7a9514c9-kube-api-access-dc4jr\") pod \"maas-api-revoke-sub-1ca28f14-26xcn\" (UID: \"08ce0dba-99e6-475e-92e5-01bc7a9514c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca28f14-26xcn" Oct 30 01:45:48.963285 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:48.963250 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca28f14-26xcn" Oct 30 01:45:49.076633 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:49.076253 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c26448-th5p8" event={"ID":"c8e92603-8f43-4815-bc95-1b88ab8dd06c","Type":"ContainerDied","Data":"45546da61ae86c9cb8b6ac9b91268b743de230771a6c1ee073425564195ce3a7"} Oct 30 01:45:49.076633 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:49.076318 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45546da61ae86c9cb8b6ac9b91268b743de230771a6c1ee073425564195ce3a7" Oct 30 01:45:49.076633 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:49.076500 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c26448-th5p8" Oct 30 01:45:49.240364 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:49.234392 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:49.271647 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:45:49.271618 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08ce0dba_99e6_475e_92e5_01bc7a9514c9.slice/crio-5ecb0db02f08792b64287ae935744177a870ff478ae8de0c9b966c5766c37841 WatchSource:0}: Error finding container 5ecb0db02f08792b64287ae935744177a870ff478ae8de0c9b966c5766c37841: Status 404 returned error can't find the container with id 5ecb0db02f08792b64287ae935744177a870ff478ae8de0c9b966c5766c37841 Oct 30 01:45:49.272150 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:49.272126 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ca28f14-26xcn"] Oct 30 01:45:49.272685 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:49.272663 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0ae258-k6wjd" Oct 30 01:45:49.401447 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:49.401413 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/822d8556-80de-405f-a0cf-8ab09bfbca3e-kube-api-access-q77jv\" (UniqueName: \"kubernetes.io/projected/822d8556-80de-405f-a0cf-8ab09bfbca3e-kube-api-access-q77jv\") pod \"822d8556-80de-405f-a0cf-8ab09bfbca3e\" (UID: \"822d8556-80de-405f-a0cf-8ab09bfbca3e\") " Oct 30 01:45:49.401615 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:49.401509 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/822d8556-80de-405f-a0cf-8ab09bfbca3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/822d8556-80de-405f-a0cf-8ab09bfbca3e-maas-api-xks-ca\") pod \"822d8556-80de-405f-a0cf-8ab09bfbca3e\" (UID: \"822d8556-80de-405f-a0cf-8ab09bfbca3e\") " Oct 30 01:45:49.401615 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:49.401547 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/822d8556-80de-405f-a0cf-8ab09bfbca3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/822d8556-80de-405f-a0cf-8ab09bfbca3e-maas-api-openshift-ca\") pod \"822d8556-80de-405f-a0cf-8ab09bfbca3e\" (UID: \"822d8556-80de-405f-a0cf-8ab09bfbca3e\") " Oct 30 01:45:49.402008 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:49.401978 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/822d8556-80de-405f-a0cf-8ab09bfbca3e-maas-api-openshift-ca" pod "822d8556-80de-405f-a0cf-8ab09bfbca3e" (UID: "822d8556-80de-405f-a0cf-8ab09bfbca3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:45:49.402151 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:49.402095 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/822d8556-80de-405f-a0cf-8ab09bfbca3e-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:49.403616 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:49.403594 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/822d8556-80de-405f-a0cf-8ab09bfbca3e-maas-api-xks-ca" pod "822d8556-80de-405f-a0cf-8ab09bfbca3e" (UID: "822d8556-80de-405f-a0cf-8ab09bfbca3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:45:49.403616 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:49.403605 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/822d8556-80de-405f-a0cf-8ab09bfbca3e-kube-api-access-q77jv" pod "822d8556-80de-405f-a0cf-8ab09bfbca3e" (UID: "822d8556-80de-405f-a0cf-8ab09bfbca3e"). InnerVolumeSpecName "kube-api-access-q77jv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:45:49.503457 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:49.503371 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q77jv\" (UniqueName: \"kubernetes.io/projected/822d8556-80de-405f-a0cf-8ab09bfbca3e-kube-api-access-q77jv\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:49.503457 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:49.503408 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/822d8556-80de-405f-a0cf-8ab09bfbca3e-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:50.086585 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:50.086556 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0ae258-k6wjd" Oct 30 01:45:50.087122 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:50.086591 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0ae258-k6wjd" event={"ID":"822d8556-80de-405f-a0cf-8ab09bfbca3e","Type":"ContainerDied","Data":"56047540a48e9fae7a4f9294afc67eb7681c9417cd74373d7f372d4720ecdd1a"} Oct 30 01:45:50.087122 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:50.086648 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56047540a48e9fae7a4f9294afc67eb7681c9417cd74373d7f372d4720ecdd1a" Oct 30 01:45:50.088579 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:50.088549 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca28f14-26xcn" event={"ID":"08ce0dba-99e6-475e-92e5-01bc7a9514c9","Type":"ContainerStarted","Data":"b910cad64710bdef0c68bacd8d4ed1ce4f5fe2e1d44639df0f9b2c5f4925b64a"} Oct 30 01:45:50.088722 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:50.088600 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca28f14-26xcn" event={"ID":"08ce0dba-99e6-475e-92e5-01bc7a9514c9","Type":"ContainerStarted","Data":"5ecb0db02f08792b64287ae935744177a870ff478ae8de0c9b966c5766c37841"} Oct 30 01:45:50.128582 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:50.128524 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca28f14-26xcn" podStartSLOduration=2.128508929 podStartE2EDuration="2.128508929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:50.125547332 +0000 UTC m=+1675.701972735" watchObservedRunningTime="2026-10-30 01:45:50.128508929 +0000 UTC m=+1675.704934331" Oct 30 01:45:50.965276 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:50.965242 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:51.099442 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:51.099402 6572 generic.go:356] "Generic (PLEG): container finished" podID="08ce0dba-99e6-475e-92e5-01bc7a9514c9" containerID="b910cad64710bdef0c68bacd8d4ed1ce4f5fe2e1d44639df0f9b2c5f4925b64a" exitCode=0 Oct 30 01:45:51.099903 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:51.099485 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca28f14-26xcn" event={"ID":"08ce0dba-99e6-475e-92e5-01bc7a9514c9","Type":"ContainerDied","Data":"b910cad64710bdef0c68bacd8d4ed1ce4f5fe2e1d44639df0f9b2c5f4925b64a"} Oct 30 01:45:52.255431 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:52.255405 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca28f14-26xcn" Oct 30 01:45:52.442915 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:52.442881 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08ce0dba-99e6-475e-92e5-01bc7a9514c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ce0dba-99e6-475e-92e5-01bc7a9514c9-maas-api-xks-ca\") pod \"08ce0dba-99e6-475e-92e5-01bc7a9514c9\" (UID: \"08ce0dba-99e6-475e-92e5-01bc7a9514c9\") " Oct 30 01:45:52.443121 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:52.443015 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08ce0dba-99e6-475e-92e5-01bc7a9514c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ce0dba-99e6-475e-92e5-01bc7a9514c9-maas-api-openshift-ca\") pod \"08ce0dba-99e6-475e-92e5-01bc7a9514c9\" (UID: \"08ce0dba-99e6-475e-92e5-01bc7a9514c9\") " Oct 30 01:45:52.443121 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:52.443112 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08ce0dba-99e6-475e-92e5-01bc7a9514c9-kube-api-access-dc4jr\" (UniqueName: \"kubernetes.io/projected/08ce0dba-99e6-475e-92e5-01bc7a9514c9-kube-api-access-dc4jr\") pod \"08ce0dba-99e6-475e-92e5-01bc7a9514c9\" (UID: \"08ce0dba-99e6-475e-92e5-01bc7a9514c9\") " Oct 30 01:45:52.443366 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:52.443322 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08ce0dba-99e6-475e-92e5-01bc7a9514c9-maas-api-openshift-ca" pod "08ce0dba-99e6-475e-92e5-01bc7a9514c9" (UID: "08ce0dba-99e6-475e-92e5-01bc7a9514c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:45:52.443469 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:52.443459 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ce0dba-99e6-475e-92e5-01bc7a9514c9-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:52.445142 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:52.445108 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08ce0dba-99e6-475e-92e5-01bc7a9514c9-maas-api-xks-ca" pod "08ce0dba-99e6-475e-92e5-01bc7a9514c9" (UID: "08ce0dba-99e6-475e-92e5-01bc7a9514c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:45:52.445262 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:52.445234 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ce0dba-99e6-475e-92e5-01bc7a9514c9-kube-api-access-dc4jr" pod "08ce0dba-99e6-475e-92e5-01bc7a9514c9" (UID: "08ce0dba-99e6-475e-92e5-01bc7a9514c9"). InnerVolumeSpecName "kube-api-access-dc4jr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:45:52.544521 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:52.544492 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc4jr\" (UniqueName: \"kubernetes.io/projected/08ce0dba-99e6-475e-92e5-01bc7a9514c9-kube-api-access-dc4jr\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:52.544521 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:52.544519 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ce0dba-99e6-475e-92e5-01bc7a9514c9-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:52.728767 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:52.728650 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:53.121688 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:53.121609 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca28f14-26xcn" Oct 30 01:45:53.121688 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:53.121648 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca28f14-26xcn" event={"ID":"08ce0dba-99e6-475e-92e5-01bc7a9514c9","Type":"ContainerDied","Data":"5ecb0db02f08792b64287ae935744177a870ff478ae8de0c9b966c5766c37841"} Oct 30 01:45:53.121907 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:53.121698 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ecb0db02f08792b64287ae935744177a870ff478ae8de0c9b966c5766c37841" Oct 30 01:45:54.398109 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:54.398059 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:55.791287 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:55.791255 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfcbcb00-5lwws"] Oct 30 01:45:55.791884 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:55.791867 6572 cpu_manager.go:414] "containerMap: removing container" podUID="08ce0dba-99e6-475e-92e5-01bc7a9514c9" containerName="revoke-keys" Oct 30 01:45:55.791884 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:55.791884 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08ce0dba-99e6-475e-92e5-01bc7a9514c9" containerName="revoke-keys" Oct 30 01:45:55.792020 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:55.791902 6572 cpu_manager.go:414] "containerMap: removing container" podUID="822d8556-80de-405f-a0cf-8ab09bfbca3e" containerName="revoke-keys" Oct 30 01:45:55.792020 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:55.791908 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="822d8556-80de-405f-a0cf-8ab09bfbca3e" containerName="revoke-keys" Oct 30 01:45:55.792020 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:55.792016 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="08ce0dba-99e6-475e-92e5-01bc7a9514c9" containerName="revoke-keys" Oct 30 01:45:55.792172 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:55.792026 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="822d8556-80de-405f-a0cf-8ab09bfbca3e" containerName="revoke-keys" Oct 30 01:45:55.795256 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:55.795233 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfcbcb00-5lwws" Oct 30 01:45:55.814801 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:55.814774 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfcbcb00-5lwws"] Oct 30 01:45:55.820436 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:55.820416 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:45:55.820593 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:55.820571 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:45:55.820593 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:55.820473 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:45:55.821555 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:55.821529 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:45:55.883395 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:55.883362 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beb287c9-9698-483d-b3f3-b749388f0eb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfcbcb00-5lwws\" (UID: \"beb287c9-9698-483d-b3f3-b749388f0eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfcbcb00-5lwws" Oct 30 01:45:55.883603 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:55.883403 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsxv9\" (UniqueName: \"kubernetes.io/projected/beb287c9-9698-483d-b3f3-b749388f0eb5-kube-api-access-nsxv9\") pod \"maas-api-revoke-sub-bfcbcb00-5lwws\" (UID: \"beb287c9-9698-483d-b3f3-b749388f0eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfcbcb00-5lwws" Oct 30 01:45:55.883603 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:55.883551 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beb287c9-9698-483d-b3f3-b749388f0eb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfcbcb00-5lwws\" (UID: \"beb287c9-9698-483d-b3f3-b749388f0eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfcbcb00-5lwws" Oct 30 01:45:55.923326 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:55.923287 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0ae258-7889569bbb-k695k"] Oct 30 01:45:55.985625 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:55.985582 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beb287c9-9698-483d-b3f3-b749388f0eb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfcbcb00-5lwws\" (UID: \"beb287c9-9698-483d-b3f3-b749388f0eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfcbcb00-5lwws" Oct 30 01:45:55.985860 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:55.985829 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beb287c9-9698-483d-b3f3-b749388f0eb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfcbcb00-5lwws\" (UID: \"beb287c9-9698-483d-b3f3-b749388f0eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfcbcb00-5lwws" Oct 30 01:45:55.985961 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:55.985876 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsxv9\" (UniqueName: \"kubernetes.io/projected/beb287c9-9698-483d-b3f3-b749388f0eb5-kube-api-access-nsxv9\") pod \"maas-api-revoke-sub-bfcbcb00-5lwws\" (UID: \"beb287c9-9698-483d-b3f3-b749388f0eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfcbcb00-5lwws" Oct 30 01:45:55.986301 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:55.986271 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beb287c9-9698-483d-b3f3-b749388f0eb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfcbcb00-5lwws\" (UID: \"beb287c9-9698-483d-b3f3-b749388f0eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfcbcb00-5lwws" Oct 30 01:45:55.988327 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:55.988298 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beb287c9-9698-483d-b3f3-b749388f0eb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfcbcb00-5lwws\" (UID: \"beb287c9-9698-483d-b3f3-b749388f0eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfcbcb00-5lwws" Oct 30 01:45:56.006434 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:56.006404 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsxv9\" (UniqueName: \"kubernetes.io/projected/beb287c9-9698-483d-b3f3-b749388f0eb5-kube-api-access-nsxv9\") pod \"maas-api-revoke-sub-bfcbcb00-5lwws\" (UID: \"beb287c9-9698-483d-b3f3-b749388f0eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfcbcb00-5lwws" Oct 30 01:45:56.108362 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:56.108278 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfcbcb00-5lwws" Oct 30 01:45:56.143419 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:56.143379 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:56.323249 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:56.323209 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:56.406771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:56.406730 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfcbcb00-5lwws"] Oct 30 01:45:56.407178 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:45:56.407158 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeb287c9_9698_483d_b3f3_b749388f0eb5.slice/crio-593576ff291fd59f9aa179e2a9a7a867ca62b21d6eef9e4b61bb07c62d692910 WatchSource:0}: Error finding container 593576ff291fd59f9aa179e2a9a7a867ca62b21d6eef9e4b61bb07c62d692910: Status 404 returned error can't find the container with id 593576ff291fd59f9aa179e2a9a7a867ca62b21d6eef9e4b61bb07c62d692910 Oct 30 01:45:56.529893 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:56.529859 6572 scope.go:122] "RemoveContainer" containerID="e860ab11401c78a99f213c39f97a5f308e3db9759277e120bc1d6e01cabf44df" Oct 30 01:45:57.157023 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:57.156937 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfcbcb00-5lwws" event={"ID":"beb287c9-9698-483d-b3f3-b749388f0eb5","Type":"ContainerStarted","Data":"616f1dc1812df540ed67d900985414e07756855907f12993b9433ba76f5f8ee9"} Oct 30 01:45:57.157023 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:57.156994 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfcbcb00-5lwws" event={"ID":"beb287c9-9698-483d-b3f3-b749388f0eb5","Type":"ContainerStarted","Data":"593576ff291fd59f9aa179e2a9a7a867ca62b21d6eef9e4b61bb07c62d692910"} Oct 30 01:45:57.195035 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:57.194986 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfcbcb00-5lwws" podStartSLOduration=2.194972616 podStartE2EDuration="2.194972616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:57.193412363 +0000 UTC m=+1682.769837765" watchObservedRunningTime="2026-10-30 01:45:57.194972616 +0000 UTC m=+1682.771398018" Oct 30 01:45:58.003278 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:58.003241 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:58.165928 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:58.165890 6572 generic.go:356] "Generic (PLEG): container finished" podID="beb287c9-9698-483d-b3f3-b749388f0eb5" containerID="616f1dc1812df540ed67d900985414e07756855907f12993b9433ba76f5f8ee9" exitCode=0 Oct 30 01:45:58.166356 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:58.165968 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfcbcb00-5lwws" event={"ID":"beb287c9-9698-483d-b3f3-b749388f0eb5","Type":"ContainerDied","Data":"616f1dc1812df540ed67d900985414e07756855907f12993b9433ba76f5f8ee9"} Oct 30 01:45:59.342698 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:59.342674 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfcbcb00-5lwws" Oct 30 01:45:59.441401 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:59.441314 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/beb287c9-9698-483d-b3f3-b749388f0eb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beb287c9-9698-483d-b3f3-b749388f0eb5-maas-api-xks-ca\") pod \"beb287c9-9698-483d-b3f3-b749388f0eb5\" (UID: \"beb287c9-9698-483d-b3f3-b749388f0eb5\") " Oct 30 01:45:59.441564 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:59.441413 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beb287c9-9698-483d-b3f3-b749388f0eb5-kube-api-access-nsxv9\" (UniqueName: \"kubernetes.io/projected/beb287c9-9698-483d-b3f3-b749388f0eb5-kube-api-access-nsxv9\") pod \"beb287c9-9698-483d-b3f3-b749388f0eb5\" (UID: \"beb287c9-9698-483d-b3f3-b749388f0eb5\") " Oct 30 01:45:59.441685 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:59.441665 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beb287c9-9698-483d-b3f3-b749388f0eb5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beb287c9-9698-483d-b3f3-b749388f0eb5-maas-api-openshift-ca\") pod \"beb287c9-9698-483d-b3f3-b749388f0eb5\" (UID: \"beb287c9-9698-483d-b3f3-b749388f0eb5\") " Oct 30 01:45:59.442101 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:59.442068 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beb287c9-9698-483d-b3f3-b749388f0eb5-maas-api-openshift-ca" pod "beb287c9-9698-483d-b3f3-b749388f0eb5" (UID: "beb287c9-9698-483d-b3f3-b749388f0eb5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:45:59.444034 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:59.444006 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beb287c9-9698-483d-b3f3-b749388f0eb5-maas-api-xks-ca" pod "beb287c9-9698-483d-b3f3-b749388f0eb5" (UID: "beb287c9-9698-483d-b3f3-b749388f0eb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:45:59.444140 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:59.444086 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beb287c9-9698-483d-b3f3-b749388f0eb5-kube-api-access-nsxv9" pod "beb287c9-9698-483d-b3f3-b749388f0eb5" (UID: "beb287c9-9698-483d-b3f3-b749388f0eb5"). InnerVolumeSpecName "kube-api-access-nsxv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:45:59.544280 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:59.544243 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beb287c9-9698-483d-b3f3-b749388f0eb5-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:59.544280 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:59.544279 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beb287c9-9698-483d-b3f3-b749388f0eb5-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:59.544497 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:59.544294 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsxv9\" (UniqueName: \"kubernetes.io/projected/beb287c9-9698-483d-b3f3-b749388f0eb5-kube-api-access-nsxv9\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:45:59.754876 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:45:59.754778 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:00.279774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:00.261691 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfcbcb00-5lwws" event={"ID":"beb287c9-9698-483d-b3f3-b749388f0eb5","Type":"ContainerDied","Data":"593576ff291fd59f9aa179e2a9a7a867ca62b21d6eef9e4b61bb07c62d692910"} Oct 30 01:46:00.279774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:00.261783 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="593576ff291fd59f9aa179e2a9a7a867ca62b21d6eef9e4b61bb07c62d692910" Oct 30 01:46:00.279774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:00.261864 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfcbcb00-5lwws" Oct 30 01:46:00.552828 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:00.550263 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:00.923856 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:00.923814 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-0ae258-7889569bbb-k695k" podUID="5b13d20c-ba82-40b2-bffe-47e7ea20c922" containerName="payload-processing" containerID="cri-o://2bb011c0be0f492e595654773c1ebf7898538c99073dcc2ce923b78f21b924f2" gracePeriod=25 Oct 30 01:46:01.507903 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:01.507860 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-0ae258-7889569bbb-k695k" podUID="5b13d20c-ba82-40b2-bffe-47e7ea20c922" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:46:01.789949 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:01.789840 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" podUID="fac3ca55-64af-42d9-90b1-0e821b893282" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:01.903797 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:01.902847 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" podUID="317505ef-238c-458e-a74e-1198806b94d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:01.999772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:01.996844 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" podUID="cb20484a-51e2-4581-a629-e92d4234784e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:01.999772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:01.998082 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" podUID="e76339df-9659-4bea-80a4-5f18c4a73fd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:02.083824 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:02.073841 6572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" podUID="fac3ca55-64af-42d9-90b1-0e821b893282" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:02.097773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:02.095052 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" podUID="0ee3f8e3-9b55-4ce5-afaf-86780a26391e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:02.761985 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:02.690909 6572 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" podUID="0ee3f8e3-9b55-4ce5-afaf-86780a26391e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded" Oct 30 01:46:02.761985 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:02.761706 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:02.802120 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:02.767881 6572 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:46:02.802120 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:02.767942 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" podUID="8b24c1c0-b61b-43fc-84b0-994045057bc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:02.832966 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:02.832909 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" podUID="fac3ca55-64af-42d9-90b1-0e821b893282" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:02.906643 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:02.906325 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" podUID="317505ef-238c-458e-a74e-1198806b94d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:03.011037 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:03.011006 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" Oct 30 01:46:03.011341 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:03.011323 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eced6ac6-xp8qt"] Oct 30 01:46:03.012796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:03.012717 6572 cpu_manager.go:414] "containerMap: removing container" podUID="beb287c9-9698-483d-b3f3-b749388f0eb5" containerName="revoke-keys" Oct 30 01:46:03.013425 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:03.013406 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beb287c9-9698-483d-b3f3-b749388f0eb5" containerName="revoke-keys" Oct 30 01:46:03.014007 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:03.013897 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="beb287c9-9698-483d-b3f3-b749388f0eb5" containerName="revoke-keys" Oct 30 01:46:03.094632 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:03.094580 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" Oct 30 01:46:03.094632 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:03.094628 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eced6ac6-xp8qt"] Oct 30 01:46:03.094959 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:03.094677 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" Oct 30 01:46:03.095567 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:03.095541 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eced6ac6-xp8qt" Oct 30 01:46:03.098303 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:03.098263 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" podUID="0ee3f8e3-9b55-4ce5-afaf-86780a26391e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:03.119855 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:03.112706 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:46:03.119855 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:03.117039 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:46:03.119855 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:03.117097 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:46:03.119855 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:03.118845 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" Oct 30 01:46:03.146989 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:03.146953 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" Oct 30 01:46:03.214984 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:03.214954 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:46:03.229473 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:03.229440 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv45j\" (UniqueName: \"kubernetes.io/projected/f4525db7-9596-46b0-bfff-95c6933db9a9-kube-api-access-qv45j\") pod \"maas-api-revoke-sub-eced6ac6-xp8qt\" (UID: \"f4525db7-9596-46b0-bfff-95c6933db9a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eced6ac6-xp8qt" Oct 30 01:46:03.229784 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:03.229736 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4525db7-9596-46b0-bfff-95c6933db9a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eced6ac6-xp8qt\" (UID: \"f4525db7-9596-46b0-bfff-95c6933db9a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eced6ac6-xp8qt" Oct 30 01:46:03.230019 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:03.230001 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4525db7-9596-46b0-bfff-95c6933db9a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eced6ac6-xp8qt\" (UID: \"f4525db7-9596-46b0-bfff-95c6933db9a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eced6ac6-xp8qt" Oct 30 01:46:03.298632 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:03.298558 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" Oct 30 01:46:03.331380 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:03.331331 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv45j\" (UniqueName: \"kubernetes.io/projected/f4525db7-9596-46b0-bfff-95c6933db9a9-kube-api-access-qv45j\") pod \"maas-api-revoke-sub-eced6ac6-xp8qt\" (UID: \"f4525db7-9596-46b0-bfff-95c6933db9a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eced6ac6-xp8qt" Oct 30 01:46:03.331579 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:03.331400 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4525db7-9596-46b0-bfff-95c6933db9a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eced6ac6-xp8qt\" (UID: \"f4525db7-9596-46b0-bfff-95c6933db9a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eced6ac6-xp8qt" Oct 30 01:46:03.331579 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:03.331461 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4525db7-9596-46b0-bfff-95c6933db9a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eced6ac6-xp8qt\" (UID: \"f4525db7-9596-46b0-bfff-95c6933db9a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eced6ac6-xp8qt" Oct 30 01:46:03.334197 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:03.334153 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4525db7-9596-46b0-bfff-95c6933db9a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eced6ac6-xp8qt\" (UID: \"f4525db7-9596-46b0-bfff-95c6933db9a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eced6ac6-xp8qt" Oct 30 01:46:03.351887 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:03.351855 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4525db7-9596-46b0-bfff-95c6933db9a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eced6ac6-xp8qt\" (UID: \"f4525db7-9596-46b0-bfff-95c6933db9a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eced6ac6-xp8qt" Oct 30 01:46:03.355479 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:03.355449 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv45j\" (UniqueName: \"kubernetes.io/projected/f4525db7-9596-46b0-bfff-95c6933db9a9-kube-api-access-qv45j\") pod \"maas-api-revoke-sub-eced6ac6-xp8qt\" (UID: \"f4525db7-9596-46b0-bfff-95c6933db9a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eced6ac6-xp8qt" Oct 30 01:46:03.479149 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:03.479119 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eced6ac6-xp8qt" Oct 30 01:46:03.784677 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:03.784626 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eced6ac6-xp8qt"] Oct 30 01:46:03.784961 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:46:03.784934 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4525db7_9596_46b0_bfff_95c6933db9a9.slice/crio-6a27288b0315779e65edf3c4f21873cf5c5f536feaa93258383cbb2cb8ff89b8 WatchSource:0}: Error finding container 6a27288b0315779e65edf3c4f21873cf5c5f536feaa93258383cbb2cb8ff89b8: Status 404 returned error can't find the container with id 6a27288b0315779e65edf3c4f21873cf5c5f536feaa93258383cbb2cb8ff89b8 Oct 30 01:46:03.906179 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:03.906136 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eced6ac6-xp8qt" event={"ID":"f4525db7-9596-46b0-bfff-95c6933db9a9","Type":"ContainerStarted","Data":"6a27288b0315779e65edf3c4f21873cf5c5f536feaa93258383cbb2cb8ff89b8"} Oct 30 01:46:04.246469 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:04.246431 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:04.924237 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:04.924204 6572 generic.go:356] "Generic (PLEG): container finished" podID="f4525db7-9596-46b0-bfff-95c6933db9a9" containerID="17ae3616f188bae0e6a9937faa7e60eb266fc7c672c10f0fbf20be36e19f2b9e" exitCode=0 Oct 30 01:46:04.924625 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:04.924283 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eced6ac6-xp8qt" event={"ID":"f4525db7-9596-46b0-bfff-95c6933db9a9","Type":"ContainerDied","Data":"17ae3616f188bae0e6a9937faa7e60eb266fc7c672c10f0fbf20be36e19f2b9e"} Oct 30 01:46:05.218779 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:05.218733 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:05.406172 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:05.406137 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:06.082898 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:06.082875 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eced6ac6-xp8qt" Oct 30 01:46:06.172842 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:06.172813 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4525db7-9596-46b0-bfff-95c6933db9a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4525db7-9596-46b0-bfff-95c6933db9a9-maas-api-openshift-ca\") pod \"f4525db7-9596-46b0-bfff-95c6933db9a9\" (UID: \"f4525db7-9596-46b0-bfff-95c6933db9a9\") " Oct 30 01:46:06.173017 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:06.172873 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4525db7-9596-46b0-bfff-95c6933db9a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4525db7-9596-46b0-bfff-95c6933db9a9-maas-api-xks-ca\") pod \"f4525db7-9596-46b0-bfff-95c6933db9a9\" (UID: \"f4525db7-9596-46b0-bfff-95c6933db9a9\") " Oct 30 01:46:06.173017 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:06.172930 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4525db7-9596-46b0-bfff-95c6933db9a9-kube-api-access-qv45j\" (UniqueName: \"kubernetes.io/projected/f4525db7-9596-46b0-bfff-95c6933db9a9-kube-api-access-qv45j\") pod \"f4525db7-9596-46b0-bfff-95c6933db9a9\" (UID: \"f4525db7-9596-46b0-bfff-95c6933db9a9\") " Oct 30 01:46:06.173253 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:06.173223 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4525db7-9596-46b0-bfff-95c6933db9a9-maas-api-openshift-ca" pod "f4525db7-9596-46b0-bfff-95c6933db9a9" (UID: "f4525db7-9596-46b0-bfff-95c6933db9a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:46:06.173379 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:06.173368 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4525db7-9596-46b0-bfff-95c6933db9a9-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:06.175000 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:06.174979 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4525db7-9596-46b0-bfff-95c6933db9a9-maas-api-xks-ca" pod "f4525db7-9596-46b0-bfff-95c6933db9a9" (UID: "f4525db7-9596-46b0-bfff-95c6933db9a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:46:06.175102 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:06.175051 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4525db7-9596-46b0-bfff-95c6933db9a9-kube-api-access-qv45j" pod "f4525db7-9596-46b0-bfff-95c6933db9a9" (UID: "f4525db7-9596-46b0-bfff-95c6933db9a9"). InnerVolumeSpecName "kube-api-access-qv45j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:46:06.274460 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:06.274425 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv45j\" (UniqueName: \"kubernetes.io/projected/f4525db7-9596-46b0-bfff-95c6933db9a9-kube-api-access-qv45j\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:06.274460 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:06.274453 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4525db7-9596-46b0-bfff-95c6933db9a9-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:06.942352 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:06.942322 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eced6ac6-xp8qt" event={"ID":"f4525db7-9596-46b0-bfff-95c6933db9a9","Type":"ContainerDied","Data":"6a27288b0315779e65edf3c4f21873cf5c5f536feaa93258383cbb2cb8ff89b8"} Oct 30 01:46:06.942352 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:06.942340 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eced6ac6-xp8qt" Oct 30 01:46:06.942560 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:06.942370 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a27288b0315779e65edf3c4f21873cf5c5f536feaa93258383cbb2cb8ff89b8" Oct 30 01:46:07.219038 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:07.218962 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:08.051910 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:08.051879 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33da9fb8-8sxpg"] Oct 30 01:46:08.052502 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:08.052488 6572 cpu_manager.go:414] "containerMap: removing container" podUID="f4525db7-9596-46b0-bfff-95c6933db9a9" containerName="revoke-keys" Oct 30 01:46:08.052562 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:08.052503 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4525db7-9596-46b0-bfff-95c6933db9a9" containerName="revoke-keys" Oct 30 01:46:08.052612 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:08.052603 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4525db7-9596-46b0-bfff-95c6933db9a9" containerName="revoke-keys" Oct 30 01:46:08.056414 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:08.056394 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33da9fb8-8sxpg" Oct 30 01:46:08.065380 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:08.065353 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:08.078587 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:08.078560 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33da9fb8-8sxpg"] Oct 30 01:46:08.081492 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:08.081468 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:46:08.081667 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:08.081499 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:46:08.082134 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:08.082113 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:46:08.082248 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:08.082161 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:46:08.198636 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:08.198605 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c227494-ff4c-4c0a-8170-8bce3a7b0f1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33da9fb8-8sxpg\" (UID: \"0c227494-ff4c-4c0a-8170-8bce3a7b0f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33da9fb8-8sxpg" Oct 30 01:46:08.198837 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:08.198665 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68nsr\" (UniqueName: \"kubernetes.io/projected/0c227494-ff4c-4c0a-8170-8bce3a7b0f1e-kube-api-access-68nsr\") pod \"maas-api-revoke-sub-33da9fb8-8sxpg\" (UID: \"0c227494-ff4c-4c0a-8170-8bce3a7b0f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33da9fb8-8sxpg" Oct 30 01:46:08.198837 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:08.198796 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c227494-ff4c-4c0a-8170-8bce3a7b0f1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33da9fb8-8sxpg\" (UID: \"0c227494-ff4c-4c0a-8170-8bce3a7b0f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33da9fb8-8sxpg" Oct 30 01:46:08.299664 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:08.299628 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c227494-ff4c-4c0a-8170-8bce3a7b0f1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33da9fb8-8sxpg\" (UID: \"0c227494-ff4c-4c0a-8170-8bce3a7b0f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33da9fb8-8sxpg" Oct 30 01:46:08.300160 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:08.299689 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68nsr\" (UniqueName: \"kubernetes.io/projected/0c227494-ff4c-4c0a-8170-8bce3a7b0f1e-kube-api-access-68nsr\") pod \"maas-api-revoke-sub-33da9fb8-8sxpg\" (UID: \"0c227494-ff4c-4c0a-8170-8bce3a7b0f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33da9fb8-8sxpg" Oct 30 01:46:08.300160 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:08.299740 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c227494-ff4c-4c0a-8170-8bce3a7b0f1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33da9fb8-8sxpg\" (UID: \"0c227494-ff4c-4c0a-8170-8bce3a7b0f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33da9fb8-8sxpg" Oct 30 01:46:08.300467 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:08.300444 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c227494-ff4c-4c0a-8170-8bce3a7b0f1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33da9fb8-8sxpg\" (UID: \"0c227494-ff4c-4c0a-8170-8bce3a7b0f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33da9fb8-8sxpg" Oct 30 01:46:08.302110 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:08.302065 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c227494-ff4c-4c0a-8170-8bce3a7b0f1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33da9fb8-8sxpg\" (UID: \"0c227494-ff4c-4c0a-8170-8bce3a7b0f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33da9fb8-8sxpg" Oct 30 01:46:08.319101 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:08.319076 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68nsr\" (UniqueName: \"kubernetes.io/projected/0c227494-ff4c-4c0a-8170-8bce3a7b0f1e-kube-api-access-68nsr\") pod \"maas-api-revoke-sub-33da9fb8-8sxpg\" (UID: \"0c227494-ff4c-4c0a-8170-8bce3a7b0f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33da9fb8-8sxpg" Oct 30 01:46:08.382771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:08.379950 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33da9fb8-8sxpg" Oct 30 01:46:08.683992 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:08.683960 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33da9fb8-8sxpg"] Oct 30 01:46:08.684605 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:46:08.684578 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c227494_ff4c_4c0a_8170_8bce3a7b0f1e.slice/crio-c5cfac47ded8a3e5a5e68bd7ebb97a3a65a1e03d0ed7102e997255f7aac7253b WatchSource:0}: Error finding container c5cfac47ded8a3e5a5e68bd7ebb97a3a65a1e03d0ed7102e997255f7aac7253b: Status 404 returned error can't find the container with id c5cfac47ded8a3e5a5e68bd7ebb97a3a65a1e03d0ed7102e997255f7aac7253b Oct 30 01:46:08.906851 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:08.906815 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:08.959951 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:08.959865 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33da9fb8-8sxpg" event={"ID":"0c227494-ff4c-4c0a-8170-8bce3a7b0f1e","Type":"ContainerStarted","Data":"c5cfac47ded8a3e5a5e68bd7ebb97a3a65a1e03d0ed7102e997255f7aac7253b"} Oct 30 01:46:09.253739 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:09.253654 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9657f785-lw8pf"] Oct 30 01:46:09.258124 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:09.258107 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657f785-lw8pf" Oct 30 01:46:09.276184 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:09.276158 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9657f785-lw8pf"] Oct 30 01:46:09.311007 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:09.310972 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n9dp\" (UniqueName: \"kubernetes.io/projected/7b04e00c-7278-4024-9352-71bea03a9b7e-kube-api-access-4n9dp\") pod \"maas-api-revoke-sub-9657f785-lw8pf\" (UID: \"7b04e00c-7278-4024-9352-71bea03a9b7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657f785-lw8pf" Oct 30 01:46:09.311498 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:09.311050 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b04e00c-7278-4024-9352-71bea03a9b7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9657f785-lw8pf\" (UID: \"7b04e00c-7278-4024-9352-71bea03a9b7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657f785-lw8pf" Oct 30 01:46:09.311498 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:09.311239 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b04e00c-7278-4024-9352-71bea03a9b7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9657f785-lw8pf\" (UID: \"7b04e00c-7278-4024-9352-71bea03a9b7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657f785-lw8pf" Oct 30 01:46:09.412247 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:09.412212 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b04e00c-7278-4024-9352-71bea03a9b7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9657f785-lw8pf\" (UID: \"7b04e00c-7278-4024-9352-71bea03a9b7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657f785-lw8pf" Oct 30 01:46:09.412436 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:09.412312 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n9dp\" (UniqueName: \"kubernetes.io/projected/7b04e00c-7278-4024-9352-71bea03a9b7e-kube-api-access-4n9dp\") pod \"maas-api-revoke-sub-9657f785-lw8pf\" (UID: \"7b04e00c-7278-4024-9352-71bea03a9b7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657f785-lw8pf" Oct 30 01:46:09.412436 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:09.412392 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b04e00c-7278-4024-9352-71bea03a9b7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9657f785-lw8pf\" (UID: \"7b04e00c-7278-4024-9352-71bea03a9b7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657f785-lw8pf" Oct 30 01:46:09.413048 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:09.413026 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b04e00c-7278-4024-9352-71bea03a9b7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9657f785-lw8pf\" (UID: \"7b04e00c-7278-4024-9352-71bea03a9b7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657f785-lw8pf" Oct 30 01:46:09.414755 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:09.414728 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b04e00c-7278-4024-9352-71bea03a9b7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9657f785-lw8pf\" (UID: \"7b04e00c-7278-4024-9352-71bea03a9b7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657f785-lw8pf" Oct 30 01:46:09.431679 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:09.431654 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n9dp\" (UniqueName: \"kubernetes.io/projected/7b04e00c-7278-4024-9352-71bea03a9b7e-kube-api-access-4n9dp\") pod \"maas-api-revoke-sub-9657f785-lw8pf\" (UID: \"7b04e00c-7278-4024-9352-71bea03a9b7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657f785-lw8pf" Oct 30 01:46:09.572804 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:09.572705 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657f785-lw8pf" Oct 30 01:46:09.615605 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:09.615574 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f197b1-57tfx"] Oct 30 01:46:09.621784 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:09.621741 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f197b1-57tfx" Oct 30 01:46:09.650266 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:09.650172 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f197b1-57tfx"] Oct 30 01:46:09.715935 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:09.715907 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e314b459-a94a-4883-a70d-899f905b48d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f197b1-57tfx\" (UID: \"e314b459-a94a-4883-a70d-899f905b48d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f197b1-57tfx" Oct 30 01:46:09.716128 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:09.716104 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e314b459-a94a-4883-a70d-899f905b48d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f197b1-57tfx\" (UID: \"e314b459-a94a-4883-a70d-899f905b48d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f197b1-57tfx" Oct 30 01:46:09.716228 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:09.716212 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brdhk\" (UniqueName: \"kubernetes.io/projected/e314b459-a94a-4883-a70d-899f905b48d2-kube-api-access-brdhk\") pod \"maas-api-revoke-keys-e2e-praxis-a-f197b1-57tfx\" (UID: \"e314b459-a94a-4883-a70d-899f905b48d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f197b1-57tfx" Oct 30 01:46:09.817315 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:09.817284 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e314b459-a94a-4883-a70d-899f905b48d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f197b1-57tfx\" (UID: \"e314b459-a94a-4883-a70d-899f905b48d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f197b1-57tfx" Oct 30 01:46:09.817504 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:09.817473 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e314b459-a94a-4883-a70d-899f905b48d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f197b1-57tfx\" (UID: \"e314b459-a94a-4883-a70d-899f905b48d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f197b1-57tfx" Oct 30 01:46:09.817590 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:09.817574 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brdhk\" (UniqueName: \"kubernetes.io/projected/e314b459-a94a-4883-a70d-899f905b48d2-kube-api-access-brdhk\") pod \"maas-api-revoke-keys-e2e-praxis-a-f197b1-57tfx\" (UID: \"e314b459-a94a-4883-a70d-899f905b48d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f197b1-57tfx" Oct 30 01:46:09.818062 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:09.818039 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e314b459-a94a-4883-a70d-899f905b48d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f197b1-57tfx\" (UID: \"e314b459-a94a-4883-a70d-899f905b48d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f197b1-57tfx" Oct 30 01:46:09.819947 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:09.819927 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e314b459-a94a-4883-a70d-899f905b48d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f197b1-57tfx\" (UID: \"e314b459-a94a-4883-a70d-899f905b48d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f197b1-57tfx" Oct 30 01:46:09.836520 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:09.836468 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brdhk\" (UniqueName: \"kubernetes.io/projected/e314b459-a94a-4883-a70d-899f905b48d2-kube-api-access-brdhk\") pod \"maas-api-revoke-keys-e2e-praxis-a-f197b1-57tfx\" (UID: \"e314b459-a94a-4883-a70d-899f905b48d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f197b1-57tfx" Oct 30 01:46:09.891267 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:46:09.891235 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b04e00c_7278_4024_9352_71bea03a9b7e.slice/crio-db22ae5ca56cf09269d1b1248fd4eb8a7b06422dc9f1709c333e347592f2ad3c WatchSource:0}: Error finding container db22ae5ca56cf09269d1b1248fd4eb8a7b06422dc9f1709c333e347592f2ad3c: Status 404 returned error can't find the container with id db22ae5ca56cf09269d1b1248fd4eb8a7b06422dc9f1709c333e347592f2ad3c Oct 30 01:46:09.891537 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:09.891516 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9657f785-lw8pf"] Oct 30 01:46:09.947800 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:09.947769 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f197b1-57tfx" Oct 30 01:46:09.971619 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:09.971590 6572 generic.go:356] "Generic (PLEG): container finished" podID="0c227494-ff4c-4c0a-8170-8bce3a7b0f1e" containerID="0d73ab548463a4ef265192ef2957aa13eea38050a8e501154f1336c41fd4c0fc" exitCode=0 Oct 30 01:46:09.971779 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:09.971672 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33da9fb8-8sxpg" event={"ID":"0c227494-ff4c-4c0a-8170-8bce3a7b0f1e","Type":"ContainerDied","Data":"0d73ab548463a4ef265192ef2957aa13eea38050a8e501154f1336c41fd4c0fc"} Oct 30 01:46:09.973287 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:09.973263 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657f785-lw8pf" event={"ID":"7b04e00c-7278-4024-9352-71bea03a9b7e","Type":"ContainerStarted","Data":"db22ae5ca56cf09269d1b1248fd4eb8a7b06422dc9f1709c333e347592f2ad3c"} Oct 30 01:46:10.254390 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:10.254347 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f197b1-57tfx"] Oct 30 01:46:10.254537 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:46:10.254499 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode314b459_a94a_4883_a70d_899f905b48d2.slice/crio-2c5eafe573011cce97b99d8ce5eb6883fcdfda78c83820ca672d5083faff3b75 WatchSource:0}: Error finding container 2c5eafe573011cce97b99d8ce5eb6883fcdfda78c83820ca672d5083faff3b75: Status 404 returned error can't find the container with id 2c5eafe573011cce97b99d8ce5eb6883fcdfda78c83820ca672d5083faff3b75 Oct 30 01:46:10.814762 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:10.814682 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:10.983204 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:10.983164 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f197b1-57tfx" event={"ID":"e314b459-a94a-4883-a70d-899f905b48d2","Type":"ContainerStarted","Data":"833e496ef6a45e6b2d937b988270eda26c2ce9bf0719f206bfbbb88fcd9ffa9b"} Oct 30 01:46:10.983412 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:10.983228 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f197b1-57tfx" event={"ID":"e314b459-a94a-4883-a70d-899f905b48d2","Type":"ContainerStarted","Data":"2c5eafe573011cce97b99d8ce5eb6883fcdfda78c83820ca672d5083faff3b75"} Oct 30 01:46:10.984953 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:10.984927 6572 generic.go:356] "Generic (PLEG): container finished" podID="7b04e00c-7278-4024-9352-71bea03a9b7e" containerID="fe8ebf8f1297e302554d4a2f43292e70eda611fe93d77b2a9cc2927df9ce3d21" exitCode=0 Oct 30 01:46:10.985083 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:10.985001 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657f785-lw8pf" event={"ID":"7b04e00c-7278-4024-9352-71bea03a9b7e","Type":"ContainerDied","Data":"fe8ebf8f1297e302554d4a2f43292e70eda611fe93d77b2a9cc2927df9ce3d21"} Oct 30 01:46:11.024995 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:11.024943 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f197b1-57tfx" podStartSLOduration=2.024925725 podStartE2EDuration="2.024925725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:46:11.021492292 +0000 UTC m=+1696.597917692" watchObservedRunningTime="2026-10-30 01:46:11.024925725 +0000 UTC m=+1696.601351127" Oct 30 01:46:11.212481 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:11.212449 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33da9fb8-8sxpg" Oct 30 01:46:11.339729 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:11.339698 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c227494-ff4c-4c0a-8170-8bce3a7b0f1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c227494-ff4c-4c0a-8170-8bce3a7b0f1e-maas-api-openshift-ca\") pod \"0c227494-ff4c-4c0a-8170-8bce3a7b0f1e\" (UID: \"0c227494-ff4c-4c0a-8170-8bce3a7b0f1e\") " Oct 30 01:46:11.339973 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:11.339914 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c227494-ff4c-4c0a-8170-8bce3a7b0f1e-kube-api-access-68nsr\" (UniqueName: \"kubernetes.io/projected/0c227494-ff4c-4c0a-8170-8bce3a7b0f1e-kube-api-access-68nsr\") pod \"0c227494-ff4c-4c0a-8170-8bce3a7b0f1e\" (UID: \"0c227494-ff4c-4c0a-8170-8bce3a7b0f1e\") " Oct 30 01:46:11.340042 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:11.339999 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c227494-ff4c-4c0a-8170-8bce3a7b0f1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c227494-ff4c-4c0a-8170-8bce3a7b0f1e-maas-api-xks-ca\") pod \"0c227494-ff4c-4c0a-8170-8bce3a7b0f1e\" (UID: \"0c227494-ff4c-4c0a-8170-8bce3a7b0f1e\") " Oct 30 01:46:11.340100 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:11.340065 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c227494-ff4c-4c0a-8170-8bce3a7b0f1e-maas-api-openshift-ca" pod "0c227494-ff4c-4c0a-8170-8bce3a7b0f1e" (UID: "0c227494-ff4c-4c0a-8170-8bce3a7b0f1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:46:11.340475 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:11.340451 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c227494-ff4c-4c0a-8170-8bce3a7b0f1e-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:11.342015 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:11.341988 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c227494-ff4c-4c0a-8170-8bce3a7b0f1e-maas-api-xks-ca" pod "0c227494-ff4c-4c0a-8170-8bce3a7b0f1e" (UID: "0c227494-ff4c-4c0a-8170-8bce3a7b0f1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:46:11.342167 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:11.342146 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c227494-ff4c-4c0a-8170-8bce3a7b0f1e-kube-api-access-68nsr" pod "0c227494-ff4c-4c0a-8170-8bce3a7b0f1e" (UID: "0c227494-ff4c-4c0a-8170-8bce3a7b0f1e"). InnerVolumeSpecName "kube-api-access-68nsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:46:11.441650 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:11.441615 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68nsr\" (UniqueName: \"kubernetes.io/projected/0c227494-ff4c-4c0a-8170-8bce3a7b0f1e-kube-api-access-68nsr\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:11.441650 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:11.441644 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c227494-ff4c-4c0a-8170-8bce3a7b0f1e-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:11.473856 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:11.473819 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-0ae258-7889569bbb-k695k" podUID="5b13d20c-ba82-40b2-bffe-47e7ea20c922" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:46:11.995210 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:11.995172 6572 generic.go:356] "Generic (PLEG): container finished" podID="e314b459-a94a-4883-a70d-899f905b48d2" containerID="833e496ef6a45e6b2d937b988270eda26c2ce9bf0719f206bfbbb88fcd9ffa9b" exitCode=0 Oct 30 01:46:11.995710 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:11.995256 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f197b1-57tfx" event={"ID":"e314b459-a94a-4883-a70d-899f905b48d2","Type":"ContainerDied","Data":"833e496ef6a45e6b2d937b988270eda26c2ce9bf0719f206bfbbb88fcd9ffa9b"} Oct 30 01:46:11.996735 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:11.996713 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33da9fb8-8sxpg" Oct 30 01:46:11.996892 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:11.996788 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33da9fb8-8sxpg" event={"ID":"0c227494-ff4c-4c0a-8170-8bce3a7b0f1e","Type":"ContainerDied","Data":"c5cfac47ded8a3e5a5e68bd7ebb97a3a65a1e03d0ed7102e997255f7aac7253b"} Oct 30 01:46:11.996892 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:11.996832 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5cfac47ded8a3e5a5e68bd7ebb97a3a65a1e03d0ed7102e997255f7aac7253b" Oct 30 01:46:12.154365 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:12.154335 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657f785-lw8pf" Oct 30 01:46:12.254380 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:12.254285 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b04e00c-7278-4024-9352-71bea03a9b7e-kube-api-access-4n9dp\" (UniqueName: \"kubernetes.io/projected/7b04e00c-7278-4024-9352-71bea03a9b7e-kube-api-access-4n9dp\") pod \"7b04e00c-7278-4024-9352-71bea03a9b7e\" (UID: \"7b04e00c-7278-4024-9352-71bea03a9b7e\") " Oct 30 01:46:12.254552 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:12.254459 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b04e00c-7278-4024-9352-71bea03a9b7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b04e00c-7278-4024-9352-71bea03a9b7e-maas-api-xks-ca\") pod \"7b04e00c-7278-4024-9352-71bea03a9b7e\" (UID: \"7b04e00c-7278-4024-9352-71bea03a9b7e\") " Oct 30 01:46:12.254552 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:12.254541 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b04e00c-7278-4024-9352-71bea03a9b7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b04e00c-7278-4024-9352-71bea03a9b7e-maas-api-openshift-ca\") pod \"7b04e00c-7278-4024-9352-71bea03a9b7e\" (UID: \"7b04e00c-7278-4024-9352-71bea03a9b7e\") " Oct 30 01:46:12.254923 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:12.254897 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b04e00c-7278-4024-9352-71bea03a9b7e-maas-api-openshift-ca" pod "7b04e00c-7278-4024-9352-71bea03a9b7e" (UID: "7b04e00c-7278-4024-9352-71bea03a9b7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:46:12.255154 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:12.255135 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b04e00c-7278-4024-9352-71bea03a9b7e-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:12.256536 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:12.256516 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b04e00c-7278-4024-9352-71bea03a9b7e-kube-api-access-4n9dp" pod "7b04e00c-7278-4024-9352-71bea03a9b7e" (UID: "7b04e00c-7278-4024-9352-71bea03a9b7e"). InnerVolumeSpecName "kube-api-access-4n9dp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:46:12.256586 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:12.256524 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b04e00c-7278-4024-9352-71bea03a9b7e-maas-api-xks-ca" pod "7b04e00c-7278-4024-9352-71bea03a9b7e" (UID: "7b04e00c-7278-4024-9352-71bea03a9b7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:46:12.356325 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:12.356288 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n9dp\" (UniqueName: \"kubernetes.io/projected/7b04e00c-7278-4024-9352-71bea03a9b7e-kube-api-access-4n9dp\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:12.356325 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:12.356319 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b04e00c-7278-4024-9352-71bea03a9b7e-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:12.492214 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:12.492174 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:13.006585 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:13.006553 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657f785-lw8pf" Oct 30 01:46:13.007108 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:13.006594 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657f785-lw8pf" event={"ID":"7b04e00c-7278-4024-9352-71bea03a9b7e","Type":"ContainerDied","Data":"db22ae5ca56cf09269d1b1248fd4eb8a7b06422dc9f1709c333e347592f2ad3c"} Oct 30 01:46:13.007108 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:13.006645 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db22ae5ca56cf09269d1b1248fd4eb8a7b06422dc9f1709c333e347592f2ad3c" Oct 30 01:46:13.173707 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:13.173677 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f197b1-57tfx" Oct 30 01:46:13.275609 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:13.275527 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e314b459-a94a-4883-a70d-899f905b48d2-kube-api-access-brdhk\" (UniqueName: \"kubernetes.io/projected/e314b459-a94a-4883-a70d-899f905b48d2-kube-api-access-brdhk\") pod \"e314b459-a94a-4883-a70d-899f905b48d2\" (UID: \"e314b459-a94a-4883-a70d-899f905b48d2\") " Oct 30 01:46:13.275800 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:13.275616 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e314b459-a94a-4883-a70d-899f905b48d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e314b459-a94a-4883-a70d-899f905b48d2-maas-api-openshift-ca\") pod \"e314b459-a94a-4883-a70d-899f905b48d2\" (UID: \"e314b459-a94a-4883-a70d-899f905b48d2\") " Oct 30 01:46:13.275800 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:13.275682 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e314b459-a94a-4883-a70d-899f905b48d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e314b459-a94a-4883-a70d-899f905b48d2-maas-api-xks-ca\") pod \"e314b459-a94a-4883-a70d-899f905b48d2\" (UID: \"e314b459-a94a-4883-a70d-899f905b48d2\") " Oct 30 01:46:13.276038 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:13.276012 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e314b459-a94a-4883-a70d-899f905b48d2-maas-api-openshift-ca" pod "e314b459-a94a-4883-a70d-899f905b48d2" (UID: "e314b459-a94a-4883-a70d-899f905b48d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:46:13.276234 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:13.276218 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e314b459-a94a-4883-a70d-899f905b48d2-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:13.277830 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:13.277804 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e314b459-a94a-4883-a70d-899f905b48d2-kube-api-access-brdhk" pod "e314b459-a94a-4883-a70d-899f905b48d2" (UID: "e314b459-a94a-4883-a70d-899f905b48d2"). InnerVolumeSpecName "kube-api-access-brdhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:46:13.277830 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:13.277809 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e314b459-a94a-4883-a70d-899f905b48d2-maas-api-xks-ca" pod "e314b459-a94a-4883-a70d-899f905b48d2" (UID: "e314b459-a94a-4883-a70d-899f905b48d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:46:13.377683 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:13.377643 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brdhk\" (UniqueName: \"kubernetes.io/projected/e314b459-a94a-4883-a70d-899f905b48d2-kube-api-access-brdhk\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:13.377683 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:13.377676 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e314b459-a94a-4883-a70d-899f905b48d2-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:14.016114 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:14.016086 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f197b1-57tfx" Oct 30 01:46:14.016508 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:14.016115 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f197b1-57tfx" event={"ID":"e314b459-a94a-4883-a70d-899f905b48d2","Type":"ContainerDied","Data":"2c5eafe573011cce97b99d8ce5eb6883fcdfda78c83820ca672d5083faff3b75"} Oct 30 01:46:14.016508 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:14.016164 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c5eafe573011cce97b99d8ce5eb6883fcdfda78c83820ca672d5083faff3b75" Oct 30 01:46:14.091276 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:14.091225 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:14.687144 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:46:14.687109 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:46:16.073983 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:16.073947 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:17.724911 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:17.724874 6572 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:46:17.725546 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:17.725494 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" podUID="8b24c1c0-b61b-43fc-84b0-994045057bc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:17.810870 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:17.810823 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" podUID="9a56da7a-9b92-40e2-b08b-43cf591b5221" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 01:46:17.898880 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:17.898833 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" podUID="317505ef-238c-458e-a74e-1198806b94d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:18.001115 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:18.000154 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" podUID="cb20484a-51e2-4581-a629-e92d4234784e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:19.177923 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:19.177891 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:19.503777 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:19.503639 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68e4f632-pbv87"] Oct 30 01:46:19.506462 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:19.506435 6572 cpu_manager.go:414] "containerMap: removing container" podUID="e314b459-a94a-4883-a70d-899f905b48d2" containerName="revoke-keys" Oct 30 01:46:19.506462 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:19.506462 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e314b459-a94a-4883-a70d-899f905b48d2" containerName="revoke-keys" Oct 30 01:46:19.506619 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:19.506483 6572 cpu_manager.go:414] "containerMap: removing container" podUID="0c227494-ff4c-4c0a-8170-8bce3a7b0f1e" containerName="revoke-keys" Oct 30 01:46:19.506619 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:19.506491 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c227494-ff4c-4c0a-8170-8bce3a7b0f1e" containerName="revoke-keys" Oct 30 01:46:19.506619 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:19.506528 6572 cpu_manager.go:414] "containerMap: removing container" podUID="7b04e00c-7278-4024-9352-71bea03a9b7e" containerName="revoke-keys" Oct 30 01:46:19.506619 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:19.506536 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b04e00c-7278-4024-9352-71bea03a9b7e" containerName="revoke-keys" Oct 30 01:46:19.506762 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:19.506687 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="e314b459-a94a-4883-a70d-899f905b48d2" containerName="revoke-keys" Oct 30 01:46:19.506762 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:19.506698 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c227494-ff4c-4c0a-8170-8bce3a7b0f1e" containerName="revoke-keys" Oct 30 01:46:19.506762 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:19.506711 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b04e00c-7278-4024-9352-71bea03a9b7e" containerName="revoke-keys" Oct 30 01:46:19.511906 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:19.511879 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e4f632-pbv87" Oct 30 01:46:19.529353 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:19.529313 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68e4f632-pbv87"] Oct 30 01:46:19.531892 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:19.531855 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:46:19.532051 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:19.531908 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:46:19.532155 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:19.531985 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:46:19.537572 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:19.537550 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:46:19.546038 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:19.546015 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e1d5968-a0a2-4e66-885d-da648cffb3f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68e4f632-pbv87\" (UID: \"6e1d5968-a0a2-4e66-885d-da648cffb3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e4f632-pbv87" Oct 30 01:46:19.546198 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:19.546176 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6d5n\" (UniqueName: \"kubernetes.io/projected/6e1d5968-a0a2-4e66-885d-da648cffb3f7-kube-api-access-n6d5n\") pod \"maas-api-revoke-sub-68e4f632-pbv87\" (UID: \"6e1d5968-a0a2-4e66-885d-da648cffb3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e4f632-pbv87" Oct 30 01:46:19.546419 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:19.546398 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e1d5968-a0a2-4e66-885d-da648cffb3f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68e4f632-pbv87\" (UID: \"6e1d5968-a0a2-4e66-885d-da648cffb3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e4f632-pbv87" Oct 30 01:46:19.648520 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:19.648484 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6d5n\" (UniqueName: \"kubernetes.io/projected/6e1d5968-a0a2-4e66-885d-da648cffb3f7-kube-api-access-n6d5n\") pod \"maas-api-revoke-sub-68e4f632-pbv87\" (UID: \"6e1d5968-a0a2-4e66-885d-da648cffb3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e4f632-pbv87" Oct 30 01:46:19.648728 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:19.648558 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e1d5968-a0a2-4e66-885d-da648cffb3f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68e4f632-pbv87\" (UID: \"6e1d5968-a0a2-4e66-885d-da648cffb3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e4f632-pbv87" Oct 30 01:46:19.648728 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:19.648684 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e1d5968-a0a2-4e66-885d-da648cffb3f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68e4f632-pbv87\" (UID: \"6e1d5968-a0a2-4e66-885d-da648cffb3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e4f632-pbv87" Oct 30 01:46:19.649388 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:19.649362 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e1d5968-a0a2-4e66-885d-da648cffb3f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68e4f632-pbv87\" (UID: \"6e1d5968-a0a2-4e66-885d-da648cffb3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e4f632-pbv87" Oct 30 01:46:19.651354 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:19.651332 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e1d5968-a0a2-4e66-885d-da648cffb3f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68e4f632-pbv87\" (UID: \"6e1d5968-a0a2-4e66-885d-da648cffb3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e4f632-pbv87" Oct 30 01:46:19.672057 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:19.672036 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6d5n\" (UniqueName: \"kubernetes.io/projected/6e1d5968-a0a2-4e66-885d-da648cffb3f7-kube-api-access-n6d5n\") pod \"maas-api-revoke-sub-68e4f632-pbv87\" (UID: \"6e1d5968-a0a2-4e66-885d-da648cffb3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e4f632-pbv87" Oct 30 01:46:19.833693 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:19.833614 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e4f632-pbv87" Oct 30 01:46:20.143948 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:20.143724 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68e4f632-pbv87"] Oct 30 01:46:20.144654 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:46:20.144608 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e1d5968_a0a2_4e66_885d_da648cffb3f7.slice/crio-bf330b6132c0d0f0a8b2feb7b83add9ad92924c550a24f4d9b96784ac4432497 WatchSource:0}: Error finding container bf330b6132c0d0f0a8b2feb7b83add9ad92924c550a24f4d9b96784ac4432497: Status 404 returned error can't find the container with id bf330b6132c0d0f0a8b2feb7b83add9ad92924c550a24f4d9b96784ac4432497 Oct 30 01:46:20.195475 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:20.195447 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f197b1-6dc895f6f6-qqnvt"] Oct 30 01:46:20.439719 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:20.439607 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:20.640602 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:20.640567 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:21.168768 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:21.167983 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:21.183400 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:21.170059 6572 generic.go:356] "Generic (PLEG): container finished" podID="5b13d20c-ba82-40b2-bffe-47e7ea20c922" containerID="2bb011c0be0f492e595654773c1ebf7898538c99073dcc2ce923b78f21b924f2" exitCode=0 Oct 30 01:46:21.183400 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:21.170254 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0ae258-7889569bbb-k695k" event={"ID":"5b13d20c-ba82-40b2-bffe-47e7ea20c922","Type":"ContainerDied","Data":"2bb011c0be0f492e595654773c1ebf7898538c99073dcc2ce923b78f21b924f2"} Oct 30 01:46:21.183400 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:21.175434 6572 generic.go:356] "Generic (PLEG): container finished" podID="6e1d5968-a0a2-4e66-885d-da648cffb3f7" containerID="1a6268881b7083bbc83fe25724523d1ecf4e0a41598f4bb55c8671371361e859" exitCode=0 Oct 30 01:46:21.183400 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:21.175647 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e4f632-pbv87" event={"ID":"6e1d5968-a0a2-4e66-885d-da648cffb3f7","Type":"ContainerDied","Data":"1a6268881b7083bbc83fe25724523d1ecf4e0a41598f4bb55c8671371361e859"} Oct 30 01:46:21.183400 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:21.175689 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e4f632-pbv87" event={"ID":"6e1d5968-a0a2-4e66-885d-da648cffb3f7","Type":"ContainerStarted","Data":"bf330b6132c0d0f0a8b2feb7b83add9ad92924c550a24f4d9b96784ac4432497"} Oct 30 01:46:21.226309 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:21.226281 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0ae258-7889569bbb-k695k" Oct 30 01:46:21.360421 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:21.360370 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b13d20c-ba82-40b2-bffe-47e7ea20c922-kube-api-access-r42wk\" (UniqueName: \"kubernetes.io/projected/5b13d20c-ba82-40b2-bffe-47e7ea20c922-kube-api-access-r42wk\") pod \"5b13d20c-ba82-40b2-bffe-47e7ea20c922\" (UID: \"5b13d20c-ba82-40b2-bffe-47e7ea20c922\") " Oct 30 01:46:21.360625 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:21.360594 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b13d20c-ba82-40b2-bffe-47e7ea20c922-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b13d20c-ba82-40b2-bffe-47e7ea20c922-plugins-config-volume\") pod \"5b13d20c-ba82-40b2-bffe-47e7ea20c922\" (UID: \"5b13d20c-ba82-40b2-bffe-47e7ea20c922\") " Oct 30 01:46:21.361995 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:21.361963 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b13d20c-ba82-40b2-bffe-47e7ea20c922-plugins-config-volume" pod "5b13d20c-ba82-40b2-bffe-47e7ea20c922" (UID: "5b13d20c-ba82-40b2-bffe-47e7ea20c922"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:46:21.373134 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:21.373081 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b13d20c-ba82-40b2-bffe-47e7ea20c922-kube-api-access-r42wk" pod "5b13d20c-ba82-40b2-bffe-47e7ea20c922" (UID: "5b13d20c-ba82-40b2-bffe-47e7ea20c922"). InnerVolumeSpecName "kube-api-access-r42wk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:46:21.468266 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:21.462888 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r42wk\" (UniqueName: \"kubernetes.io/projected/5b13d20c-ba82-40b2-bffe-47e7ea20c922-kube-api-access-r42wk\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:21.468266 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:21.462925 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b13d20c-ba82-40b2-bffe-47e7ea20c922-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:21.810652 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:21.810566 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:21.811231 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:21.811211 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c8739dd-74v8c"] Oct 30 01:46:21.811864 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:21.811842 6572 cpu_manager.go:414] "containerMap: removing container" podUID="5b13d20c-ba82-40b2-bffe-47e7ea20c922" containerName="payload-processing" Oct 30 01:46:21.811864 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:21.811859 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b13d20c-ba82-40b2-bffe-47e7ea20c922" containerName="payload-processing" Oct 30 01:46:21.812076 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:21.811979 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b13d20c-ba82-40b2-bffe-47e7ea20c922" containerName="payload-processing" Oct 30 01:46:21.815843 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:21.815825 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8739dd-74v8c" Oct 30 01:46:21.839886 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:21.839001 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c8739dd-74v8c"] Oct 30 01:46:21.867469 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:21.867432 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5cf289d-5797-422d-93a3-6904f21eb4c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c8739dd-74v8c\" (UID: \"c5cf289d-5797-422d-93a3-6904f21eb4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8739dd-74v8c" Oct 30 01:46:21.867469 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:21.867475 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7hl5\" (UniqueName: \"kubernetes.io/projected/c5cf289d-5797-422d-93a3-6904f21eb4c8-kube-api-access-x7hl5\") pod \"maas-api-revoke-sub-8c8739dd-74v8c\" (UID: \"c5cf289d-5797-422d-93a3-6904f21eb4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8739dd-74v8c" Oct 30 01:46:21.867803 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:21.867607 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5cf289d-5797-422d-93a3-6904f21eb4c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c8739dd-74v8c\" (UID: \"c5cf289d-5797-422d-93a3-6904f21eb4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8739dd-74v8c" Oct 30 01:46:21.905593 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:21.905534 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0ae258-k6wjd"] Oct 30 01:46:21.911146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:21.911114 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0ae258-k6wjd"] Oct 30 01:46:21.968798 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:21.968757 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5cf289d-5797-422d-93a3-6904f21eb4c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c8739dd-74v8c\" (UID: \"c5cf289d-5797-422d-93a3-6904f21eb4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8739dd-74v8c" Oct 30 01:46:21.968972 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:21.968813 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7hl5\" (UniqueName: \"kubernetes.io/projected/c5cf289d-5797-422d-93a3-6904f21eb4c8-kube-api-access-x7hl5\") pod \"maas-api-revoke-sub-8c8739dd-74v8c\" (UID: \"c5cf289d-5797-422d-93a3-6904f21eb4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8739dd-74v8c" Oct 30 01:46:21.968972 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:21.968864 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5cf289d-5797-422d-93a3-6904f21eb4c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c8739dd-74v8c\" (UID: \"c5cf289d-5797-422d-93a3-6904f21eb4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8739dd-74v8c" Oct 30 01:46:21.969419 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:21.969400 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5cf289d-5797-422d-93a3-6904f21eb4c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c8739dd-74v8c\" (UID: \"c5cf289d-5797-422d-93a3-6904f21eb4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8739dd-74v8c" Oct 30 01:46:21.971420 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:21.971397 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5cf289d-5797-422d-93a3-6904f21eb4c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c8739dd-74v8c\" (UID: \"c5cf289d-5797-422d-93a3-6904f21eb4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8739dd-74v8c" Oct 30 01:46:21.987702 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:21.987676 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7hl5\" (UniqueName: \"kubernetes.io/projected/c5cf289d-5797-422d-93a3-6904f21eb4c8-kube-api-access-x7hl5\") pod \"maas-api-revoke-sub-8c8739dd-74v8c\" (UID: \"c5cf289d-5797-422d-93a3-6904f21eb4c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8739dd-74v8c" Oct 30 01:46:22.005457 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:22.005431 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:22.134853 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:22.134758 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8739dd-74v8c" Oct 30 01:46:22.187909 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:22.187876 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0ae258-7889569bbb-k695k" event={"ID":"5b13d20c-ba82-40b2-bffe-47e7ea20c922","Type":"ContainerDied","Data":"c90e5786277c3d19645fd9214325f68e8105a2c9b7267f59b7251e9155847221"} Oct 30 01:46:22.188106 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:22.187955 6572 scope.go:122] "RemoveContainer" containerID="2bb011c0be0f492e595654773c1ebf7898538c99073dcc2ce923b78f21b924f2" Oct 30 01:46:22.188106 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:22.187882 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0ae258-7889569bbb-k695k" Oct 30 01:46:22.228960 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:22.228928 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:22.268037 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:22.268000 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0ae258-7889569bbb-k695k"] Oct 30 01:46:22.275263 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:22.274836 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0ae258-7889569bbb-k695k"] Oct 30 01:46:22.414374 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:22.414349 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e4f632-pbv87" Oct 30 01:46:22.443806 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:46:22.443267 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5cf289d_5797_422d_93a3_6904f21eb4c8.slice/crio-379ea323a276a5afe83f9e100705c7a74f3be57a73b771518ad3ce1a84535f77 WatchSource:0}: Error finding container 379ea323a276a5afe83f9e100705c7a74f3be57a73b771518ad3ce1a84535f77: Status 404 returned error can't find the container with id 379ea323a276a5afe83f9e100705c7a74f3be57a73b771518ad3ce1a84535f77 Oct 30 01:46:22.443806 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:22.443353 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c8739dd-74v8c"] Oct 30 01:46:22.476391 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:22.476366 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e1d5968-a0a2-4e66-885d-da648cffb3f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e1d5968-a0a2-4e66-885d-da648cffb3f7-maas-api-xks-ca\") pod \"6e1d5968-a0a2-4e66-885d-da648cffb3f7\" (UID: \"6e1d5968-a0a2-4e66-885d-da648cffb3f7\") " Oct 30 01:46:22.476515 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:22.476504 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e1d5968-a0a2-4e66-885d-da648cffb3f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e1d5968-a0a2-4e66-885d-da648cffb3f7-maas-api-openshift-ca\") pod \"6e1d5968-a0a2-4e66-885d-da648cffb3f7\" (UID: \"6e1d5968-a0a2-4e66-885d-da648cffb3f7\") " Oct 30 01:46:22.476563 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:22.476540 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e1d5968-a0a2-4e66-885d-da648cffb3f7-kube-api-access-n6d5n\" (UniqueName: \"kubernetes.io/projected/6e1d5968-a0a2-4e66-885d-da648cffb3f7-kube-api-access-n6d5n\") pod \"6e1d5968-a0a2-4e66-885d-da648cffb3f7\" (UID: \"6e1d5968-a0a2-4e66-885d-da648cffb3f7\") " Oct 30 01:46:22.476950 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:22.476924 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e1d5968-a0a2-4e66-885d-da648cffb3f7-maas-api-openshift-ca" pod "6e1d5968-a0a2-4e66-885d-da648cffb3f7" (UID: "6e1d5968-a0a2-4e66-885d-da648cffb3f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:46:22.477105 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:22.477090 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e1d5968-a0a2-4e66-885d-da648cffb3f7-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:22.478422 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:22.478403 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e1d5968-a0a2-4e66-885d-da648cffb3f7-maas-api-xks-ca" pod "6e1d5968-a0a2-4e66-885d-da648cffb3f7" (UID: "6e1d5968-a0a2-4e66-885d-da648cffb3f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:46:22.478577 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:22.478557 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e1d5968-a0a2-4e66-885d-da648cffb3f7-kube-api-access-n6d5n" pod "6e1d5968-a0a2-4e66-885d-da648cffb3f7" (UID: "6e1d5968-a0a2-4e66-885d-da648cffb3f7"). InnerVolumeSpecName "kube-api-access-n6d5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:46:22.534530 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:22.534496 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:22.542933 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:22.542906 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b13d20c-ba82-40b2-bffe-47e7ea20c922" path="/var/lib/kubelet/pods/5b13d20c-ba82-40b2-bffe-47e7ea20c922/volumes" Oct 30 01:46:22.543430 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:22.543415 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="822d8556-80de-405f-a0cf-8ab09bfbca3e" path="/var/lib/kubelet/pods/822d8556-80de-405f-a0cf-8ab09bfbca3e/volumes" Oct 30 01:46:22.578926 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:22.578896 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6d5n\" (UniqueName: \"kubernetes.io/projected/6e1d5968-a0a2-4e66-885d-da648cffb3f7-kube-api-access-n6d5n\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:22.578926 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:22.578920 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e1d5968-a0a2-4e66-885d-da648cffb3f7-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:22.728856 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:22.728823 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:23.032953 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.031118 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k"] Oct 30 01:46:23.032953 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.031850 6572 cpu_manager.go:414] "containerMap: removing container" podUID="6e1d5968-a0a2-4e66-885d-da648cffb3f7" containerName="revoke-keys" Oct 30 01:46:23.032953 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.031869 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e1d5968-a0a2-4e66-885d-da648cffb3f7" containerName="revoke-keys" Oct 30 01:46:23.032953 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.032118 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e1d5968-a0a2-4e66-885d-da648cffb3f7" containerName="revoke-keys" Oct 30 01:46:23.036503 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.036125 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 01:46:23.063112 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.063084 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k"] Oct 30 01:46:23.072309 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.072277 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-396f8e-openshift-default-dockercfg-lmvlx\"" Oct 30 01:46:23.089185 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.089152 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-workload-certs\") pod \"e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k\" (UID: \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\") " pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 01:46:23.089372 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.089196 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-istio-token\") pod \"e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k\" (UID: \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\") " pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 01:46:23.089372 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.089291 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-istio-data\") pod \"e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k\" (UID: \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\") " pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 01:46:23.089372 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.089347 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-istiod-ca-cert\") pod \"e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k\" (UID: \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\") " pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 01:46:23.089560 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.089409 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-workload-socket\") pod \"e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k\" (UID: \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\") " pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 01:46:23.089560 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.089471 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xql4f\" (UniqueName: \"kubernetes.io/projected/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-kube-api-access-xql4f\") pod \"e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k\" (UID: \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\") " pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 01:46:23.089954 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.089842 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-istio-podinfo\") pod \"e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k\" (UID: \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\") " pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 01:46:23.089954 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.089908 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-credential-socket\") pod \"e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k\" (UID: \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\") " pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 01:46:23.090684 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.090088 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-istio-envoy\") pod \"e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k\" (UID: \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\") " pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 01:46:23.170424 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.170385 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:23.193045 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.193010 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-workload-certs\") pod \"e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k\" (UID: \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\") " pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 01:46:23.193045 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.193054 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-istio-token\") pod \"e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k\" (UID: \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\") " pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 01:46:23.193566 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.193124 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-istio-data\") pod \"e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k\" (UID: \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\") " pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 01:46:23.193566 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.193165 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-istiod-ca-cert\") pod \"e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k\" (UID: \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\") " pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 01:46:23.193566 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.193212 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-workload-socket\") pod \"e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k\" (UID: \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\") " pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 01:46:23.193566 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.193264 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xql4f\" (UniqueName: \"kubernetes.io/projected/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-kube-api-access-xql4f\") pod \"e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k\" (UID: \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\") " pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 01:46:23.193566 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.193329 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-istio-podinfo\") pod \"e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k\" (UID: \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\") " pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 01:46:23.193566 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.193363 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-credential-socket\") pod \"e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k\" (UID: \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\") " pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 01:46:23.193566 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.193388 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-istio-envoy\") pod \"e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k\" (UID: \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\") " pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 01:46:23.194139 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.193697 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-workload-socket\") pod \"e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k\" (UID: \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\") " pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 01:46:23.194139 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.193775 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-istio-data\") pod \"e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k\" (UID: \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\") " pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 01:46:23.194139 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.193849 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-workload-certs\") pod \"e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k\" (UID: \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\") " pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 01:46:23.194139 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.194041 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-credential-socket\") pod \"e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k\" (UID: \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\") " pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 01:46:23.194392 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.194368 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-istiod-ca-cert\") pod \"e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k\" (UID: \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\") " pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 01:46:23.195955 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.195934 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-istio-envoy\") pod \"e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k\" (UID: \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\") " pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 01:46:23.196051 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.195980 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-istio-podinfo\") pod \"e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k\" (UID: \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\") " pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 01:46:23.201382 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.201363 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e4f632-pbv87" Oct 30 01:46:23.201382 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.201371 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e4f632-pbv87" event={"ID":"6e1d5968-a0a2-4e66-885d-da648cffb3f7","Type":"ContainerDied","Data":"bf330b6132c0d0f0a8b2feb7b83add9ad92924c550a24f4d9b96784ac4432497"} Oct 30 01:46:23.201557 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.201454 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf330b6132c0d0f0a8b2feb7b83add9ad92924c550a24f4d9b96784ac4432497" Oct 30 01:46:23.203255 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.203231 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8739dd-74v8c" event={"ID":"c5cf289d-5797-422d-93a3-6904f21eb4c8","Type":"ContainerStarted","Data":"c582eb740d2e16dde0a35ede24ae9408572442fc9bafb6ceeaa484bd54389c51"} Oct 30 01:46:23.203358 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.203323 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8739dd-74v8c" event={"ID":"c5cf289d-5797-422d-93a3-6904f21eb4c8","Type":"ContainerStarted","Data":"379ea323a276a5afe83f9e100705c7a74f3be57a73b771518ad3ce1a84535f77"} Oct 30 01:46:23.211962 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.211936 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-istio-token\") pod \"e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k\" (UID: \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\") " pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 01:46:23.212059 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.211985 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xql4f\" (UniqueName: \"kubernetes.io/projected/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-kube-api-access-xql4f\") pod \"e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k\" (UID: \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\") " pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 01:46:23.253011 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.252961 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8739dd-74v8c" podStartSLOduration=2.25294785 podStartE2EDuration="2.25294785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:46:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:46:23.247526783 +0000 UTC m=+1708.823952186" watchObservedRunningTime="2026-10-30 01:46:23.25294785 +0000 UTC m=+1708.829373251" Oct 30 01:46:23.354155 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.354041 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 01:46:23.467983 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.466709 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:23.747238 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.747209 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k"] Oct 30 01:46:23.747839 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:46:23.747759 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e60f1dc_1188_4053_8f31_fcd08da0ef1c.slice/crio-a0c1643fe9ac9146f6494e41f7a53f5c7281bc69ed8f11725457f67e91f20437 WatchSource:0}: Error finding container a0c1643fe9ac9146f6494e41f7a53f5c7281bc69ed8f11725457f67e91f20437: Status 404 returned error can't find the container with id a0c1643fe9ac9146f6494e41f7a53f5c7281bc69ed8f11725457f67e91f20437 Oct 30 01:46:23.750476 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.750440 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:46:23.750594 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.750507 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:46:23.750594 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:23.750539 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:46:24.108510 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:24.108420 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:24.230980 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:24.230943 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" event={"ID":"7e60f1dc-1188-4053-8f31-fcd08da0ef1c","Type":"ContainerStarted","Data":"63e3d2a4a936608d5d3466dd4111cb9af461b13e9ee684ef2b45ac6229096912"} Oct 30 01:46:24.231181 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:24.231009 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" event={"ID":"7e60f1dc-1188-4053-8f31-fcd08da0ef1c","Type":"ContainerStarted","Data":"a0c1643fe9ac9146f6494e41f7a53f5c7281bc69ed8f11725457f67e91f20437"} Oct 30 01:46:24.233879 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:24.233730 6572 generic.go:356] "Generic (PLEG): container finished" podID="c5cf289d-5797-422d-93a3-6904f21eb4c8" containerID="c582eb740d2e16dde0a35ede24ae9408572442fc9bafb6ceeaa484bd54389c51" exitCode=0 Oct 30 01:46:24.233879 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:24.233880 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8739dd-74v8c" event={"ID":"c5cf289d-5797-422d-93a3-6904f21eb4c8","Type":"ContainerDied","Data":"c582eb740d2e16dde0a35ede24ae9408572442fc9bafb6ceeaa484bd54389c51"} Oct 30 01:46:24.275499 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:24.275444 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" podStartSLOduration=2.275428378 podStartE2EDuration="2.275428378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:46:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:46:24.273260278 +0000 UTC m=+1709.849685703" watchObservedRunningTime="2026-10-30 01:46:24.275428378 +0000 UTC m=+1709.851853777" Oct 30 01:46:24.355121 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:24.355081 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 01:46:24.361523 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:24.361441 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 01:46:25.195924 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:25.195877 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-f197b1-6dc895f6f6-qqnvt" podUID="8136562a-9ad1-41c5-9139-677268c5da39" containerName="payload-processing" containerID="cri-o://aab9a4a191b245c7dc3acae84840cfb44e08df0702c286e92480a8c243a3cb90" gracePeriod=25 Oct 30 01:46:25.242545 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:25.242515 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 01:46:25.243724 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:25.243701 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 01:46:25.462643 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:25.462549 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8739dd-74v8c" Oct 30 01:46:25.521615 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:25.521583 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f197b1-6dc895f6f6-qqnvt" Oct 30 01:46:25.523618 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:25.523588 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5cf289d-5797-422d-93a3-6904f21eb4c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5cf289d-5797-422d-93a3-6904f21eb4c8-maas-api-xks-ca\") pod \"c5cf289d-5797-422d-93a3-6904f21eb4c8\" (UID: \"c5cf289d-5797-422d-93a3-6904f21eb4c8\") " Oct 30 01:46:25.523786 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:25.523661 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5cf289d-5797-422d-93a3-6904f21eb4c8-kube-api-access-x7hl5\" (UniqueName: \"kubernetes.io/projected/c5cf289d-5797-422d-93a3-6904f21eb4c8-kube-api-access-x7hl5\") pod \"c5cf289d-5797-422d-93a3-6904f21eb4c8\" (UID: \"c5cf289d-5797-422d-93a3-6904f21eb4c8\") " Oct 30 01:46:25.523786 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:25.523765 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5cf289d-5797-422d-93a3-6904f21eb4c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5cf289d-5797-422d-93a3-6904f21eb4c8-maas-api-openshift-ca\") pod \"c5cf289d-5797-422d-93a3-6904f21eb4c8\" (UID: \"c5cf289d-5797-422d-93a3-6904f21eb4c8\") " Oct 30 01:46:25.524115 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:25.524091 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5cf289d-5797-422d-93a3-6904f21eb4c8-maas-api-openshift-ca" pod "c5cf289d-5797-422d-93a3-6904f21eb4c8" (UID: "c5cf289d-5797-422d-93a3-6904f21eb4c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:46:25.524521 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:25.524496 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5cf289d-5797-422d-93a3-6904f21eb4c8-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:25.526325 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:25.526301 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5cf289d-5797-422d-93a3-6904f21eb4c8-kube-api-access-x7hl5" pod "c5cf289d-5797-422d-93a3-6904f21eb4c8" (UID: "c5cf289d-5797-422d-93a3-6904f21eb4c8"). InnerVolumeSpecName "kube-api-access-x7hl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:46:25.526325 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:25.526307 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5cf289d-5797-422d-93a3-6904f21eb4c8-maas-api-xks-ca" pod "c5cf289d-5797-422d-93a3-6904f21eb4c8" (UID: "c5cf289d-5797-422d-93a3-6904f21eb4c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:46:25.625708 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:25.625664 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8136562a-9ad1-41c5-9139-677268c5da39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8136562a-9ad1-41c5-9139-677268c5da39-plugins-config-volume\") pod \"8136562a-9ad1-41c5-9139-677268c5da39\" (UID: \"8136562a-9ad1-41c5-9139-677268c5da39\") " Oct 30 01:46:25.625894 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:25.625788 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8136562a-9ad1-41c5-9139-677268c5da39-kube-api-access-zzzs6\" (UniqueName: \"kubernetes.io/projected/8136562a-9ad1-41c5-9139-677268c5da39-kube-api-access-zzzs6\") pod \"8136562a-9ad1-41c5-9139-677268c5da39\" (UID: \"8136562a-9ad1-41c5-9139-677268c5da39\") " Oct 30 01:46:25.626064 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:25.626041 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8136562a-9ad1-41c5-9139-677268c5da39-plugins-config-volume" pod "8136562a-9ad1-41c5-9139-677268c5da39" (UID: "8136562a-9ad1-41c5-9139-677268c5da39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:46:25.626324 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:25.626300 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5cf289d-5797-422d-93a3-6904f21eb4c8-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:25.626324 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:25.626323 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7hl5\" (UniqueName: \"kubernetes.io/projected/c5cf289d-5797-422d-93a3-6904f21eb4c8-kube-api-access-x7hl5\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:25.626552 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:25.626336 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8136562a-9ad1-41c5-9139-677268c5da39-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:25.628003 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:25.627971 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8136562a-9ad1-41c5-9139-677268c5da39-kube-api-access-zzzs6" pod "8136562a-9ad1-41c5-9139-677268c5da39" (UID: "8136562a-9ad1-41c5-9139-677268c5da39"). InnerVolumeSpecName "kube-api-access-zzzs6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:46:25.727663 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:25.727570 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzzs6\" (UniqueName: \"kubernetes.io/projected/8136562a-9ad1-41c5-9139-677268c5da39-kube-api-access-zzzs6\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:25.798340 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:25.796984 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:26.251645 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:26.251615 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8739dd-74v8c" Oct 30 01:46:26.251865 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:26.251651 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8739dd-74v8c" event={"ID":"c5cf289d-5797-422d-93a3-6904f21eb4c8","Type":"ContainerDied","Data":"379ea323a276a5afe83f9e100705c7a74f3be57a73b771518ad3ce1a84535f77"} Oct 30 01:46:26.251865 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:26.251700 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="379ea323a276a5afe83f9e100705c7a74f3be57a73b771518ad3ce1a84535f77" Oct 30 01:46:26.253391 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:26.253359 6572 generic.go:356] "Generic (PLEG): container finished" podID="8136562a-9ad1-41c5-9139-677268c5da39" containerID="aab9a4a191b245c7dc3acae84840cfb44e08df0702c286e92480a8c243a3cb90" exitCode=0 Oct 30 01:46:26.253498 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:26.253403 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f197b1-6dc895f6f6-qqnvt" Oct 30 01:46:26.253498 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:26.253406 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f197b1-6dc895f6f6-qqnvt" event={"ID":"8136562a-9ad1-41c5-9139-677268c5da39","Type":"ContainerDied","Data":"aab9a4a191b245c7dc3acae84840cfb44e08df0702c286e92480a8c243a3cb90"} Oct 30 01:46:26.253498 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:26.253456 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f197b1-6dc895f6f6-qqnvt" event={"ID":"8136562a-9ad1-41c5-9139-677268c5da39","Type":"ContainerDied","Data":"b8f7748c4f92b577dd4c85f99a88a7fe3ac506d21313a78e7d048cbd93a5cb01"} Oct 30 01:46:26.253498 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:26.253485 6572 scope.go:122] "RemoveContainer" containerID="aab9a4a191b245c7dc3acae84840cfb44e08df0702c286e92480a8c243a3cb90" Oct 30 01:46:26.265064 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:26.265046 6572 scope.go:122] "RemoveContainer" containerID="aab9a4a191b245c7dc3acae84840cfb44e08df0702c286e92480a8c243a3cb90" Oct 30 01:46:26.265326 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:46:26.265310 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aab9a4a191b245c7dc3acae84840cfb44e08df0702c286e92480a8c243a3cb90\": container with ID starting with aab9a4a191b245c7dc3acae84840cfb44e08df0702c286e92480a8c243a3cb90 not found: ID does not exist" containerID="aab9a4a191b245c7dc3acae84840cfb44e08df0702c286e92480a8c243a3cb90" Oct 30 01:46:26.265372 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:26.265334 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aab9a4a191b245c7dc3acae84840cfb44e08df0702c286e92480a8c243a3cb90"} err="failed to get container status \"aab9a4a191b245c7dc3acae84840cfb44e08df0702c286e92480a8c243a3cb90\": rpc error: code = NotFound desc = could not find container \"aab9a4a191b245c7dc3acae84840cfb44e08df0702c286e92480a8c243a3cb90\": container with ID starting with aab9a4a191b245c7dc3acae84840cfb44e08df0702c286e92480a8c243a3cb90 not found: ID does not exist" Oct 30 01:46:26.325654 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:26.325607 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f197b1-6dc895f6f6-qqnvt"] Oct 30 01:46:26.330459 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:26.330408 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f197b1-6dc895f6f6-qqnvt"] Oct 30 01:46:26.542758 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:26.542664 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8136562a-9ad1-41c5-9139-677268c5da39" path="/var/lib/kubelet/pods/8136562a-9ad1-41c5-9139-677268c5da39/volumes" Oct 30 01:46:26.978420 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:26.978379 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:27.190976 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:27.190937 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:27.598718 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:27.598667 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:27.817338 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:27.817307 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:28.273720 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.273507 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8"] Oct 30 01:46:28.274497 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.274473 6572 cpu_manager.go:414] "containerMap: removing container" podUID="8136562a-9ad1-41c5-9139-677268c5da39" containerName="payload-processing" Oct 30 01:46:28.274587 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.274501 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8136562a-9ad1-41c5-9139-677268c5da39" containerName="payload-processing" Oct 30 01:46:28.274587 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.274574 6572 cpu_manager.go:414] "containerMap: removing container" podUID="c5cf289d-5797-422d-93a3-6904f21eb4c8" containerName="revoke-keys" Oct 30 01:46:28.274587 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.274580 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5cf289d-5797-422d-93a3-6904f21eb4c8" containerName="revoke-keys" Oct 30 01:46:28.274789 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.274683 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5cf289d-5797-422d-93a3-6904f21eb4c8" containerName="revoke-keys" Oct 30 01:46:28.274789 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.274693 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="8136562a-9ad1-41c5-9139-677268c5da39" containerName="payload-processing" Oct 30 01:46:28.279884 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.279857 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" Oct 30 01:46:28.313835 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.311224 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8"] Oct 30 01:46:28.361458 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.361419 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00464688-4494-49d1-b56d-0b7baa8ca9c5-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8\" (UID: \"00464688-4494-49d1-b56d-0b7baa8ca9c5\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" Oct 30 01:46:28.361458 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.361460 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00464688-4494-49d1-b56d-0b7baa8ca9c5-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8\" (UID: \"00464688-4494-49d1-b56d-0b7baa8ca9c5\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" Oct 30 01:46:28.361683 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.361592 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00464688-4494-49d1-b56d-0b7baa8ca9c5-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8\" (UID: \"00464688-4494-49d1-b56d-0b7baa8ca9c5\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" Oct 30 01:46:28.361683 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.361636 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkl9q\" (UniqueName: \"kubernetes.io/projected/00464688-4494-49d1-b56d-0b7baa8ca9c5-kube-api-access-kkl9q\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8\" (UID: \"00464688-4494-49d1-b56d-0b7baa8ca9c5\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" Oct 30 01:46:28.361776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.361692 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00464688-4494-49d1-b56d-0b7baa8ca9c5-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8\" (UID: \"00464688-4494-49d1-b56d-0b7baa8ca9c5\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" Oct 30 01:46:28.361776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.361721 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00464688-4494-49d1-b56d-0b7baa8ca9c5-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8\" (UID: \"00464688-4494-49d1-b56d-0b7baa8ca9c5\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" Oct 30 01:46:28.361776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.361741 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00464688-4494-49d1-b56d-0b7baa8ca9c5-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8\" (UID: \"00464688-4494-49d1-b56d-0b7baa8ca9c5\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" Oct 30 01:46:28.361963 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.361948 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00464688-4494-49d1-b56d-0b7baa8ca9c5-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8\" (UID: \"00464688-4494-49d1-b56d-0b7baa8ca9c5\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" Oct 30 01:46:28.362008 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.361987 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00464688-4494-49d1-b56d-0b7baa8ca9c5-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8\" (UID: \"00464688-4494-49d1-b56d-0b7baa8ca9c5\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" Oct 30 01:46:28.463223 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.463188 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00464688-4494-49d1-b56d-0b7baa8ca9c5-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8\" (UID: \"00464688-4494-49d1-b56d-0b7baa8ca9c5\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" Oct 30 01:46:28.463417 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.463229 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00464688-4494-49d1-b56d-0b7baa8ca9c5-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8\" (UID: \"00464688-4494-49d1-b56d-0b7baa8ca9c5\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" Oct 30 01:46:28.463417 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.463289 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00464688-4494-49d1-b56d-0b7baa8ca9c5-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8\" (UID: \"00464688-4494-49d1-b56d-0b7baa8ca9c5\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" Oct 30 01:46:28.463417 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.463326 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkl9q\" (UniqueName: \"kubernetes.io/projected/00464688-4494-49d1-b56d-0b7baa8ca9c5-kube-api-access-kkl9q\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8\" (UID: \"00464688-4494-49d1-b56d-0b7baa8ca9c5\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" Oct 30 01:46:28.463417 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.463353 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00464688-4494-49d1-b56d-0b7baa8ca9c5-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8\" (UID: \"00464688-4494-49d1-b56d-0b7baa8ca9c5\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" Oct 30 01:46:28.463417 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.463380 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00464688-4494-49d1-b56d-0b7baa8ca9c5-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8\" (UID: \"00464688-4494-49d1-b56d-0b7baa8ca9c5\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" Oct 30 01:46:28.463417 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.463411 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00464688-4494-49d1-b56d-0b7baa8ca9c5-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8\" (UID: \"00464688-4494-49d1-b56d-0b7baa8ca9c5\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" Oct 30 01:46:28.463819 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.463551 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00464688-4494-49d1-b56d-0b7baa8ca9c5-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8\" (UID: \"00464688-4494-49d1-b56d-0b7baa8ca9c5\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" Oct 30 01:46:28.463819 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.463596 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00464688-4494-49d1-b56d-0b7baa8ca9c5-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8\" (UID: \"00464688-4494-49d1-b56d-0b7baa8ca9c5\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" Oct 30 01:46:28.463819 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.463667 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00464688-4494-49d1-b56d-0b7baa8ca9c5-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8\" (UID: \"00464688-4494-49d1-b56d-0b7baa8ca9c5\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" Oct 30 01:46:28.463819 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.463717 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00464688-4494-49d1-b56d-0b7baa8ca9c5-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8\" (UID: \"00464688-4494-49d1-b56d-0b7baa8ca9c5\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" Oct 30 01:46:28.464087 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.464067 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00464688-4494-49d1-b56d-0b7baa8ca9c5-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8\" (UID: \"00464688-4494-49d1-b56d-0b7baa8ca9c5\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" Oct 30 01:46:28.464184 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.464087 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00464688-4494-49d1-b56d-0b7baa8ca9c5-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8\" (UID: \"00464688-4494-49d1-b56d-0b7baa8ca9c5\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" Oct 30 01:46:28.464184 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.464096 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00464688-4494-49d1-b56d-0b7baa8ca9c5-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8\" (UID: \"00464688-4494-49d1-b56d-0b7baa8ca9c5\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" Oct 30 01:46:28.465863 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.465837 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00464688-4494-49d1-b56d-0b7baa8ca9c5-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8\" (UID: \"00464688-4494-49d1-b56d-0b7baa8ca9c5\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" Oct 30 01:46:28.466022 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.466006 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00464688-4494-49d1-b56d-0b7baa8ca9c5-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8\" (UID: \"00464688-4494-49d1-b56d-0b7baa8ca9c5\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" Oct 30 01:46:28.486342 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.486307 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkl9q\" (UniqueName: \"kubernetes.io/projected/00464688-4494-49d1-b56d-0b7baa8ca9c5-kube-api-access-kkl9q\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8\" (UID: \"00464688-4494-49d1-b56d-0b7baa8ca9c5\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" Oct 30 01:46:28.486476 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.486355 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00464688-4494-49d1-b56d-0b7baa8ca9c5-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8\" (UID: \"00464688-4494-49d1-b56d-0b7baa8ca9c5\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" Oct 30 01:46:28.598532 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.598446 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" Oct 30 01:46:28.722027 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.721944 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:28.992697 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.992664 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:28.998241 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.996686 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:46:28.998241 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.996807 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:46:28.998241 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:28.996855 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:46:29.033385 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:29.033033 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8"] Oct 30 01:46:29.288796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:29.288682 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" event={"ID":"00464688-4494-49d1-b56d-0b7baa8ca9c5","Type":"ContainerStarted","Data":"a4490045fc342f05c8dad5df2e0024ea64832221014d10845b9d0e70724e662c"} Oct 30 01:46:29.288796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:29.288782 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" event={"ID":"00464688-4494-49d1-b56d-0b7baa8ca9c5","Type":"ContainerStarted","Data":"ab46a6b24c2d29e0dd42fcf7dcb25c6231844966a45d5844f07d116c643bf8ab"} Oct 30 01:46:29.337834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:29.337730 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" podStartSLOduration=1.337710912 podStartE2EDuration="1.337710912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:46:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:46:29.33587893 +0000 UTC m=+1714.912304331" watchObservedRunningTime="2026-10-30 01:46:29.337710912 +0000 UTC m=+1714.914136314" Oct 30 01:46:29.598948 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:29.598858 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" Oct 30 01:46:29.737775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:29.736365 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:30.600575 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:30.600535 6572 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Oct 30 01:46:30.600819 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:30.600592 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" podUID="00464688-4494-49d1-b56d-0b7baa8ca9c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Oct 30 01:46:31.437290 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:31.437250 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:31.603404 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:31.603353 6572 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Oct 30 01:46:31.603647 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:31.603424 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" podUID="00464688-4494-49d1-b56d-0b7baa8ca9c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Oct 30 01:46:32.123775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:32.122542 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:32.175062 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:32.175028 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f197b1-57tfx"] Oct 30 01:46:32.186914 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:32.186877 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f197b1-57tfx"] Oct 30 01:46:32.264767 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:32.264662 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd559747-z2t4k"] Oct 30 01:46:32.282798 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:32.282253 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd559747-z2t4k"] Oct 30 01:46:32.282798 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:32.282514 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd559747-z2t4k" Oct 30 01:46:32.289041 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:32.289005 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpvz5\" (UniqueName: \"kubernetes.io/projected/0f64ef38-8649-4f43-a27e-2ad93b5fc9dd-kube-api-access-rpvz5\") pod \"maas-api-revoke-sub-cd559747-z2t4k\" (UID: \"0f64ef38-8649-4f43-a27e-2ad93b5fc9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd559747-z2t4k" Oct 30 01:46:32.289230 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:32.289055 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f64ef38-8649-4f43-a27e-2ad93b5fc9dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd559747-z2t4k\" (UID: \"0f64ef38-8649-4f43-a27e-2ad93b5fc9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd559747-z2t4k" Oct 30 01:46:32.289230 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:32.289076 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f64ef38-8649-4f43-a27e-2ad93b5fc9dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd559747-z2t4k\" (UID: \"0f64ef38-8649-4f43-a27e-2ad93b5fc9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd559747-z2t4k" Oct 30 01:46:32.300258 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:32.299971 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:46:32.300258 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:32.300113 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:46:32.300258 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:32.300209 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:46:32.300564 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:32.300272 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:46:32.338776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:32.338649 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:32.390452 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:32.390356 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f64ef38-8649-4f43-a27e-2ad93b5fc9dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd559747-z2t4k\" (UID: \"0f64ef38-8649-4f43-a27e-2ad93b5fc9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd559747-z2t4k" Oct 30 01:46:32.390452 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:32.390396 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f64ef38-8649-4f43-a27e-2ad93b5fc9dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd559747-z2t4k\" (UID: \"0f64ef38-8649-4f43-a27e-2ad93b5fc9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd559747-z2t4k" Oct 30 01:46:32.390711 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:32.390554 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpvz5\" (UniqueName: \"kubernetes.io/projected/0f64ef38-8649-4f43-a27e-2ad93b5fc9dd-kube-api-access-rpvz5\") pod \"maas-api-revoke-sub-cd559747-z2t4k\" (UID: \"0f64ef38-8649-4f43-a27e-2ad93b5fc9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd559747-z2t4k" Oct 30 01:46:32.391088 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:32.391057 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f64ef38-8649-4f43-a27e-2ad93b5fc9dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd559747-z2t4k\" (UID: \"0f64ef38-8649-4f43-a27e-2ad93b5fc9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd559747-z2t4k" Oct 30 01:46:32.392935 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:32.392911 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f64ef38-8649-4f43-a27e-2ad93b5fc9dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd559747-z2t4k\" (UID: \"0f64ef38-8649-4f43-a27e-2ad93b5fc9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd559747-z2t4k" Oct 30 01:46:32.429773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:32.429231 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpvz5\" (UniqueName: \"kubernetes.io/projected/0f64ef38-8649-4f43-a27e-2ad93b5fc9dd-kube-api-access-rpvz5\") pod \"maas-api-revoke-sub-cd559747-z2t4k\" (UID: \"0f64ef38-8649-4f43-a27e-2ad93b5fc9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd559747-z2t4k" Oct 30 01:46:32.543286 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:32.543253 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e314b459-a94a-4883-a70d-899f905b48d2" path="/var/lib/kubelet/pods/e314b459-a94a-4883-a70d-899f905b48d2/volumes" Oct 30 01:46:32.549358 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:32.549321 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:32.599130 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:32.599087 6572 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Oct 30 01:46:32.599322 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:32.599148 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" podUID="00464688-4494-49d1-b56d-0b7baa8ca9c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Oct 30 01:46:32.638540 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:32.638502 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd559747-z2t4k" Oct 30 01:46:32.790086 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:32.790052 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:32.947607 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:32.947576 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd559747-z2t4k"] Oct 30 01:46:32.947732 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:46:32.947679 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f64ef38_8649_4f43_a27e_2ad93b5fc9dd.slice/crio-2130bca525a4c5892891c766ff9aec87cc9749423b1288226ff76a416795e862 WatchSource:0}: Error finding container 2130bca525a4c5892891c766ff9aec87cc9749423b1288226ff76a416795e862: Status 404 returned error can't find the container with id 2130bca525a4c5892891c766ff9aec87cc9749423b1288226ff76a416795e862 Oct 30 01:46:33.356881 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:33.356787 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd559747-z2t4k" event={"ID":"0f64ef38-8649-4f43-a27e-2ad93b5fc9dd","Type":"ContainerStarted","Data":"2130bca525a4c5892891c766ff9aec87cc9749423b1288226ff76a416795e862"} Oct 30 01:46:33.599165 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:33.599125 6572 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Oct 30 01:46:33.599579 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:33.599196 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" podUID="00464688-4494-49d1-b56d-0b7baa8ca9c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Oct 30 01:46:34.149615 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:34.149442 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:34.367296 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:34.367193 6572 generic.go:356] "Generic (PLEG): container finished" podID="0f64ef38-8649-4f43-a27e-2ad93b5fc9dd" containerID="d6daf7d3fe4414366717d7023ab628b8112d32d19ba4bd5853cf3a7b438e9869" exitCode=0 Oct 30 01:46:34.367296 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:34.367246 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd559747-z2t4k" event={"ID":"0f64ef38-8649-4f43-a27e-2ad93b5fc9dd","Type":"ContainerDied","Data":"d6daf7d3fe4414366717d7023ab628b8112d32d19ba4bd5853cf3a7b438e9869"} Oct 30 01:46:34.599493 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:34.599441 6572 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Oct 30 01:46:34.599958 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:34.599502 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" podUID="00464688-4494-49d1-b56d-0b7baa8ca9c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Oct 30 01:46:35.553894 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:35.553865 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd559747-z2t4k" Oct 30 01:46:35.599818 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:35.599785 6572 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Oct 30 01:46:35.600322 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:35.599847 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" podUID="00464688-4494-49d1-b56d-0b7baa8ca9c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Oct 30 01:46:35.663344 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:35.663313 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f64ef38-8649-4f43-a27e-2ad93b5fc9dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f64ef38-8649-4f43-a27e-2ad93b5fc9dd-maas-api-xks-ca\") pod \"0f64ef38-8649-4f43-a27e-2ad93b5fc9dd\" (UID: \"0f64ef38-8649-4f43-a27e-2ad93b5fc9dd\") " Oct 30 01:46:35.663563 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:35.663544 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f64ef38-8649-4f43-a27e-2ad93b5fc9dd-kube-api-access-rpvz5\" (UniqueName: \"kubernetes.io/projected/0f64ef38-8649-4f43-a27e-2ad93b5fc9dd-kube-api-access-rpvz5\") pod \"0f64ef38-8649-4f43-a27e-2ad93b5fc9dd\" (UID: \"0f64ef38-8649-4f43-a27e-2ad93b5fc9dd\") " Oct 30 01:46:35.663627 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:35.663610 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f64ef38-8649-4f43-a27e-2ad93b5fc9dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f64ef38-8649-4f43-a27e-2ad93b5fc9dd-maas-api-openshift-ca\") pod \"0f64ef38-8649-4f43-a27e-2ad93b5fc9dd\" (UID: \"0f64ef38-8649-4f43-a27e-2ad93b5fc9dd\") " Oct 30 01:46:35.664014 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:35.663982 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f64ef38-8649-4f43-a27e-2ad93b5fc9dd-maas-api-openshift-ca" pod "0f64ef38-8649-4f43-a27e-2ad93b5fc9dd" (UID: "0f64ef38-8649-4f43-a27e-2ad93b5fc9dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:46:35.664191 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:35.664168 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f64ef38-8649-4f43-a27e-2ad93b5fc9dd-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:35.665541 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:35.665517 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f64ef38-8649-4f43-a27e-2ad93b5fc9dd-maas-api-xks-ca" pod "0f64ef38-8649-4f43-a27e-2ad93b5fc9dd" (UID: "0f64ef38-8649-4f43-a27e-2ad93b5fc9dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:46:35.666222 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:35.666189 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f64ef38-8649-4f43-a27e-2ad93b5fc9dd-kube-api-access-rpvz5" pod "0f64ef38-8649-4f43-a27e-2ad93b5fc9dd" (UID: "0f64ef38-8649-4f43-a27e-2ad93b5fc9dd"). InnerVolumeSpecName "kube-api-access-rpvz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:46:35.765166 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:35.765133 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpvz5\" (UniqueName: \"kubernetes.io/projected/0f64ef38-8649-4f43-a27e-2ad93b5fc9dd-kube-api-access-rpvz5\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:35.765166 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:35.765163 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f64ef38-8649-4f43-a27e-2ad93b5fc9dd-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:35.985711 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:35.985429 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:36.395203 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:36.395121 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd559747-z2t4k" Oct 30 01:46:36.395203 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:36.395156 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd559747-z2t4k" event={"ID":"0f64ef38-8649-4f43-a27e-2ad93b5fc9dd","Type":"ContainerDied","Data":"2130bca525a4c5892891c766ff9aec87cc9749423b1288226ff76a416795e862"} Oct 30 01:46:36.395203 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:36.395200 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2130bca525a4c5892891c766ff9aec87cc9749423b1288226ff76a416795e862" Oct 30 01:46:36.604155 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:36.604126 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" Oct 30 01:46:36.604648 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:36.604624 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" Oct 30 01:46:36.605402 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:36.605386 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" Oct 30 01:46:37.590475 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:37.590443 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:38.749490 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:38.749455 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:38.965470 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:38.965437 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl"] Oct 30 01:46:38.965796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:38.965737 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" podUID="2c1aa377-ab2c-4653-a630-3d4eef49526d" containerName="istio-proxy" containerID="cri-o://72cc259568b451aac5ebe330f41c573ac8d2b1c09862bcf3a6962ec910780676" gracePeriod=30 Oct 30 01:46:38.980979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:38.980918 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:39.177351 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:39.172299 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:39.352916 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:39.352881 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07406684-6pkdb"] Oct 30 01:46:39.353527 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:39.353509 6572 cpu_manager.go:414] "containerMap: removing container" podUID="0f64ef38-8649-4f43-a27e-2ad93b5fc9dd" containerName="revoke-keys" Oct 30 01:46:39.353591 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:39.353528 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f64ef38-8649-4f43-a27e-2ad93b5fc9dd" containerName="revoke-keys" Oct 30 01:46:39.353676 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:39.353666 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f64ef38-8649-4f43-a27e-2ad93b5fc9dd" containerName="revoke-keys" Oct 30 01:46:39.357784 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:39.357759 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07406684-6pkdb" Oct 30 01:46:39.372467 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:39.372437 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:46:39.372641 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:39.372519 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:46:39.377823 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:39.377786 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07406684-6pkdb"] Oct 30 01:46:39.380823 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:39.380646 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:39.416079 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:39.416042 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93e155d3-0a3e-49bf-8951-193b216ec87c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07406684-6pkdb\" (UID: \"93e155d3-0a3e-49bf-8951-193b216ec87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07406684-6pkdb" Oct 30 01:46:39.416282 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:39.416108 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmhzr\" (UniqueName: \"kubernetes.io/projected/93e155d3-0a3e-49bf-8951-193b216ec87c-kube-api-access-wmhzr\") pod \"maas-api-revoke-sub-07406684-6pkdb\" (UID: \"93e155d3-0a3e-49bf-8951-193b216ec87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07406684-6pkdb" Oct 30 01:46:39.416282 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:39.416215 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93e155d3-0a3e-49bf-8951-193b216ec87c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07406684-6pkdb\" (UID: \"93e155d3-0a3e-49bf-8951-193b216ec87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07406684-6pkdb" Oct 30 01:46:39.447822 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:39.447728 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:46:39.447980 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:39.447936 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:46:39.517907 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:39.517871 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93e155d3-0a3e-49bf-8951-193b216ec87c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07406684-6pkdb\" (UID: \"93e155d3-0a3e-49bf-8951-193b216ec87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07406684-6pkdb" Oct 30 01:46:39.517907 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:39.517915 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmhzr\" (UniqueName: \"kubernetes.io/projected/93e155d3-0a3e-49bf-8951-193b216ec87c-kube-api-access-wmhzr\") pod \"maas-api-revoke-sub-07406684-6pkdb\" (UID: \"93e155d3-0a3e-49bf-8951-193b216ec87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07406684-6pkdb" Oct 30 01:46:39.518204 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:39.517940 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93e155d3-0a3e-49bf-8951-193b216ec87c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07406684-6pkdb\" (UID: \"93e155d3-0a3e-49bf-8951-193b216ec87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07406684-6pkdb" Oct 30 01:46:39.518705 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:39.518681 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93e155d3-0a3e-49bf-8951-193b216ec87c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07406684-6pkdb\" (UID: \"93e155d3-0a3e-49bf-8951-193b216ec87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07406684-6pkdb" Oct 30 01:46:39.520390 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:39.520368 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93e155d3-0a3e-49bf-8951-193b216ec87c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07406684-6pkdb\" (UID: \"93e155d3-0a3e-49bf-8951-193b216ec87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07406684-6pkdb" Oct 30 01:46:39.537204 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:39.537172 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmhzr\" (UniqueName: \"kubernetes.io/projected/93e155d3-0a3e-49bf-8951-193b216ec87c-kube-api-access-wmhzr\") pod \"maas-api-revoke-sub-07406684-6pkdb\" (UID: \"93e155d3-0a3e-49bf-8951-193b216ec87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07406684-6pkdb" Oct 30 01:46:39.673177 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:39.673143 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07406684-6pkdb" Oct 30 01:46:39.986285 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:46:39.986252 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93e155d3_0a3e_49bf_8951_193b216ec87c.slice/crio-28b30e35920cd576d23c9245dccf9a333a03cbcfe66a01b90e1c9f8aa21d53d9 WatchSource:0}: Error finding container 28b30e35920cd576d23c9245dccf9a333a03cbcfe66a01b90e1c9f8aa21d53d9: Status 404 returned error can't find the container with id 28b30e35920cd576d23c9245dccf9a333a03cbcfe66a01b90e1c9f8aa21d53d9 Oct 30 01:46:39.987685 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:39.987651 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07406684-6pkdb"] Oct 30 01:46:40.499318 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:40.499253 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07406684-6pkdb" event={"ID":"93e155d3-0a3e-49bf-8951-193b216ec87c","Type":"ContainerStarted","Data":"28b30e35920cd576d23c9245dccf9a333a03cbcfe66a01b90e1c9f8aa21d53d9"} Oct 30 01:46:41.408889 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:41.408818 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:41.555033 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:41.554903 6572 generic.go:356] "Generic (PLEG): container finished" podID="93e155d3-0a3e-49bf-8951-193b216ec87c" containerID="61cafa0ca98efd48ff866256aa1b3f80bdffa318324da945568c503a7092df4e" exitCode=0 Oct 30 01:46:41.555442 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:41.555413 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07406684-6pkdb" event={"ID":"93e155d3-0a3e-49bf-8951-193b216ec87c","Type":"ContainerDied","Data":"61cafa0ca98efd48ff866256aa1b3f80bdffa318324da945568c503a7092df4e"} Oct 30 01:46:41.616597 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:41.616554 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:41.820840 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:41.817325 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:42.877193 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:42.877109 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07406684-6pkdb" Oct 30 01:46:42.941937 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:42.941816 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93e155d3-0a3e-49bf-8951-193b216ec87c-kube-api-access-wmhzr\" (UniqueName: \"kubernetes.io/projected/93e155d3-0a3e-49bf-8951-193b216ec87c-kube-api-access-wmhzr\") pod \"93e155d3-0a3e-49bf-8951-193b216ec87c\" (UID: \"93e155d3-0a3e-49bf-8951-193b216ec87c\") " Oct 30 01:46:42.942159 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:42.942025 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93e155d3-0a3e-49bf-8951-193b216ec87c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93e155d3-0a3e-49bf-8951-193b216ec87c-maas-api-xks-ca\") pod \"93e155d3-0a3e-49bf-8951-193b216ec87c\" (UID: \"93e155d3-0a3e-49bf-8951-193b216ec87c\") " Oct 30 01:46:42.942159 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:42.942064 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93e155d3-0a3e-49bf-8951-193b216ec87c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93e155d3-0a3e-49bf-8951-193b216ec87c-maas-api-openshift-ca\") pod \"93e155d3-0a3e-49bf-8951-193b216ec87c\" (UID: \"93e155d3-0a3e-49bf-8951-193b216ec87c\") " Oct 30 01:46:42.943248 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:42.943213 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93e155d3-0a3e-49bf-8951-193b216ec87c-maas-api-openshift-ca" pod "93e155d3-0a3e-49bf-8951-193b216ec87c" (UID: "93e155d3-0a3e-49bf-8951-193b216ec87c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:46:42.944634 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:42.944604 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93e155d3-0a3e-49bf-8951-193b216ec87c-maas-api-xks-ca" pod "93e155d3-0a3e-49bf-8951-193b216ec87c" (UID: "93e155d3-0a3e-49bf-8951-193b216ec87c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:46:42.944828 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:42.944808 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93e155d3-0a3e-49bf-8951-193b216ec87c-kube-api-access-wmhzr" pod "93e155d3-0a3e-49bf-8951-193b216ec87c" (UID: "93e155d3-0a3e-49bf-8951-193b216ec87c"). InnerVolumeSpecName "kube-api-access-wmhzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:46:43.044397 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:43.044352 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmhzr\" (UniqueName: \"kubernetes.io/projected/93e155d3-0a3e-49bf-8951-193b216ec87c-kube-api-access-wmhzr\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:43.044397 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:43.044398 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93e155d3-0a3e-49bf-8951-193b216ec87c-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:43.044661 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:43.044410 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93e155d3-0a3e-49bf-8951-193b216ec87c-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:43.612473 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:43.612437 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07406684-6pkdb" Oct 30 01:46:43.612473 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:43.612464 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07406684-6pkdb" event={"ID":"93e155d3-0a3e-49bf-8951-193b216ec87c","Type":"ContainerDied","Data":"28b30e35920cd576d23c9245dccf9a333a03cbcfe66a01b90e1c9f8aa21d53d9"} Oct 30 01:46:43.612728 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:43.612529 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28b30e35920cd576d23c9245dccf9a333a03cbcfe66a01b90e1c9f8aa21d53d9" Oct 30 01:46:44.237557 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.237187 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:44.284712 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.284687 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:46:44.362733 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.362697 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c1aa377-ab2c-4653-a630-3d4eef49526d-istio-token\" (UniqueName: \"kubernetes.io/projected/2c1aa377-ab2c-4653-a630-3d4eef49526d-istio-token\") pod \"2c1aa377-ab2c-4653-a630-3d4eef49526d\" (UID: \"2c1aa377-ab2c-4653-a630-3d4eef49526d\") " Oct 30 01:46:44.362927 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.362825 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c1aa377-ab2c-4653-a630-3d4eef49526d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c1aa377-ab2c-4653-a630-3d4eef49526d-workload-certs\") pod \"2c1aa377-ab2c-4653-a630-3d4eef49526d\" (UID: \"2c1aa377-ab2c-4653-a630-3d4eef49526d\") " Oct 30 01:46:44.362927 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.362869 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c1aa377-ab2c-4653-a630-3d4eef49526d-kube-api-access-9n58v\" (UniqueName: \"kubernetes.io/projected/2c1aa377-ab2c-4653-a630-3d4eef49526d-kube-api-access-9n58v\") pod \"2c1aa377-ab2c-4653-a630-3d4eef49526d\" (UID: \"2c1aa377-ab2c-4653-a630-3d4eef49526d\") " Oct 30 01:46:44.362927 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.362913 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c1aa377-ab2c-4653-a630-3d4eef49526d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c1aa377-ab2c-4653-a630-3d4eef49526d-istio-podinfo\") pod \"2c1aa377-ab2c-4653-a630-3d4eef49526d\" (UID: \"2c1aa377-ab2c-4653-a630-3d4eef49526d\") " Oct 30 01:46:44.363188 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.362945 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c1aa377-ab2c-4653-a630-3d4eef49526d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1aa377-ab2c-4653-a630-3d4eef49526d-workload-socket\") pod \"2c1aa377-ab2c-4653-a630-3d4eef49526d\" (UID: \"2c1aa377-ab2c-4653-a630-3d4eef49526d\") " Oct 30 01:46:44.363188 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.363061 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c1aa377-ab2c-4653-a630-3d4eef49526d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c1aa377-ab2c-4653-a630-3d4eef49526d-istio-envoy\") pod \"2c1aa377-ab2c-4653-a630-3d4eef49526d\" (UID: \"2c1aa377-ab2c-4653-a630-3d4eef49526d\") " Oct 30 01:46:44.363316 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.363198 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c1aa377-ab2c-4653-a630-3d4eef49526d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1aa377-ab2c-4653-a630-3d4eef49526d-credential-socket\") pod \"2c1aa377-ab2c-4653-a630-3d4eef49526d\" (UID: \"2c1aa377-ab2c-4653-a630-3d4eef49526d\") " Oct 30 01:46:44.363316 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.363234 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c1aa377-ab2c-4653-a630-3d4eef49526d-workload-certs" pod "2c1aa377-ab2c-4653-a630-3d4eef49526d" (UID: "2c1aa377-ab2c-4653-a630-3d4eef49526d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:46:44.363316 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.363246 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c1aa377-ab2c-4653-a630-3d4eef49526d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c1aa377-ab2c-4653-a630-3d4eef49526d-istiod-ca-cert\") pod \"2c1aa377-ab2c-4653-a630-3d4eef49526d\" (UID: \"2c1aa377-ab2c-4653-a630-3d4eef49526d\") " Oct 30 01:46:44.363316 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.363289 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c1aa377-ab2c-4653-a630-3d4eef49526d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c1aa377-ab2c-4653-a630-3d4eef49526d-istio-data\") pod \"2c1aa377-ab2c-4653-a630-3d4eef49526d\" (UID: \"2c1aa377-ab2c-4653-a630-3d4eef49526d\") " Oct 30 01:46:44.364296 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.363970 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c1aa377-ab2c-4653-a630-3d4eef49526d-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:44.364296 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.364067 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c1aa377-ab2c-4653-a630-3d4eef49526d-workload-socket" pod "2c1aa377-ab2c-4653-a630-3d4eef49526d" (UID: "2c1aa377-ab2c-4653-a630-3d4eef49526d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:46:44.364296 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.364008 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c1aa377-ab2c-4653-a630-3d4eef49526d-credential-socket" pod "2c1aa377-ab2c-4653-a630-3d4eef49526d" (UID: "2c1aa377-ab2c-4653-a630-3d4eef49526d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:46:44.364296 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.364226 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c1aa377-ab2c-4653-a630-3d4eef49526d-istio-data" pod "2c1aa377-ab2c-4653-a630-3d4eef49526d" (UID: "2c1aa377-ab2c-4653-a630-3d4eef49526d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:46:44.364702 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.364678 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c1aa377-ab2c-4653-a630-3d4eef49526d-istiod-ca-cert" pod "2c1aa377-ab2c-4653-a630-3d4eef49526d" (UID: "2c1aa377-ab2c-4653-a630-3d4eef49526d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:46:44.365704 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.365676 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c1aa377-ab2c-4653-a630-3d4eef49526d-istio-token" pod "2c1aa377-ab2c-4653-a630-3d4eef49526d" (UID: "2c1aa377-ab2c-4653-a630-3d4eef49526d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:46:44.365704 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.365690 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c1aa377-ab2c-4653-a630-3d4eef49526d-istio-envoy" pod "2c1aa377-ab2c-4653-a630-3d4eef49526d" (UID: "2c1aa377-ab2c-4653-a630-3d4eef49526d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:46:44.368171 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.368136 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c1aa377-ab2c-4653-a630-3d4eef49526d-kube-api-access-9n58v" pod "2c1aa377-ab2c-4653-a630-3d4eef49526d" (UID: "2c1aa377-ab2c-4653-a630-3d4eef49526d"). InnerVolumeSpecName "kube-api-access-9n58v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:46:44.368737 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.368715 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c1aa377-ab2c-4653-a630-3d4eef49526d-istio-podinfo" pod "2c1aa377-ab2c-4653-a630-3d4eef49526d" (UID: "2c1aa377-ab2c-4653-a630-3d4eef49526d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:46:44.465649 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.465612 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n58v\" (UniqueName: \"kubernetes.io/projected/2c1aa377-ab2c-4653-a630-3d4eef49526d-kube-api-access-9n58v\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:44.465649 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.465645 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c1aa377-ab2c-4653-a630-3d4eef49526d-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:44.465649 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.465655 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1aa377-ab2c-4653-a630-3d4eef49526d-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:44.465966 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.465664 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c1aa377-ab2c-4653-a630-3d4eef49526d-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:44.465966 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.465673 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c1aa377-ab2c-4653-a630-3d4eef49526d-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:44.465966 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.465680 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c1aa377-ab2c-4653-a630-3d4eef49526d-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:44.465966 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.465690 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c1aa377-ab2c-4653-a630-3d4eef49526d-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:44.465966 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.465697 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c1aa377-ab2c-4653-a630-3d4eef49526d-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:44.622408 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.622374 6572 generic.go:356] "Generic (PLEG): container finished" podID="2c1aa377-ab2c-4653-a630-3d4eef49526d" containerID="72cc259568b451aac5ebe330f41c573ac8d2b1c09862bcf3a6962ec910780676" exitCode=0 Oct 30 01:46:44.622588 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.622434 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" Oct 30 01:46:44.622588 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.622469 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" event={"ID":"2c1aa377-ab2c-4653-a630-3d4eef49526d","Type":"ContainerDied","Data":"72cc259568b451aac5ebe330f41c573ac8d2b1c09862bcf3a6962ec910780676"} Oct 30 01:46:44.622588 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.622535 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl" event={"ID":"2c1aa377-ab2c-4653-a630-3d4eef49526d","Type":"ContainerDied","Data":"2aabc151dcdab27d124e5748d10ca31228a03cbec38b93914f0353c878dfe7f1"} Oct 30 01:46:44.622588 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.622572 6572 scope.go:122] "RemoveContainer" containerID="72cc259568b451aac5ebe330f41c573ac8d2b1c09862bcf3a6962ec910780676" Oct 30 01:46:44.634920 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.634899 6572 scope.go:122] "RemoveContainer" containerID="14fa9a4fa5bce69078f1ee86094df3c5f5a8a02dd48631a01adacb880f7bfdf8" Oct 30 01:46:44.646123 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.646100 6572 scope.go:122] "RemoveContainer" containerID="72cc259568b451aac5ebe330f41c573ac8d2b1c09862bcf3a6962ec910780676" Oct 30 01:46:44.646478 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:46:44.646446 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72cc259568b451aac5ebe330f41c573ac8d2b1c09862bcf3a6962ec910780676\": container with ID starting with 72cc259568b451aac5ebe330f41c573ac8d2b1c09862bcf3a6962ec910780676 not found: ID does not exist" containerID="72cc259568b451aac5ebe330f41c573ac8d2b1c09862bcf3a6962ec910780676" Oct 30 01:46:44.646596 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.646487 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72cc259568b451aac5ebe330f41c573ac8d2b1c09862bcf3a6962ec910780676"} err="failed to get container status \"72cc259568b451aac5ebe330f41c573ac8d2b1c09862bcf3a6962ec910780676\": rpc error: code = NotFound desc = could not find container \"72cc259568b451aac5ebe330f41c573ac8d2b1c09862bcf3a6962ec910780676\": container with ID starting with 72cc259568b451aac5ebe330f41c573ac8d2b1c09862bcf3a6962ec910780676 not found: ID does not exist" Oct 30 01:46:44.646596 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.646517 6572 scope.go:122] "RemoveContainer" containerID="14fa9a4fa5bce69078f1ee86094df3c5f5a8a02dd48631a01adacb880f7bfdf8" Oct 30 01:46:44.646855 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:46:44.646827 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14fa9a4fa5bce69078f1ee86094df3c5f5a8a02dd48631a01adacb880f7bfdf8\": container with ID starting with 14fa9a4fa5bce69078f1ee86094df3c5f5a8a02dd48631a01adacb880f7bfdf8 not found: ID does not exist" containerID="14fa9a4fa5bce69078f1ee86094df3c5f5a8a02dd48631a01adacb880f7bfdf8" Oct 30 01:46:44.646972 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.646859 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14fa9a4fa5bce69078f1ee86094df3c5f5a8a02dd48631a01adacb880f7bfdf8"} err="failed to get container status \"14fa9a4fa5bce69078f1ee86094df3c5f5a8a02dd48631a01adacb880f7bfdf8\": rpc error: code = NotFound desc = could not find container \"14fa9a4fa5bce69078f1ee86094df3c5f5a8a02dd48631a01adacb880f7bfdf8\": container with ID starting with 14fa9a4fa5bce69078f1ee86094df3c5f5a8a02dd48631a01adacb880f7bfdf8 not found: ID does not exist" Oct 30 01:46:44.676526 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.676493 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl"] Oct 30 01:46:44.688147 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:44.688118 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-f197b1-openshift-default-5c59d4665-vh9hl"] Oct 30 01:46:44.690399 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:46:44.690373 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:46:45.897813 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:45.897597 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:46.543208 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:46.543160 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c1aa377-ab2c-4653-a630-3d4eef49526d" path="/var/lib/kubelet/pods/2c1aa377-ab2c-4653-a630-3d4eef49526d/volumes" Oct 30 01:46:47.601723 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:47.601011 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:49.036245 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:49.036204 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cc3986b-q5gw2"] Oct 30 01:46:49.036837 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:49.036819 6572 cpu_manager.go:414] "containerMap: removing container" podUID="2c1aa377-ab2c-4653-a630-3d4eef49526d" containerName="istio-proxy" Oct 30 01:46:49.036914 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:49.036839 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c1aa377-ab2c-4653-a630-3d4eef49526d" containerName="istio-proxy" Oct 30 01:46:49.036914 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:49.036872 6572 cpu_manager.go:414] "containerMap: removing container" podUID="93e155d3-0a3e-49bf-8951-193b216ec87c" containerName="revoke-keys" Oct 30 01:46:49.036914 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:49.036878 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93e155d3-0a3e-49bf-8951-193b216ec87c" containerName="revoke-keys" Oct 30 01:46:49.037010 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:49.036963 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c1aa377-ab2c-4653-a630-3d4eef49526d" containerName="istio-proxy" Oct 30 01:46:49.037010 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:49.036975 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c1aa377-ab2c-4653-a630-3d4eef49526d" containerName="istio-proxy" Oct 30 01:46:49.037010 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:49.036981 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="93e155d3-0a3e-49bf-8951-193b216ec87c" containerName="revoke-keys" Oct 30 01:46:49.042171 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:49.042151 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cc3986b-q5gw2" Oct 30 01:46:49.062198 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:49.062164 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cc3986b-q5gw2"] Oct 30 01:46:49.066971 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:49.066927 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:46:49.067169 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:49.067080 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:46:49.067169 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:49.067107 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:46:49.067169 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:49.067144 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:46:49.128775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:49.128721 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shdpl\" (UniqueName: \"kubernetes.io/projected/db6add3a-0d1b-462f-bee9-09d022cd41ab-kube-api-access-shdpl\") pod \"maas-api-revoke-sub-5cc3986b-q5gw2\" (UID: \"db6add3a-0d1b-462f-bee9-09d022cd41ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cc3986b-q5gw2" Oct 30 01:46:49.128961 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:49.128901 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db6add3a-0d1b-462f-bee9-09d022cd41ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cc3986b-q5gw2\" (UID: \"db6add3a-0d1b-462f-bee9-09d022cd41ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cc3986b-q5gw2" Oct 30 01:46:49.129169 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:49.129144 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db6add3a-0d1b-462f-bee9-09d022cd41ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cc3986b-q5gw2\" (UID: \"db6add3a-0d1b-462f-bee9-09d022cd41ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cc3986b-q5gw2" Oct 30 01:46:49.230356 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:49.230319 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db6add3a-0d1b-462f-bee9-09d022cd41ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cc3986b-q5gw2\" (UID: \"db6add3a-0d1b-462f-bee9-09d022cd41ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cc3986b-q5gw2" Oct 30 01:46:49.230558 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:49.230386 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shdpl\" (UniqueName: \"kubernetes.io/projected/db6add3a-0d1b-462f-bee9-09d022cd41ab-kube-api-access-shdpl\") pod \"maas-api-revoke-sub-5cc3986b-q5gw2\" (UID: \"db6add3a-0d1b-462f-bee9-09d022cd41ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cc3986b-q5gw2" Oct 30 01:46:49.230558 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:49.230441 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db6add3a-0d1b-462f-bee9-09d022cd41ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cc3986b-q5gw2\" (UID: \"db6add3a-0d1b-462f-bee9-09d022cd41ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cc3986b-q5gw2" Oct 30 01:46:49.231305 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:49.231276 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db6add3a-0d1b-462f-bee9-09d022cd41ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cc3986b-q5gw2\" (UID: \"db6add3a-0d1b-462f-bee9-09d022cd41ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cc3986b-q5gw2" Oct 30 01:46:49.235577 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:49.235553 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db6add3a-0d1b-462f-bee9-09d022cd41ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cc3986b-q5gw2\" (UID: \"db6add3a-0d1b-462f-bee9-09d022cd41ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cc3986b-q5gw2" Oct 30 01:46:49.262190 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:49.262122 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shdpl\" (UniqueName: \"kubernetes.io/projected/db6add3a-0d1b-462f-bee9-09d022cd41ab-kube-api-access-shdpl\") pod \"maas-api-revoke-sub-5cc3986b-q5gw2\" (UID: \"db6add3a-0d1b-462f-bee9-09d022cd41ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cc3986b-q5gw2" Oct 30 01:46:49.308142 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:49.308054 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:49.357224 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:49.357190 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cc3986b-q5gw2" Oct 30 01:46:49.666621 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:49.666589 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cc3986b-q5gw2"] Oct 30 01:46:49.666824 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:46:49.666648 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb6add3a_0d1b_462f_bee9_09d022cd41ab.slice/crio-c285eec055aee1723037c2edb2f51046d69f7902eee1d51e04b8554818120b50 WatchSource:0}: Error finding container c285eec055aee1723037c2edb2f51046d69f7902eee1d51e04b8554818120b50: Status 404 returned error can't find the container with id c285eec055aee1723037c2edb2f51046d69f7902eee1d51e04b8554818120b50 Oct 30 01:46:50.056151 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:50.056114 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-8f68ab-57c584d79b-n88mc"] Oct 30 01:46:50.096378 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:50.096340 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-8f68ab-7dbc58dc79-zl8qv"] Oct 30 01:46:50.300290 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:50.300202 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:50.512629 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:50.512600 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:50.683156 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:50.683120 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cc3986b-q5gw2" event={"ID":"db6add3a-0d1b-462f-bee9-09d022cd41ab","Type":"ContainerStarted","Data":"e70f26ded9b2648143e8a4bd32980443d1322656c0e2ecf05cb6a336c318c207"} Oct 30 01:46:50.683337 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:50.683180 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cc3986b-q5gw2" event={"ID":"db6add3a-0d1b-462f-bee9-09d022cd41ab","Type":"ContainerStarted","Data":"c285eec055aee1723037c2edb2f51046d69f7902eee1d51e04b8554818120b50"} Oct 30 01:46:50.728387 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:50.728329 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cc3986b-q5gw2" podStartSLOduration=1.728313134 podStartE2EDuration="1.728313134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:46:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:46:50.726061116 +0000 UTC m=+1736.302486518" watchObservedRunningTime="2026-10-30 01:46:50.728313134 +0000 UTC m=+1736.304738536" Oct 30 01:46:50.883698 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:50.883661 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p"] Oct 30 01:46:50.885420 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:50.885391 6572 cpu_manager.go:414] "containerMap: removing container" podUID="2c1aa377-ab2c-4653-a630-3d4eef49526d" containerName="istio-proxy" Oct 30 01:46:50.885420 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:50.885415 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c1aa377-ab2c-4653-a630-3d4eef49526d" containerName="istio-proxy" Oct 30 01:46:50.922776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:50.893100 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:46:50.922776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:50.914937 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p"] Oct 30 01:46:50.922776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:50.918941 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-abb30f-openshift-default-dockercfg-s7f6g\"" Oct 30 01:46:51.047380 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.047268 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:51.066055 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.066016 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/414b871d-0343-42a6-b560-0be9f9ff4fce-istio-envoy\") pod \"e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p\" (UID: \"414b871d-0343-42a6-b560-0be9f9ff4fce\") " pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:46:51.067090 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.066074 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/414b871d-0343-42a6-b560-0be9f9ff4fce-workload-socket\") pod \"e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p\" (UID: \"414b871d-0343-42a6-b560-0be9f9ff4fce\") " pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:46:51.067090 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.066155 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/414b871d-0343-42a6-b560-0be9f9ff4fce-istio-token\") pod \"e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p\" (UID: \"414b871d-0343-42a6-b560-0be9f9ff4fce\") " pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:46:51.067090 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.066209 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/414b871d-0343-42a6-b560-0be9f9ff4fce-istio-data\") pod \"e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p\" (UID: \"414b871d-0343-42a6-b560-0be9f9ff4fce\") " pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:46:51.067090 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.066237 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/414b871d-0343-42a6-b560-0be9f9ff4fce-istiod-ca-cert\") pod \"e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p\" (UID: \"414b871d-0343-42a6-b560-0be9f9ff4fce\") " pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:46:51.067090 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.066288 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/414b871d-0343-42a6-b560-0be9f9ff4fce-istio-podinfo\") pod \"e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p\" (UID: \"414b871d-0343-42a6-b560-0be9f9ff4fce\") " pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:46:51.067090 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.066356 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/414b871d-0343-42a6-b560-0be9f9ff4fce-workload-certs\") pod \"e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p\" (UID: \"414b871d-0343-42a6-b560-0be9f9ff4fce\") " pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:46:51.067090 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.066458 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/414b871d-0343-42a6-b560-0be9f9ff4fce-credential-socket\") pod \"e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p\" (UID: \"414b871d-0343-42a6-b560-0be9f9ff4fce\") " pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:46:51.067090 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.066523 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghnnn\" (UniqueName: \"kubernetes.io/projected/414b871d-0343-42a6-b560-0be9f9ff4fce-kube-api-access-ghnnn\") pod \"e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p\" (UID: \"414b871d-0343-42a6-b560-0be9f9ff4fce\") " pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:46:51.168010 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.167969 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/414b871d-0343-42a6-b560-0be9f9ff4fce-istio-token\") pod \"e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p\" (UID: \"414b871d-0343-42a6-b560-0be9f9ff4fce\") " pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:46:51.168214 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.168032 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/414b871d-0343-42a6-b560-0be9f9ff4fce-istio-data\") pod \"e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p\" (UID: \"414b871d-0343-42a6-b560-0be9f9ff4fce\") " pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:46:51.168214 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.168051 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/414b871d-0343-42a6-b560-0be9f9ff4fce-istiod-ca-cert\") pod \"e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p\" (UID: \"414b871d-0343-42a6-b560-0be9f9ff4fce\") " pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:46:51.168214 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.168086 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/414b871d-0343-42a6-b560-0be9f9ff4fce-istio-podinfo\") pod \"e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p\" (UID: \"414b871d-0343-42a6-b560-0be9f9ff4fce\") " pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:46:51.168214 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.168151 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/414b871d-0343-42a6-b560-0be9f9ff4fce-workload-certs\") pod \"e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p\" (UID: \"414b871d-0343-42a6-b560-0be9f9ff4fce\") " pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:46:51.168420 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.168257 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/414b871d-0343-42a6-b560-0be9f9ff4fce-credential-socket\") pod \"e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p\" (UID: \"414b871d-0343-42a6-b560-0be9f9ff4fce\") " pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:46:51.168420 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.168320 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghnnn\" (UniqueName: \"kubernetes.io/projected/414b871d-0343-42a6-b560-0be9f9ff4fce-kube-api-access-ghnnn\") pod \"e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p\" (UID: \"414b871d-0343-42a6-b560-0be9f9ff4fce\") " pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:46:51.168420 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.168364 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/414b871d-0343-42a6-b560-0be9f9ff4fce-istio-envoy\") pod \"e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p\" (UID: \"414b871d-0343-42a6-b560-0be9f9ff4fce\") " pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:46:51.168420 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.168397 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/414b871d-0343-42a6-b560-0be9f9ff4fce-workload-socket\") pod \"e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p\" (UID: \"414b871d-0343-42a6-b560-0be9f9ff4fce\") " pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:46:51.168659 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.168541 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/414b871d-0343-42a6-b560-0be9f9ff4fce-istio-data\") pod \"e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p\" (UID: \"414b871d-0343-42a6-b560-0be9f9ff4fce\") " pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:46:51.168659 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.168626 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/414b871d-0343-42a6-b560-0be9f9ff4fce-workload-certs\") pod \"e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p\" (UID: \"414b871d-0343-42a6-b560-0be9f9ff4fce\") " pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:46:51.168811 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.168711 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/414b871d-0343-42a6-b560-0be9f9ff4fce-credential-socket\") pod \"e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p\" (UID: \"414b871d-0343-42a6-b560-0be9f9ff4fce\") " pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:46:51.168861 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.168840 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/414b871d-0343-42a6-b560-0be9f9ff4fce-workload-socket\") pod \"e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p\" (UID: \"414b871d-0343-42a6-b560-0be9f9ff4fce\") " pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:46:51.169015 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.168996 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/414b871d-0343-42a6-b560-0be9f9ff4fce-istiod-ca-cert\") pod \"e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p\" (UID: \"414b871d-0343-42a6-b560-0be9f9ff4fce\") " pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:46:51.170796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.170775 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/414b871d-0343-42a6-b560-0be9f9ff4fce-istio-envoy\") pod \"e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p\" (UID: \"414b871d-0343-42a6-b560-0be9f9ff4fce\") " pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:46:51.171056 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.171030 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/414b871d-0343-42a6-b560-0be9f9ff4fce-istio-podinfo\") pod \"e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p\" (UID: \"414b871d-0343-42a6-b560-0be9f9ff4fce\") " pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:46:51.189547 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.189511 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/414b871d-0343-42a6-b560-0be9f9ff4fce-istio-token\") pod \"e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p\" (UID: \"414b871d-0343-42a6-b560-0be9f9ff4fce\") " pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:46:51.189697 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.189607 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghnnn\" (UniqueName: \"kubernetes.io/projected/414b871d-0343-42a6-b560-0be9f9ff4fce-kube-api-access-ghnnn\") pod \"e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p\" (UID: \"414b871d-0343-42a6-b560-0be9f9ff4fce\") " pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:46:51.269531 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.269486 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:46:51.321166 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.321117 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:51.560598 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.560491 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:51.629391 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:46:51.629353 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod414b871d_0343_42a6_b560_0be9f9ff4fce.slice/crio-8d74e2be1beddcf7980a28d5b40d0c2b4c665a270a8ae6c77803a08021693ebd WatchSource:0}: Error finding container 8d74e2be1beddcf7980a28d5b40d0c2b4c665a270a8ae6c77803a08021693ebd: Status 404 returned error can't find the container with id 8d74e2be1beddcf7980a28d5b40d0c2b4c665a270a8ae6c77803a08021693ebd Oct 30 01:46:51.630054 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.630033 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p"] Oct 30 01:46:51.632147 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.632111 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:46:51.632247 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.632180 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:46:51.632247 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.632212 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:46:51.694896 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.694859 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" event={"ID":"414b871d-0343-42a6-b560-0be9f9ff4fce","Type":"ContainerStarted","Data":"8d74e2be1beddcf7980a28d5b40d0c2b4c665a270a8ae6c77803a08021693ebd"} Oct 30 01:46:51.696760 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.696718 6572 generic.go:356] "Generic (PLEG): container finished" podID="db6add3a-0d1b-462f-bee9-09d022cd41ab" containerID="e70f26ded9b2648143e8a4bd32980443d1322656c0e2ecf05cb6a336c318c207" exitCode=0 Oct 30 01:46:51.696910 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:51.696826 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cc3986b-q5gw2" event={"ID":"db6add3a-0d1b-462f-bee9-09d022cd41ab","Type":"ContainerDied","Data":"e70f26ded9b2648143e8a4bd32980443d1322656c0e2ecf05cb6a336c318c207"} Oct 30 01:46:52.717406 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:52.717359 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" event={"ID":"414b871d-0343-42a6-b560-0be9f9ff4fce","Type":"ContainerStarted","Data":"11c56fe9916e904a2d2119cedb366a949764a0794f4ea155e5fd1c4223fcf754"} Oct 30 01:46:52.840176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:52.840100 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" podStartSLOduration=2.840076422 podStartE2EDuration="2.840076422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:46:52.834089773 +0000 UTC m=+1738.410515176" watchObservedRunningTime="2026-10-30 01:46:52.840076422 +0000 UTC m=+1738.416501825" Oct 30 01:46:52.946514 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:52.946477 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:52.958037 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:52.958013 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cc3986b-q5gw2" Oct 30 01:46:53.122945 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:53.119458 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db6add3a-0d1b-462f-bee9-09d022cd41ab-kube-api-access-shdpl\" (UniqueName: \"kubernetes.io/projected/db6add3a-0d1b-462f-bee9-09d022cd41ab-kube-api-access-shdpl\") pod \"db6add3a-0d1b-462f-bee9-09d022cd41ab\" (UID: \"db6add3a-0d1b-462f-bee9-09d022cd41ab\") " Oct 30 01:46:53.122945 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:53.119695 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db6add3a-0d1b-462f-bee9-09d022cd41ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db6add3a-0d1b-462f-bee9-09d022cd41ab-maas-api-xks-ca\") pod \"db6add3a-0d1b-462f-bee9-09d022cd41ab\" (UID: \"db6add3a-0d1b-462f-bee9-09d022cd41ab\") " Oct 30 01:46:53.122945 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:53.119731 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db6add3a-0d1b-462f-bee9-09d022cd41ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db6add3a-0d1b-462f-bee9-09d022cd41ab-maas-api-openshift-ca\") pod \"db6add3a-0d1b-462f-bee9-09d022cd41ab\" (UID: \"db6add3a-0d1b-462f-bee9-09d022cd41ab\") " Oct 30 01:46:53.122945 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:53.120576 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db6add3a-0d1b-462f-bee9-09d022cd41ab-maas-api-openshift-ca" pod "db6add3a-0d1b-462f-bee9-09d022cd41ab" (UID: "db6add3a-0d1b-462f-bee9-09d022cd41ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:46:53.132558 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:53.132515 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db6add3a-0d1b-462f-bee9-09d022cd41ab-kube-api-access-shdpl" pod "db6add3a-0d1b-462f-bee9-09d022cd41ab" (UID: "db6add3a-0d1b-462f-bee9-09d022cd41ab"). InnerVolumeSpecName "kube-api-access-shdpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:46:53.133057 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:53.133020 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db6add3a-0d1b-462f-bee9-09d022cd41ab-maas-api-xks-ca" pod "db6add3a-0d1b-462f-bee9-09d022cd41ab" (UID: "db6add3a-0d1b-462f-bee9-09d022cd41ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:46:53.221045 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:53.221005 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shdpl\" (UniqueName: \"kubernetes.io/projected/db6add3a-0d1b-462f-bee9-09d022cd41ab-kube-api-access-shdpl\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:53.221045 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:53.221041 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db6add3a-0d1b-462f-bee9-09d022cd41ab-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:53.221045 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:53.221053 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db6add3a-0d1b-462f-bee9-09d022cd41ab-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:53.269679 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:53.269639 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:46:53.376952 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:53.376803 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:46:53.600604 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:53.600565 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:53.751546 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:53.751512 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cc3986b-q5gw2" Oct 30 01:46:53.751546 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:53.751527 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cc3986b-q5gw2" event={"ID":"db6add3a-0d1b-462f-bee9-09d022cd41ab","Type":"ContainerDied","Data":"c285eec055aee1723037c2edb2f51046d69f7902eee1d51e04b8554818120b50"} Oct 30 01:46:53.752106 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:53.751598 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c285eec055aee1723037c2edb2f51046d69f7902eee1d51e04b8554818120b50" Oct 30 01:46:53.752106 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:53.751901 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:46:53.755890 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:53.755859 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:46:53.809693 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:53.809659 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:54.224272 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:54.224242 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:55.056465 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:55.056409 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-8f68ab-57c584d79b-n88mc" podUID="0731d0f8-7e67-4fce-84ea-84441098c02d" containerName="payload-processing" containerID="cri-o://1a274491c04dce642c0d9007b015b44bdf0f0bf84120362fb9e8812706004cd3" gracePeriod=25 Oct 30 01:46:55.096879 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:55.096812 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8f68ab-7dbc58dc79-zl8qv" podUID="26657c68-1439-4806-b039-13baea561144" containerName="payload-pre-processing" containerID="cri-o://171f11a91479f0ddb23834903a3742cec4df8ffa6ce8e28b7101017e8ae15e5f" gracePeriod=25 Oct 30 01:46:55.375234 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:55.375207 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-8f68ab-57c584d79b-n88mc" Oct 30 01:46:55.427965 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:55.427937 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8f68ab-7dbc58dc79-zl8qv" Oct 30 01:46:55.450800 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:55.450771 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0731d0f8-7e67-4fce-84ea-84441098c02d-kube-api-access-mkrsk\" (UniqueName: \"kubernetes.io/projected/0731d0f8-7e67-4fce-84ea-84441098c02d-kube-api-access-mkrsk\") pod \"0731d0f8-7e67-4fce-84ea-84441098c02d\" (UID: \"0731d0f8-7e67-4fce-84ea-84441098c02d\") " Oct 30 01:46:55.450998 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:55.450872 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26657c68-1439-4806-b039-13baea561144-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26657c68-1439-4806-b039-13baea561144-plugins-config-volume\") pod \"26657c68-1439-4806-b039-13baea561144\" (UID: \"26657c68-1439-4806-b039-13baea561144\") " Oct 30 01:46:55.450998 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:55.450902 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26657c68-1439-4806-b039-13baea561144-kube-api-access-2vhsn\" (UniqueName: \"kubernetes.io/projected/26657c68-1439-4806-b039-13baea561144-kube-api-access-2vhsn\") pod \"26657c68-1439-4806-b039-13baea561144\" (UID: \"26657c68-1439-4806-b039-13baea561144\") " Oct 30 01:46:55.450998 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:55.450972 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0731d0f8-7e67-4fce-84ea-84441098c02d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0731d0f8-7e67-4fce-84ea-84441098c02d-plugins-config-volume\") pod \"0731d0f8-7e67-4fce-84ea-84441098c02d\" (UID: \"0731d0f8-7e67-4fce-84ea-84441098c02d\") " Oct 30 01:46:55.451307 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:55.451279 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26657c68-1439-4806-b039-13baea561144-plugins-config-volume" pod "26657c68-1439-4806-b039-13baea561144" (UID: "26657c68-1439-4806-b039-13baea561144"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:46:55.451783 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:55.451740 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26657c68-1439-4806-b039-13baea561144-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:55.452475 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:55.452433 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0731d0f8-7e67-4fce-84ea-84441098c02d-plugins-config-volume" pod "0731d0f8-7e67-4fce-84ea-84441098c02d" (UID: "0731d0f8-7e67-4fce-84ea-84441098c02d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:46:55.454805 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:55.454732 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0731d0f8-7e67-4fce-84ea-84441098c02d-kube-api-access-mkrsk" pod "0731d0f8-7e67-4fce-84ea-84441098c02d" (UID: "0731d0f8-7e67-4fce-84ea-84441098c02d"). InnerVolumeSpecName "kube-api-access-mkrsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:46:55.456105 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:55.456065 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26657c68-1439-4806-b039-13baea561144-kube-api-access-2vhsn" pod "26657c68-1439-4806-b039-13baea561144" (UID: "26657c68-1439-4806-b039-13baea561144"). InnerVolumeSpecName "kube-api-access-2vhsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:46:55.487482 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:55.487448 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:55.553841 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:55.553805 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkrsk\" (UniqueName: \"kubernetes.io/projected/0731d0f8-7e67-4fce-84ea-84441098c02d-kube-api-access-mkrsk\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:55.553841 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:55.553841 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vhsn\" (UniqueName: \"kubernetes.io/projected/26657c68-1439-4806-b039-13baea561144-kube-api-access-2vhsn\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:55.554048 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:55.553857 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0731d0f8-7e67-4fce-84ea-84441098c02d-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:46:55.770913 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:55.770880 6572 generic.go:356] "Generic (PLEG): container finished" podID="0731d0f8-7e67-4fce-84ea-84441098c02d" containerID="1a274491c04dce642c0d9007b015b44bdf0f0bf84120362fb9e8812706004cd3" exitCode=0 Oct 30 01:46:55.771117 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:55.770940 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-8f68ab-57c584d79b-n88mc" Oct 30 01:46:55.771117 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:55.770968 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-8f68ab-57c584d79b-n88mc" event={"ID":"0731d0f8-7e67-4fce-84ea-84441098c02d","Type":"ContainerDied","Data":"1a274491c04dce642c0d9007b015b44bdf0f0bf84120362fb9e8812706004cd3"} Oct 30 01:46:55.771117 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:55.771030 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-8f68ab-57c584d79b-n88mc" event={"ID":"0731d0f8-7e67-4fce-84ea-84441098c02d","Type":"ContainerDied","Data":"b63d79e1441c85134de219f1302dc9f1793cd43521a74781bf1fe94dc985af0d"} Oct 30 01:46:55.771117 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:55.771047 6572 scope.go:122] "RemoveContainer" containerID="1a274491c04dce642c0d9007b015b44bdf0f0bf84120362fb9e8812706004cd3" Oct 30 01:46:55.773083 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:55.773057 6572 generic.go:356] "Generic (PLEG): container finished" podID="26657c68-1439-4806-b039-13baea561144" containerID="171f11a91479f0ddb23834903a3742cec4df8ffa6ce8e28b7101017e8ae15e5f" exitCode=0 Oct 30 01:46:55.773214 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:55.773111 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8f68ab-7dbc58dc79-zl8qv" Oct 30 01:46:55.773214 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:55.773126 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8f68ab-7dbc58dc79-zl8qv" event={"ID":"26657c68-1439-4806-b039-13baea561144","Type":"ContainerDied","Data":"171f11a91479f0ddb23834903a3742cec4df8ffa6ce8e28b7101017e8ae15e5f"} Oct 30 01:46:55.773214 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:55.773174 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-8f68ab-7dbc58dc79-zl8qv" event={"ID":"26657c68-1439-4806-b039-13baea561144","Type":"ContainerDied","Data":"3894c641bf534def77b58e2110715c978f3466f43172af6a728e90791edd96b9"} Oct 30 01:46:55.786273 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:55.786256 6572 scope.go:122] "RemoveContainer" containerID="1a274491c04dce642c0d9007b015b44bdf0f0bf84120362fb9e8812706004cd3" Oct 30 01:46:55.786603 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:46:55.786585 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a274491c04dce642c0d9007b015b44bdf0f0bf84120362fb9e8812706004cd3\": container with ID starting with 1a274491c04dce642c0d9007b015b44bdf0f0bf84120362fb9e8812706004cd3 not found: ID does not exist" containerID="1a274491c04dce642c0d9007b015b44bdf0f0bf84120362fb9e8812706004cd3" Oct 30 01:46:55.786687 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:55.786613 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a274491c04dce642c0d9007b015b44bdf0f0bf84120362fb9e8812706004cd3"} err="failed to get container status \"1a274491c04dce642c0d9007b015b44bdf0f0bf84120362fb9e8812706004cd3\": rpc error: code = NotFound desc = could not find container \"1a274491c04dce642c0d9007b015b44bdf0f0bf84120362fb9e8812706004cd3\": container with ID starting with 1a274491c04dce642c0d9007b015b44bdf0f0bf84120362fb9e8812706004cd3 not found: ID does not exist" Oct 30 01:46:55.786687 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:55.786634 6572 scope.go:122] "RemoveContainer" containerID="171f11a91479f0ddb23834903a3742cec4df8ffa6ce8e28b7101017e8ae15e5f" Oct 30 01:46:55.798375 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:55.798347 6572 scope.go:122] "RemoveContainer" containerID="171f11a91479f0ddb23834903a3742cec4df8ffa6ce8e28b7101017e8ae15e5f" Oct 30 01:46:55.798872 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:46:55.798728 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"171f11a91479f0ddb23834903a3742cec4df8ffa6ce8e28b7101017e8ae15e5f\": container with ID starting with 171f11a91479f0ddb23834903a3742cec4df8ffa6ce8e28b7101017e8ae15e5f not found: ID does not exist" containerID="171f11a91479f0ddb23834903a3742cec4df8ffa6ce8e28b7101017e8ae15e5f" Oct 30 01:46:55.798872 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:55.798829 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"171f11a91479f0ddb23834903a3742cec4df8ffa6ce8e28b7101017e8ae15e5f"} err="failed to get container status \"171f11a91479f0ddb23834903a3742cec4df8ffa6ce8e28b7101017e8ae15e5f\": rpc error: code = NotFound desc = could not find container \"171f11a91479f0ddb23834903a3742cec4df8ffa6ce8e28b7101017e8ae15e5f\": container with ID starting with 171f11a91479f0ddb23834903a3742cec4df8ffa6ce8e28b7101017e8ae15e5f not found: ID does not exist" Oct 30 01:46:55.838558 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:55.838524 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-8f68ab-57c584d79b-n88mc"] Oct 30 01:46:55.853457 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:55.853418 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-8f68ab-57c584d79b-n88mc"] Oct 30 01:46:55.883903 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:55.883870 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-8f68ab-7dbc58dc79-zl8qv"] Oct 30 01:46:55.888322 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:55.888292 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-8f68ab-7dbc58dc79-zl8qv"] Oct 30 01:46:56.542619 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:56.542581 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0731d0f8-7e67-4fce-84ea-84441098c02d" path="/var/lib/kubelet/pods/0731d0f8-7e67-4fce-84ea-84441098c02d/volumes" Oct 30 01:46:56.543199 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:56.543178 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26657c68-1439-4806-b039-13baea561144" path="/var/lib/kubelet/pods/26657c68-1439-4806-b039-13baea561144/volumes" Oct 30 01:46:56.671969 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:56.671939 6572 scope.go:122] "RemoveContainer" containerID="996d63491e9c5e2f791889c7b994eafc1f0873a1b3e2e12d5abef087348815ae" Oct 30 01:46:56.682986 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:56.682962 6572 scope.go:122] "RemoveContainer" containerID="cae2e61e4343e7b48804ce1812c41a9ec81782033838c25e75712fdd6963f4dc" Oct 30 01:46:56.694449 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:56.694422 6572 scope.go:122] "RemoveContainer" containerID="e9dd87355e8fda4cd0bf8a1c2d0761e7bcbaed9127e8353b7dbe57dc7b3100a1" Oct 30 01:46:58.398792 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:58.398730 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:59.224630 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:46:59.224589 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:00.051402 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:00.051351 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:00.624380 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:00.624301 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:00.836259 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:00.836223 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:01.047290 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:01.047259 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:01.230784 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:01.230722 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:01.411334 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:01.411298 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:02.418386 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:02.418345 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:02.866829 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:02.866672 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:03.053538 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.053502 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:03.238774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.237980 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq"] Oct 30 01:47:03.239013 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.238878 6572 cpu_manager.go:414] "containerMap: removing container" podUID="0731d0f8-7e67-4fce-84ea-84441098c02d" containerName="payload-processing" Oct 30 01:47:03.239013 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.238900 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0731d0f8-7e67-4fce-84ea-84441098c02d" containerName="payload-processing" Oct 30 01:47:03.239013 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.238956 6572 cpu_manager.go:414] "containerMap: removing container" podUID="26657c68-1439-4806-b039-13baea561144" containerName="payload-pre-processing" Oct 30 01:47:03.239013 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.238964 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26657c68-1439-4806-b039-13baea561144" containerName="payload-pre-processing" Oct 30 01:47:03.239013 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.239000 6572 cpu_manager.go:414] "containerMap: removing container" podUID="db6add3a-0d1b-462f-bee9-09d022cd41ab" containerName="revoke-keys" Oct 30 01:47:03.239013 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.239008 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db6add3a-0d1b-462f-bee9-09d022cd41ab" containerName="revoke-keys" Oct 30 01:47:03.239384 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.239159 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="0731d0f8-7e67-4fce-84ea-84441098c02d" containerName="payload-processing" Oct 30 01:47:03.239384 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.239180 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="26657c68-1439-4806-b039-13baea561144" containerName="payload-pre-processing" Oct 30 01:47:03.239384 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.239194 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="db6add3a-0d1b-462f-bee9-09d022cd41ab" containerName="revoke-keys" Oct 30 01:47:03.245542 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.245508 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 01:47:03.268599 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.268566 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq"] Oct 30 01:47:03.277121 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.277087 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-c8d376-openshift-default-dockercfg-lmdrw\"" Oct 30 01:47:03.398570 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.398527 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/322cef03-4f70-42c8-8571-42c939fba68b-istio-token\") pod \"e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq\" (UID: \"322cef03-4f70-42c8-8571-42c939fba68b\") " pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 01:47:03.399565 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.399534 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwsvk\" (UniqueName: \"kubernetes.io/projected/322cef03-4f70-42c8-8571-42c939fba68b-kube-api-access-qwsvk\") pod \"e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq\" (UID: \"322cef03-4f70-42c8-8571-42c939fba68b\") " pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 01:47:03.399854 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.399816 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/322cef03-4f70-42c8-8571-42c939fba68b-credential-socket\") pod \"e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq\" (UID: \"322cef03-4f70-42c8-8571-42c939fba68b\") " pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 01:47:03.399994 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.399869 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/322cef03-4f70-42c8-8571-42c939fba68b-istio-data\") pod \"e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq\" (UID: \"322cef03-4f70-42c8-8571-42c939fba68b\") " pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 01:47:03.399994 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.399962 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/322cef03-4f70-42c8-8571-42c939fba68b-workload-socket\") pod \"e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq\" (UID: \"322cef03-4f70-42c8-8571-42c939fba68b\") " pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 01:47:03.400133 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.400015 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/322cef03-4f70-42c8-8571-42c939fba68b-istio-envoy\") pod \"e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq\" (UID: \"322cef03-4f70-42c8-8571-42c939fba68b\") " pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 01:47:03.400133 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.400050 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/322cef03-4f70-42c8-8571-42c939fba68b-workload-certs\") pod \"e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq\" (UID: \"322cef03-4f70-42c8-8571-42c939fba68b\") " pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 01:47:03.400248 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.400131 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/322cef03-4f70-42c8-8571-42c939fba68b-istio-podinfo\") pod \"e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq\" (UID: \"322cef03-4f70-42c8-8571-42c939fba68b\") " pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 01:47:03.400311 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.400255 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/322cef03-4f70-42c8-8571-42c939fba68b-istiod-ca-cert\") pod \"e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq\" (UID: \"322cef03-4f70-42c8-8571-42c939fba68b\") " pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 01:47:03.441371 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.433930 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:03.508090 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.505670 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/322cef03-4f70-42c8-8571-42c939fba68b-istio-envoy\") pod \"e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq\" (UID: \"322cef03-4f70-42c8-8571-42c939fba68b\") " pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 01:47:03.508090 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.505731 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/322cef03-4f70-42c8-8571-42c939fba68b-workload-certs\") pod \"e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq\" (UID: \"322cef03-4f70-42c8-8571-42c939fba68b\") " pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 01:47:03.508090 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.505828 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/322cef03-4f70-42c8-8571-42c939fba68b-istio-podinfo\") pod \"e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq\" (UID: \"322cef03-4f70-42c8-8571-42c939fba68b\") " pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 01:47:03.508090 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.505927 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/322cef03-4f70-42c8-8571-42c939fba68b-istiod-ca-cert\") pod \"e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq\" (UID: \"322cef03-4f70-42c8-8571-42c939fba68b\") " pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 01:47:03.508090 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.505994 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/322cef03-4f70-42c8-8571-42c939fba68b-istio-token\") pod \"e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq\" (UID: \"322cef03-4f70-42c8-8571-42c939fba68b\") " pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 01:47:03.508090 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.506025 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwsvk\" (UniqueName: \"kubernetes.io/projected/322cef03-4f70-42c8-8571-42c939fba68b-kube-api-access-qwsvk\") pod \"e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq\" (UID: \"322cef03-4f70-42c8-8571-42c939fba68b\") " pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 01:47:03.508090 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.506081 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/322cef03-4f70-42c8-8571-42c939fba68b-credential-socket\") pod \"e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq\" (UID: \"322cef03-4f70-42c8-8571-42c939fba68b\") " pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 01:47:03.508090 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.506117 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/322cef03-4f70-42c8-8571-42c939fba68b-istio-data\") pod \"e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq\" (UID: \"322cef03-4f70-42c8-8571-42c939fba68b\") " pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 01:47:03.508090 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.506197 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/322cef03-4f70-42c8-8571-42c939fba68b-workload-socket\") pod \"e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq\" (UID: \"322cef03-4f70-42c8-8571-42c939fba68b\") " pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 01:47:03.508090 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.506682 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/322cef03-4f70-42c8-8571-42c939fba68b-workload-socket\") pod \"e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq\" (UID: \"322cef03-4f70-42c8-8571-42c939fba68b\") " pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 01:47:03.508792 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.508207 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/322cef03-4f70-42c8-8571-42c939fba68b-istiod-ca-cert\") pod \"e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq\" (UID: \"322cef03-4f70-42c8-8571-42c939fba68b\") " pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 01:47:03.508792 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.508455 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/322cef03-4f70-42c8-8571-42c939fba68b-workload-certs\") pod \"e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq\" (UID: \"322cef03-4f70-42c8-8571-42c939fba68b\") " pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 01:47:03.508792 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.508675 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/322cef03-4f70-42c8-8571-42c939fba68b-credential-socket\") pod \"e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq\" (UID: \"322cef03-4f70-42c8-8571-42c939fba68b\") " pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 01:47:03.514028 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.509116 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/322cef03-4f70-42c8-8571-42c939fba68b-istio-data\") pod \"e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq\" (UID: \"322cef03-4f70-42c8-8571-42c939fba68b\") " pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 01:47:03.552962 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.543012 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/322cef03-4f70-42c8-8571-42c939fba68b-istio-podinfo\") pod \"e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq\" (UID: \"322cef03-4f70-42c8-8571-42c939fba68b\") " pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 01:47:03.566375 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.565351 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwsvk\" (UniqueName: \"kubernetes.io/projected/322cef03-4f70-42c8-8571-42c939fba68b-kube-api-access-qwsvk\") pod \"e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq\" (UID: \"322cef03-4f70-42c8-8571-42c939fba68b\") " pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 01:47:03.567202 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.567172 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/322cef03-4f70-42c8-8571-42c939fba68b-istio-envoy\") pod \"e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq\" (UID: \"322cef03-4f70-42c8-8571-42c939fba68b\") " pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 01:47:03.581142 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.581076 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/322cef03-4f70-42c8-8571-42c939fba68b-istio-token\") pod \"e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq\" (UID: \"322cef03-4f70-42c8-8571-42c939fba68b\") " pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 01:47:03.703620 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.703580 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:03.867022 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.866941 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 01:47:03.965234 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.965187 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc614d4b-hcvf8"] Oct 30 01:47:03.980143 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.980109 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc614d4b-hcvf8" Oct 30 01:47:03.982350 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.982321 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc614d4b-hcvf8"] Oct 30 01:47:03.996143 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.996120 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:47:03.996279 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:03.996160 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:47:04.007917 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:04.007883 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:47:04.096849 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:04.096814 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:47:04.116449 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:04.116420 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10cff785-4b67-4c00-9ae3-135985d56904-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc614d4b-hcvf8\" (UID: \"10cff785-4b67-4c00-9ae3-135985d56904\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc614d4b-hcvf8" Oct 30 01:47:04.116585 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:04.116486 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10cff785-4b67-4c00-9ae3-135985d56904-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc614d4b-hcvf8\" (UID: \"10cff785-4b67-4c00-9ae3-135985d56904\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc614d4b-hcvf8" Oct 30 01:47:04.116633 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:04.116621 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kphwm\" (UniqueName: \"kubernetes.io/projected/10cff785-4b67-4c00-9ae3-135985d56904-kube-api-access-kphwm\") pod \"maas-api-revoke-sub-cc614d4b-hcvf8\" (UID: \"10cff785-4b67-4c00-9ae3-135985d56904\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc614d4b-hcvf8" Oct 30 01:47:04.200659 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:04.200163 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:04.202797 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:04.202771 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq"] Oct 30 01:47:04.203150 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:47:04.203122 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod322cef03_4f70_42c8_8571_42c939fba68b.slice/crio-28d64368f89b4a1d196e36d4c40ce1955a6836b3a1c129de1a975716adf52856 WatchSource:0}: Error finding container 28d64368f89b4a1d196e36d4c40ce1955a6836b3a1c129de1a975716adf52856: Status 404 returned error can't find the container with id 28d64368f89b4a1d196e36d4c40ce1955a6836b3a1c129de1a975716adf52856 Oct 30 01:47:04.206113 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:04.206067 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:47:04.206218 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:04.206161 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:47:04.206218 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:04.206209 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:47:04.217859 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:04.217825 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10cff785-4b67-4c00-9ae3-135985d56904-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc614d4b-hcvf8\" (UID: \"10cff785-4b67-4c00-9ae3-135985d56904\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc614d4b-hcvf8" Oct 30 01:47:04.218017 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:04.217938 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10cff785-4b67-4c00-9ae3-135985d56904-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc614d4b-hcvf8\" (UID: \"10cff785-4b67-4c00-9ae3-135985d56904\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc614d4b-hcvf8" Oct 30 01:47:04.218090 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:04.218042 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kphwm\" (UniqueName: \"kubernetes.io/projected/10cff785-4b67-4c00-9ae3-135985d56904-kube-api-access-kphwm\") pod \"maas-api-revoke-sub-cc614d4b-hcvf8\" (UID: \"10cff785-4b67-4c00-9ae3-135985d56904\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc614d4b-hcvf8" Oct 30 01:47:04.218514 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:04.218494 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10cff785-4b67-4c00-9ae3-135985d56904-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc614d4b-hcvf8\" (UID: \"10cff785-4b67-4c00-9ae3-135985d56904\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc614d4b-hcvf8" Oct 30 01:47:04.220459 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:04.220430 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10cff785-4b67-4c00-9ae3-135985d56904-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc614d4b-hcvf8\" (UID: \"10cff785-4b67-4c00-9ae3-135985d56904\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc614d4b-hcvf8" Oct 30 01:47:04.238460 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:04.238431 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kphwm\" (UniqueName: \"kubernetes.io/projected/10cff785-4b67-4c00-9ae3-135985d56904-kube-api-access-kphwm\") pod \"maas-api-revoke-sub-cc614d4b-hcvf8\" (UID: \"10cff785-4b67-4c00-9ae3-135985d56904\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc614d4b-hcvf8" Oct 30 01:47:04.299383 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:04.299348 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc614d4b-hcvf8" Oct 30 01:47:04.440771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:04.439374 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:04.873596 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:04.872840 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" event={"ID":"322cef03-4f70-42c8-8571-42c939fba68b","Type":"ContainerStarted","Data":"fe9b85c6a37107c277e18fb1db2f2d60c35aa9f585467f2d4ef2cae370a26036"} Oct 30 01:47:04.873596 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:04.872913 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" event={"ID":"322cef03-4f70-42c8-8571-42c939fba68b","Type":"ContainerStarted","Data":"28d64368f89b4a1d196e36d4c40ce1955a6836b3a1c129de1a975716adf52856"} Oct 30 01:47:04.897905 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:04.897867 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc614d4b-hcvf8"] Oct 30 01:47:04.960012 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:04.959966 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" podStartSLOduration=1.959947932 podStartE2EDuration="1.959947932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:04.955968502 +0000 UTC m=+1750.532393903" watchObservedRunningTime="2026-10-30 01:47:04.959947932 +0000 UTC m=+1750.536373351" Oct 30 01:47:05.454051 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:05.452329 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:05.640399 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:05.640364 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:05.886902 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:05.886802 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 01:47:05.932097 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:05.932062 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 01:47:06.018176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:06.018135 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc614d4b-hcvf8" event={"ID":"10cff785-4b67-4c00-9ae3-135985d56904","Type":"ContainerStarted","Data":"ca423b3b178354a751abc2ca88e6db907cd24ca68c3dd913def338870d1c046b"} Oct 30 01:47:06.019547 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:06.019523 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 01:47:06.019733 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:06.019712 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc614d4b-hcvf8" event={"ID":"10cff785-4b67-4c00-9ae3-135985d56904","Type":"ContainerStarted","Data":"cce8dd61873c9512b61dd6c0e6fc3c10db4b8ee7fea7559be9b33c342c603c9e"} Oct 30 01:47:06.023012 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:06.022985 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 01:47:06.168101 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:06.168032 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc614d4b-hcvf8" podStartSLOduration=3.168008513 podStartE2EDuration="3.168008513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:06.087124306 +0000 UTC m=+1751.663549708" watchObservedRunningTime="2026-10-30 01:47:06.168008513 +0000 UTC m=+1751.744433915" Oct 30 01:47:06.459094 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:06.459006 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:06.701778 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:06.701460 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:06.904292 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:06.904198 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:07.034029 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:07.033991 6572 generic.go:356] "Generic (PLEG): container finished" podID="10cff785-4b67-4c00-9ae3-135985d56904" containerID="ca423b3b178354a751abc2ca88e6db907cd24ca68c3dd913def338870d1c046b" exitCode=0 Oct 30 01:47:07.034211 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:07.034118 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc614d4b-hcvf8" event={"ID":"10cff785-4b67-4c00-9ae3-135985d56904","Type":"ContainerDied","Data":"ca423b3b178354a751abc2ca88e6db907cd24ca68c3dd913def338870d1c046b"} Oct 30 01:47:07.592653 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:07.592621 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:07.804122 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:07.804090 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:08.268717 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:08.268689 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc614d4b-hcvf8" Oct 30 01:47:08.344775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:08.344662 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10cff785-4b67-4c00-9ae3-135985d56904-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10cff785-4b67-4c00-9ae3-135985d56904-maas-api-openshift-ca\") pod \"10cff785-4b67-4c00-9ae3-135985d56904\" (UID: \"10cff785-4b67-4c00-9ae3-135985d56904\") " Oct 30 01:47:08.344775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:08.344743 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10cff785-4b67-4c00-9ae3-135985d56904-kube-api-access-kphwm\" (UniqueName: \"kubernetes.io/projected/10cff785-4b67-4c00-9ae3-135985d56904-kube-api-access-kphwm\") pod \"10cff785-4b67-4c00-9ae3-135985d56904\" (UID: \"10cff785-4b67-4c00-9ae3-135985d56904\") " Oct 30 01:47:08.345040 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:08.344857 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10cff785-4b67-4c00-9ae3-135985d56904-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10cff785-4b67-4c00-9ae3-135985d56904-maas-api-xks-ca\") pod \"10cff785-4b67-4c00-9ae3-135985d56904\" (UID: \"10cff785-4b67-4c00-9ae3-135985d56904\") " Oct 30 01:47:08.345040 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:08.345017 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10cff785-4b67-4c00-9ae3-135985d56904-maas-api-openshift-ca" pod "10cff785-4b67-4c00-9ae3-135985d56904" (UID: "10cff785-4b67-4c00-9ae3-135985d56904"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:47:08.345410 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:08.345386 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10cff785-4b67-4c00-9ae3-135985d56904-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:47:08.347136 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:08.347111 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10cff785-4b67-4c00-9ae3-135985d56904-kube-api-access-kphwm" pod "10cff785-4b67-4c00-9ae3-135985d56904" (UID: "10cff785-4b67-4c00-9ae3-135985d56904"). InnerVolumeSpecName "kube-api-access-kphwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:47:08.347373 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:08.347351 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10cff785-4b67-4c00-9ae3-135985d56904-maas-api-xks-ca" pod "10cff785-4b67-4c00-9ae3-135985d56904" (UID: "10cff785-4b67-4c00-9ae3-135985d56904"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:47:08.446406 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:08.446357 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kphwm\" (UniqueName: \"kubernetes.io/projected/10cff785-4b67-4c00-9ae3-135985d56904-kube-api-access-kphwm\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:47:08.446406 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:08.446402 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10cff785-4b67-4c00-9ae3-135985d56904-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:47:09.052828 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:09.052793 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc614d4b-hcvf8" Oct 30 01:47:09.053026 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:09.052798 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc614d4b-hcvf8" event={"ID":"10cff785-4b67-4c00-9ae3-135985d56904","Type":"ContainerDied","Data":"cce8dd61873c9512b61dd6c0e6fc3c10db4b8ee7fea7559be9b33c342c603c9e"} Oct 30 01:47:09.053026 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:09.052928 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cce8dd61873c9512b61dd6c0e6fc3c10db4b8ee7fea7559be9b33c342c603c9e" Oct 30 01:47:09.240090 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:09.240042 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:09.477628 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:09.477589 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:09.674839 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:09.674803 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:11.320394 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:11.320356 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:11.508950 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:11.508918 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:13.115814 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:13.115729 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:13.326581 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:13.326534 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:13.523102 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:13.523064 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:13.731908 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:13.731868 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:14.190557 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:14.190514 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-58632b-fqnfb"] Oct 30 01:47:14.191402 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:14.191376 6572 cpu_manager.go:414] "containerMap: removing container" podUID="10cff785-4b67-4c00-9ae3-135985d56904" containerName="revoke-keys" Oct 30 01:47:14.191506 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:14.191404 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10cff785-4b67-4c00-9ae3-135985d56904" containerName="revoke-keys" Oct 30 01:47:14.191615 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:14.191600 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="10cff785-4b67-4c00-9ae3-135985d56904" containerName="revoke-keys" Oct 30 01:47:14.195400 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:14.195376 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-58632b-fqnfb" Oct 30 01:47:14.213295 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:14.213265 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-58632b-fqnfb"] Oct 30 01:47:14.234216 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:14.219440 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:47:14.234216 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:14.219579 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:47:14.254437 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:14.254404 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:47:14.254618 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:14.254441 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:47:14.304108 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:14.304072 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90cf234a-25a2-4705-a8f4-8efe70f9f059-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-58632b-fqnfb\" (UID: \"90cf234a-25a2-4705-a8f4-8efe70f9f059\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-58632b-fqnfb" Oct 30 01:47:14.304327 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:14.304167 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpkfw\" (UniqueName: \"kubernetes.io/projected/90cf234a-25a2-4705-a8f4-8efe70f9f059-kube-api-access-mpkfw\") pod \"maas-api-revoke-keys-e2e-praxis-b-58632b-fqnfb\" (UID: \"90cf234a-25a2-4705-a8f4-8efe70f9f059\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-58632b-fqnfb" Oct 30 01:47:14.304327 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:14.304257 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90cf234a-25a2-4705-a8f4-8efe70f9f059-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-58632b-fqnfb\" (UID: \"90cf234a-25a2-4705-a8f4-8efe70f9f059\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-58632b-fqnfb" Oct 30 01:47:14.406181 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:14.406135 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90cf234a-25a2-4705-a8f4-8efe70f9f059-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-58632b-fqnfb\" (UID: \"90cf234a-25a2-4705-a8f4-8efe70f9f059\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-58632b-fqnfb" Oct 30 01:47:14.406408 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:14.406301 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90cf234a-25a2-4705-a8f4-8efe70f9f059-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-58632b-fqnfb\" (UID: \"90cf234a-25a2-4705-a8f4-8efe70f9f059\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-58632b-fqnfb" Oct 30 01:47:14.406408 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:14.406330 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpkfw\" (UniqueName: \"kubernetes.io/projected/90cf234a-25a2-4705-a8f4-8efe70f9f059-kube-api-access-mpkfw\") pod \"maas-api-revoke-keys-e2e-praxis-b-58632b-fqnfb\" (UID: \"90cf234a-25a2-4705-a8f4-8efe70f9f059\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-58632b-fqnfb" Oct 30 01:47:14.406976 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:14.406950 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90cf234a-25a2-4705-a8f4-8efe70f9f059-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-58632b-fqnfb\" (UID: \"90cf234a-25a2-4705-a8f4-8efe70f9f059\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-58632b-fqnfb" Oct 30 01:47:14.408727 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:14.408695 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90cf234a-25a2-4705-a8f4-8efe70f9f059-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-58632b-fqnfb\" (UID: \"90cf234a-25a2-4705-a8f4-8efe70f9f059\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-58632b-fqnfb" Oct 30 01:47:14.424708 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:14.424676 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpkfw\" (UniqueName: \"kubernetes.io/projected/90cf234a-25a2-4705-a8f4-8efe70f9f059-kube-api-access-mpkfw\") pod \"maas-api-revoke-keys-e2e-praxis-b-58632b-fqnfb\" (UID: \"90cf234a-25a2-4705-a8f4-8efe70f9f059\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-58632b-fqnfb" Oct 30 01:47:14.513045 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:14.513004 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-58632b-fqnfb" Oct 30 01:47:14.690485 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:47:14.690443 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:47:14.823168 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:47:14.823129 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90cf234a_25a2_4705_a8f4_8efe70f9f059.slice/crio-ca8a214fbda37e62482b0ed46aebb1469a97d04d59e0ee48c8e1af7a35c571de WatchSource:0}: Error finding container ca8a214fbda37e62482b0ed46aebb1469a97d04d59e0ee48c8e1af7a35c571de: Status 404 returned error can't find the container with id ca8a214fbda37e62482b0ed46aebb1469a97d04d59e0ee48c8e1af7a35c571de Oct 30 01:47:14.824710 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:14.824678 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-58632b-fqnfb"] Oct 30 01:47:15.039295 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:15.039264 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:15.133899 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:15.133864 6572 generic.go:356] "Generic (PLEG): container finished" podID="7e60f1dc-1188-4053-8f31-fcd08da0ef1c" containerID="63e3d2a4a936608d5d3466dd4111cb9af461b13e9ee684ef2b45ac6229096912" exitCode=0 Oct 30 01:47:15.134079 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:15.133937 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" event={"ID":"7e60f1dc-1188-4053-8f31-fcd08da0ef1c","Type":"ContainerDied","Data":"63e3d2a4a936608d5d3466dd4111cb9af461b13e9ee684ef2b45ac6229096912"} Oct 30 01:47:15.134479 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:15.134459 6572 scope.go:122] "RemoveContainer" containerID="63e3d2a4a936608d5d3466dd4111cb9af461b13e9ee684ef2b45ac6229096912" Oct 30 01:47:15.136020 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:15.135991 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-58632b-fqnfb" event={"ID":"90cf234a-25a2-4705-a8f4-8efe70f9f059","Type":"ContainerStarted","Data":"15f75a99e926a11445878ef36f2c81e0051c5076536c0c708494cb95ec58475f"} Oct 30 01:47:15.136123 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:15.136039 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-58632b-fqnfb" event={"ID":"90cf234a-25a2-4705-a8f4-8efe70f9f059","Type":"ContainerStarted","Data":"ca8a214fbda37e62482b0ed46aebb1469a97d04d59e0ee48c8e1af7a35c571de"} Oct 30 01:47:15.136190 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:15.136163 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:47:15.136243 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:15.136230 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:47:15.136285 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:15.136266 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:47:15.232426 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:15.232371 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-58632b-fqnfb" podStartSLOduration=1.23235673 podStartE2EDuration="1.23235673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:15.228290131 +0000 UTC m=+1760.804715531" watchObservedRunningTime="2026-10-30 01:47:15.23235673 +0000 UTC m=+1760.808782191" Oct 30 01:47:15.620779 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:15.618551 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:16.127834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:16.127521 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:16.254825 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:16.239857 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" event={"ID":"7e60f1dc-1188-4053-8f31-fcd08da0ef1c","Type":"ContainerStarted","Data":"801d83f6ed507e7b49a1d63784214ef842b44a7c295e4d9a9ada9963d1a2367e"} Oct 30 01:47:16.289772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:16.285343 6572 generic.go:356] "Generic (PLEG): container finished" podID="90cf234a-25a2-4705-a8f4-8efe70f9f059" containerID="15f75a99e926a11445878ef36f2c81e0051c5076536c0c708494cb95ec58475f" exitCode=0 Oct 30 01:47:16.289772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:16.285675 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-58632b-fqnfb" event={"ID":"90cf234a-25a2-4705-a8f4-8efe70f9f059","Type":"ContainerDied","Data":"15f75a99e926a11445878ef36f2c81e0051c5076536c0c708494cb95ec58475f"} Oct 30 01:47:16.334773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:16.323831 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:16.364655 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:16.364189 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 01:47:16.364655 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:16.364343 6572 patch_prober.go:29] interesting pod/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body= Oct 30 01:47:16.364655 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:16.364387 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" podUID="7e60f1dc-1188-4053-8f31-fcd08da0ef1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" Oct 30 01:47:17.009772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:17.006514 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:17.356299 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:17.356262 6572 patch_prober.go:29] interesting pod/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body= Oct 30 01:47:17.356735 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:17.356333 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" podUID="7e60f1dc-1188-4053-8f31-fcd08da0ef1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" Oct 30 01:47:17.545450 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:17.545366 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-58632b-fqnfb" Oct 30 01:47:17.584003 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:17.583966 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90cf234a-25a2-4705-a8f4-8efe70f9f059-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90cf234a-25a2-4705-a8f4-8efe70f9f059-maas-api-openshift-ca\") pod \"90cf234a-25a2-4705-a8f4-8efe70f9f059\" (UID: \"90cf234a-25a2-4705-a8f4-8efe70f9f059\") " Oct 30 01:47:17.584191 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:17.584101 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90cf234a-25a2-4705-a8f4-8efe70f9f059-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90cf234a-25a2-4705-a8f4-8efe70f9f059-maas-api-xks-ca\") pod \"90cf234a-25a2-4705-a8f4-8efe70f9f059\" (UID: \"90cf234a-25a2-4705-a8f4-8efe70f9f059\") " Oct 30 01:47:17.584191 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:17.584140 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90cf234a-25a2-4705-a8f4-8efe70f9f059-kube-api-access-mpkfw\" (UniqueName: \"kubernetes.io/projected/90cf234a-25a2-4705-a8f4-8efe70f9f059-kube-api-access-mpkfw\") pod \"90cf234a-25a2-4705-a8f4-8efe70f9f059\" (UID: \"90cf234a-25a2-4705-a8f4-8efe70f9f059\") " Oct 30 01:47:17.584525 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:17.584492 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90cf234a-25a2-4705-a8f4-8efe70f9f059-maas-api-openshift-ca" pod "90cf234a-25a2-4705-a8f4-8efe70f9f059" (UID: "90cf234a-25a2-4705-a8f4-8efe70f9f059"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:47:17.589097 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:17.589058 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90cf234a-25a2-4705-a8f4-8efe70f9f059-maas-api-xks-ca" pod "90cf234a-25a2-4705-a8f4-8efe70f9f059" (UID: "90cf234a-25a2-4705-a8f4-8efe70f9f059"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:47:17.589533 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:17.589491 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90cf234a-25a2-4705-a8f4-8efe70f9f059-kube-api-access-mpkfw" pod "90cf234a-25a2-4705-a8f4-8efe70f9f059" (UID: "90cf234a-25a2-4705-a8f4-8efe70f9f059"). InnerVolumeSpecName "kube-api-access-mpkfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:47:17.685388 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:17.685342 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90cf234a-25a2-4705-a8f4-8efe70f9f059-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:47:17.685388 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:17.685378 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpkfw\" (UniqueName: \"kubernetes.io/projected/90cf234a-25a2-4705-a8f4-8efe70f9f059-kube-api-access-mpkfw\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:47:17.685388 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:17.685388 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90cf234a-25a2-4705-a8f4-8efe70f9f059-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:47:18.303998 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:18.303958 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-58632b-fqnfb" event={"ID":"90cf234a-25a2-4705-a8f4-8efe70f9f059","Type":"ContainerDied","Data":"ca8a214fbda37e62482b0ed46aebb1469a97d04d59e0ee48c8e1af7a35c571de"} Oct 30 01:47:18.303998 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:18.303981 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-58632b-fqnfb" Oct 30 01:47:18.304274 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:18.304042 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca8a214fbda37e62482b0ed46aebb1469a97d04d59e0ee48c8e1af7a35c571de" Oct 30 01:47:18.355574 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:18.355542 6572 patch_prober.go:29] interesting pod/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body= Oct 30 01:47:18.355782 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:18.355605 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" podUID="7e60f1dc-1188-4053-8f31-fcd08da0ef1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" Oct 30 01:47:18.987121 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:18.987081 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:19.355926 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:19.355837 6572 patch_prober.go:29] interesting pod/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body= Oct 30 01:47:19.355926 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:19.355910 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" podUID="7e60f1dc-1188-4053-8f31-fcd08da0ef1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" Oct 30 01:47:19.536912 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:19.536790 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:19.746631 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:19.746592 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:20.355645 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:20.355613 6572 patch_prober.go:29] interesting pod/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body= Oct 30 01:47:20.356105 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:20.355673 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" podUID="7e60f1dc-1188-4053-8f31-fcd08da0ef1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" Oct 30 01:47:20.988489 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:20.988442 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:21.191095 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:21.191049 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:21.389280 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:21.389192 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:21.482116 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:21.482077 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 01:47:21.483660 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:21.483637 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 01:47:21.484580 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:21.484557 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 01:47:21.878097 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:21.878055 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:21.969630 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:21.969597 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-292018-f884cbb67-mvk86"] Oct 30 01:47:21.970870 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:21.970845 6572 cpu_manager.go:414] "containerMap: removing container" podUID="90cf234a-25a2-4705-a8f4-8efe70f9f059" containerName="revoke-keys" Oct 30 01:47:21.970870 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:21.970869 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90cf234a-25a2-4705-a8f4-8efe70f9f059" containerName="revoke-keys" Oct 30 01:47:21.971089 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:21.971078 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="90cf234a-25a2-4705-a8f4-8efe70f9f059" containerName="revoke-keys" Oct 30 01:47:21.975767 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:21.975732 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-292018-f884cbb67-mvk86" Oct 30 01:47:21.993371 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:21.993344 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-292018-dockercfg-s6vmr\"" Oct 30 01:47:21.993524 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:21.993406 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-292018\"" Oct 30 01:47:21.997455 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:21.997416 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-292018-f884cbb67-mvk86"] Oct 30 01:47:22.047056 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:22.047016 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-292018-6fc6bf57c6-nslxv"] Oct 30 01:47:22.051900 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:22.051876 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-292018-6fc6bf57c6-nslxv" Oct 30 01:47:22.073953 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:22.073924 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-292018-6fc6bf57c6-nslxv"] Oct 30 01:47:22.078842 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:22.078779 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a98e88-a93c-462b-b2f5-68c61b386789-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-292018-f884cbb67-mvk86\" (UID: \"d0a98e88-a93c-462b-b2f5-68c61b386789\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-292018-f884cbb67-mvk86" Oct 30 01:47:22.079010 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:22.078947 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h49cm\" (UniqueName: \"kubernetes.io/projected/d0a98e88-a93c-462b-b2f5-68c61b386789-kube-api-access-h49cm\") pod \"payload-pre-processing-e2e-shared-a-w4-292018-f884cbb67-mvk86\" (UID: \"d0a98e88-a93c-462b-b2f5-68c61b386789\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-292018-f884cbb67-mvk86" Oct 30 01:47:22.082315 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:22.082291 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:22.198484 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:22.193623 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd10ed97-4e22-4efe-a0c6-7cff1ca2f157-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-292018-6fc6bf57c6-nslxv\" (UID: \"cd10ed97-4e22-4efe-a0c6-7cff1ca2f157\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-292018-6fc6bf57c6-nslxv" Oct 30 01:47:22.198484 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:22.193694 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h49cm\" (UniqueName: \"kubernetes.io/projected/d0a98e88-a93c-462b-b2f5-68c61b386789-kube-api-access-h49cm\") pod \"payload-pre-processing-e2e-shared-a-w4-292018-f884cbb67-mvk86\" (UID: \"d0a98e88-a93c-462b-b2f5-68c61b386789\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-292018-f884cbb67-mvk86" Oct 30 01:47:22.198484 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:22.193851 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d7br\" (UniqueName: \"kubernetes.io/projected/cd10ed97-4e22-4efe-a0c6-7cff1ca2f157-kube-api-access-6d7br\") pod \"payload-processing-e2e-shared-a-w4-292018-6fc6bf57c6-nslxv\" (UID: \"cd10ed97-4e22-4efe-a0c6-7cff1ca2f157\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-292018-6fc6bf57c6-nslxv" Oct 30 01:47:22.198484 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:22.193989 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a98e88-a93c-462b-b2f5-68c61b386789-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-292018-f884cbb67-mvk86\" (UID: \"d0a98e88-a93c-462b-b2f5-68c61b386789\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-292018-f884cbb67-mvk86" Oct 30 01:47:22.198484 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:22.195020 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a98e88-a93c-462b-b2f5-68c61b386789-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-292018-f884cbb67-mvk86\" (UID: \"d0a98e88-a93c-462b-b2f5-68c61b386789\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-292018-f884cbb67-mvk86" Oct 30 01:47:22.246972 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:22.246933 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h49cm\" (UniqueName: \"kubernetes.io/projected/d0a98e88-a93c-462b-b2f5-68c61b386789-kube-api-access-h49cm\") pod \"payload-pre-processing-e2e-shared-a-w4-292018-f884cbb67-mvk86\" (UID: \"d0a98e88-a93c-462b-b2f5-68c61b386789\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-292018-f884cbb67-mvk86" Oct 30 01:47:22.295808 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:22.295764 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d7br\" (UniqueName: \"kubernetes.io/projected/cd10ed97-4e22-4efe-a0c6-7cff1ca2f157-kube-api-access-6d7br\") pod \"payload-processing-e2e-shared-a-w4-292018-6fc6bf57c6-nslxv\" (UID: \"cd10ed97-4e22-4efe-a0c6-7cff1ca2f157\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-292018-6fc6bf57c6-nslxv" Oct 30 01:47:22.296092 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:22.296024 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd10ed97-4e22-4efe-a0c6-7cff1ca2f157-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-292018-6fc6bf57c6-nslxv\" (UID: \"cd10ed97-4e22-4efe-a0c6-7cff1ca2f157\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-292018-6fc6bf57c6-nslxv" Oct 30 01:47:22.296600 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:22.296580 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-292018-f884cbb67-mvk86" Oct 30 01:47:22.296946 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:22.296884 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd10ed97-4e22-4efe-a0c6-7cff1ca2f157-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-292018-6fc6bf57c6-nslxv\" (UID: \"cd10ed97-4e22-4efe-a0c6-7cff1ca2f157\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-292018-6fc6bf57c6-nslxv" Oct 30 01:47:22.315781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:22.315717 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d7br\" (UniqueName: \"kubernetes.io/projected/cd10ed97-4e22-4efe-a0c6-7cff1ca2f157-kube-api-access-6d7br\") pod \"payload-processing-e2e-shared-a-w4-292018-6fc6bf57c6-nslxv\" (UID: \"cd10ed97-4e22-4efe-a0c6-7cff1ca2f157\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-292018-6fc6bf57c6-nslxv" Oct 30 01:47:22.366119 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:22.366091 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-292018-6fc6bf57c6-nslxv" Oct 30 01:47:22.603289 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:47:22.603262 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a98e88_a93c_462b_b2f5_68c61b386789.slice/crio-90ffc21ab3b5915567c2024b8831c1ae5928c4f017a905d5cc439b1fd2199838 WatchSource:0}: Error finding container 90ffc21ab3b5915567c2024b8831c1ae5928c4f017a905d5cc439b1fd2199838: Status 404 returned error can't find the container with id 90ffc21ab3b5915567c2024b8831c1ae5928c4f017a905d5cc439b1fd2199838 Oct 30 01:47:22.604825 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:22.604786 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-292018-f884cbb67-mvk86"] Oct 30 01:47:22.693152 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:47:22.693122 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd10ed97_4e22_4efe_a0c6_7cff1ca2f157.slice/crio-6eef2ec45c3ff2268700e34182f2aa69bed060ef22ad6e7e6dc77073eb73e7c8 WatchSource:0}: Error finding container 6eef2ec45c3ff2268700e34182f2aa69bed060ef22ad6e7e6dc77073eb73e7c8: Status 404 returned error can't find the container with id 6eef2ec45c3ff2268700e34182f2aa69bed060ef22ad6e7e6dc77073eb73e7c8 Oct 30 01:47:22.694675 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:22.694654 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-292018-6fc6bf57c6-nslxv"] Oct 30 01:47:23.006212 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:23.006112 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:23.397265 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:23.397170 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-292018-6fc6bf57c6-nslxv" event={"ID":"cd10ed97-4e22-4efe-a0c6-7cff1ca2f157","Type":"ContainerStarted","Data":"2afe31e5495d6d4ce4a3f9834325d50ea2852c2fcc83681af8d02959cb59482a"} Oct 30 01:47:23.397265 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:23.397234 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-292018-6fc6bf57c6-nslxv" event={"ID":"cd10ed97-4e22-4efe-a0c6-7cff1ca2f157","Type":"ContainerStarted","Data":"6eef2ec45c3ff2268700e34182f2aa69bed060ef22ad6e7e6dc77073eb73e7c8"} Oct 30 01:47:23.397265 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:23.397264 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-292018-6fc6bf57c6-nslxv" Oct 30 01:47:23.399337 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:23.399311 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-292018-f884cbb67-mvk86" event={"ID":"d0a98e88-a93c-462b-b2f5-68c61b386789","Type":"ContainerStarted","Data":"4625e5617338c40526823aca86dbe2fb3e8217e4e4c1c41f1bb25b44c70ac2a2"} Oct 30 01:47:23.399458 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:23.399354 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-292018-f884cbb67-mvk86" event={"ID":"d0a98e88-a93c-462b-b2f5-68c61b386789","Type":"ContainerStarted","Data":"90ffc21ab3b5915567c2024b8831c1ae5928c4f017a905d5cc439b1fd2199838"} Oct 30 01:47:23.399458 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:23.399398 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-292018-f884cbb67-mvk86" Oct 30 01:47:23.439124 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:23.439069 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-292018-6fc6bf57c6-nslxv" podStartSLOduration=1.439054887 podStartE2EDuration="1.439054887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:23.435118717 +0000 UTC m=+1769.011544119" watchObservedRunningTime="2026-10-30 01:47:23.439054887 +0000 UTC m=+1769.015480288" Oct 30 01:47:23.479415 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:23.479351 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-292018-f884cbb67-mvk86" podStartSLOduration=2.479334847 podStartE2EDuration="2.479334847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:23.474797521 +0000 UTC m=+1769.051222919" watchObservedRunningTime="2026-10-30 01:47:23.479334847 +0000 UTC m=+1769.055760550" Oct 30 01:47:24.696425 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:24.696385 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:24.903454 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:24.903417 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:26.369647 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:26.369605 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:26.564548 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:26.564507 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:28.509714 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:28.509675 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:30.432983 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:30.429968 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:32.227773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:32.225165 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:34.071555 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:34.071517 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:34.411527 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:34.411502 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-292018-f884cbb67-mvk86" Oct 30 01:47:34.411692 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:34.411594 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-292018-6fc6bf57c6-nslxv" Oct 30 01:47:35.803450 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:35.803410 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:36.077274 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:36.077180 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:36.549254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:36.549223 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:37.058356 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:37.058306 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:37.622560 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:37.622524 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:37.825448 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:37.825403 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:38.626931 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:38.626895 6572 generic.go:356] "Generic (PLEG): container finished" podID="414b871d-0343-42a6-b560-0be9f9ff4fce" containerID="11c56fe9916e904a2d2119cedb366a949764a0794f4ea155e5fd1c4223fcf754" exitCode=0 Oct 30 01:47:38.627345 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:38.626967 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" event={"ID":"414b871d-0343-42a6-b560-0be9f9ff4fce","Type":"ContainerDied","Data":"11c56fe9916e904a2d2119cedb366a949764a0794f4ea155e5fd1c4223fcf754"} Oct 30 01:47:38.627419 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:38.627402 6572 scope.go:122] "RemoveContainer" containerID="11c56fe9916e904a2d2119cedb366a949764a0794f4ea155e5fd1c4223fcf754" Oct 30 01:47:38.629431 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:38.629397 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:47:38.629535 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:38.629479 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:47:38.629535 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:38.629518 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:47:38.752416 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:38.752375 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:47:39.331975 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:39.331937 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:39.532424 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:39.532381 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:39.642053 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:39.641969 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" event={"ID":"414b871d-0343-42a6-b560-0be9f9ff4fce","Type":"ContainerStarted","Data":"8f2d924a182f67c7f12aab304b2d722eac721eced21b25f79feab7136924d149"} Oct 30 01:47:40.270337 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:40.270305 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:47:40.271692 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:40.271667 6572 patch_prober.go:29] interesting pod/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 01:47:40.271894 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:40.271723 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" podUID="414b871d-0343-42a6-b560-0be9f9ff4fce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 01:47:40.894056 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:40.894009 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:41.101709 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:41.101666 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:41.271378 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:41.270804 6572 patch_prober.go:29] interesting pod/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 01:47:41.271378 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:41.270893 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" podUID="414b871d-0343-42a6-b560-0be9f9ff4fce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 01:47:41.324614 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:41.324577 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:42.270531 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:42.270489 6572 patch_prober.go:29] interesting pod/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 01:47:42.270999 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:42.270548 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" podUID="414b871d-0343-42a6-b560-0be9f9ff4fce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 01:47:42.532633 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:42.532538 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:42.639606 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:42.639574 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-861cd2-6d4589fbd7-hr89s"] Oct 30 01:47:42.644474 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:42.644432 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-861cd2-6d4589fbd7-hr89s" Oct 30 01:47:42.645315 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:42.645283 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-861cd2-6d4589fbd7-hr89s"] Oct 30 01:47:42.661322 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:42.660975 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-861cd2\"" Oct 30 01:47:42.702206 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:42.680217 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-861cd2-dockercfg-vqmqv\"" Oct 30 01:47:42.743171 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:42.735935 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/963a53c2-66a3-415b-a7b0-2dcf1b48daf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-861cd2-6d4589fbd7-hr89s\" (UID: \"963a53c2-66a3-415b-a7b0-2dcf1b48daf0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-861cd2-6d4589fbd7-hr89s" Oct 30 01:47:42.743171 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:42.736063 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ptwj\" (UniqueName: \"kubernetes.io/projected/963a53c2-66a3-415b-a7b0-2dcf1b48daf0-kube-api-access-6ptwj\") pod \"payload-pre-processing-e2e-shared-b-w1-861cd2-6d4589fbd7-hr89s\" (UID: \"963a53c2-66a3-415b-a7b0-2dcf1b48daf0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-861cd2-6d4589fbd7-hr89s" Oct 30 01:47:42.781163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:42.781108 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:42.920988 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:42.919831 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/963a53c2-66a3-415b-a7b0-2dcf1b48daf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-861cd2-6d4589fbd7-hr89s\" (UID: \"963a53c2-66a3-415b-a7b0-2dcf1b48daf0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-861cd2-6d4589fbd7-hr89s" Oct 30 01:47:42.920988 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:42.919924 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ptwj\" (UniqueName: \"kubernetes.io/projected/963a53c2-66a3-415b-a7b0-2dcf1b48daf0-kube-api-access-6ptwj\") pod \"payload-pre-processing-e2e-shared-b-w1-861cd2-6d4589fbd7-hr89s\" (UID: \"963a53c2-66a3-415b-a7b0-2dcf1b48daf0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-861cd2-6d4589fbd7-hr89s" Oct 30 01:47:42.947974 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:42.946952 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/963a53c2-66a3-415b-a7b0-2dcf1b48daf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-861cd2-6d4589fbd7-hr89s\" (UID: \"963a53c2-66a3-415b-a7b0-2dcf1b48daf0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-861cd2-6d4589fbd7-hr89s" Oct 30 01:47:42.948234 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:42.948206 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ptwj\" (UniqueName: \"kubernetes.io/projected/963a53c2-66a3-415b-a7b0-2dcf1b48daf0-kube-api-access-6ptwj\") pod \"payload-pre-processing-e2e-shared-b-w1-861cd2-6d4589fbd7-hr89s\" (UID: \"963a53c2-66a3-415b-a7b0-2dcf1b48daf0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-861cd2-6d4589fbd7-hr89s" Oct 30 01:47:43.033287 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:43.033226 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-861cd2-6d4589fbd7-hr89s" Oct 30 01:47:43.265840 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:43.264794 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:43.270586 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:43.270124 6572 patch_prober.go:29] interesting pod/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 01:47:43.270586 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:43.270192 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" podUID="414b871d-0343-42a6-b560-0be9f9ff4fce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 01:47:43.344228 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:47:43.344192 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod963a53c2_66a3_415b_a7b0_2dcf1b48daf0.slice/crio-f1079eb7650a7bcb198854ddfaa053c91ca89f4cb7c59f8e85b869d4a90adea9 WatchSource:0}: Error finding container f1079eb7650a7bcb198854ddfaa053c91ca89f4cb7c59f8e85b869d4a90adea9: Status 404 returned error can't find the container with id f1079eb7650a7bcb198854ddfaa053c91ca89f4cb7c59f8e85b869d4a90adea9 Oct 30 01:47:43.344519 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:43.344499 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-861cd2-6d4589fbd7-hr89s"] Oct 30 01:47:43.799714 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:43.799683 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-861cd2-6d4589fbd7-hr89s" event={"ID":"963a53c2-66a3-415b-a7b0-2dcf1b48daf0","Type":"ContainerStarted","Data":"97c74579079ad79aa475823bfa55533b63d4d5696c92c0966e6ac3ef93f47fbf"} Oct 30 01:47:43.799941 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:43.799733 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-861cd2-6d4589fbd7-hr89s" event={"ID":"963a53c2-66a3-415b-a7b0-2dcf1b48daf0","Type":"ContainerStarted","Data":"f1079eb7650a7bcb198854ddfaa053c91ca89f4cb7c59f8e85b869d4a90adea9"} Oct 30 01:47:43.799941 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:43.799911 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-861cd2-6d4589fbd7-hr89s" Oct 30 01:47:43.848400 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:43.848347 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-861cd2-6d4589fbd7-hr89s" podStartSLOduration=1.848332998 podStartE2EDuration="1.848332998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:43.842809854 +0000 UTC m=+1789.419235253" watchObservedRunningTime="2026-10-30 01:47:43.848332998 +0000 UTC m=+1789.424758425" Oct 30 01:47:44.270021 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:44.269987 6572 patch_prober.go:29] interesting pod/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 01:47:44.270206 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:44.270037 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" podUID="414b871d-0343-42a6-b560-0be9f9ff4fce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 01:47:44.689620 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:47:44.689586 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:47:45.300187 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:45.300064 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:47:45.302296 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:45.301130 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:47:45.304777 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:45.304547 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 01:47:45.311978 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:45.311950 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:46.879182 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:46.879153 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:47.589240 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:47.589204 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:47.695351 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:47.695313 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-58632b-fqnfb"] Oct 30 01:47:47.707005 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:47.706966 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-58632b-fqnfb"] Oct 30 01:47:47.773532 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:47.773495 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:47.957915 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:47.957874 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:48.300917 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:48.300816 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:48.543341 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:48.543306 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90cf234a-25a2-4705-a8f4-8efe70f9f059" path="/var/lib/kubelet/pods/90cf234a-25a2-4705-a8f4-8efe70f9f059/volumes" Oct 30 01:47:49.949405 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:49.949363 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:50.137491 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:50.137454 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:50.426781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:50.424370 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:50.891228 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:50.891142 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:51.093772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:51.090657 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:51.890046 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:51.890010 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:53.258848 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:53.258810 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:53.520838 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:53.517412 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:54.846542 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:54.845878 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:54.850486 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:54.850458 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-861cd2-6d4589fbd7-hr89s" Oct 30 01:47:55.095081 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:55.095047 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:55.314775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:55.311456 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:55.701211 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:55.701174 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:56.365620 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:56.365585 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:56.771861 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:56.771831 6572 scope.go:122] "RemoveContainer" containerID="f225611cfa0dfdd95882eb70ebd04f8b7025ef330a9f62674862f0c64072bf81" Oct 30 01:47:56.789303 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:56.789166 6572 scope.go:122] "RemoveContainer" containerID="8d6eb7b6247f6462e7bad9fa1295d693a2c24278c52154180cb8a3301e062341" Oct 30 01:47:56.805851 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:56.805819 6572 scope.go:122] "RemoveContainer" containerID="bbeb048e14d009062948fea6dffcc284be9d88aad6fa6723c75df81a6c6bf5ad" Oct 30 01:47:56.817037 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:56.817007 6572 scope.go:122] "RemoveContainer" containerID="237e7c18b521415c115447c553699fba789fbbabd9ade9a7938126921c65ca9f" Oct 30 01:47:56.820054 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:56.820028 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:57.026190 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:57.026066 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-264bdd-65d6c7bf46-cq56s"] Oct 30 01:47:57.032462 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:57.032438 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-264bdd-65d6c7bf46-cq56s" Oct 30 01:47:57.046789 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:57.043437 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:57.048149 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:57.048125 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-264bdd\"" Oct 30 01:47:57.059628 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:57.059602 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-264bdd-dockercfg-qbkjv\"" Oct 30 01:47:57.075156 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:57.075124 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-264bdd-65d6c7bf46-cq56s"] Oct 30 01:47:57.123669 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:57.123632 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dbd7a2c-c776-496e-ac5a-6a97046ab3f7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-264bdd-65d6c7bf46-cq56s\" (UID: \"6dbd7a2c-c776-496e-ac5a-6a97046ab3f7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-264bdd-65d6c7bf46-cq56s" Oct 30 01:47:57.123878 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:57.123757 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsbxl\" (UniqueName: \"kubernetes.io/projected/6dbd7a2c-c776-496e-ac5a-6a97046ab3f7-kube-api-access-fsbxl\") pod \"payload-processing-e2e-shared-b-w2-264bdd-65d6c7bf46-cq56s\" (UID: \"6dbd7a2c-c776-496e-ac5a-6a97046ab3f7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-264bdd-65d6c7bf46-cq56s" Oct 30 01:47:57.225584 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:57.225546 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dbd7a2c-c776-496e-ac5a-6a97046ab3f7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-264bdd-65d6c7bf46-cq56s\" (UID: \"6dbd7a2c-c776-496e-ac5a-6a97046ab3f7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-264bdd-65d6c7bf46-cq56s" Oct 30 01:47:57.225797 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:57.225655 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsbxl\" (UniqueName: \"kubernetes.io/projected/6dbd7a2c-c776-496e-ac5a-6a97046ab3f7-kube-api-access-fsbxl\") pod \"payload-processing-e2e-shared-b-w2-264bdd-65d6c7bf46-cq56s\" (UID: \"6dbd7a2c-c776-496e-ac5a-6a97046ab3f7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-264bdd-65d6c7bf46-cq56s" Oct 30 01:47:57.226546 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:57.226520 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dbd7a2c-c776-496e-ac5a-6a97046ab3f7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-264bdd-65d6c7bf46-cq56s\" (UID: \"6dbd7a2c-c776-496e-ac5a-6a97046ab3f7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-264bdd-65d6c7bf46-cq56s" Oct 30 01:47:57.239947 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:57.239917 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:57.245997 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:57.245970 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsbxl\" (UniqueName: \"kubernetes.io/projected/6dbd7a2c-c776-496e-ac5a-6a97046ab3f7-kube-api-access-fsbxl\") pod \"payload-processing-e2e-shared-b-w2-264bdd-65d6c7bf46-cq56s\" (UID: \"6dbd7a2c-c776-496e-ac5a-6a97046ab3f7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-264bdd-65d6c7bf46-cq56s" Oct 30 01:47:57.350821 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:57.350705 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-264bdd-65d6c7bf46-cq56s" Oct 30 01:47:57.498774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:57.493578 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:57.825824 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:57.823159 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:57.867570 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:57.867536 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-264bdd-65d6c7bf46-cq56s"] Oct 30 01:47:57.870091 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:47:57.870063 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dbd7a2c_c776_496e_ac5a_6a97046ab3f7.slice/crio-c63946815fef4f4082a770fac17c9ee0dbbcdc5ef634206e0dfcbd2ab4128554 WatchSource:0}: Error finding container c63946815fef4f4082a770fac17c9ee0dbbcdc5ef634206e0dfcbd2ab4128554: Status 404 returned error can't find the container with id c63946815fef4f4082a770fac17c9ee0dbbcdc5ef634206e0dfcbd2ab4128554 Oct 30 01:47:58.131411 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:58.131333 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-264bdd-65d6c7bf46-cq56s" event={"ID":"6dbd7a2c-c776-496e-ac5a-6a97046ab3f7","Type":"ContainerStarted","Data":"1aea154994541a3bfe5be6dda7f2b5680d88761896e8d4b8361a9b694fa7d032"} Oct 30 01:47:58.131411 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:58.131388 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-264bdd-65d6c7bf46-cq56s" event={"ID":"6dbd7a2c-c776-496e-ac5a-6a97046ab3f7","Type":"ContainerStarted","Data":"c63946815fef4f4082a770fac17c9ee0dbbcdc5ef634206e0dfcbd2ab4128554"} Oct 30 01:47:58.131621 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:58.131486 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-264bdd-65d6c7bf46-cq56s" Oct 30 01:47:58.180758 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:58.180694 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-264bdd-65d6c7bf46-cq56s" podStartSLOduration=2.18067519 podStartE2EDuration="2.18067519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:58.172690402 +0000 UTC m=+1803.749115814" watchObservedRunningTime="2026-10-30 01:47:58.18067519 +0000 UTC m=+1803.757100599" Oct 30 01:47:59.702631 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:47:59.702540 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:00.029904 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:00.029830 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9477f07e-mfh26"] Oct 30 01:48:00.041575 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:00.041549 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9477f07e-mfh26"] Oct 30 01:48:00.156742 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:00.156707 6572 generic.go:356] "Generic (PLEG): container finished" podID="322cef03-4f70-42c8-8571-42c939fba68b" containerID="fe9b85c6a37107c277e18fb1db2f2d60c35aa9f585467f2d4ef2cae370a26036" exitCode=0 Oct 30 01:48:00.156939 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:00.156789 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" event={"ID":"322cef03-4f70-42c8-8571-42c939fba68b","Type":"ContainerDied","Data":"fe9b85c6a37107c277e18fb1db2f2d60c35aa9f585467f2d4ef2cae370a26036"} Oct 30 01:48:00.157192 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:00.157174 6572 scope.go:122] "RemoveContainer" containerID="fe9b85c6a37107c277e18fb1db2f2d60c35aa9f585467f2d4ef2cae370a26036" Oct 30 01:48:00.158648 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:00.158618 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:48:00.158732 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:00.158689 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:48:00.158732 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:00.158720 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:48:00.544797 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:00.544758 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="806c6aa2-26f5-460b-be1e-e7112f9b679d" path="/var/lib/kubelet/pods/806c6aa2-26f5-460b-be1e-e7112f9b679d/volumes" Oct 30 01:48:00.758838 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:00.758788 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:00.947976 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:00.947937 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:01.168305 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:01.168271 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" event={"ID":"322cef03-4f70-42c8-8571-42c939fba68b","Type":"ContainerStarted","Data":"a47a8d4c2caa136ea3ab6db923cb02b3b0bc5dee668d52aae61b5d4773314b0f"} Oct 30 01:48:01.598804 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:01.598378 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:01.867987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:01.867909 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 01:48:01.869384 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:01.869356 6572 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 30 01:48:01.869520 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:01.869419 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" podUID="322cef03-4f70-42c8-8571-42c939fba68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 30 01:48:02.458392 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:02.458357 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:02.549526 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:02.549494 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-389dba-859b76bcbc-vv6jx"] Oct 30 01:48:02.555021 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:02.554996 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-389dba-859b76bcbc-vv6jx" Oct 30 01:48:02.572042 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:02.572016 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-389dba-859b76bcbc-vv6jx"] Oct 30 01:48:02.575604 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:02.575577 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-389dba\"" Oct 30 01:48:02.575741 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:02.575655 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-389dba-dockercfg-sghn6\"" Oct 30 01:48:02.652505 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:02.652472 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:02.686808 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:02.686766 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4099303a-6f82-4f5d-a962-7ccb1251ca9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-389dba-859b76bcbc-vv6jx\" (UID: \"4099303a-6f82-4f5d-a962-7ccb1251ca9a\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-389dba-859b76bcbc-vv6jx" Oct 30 01:48:02.687201 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:02.687178 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp64q\" (UniqueName: \"kubernetes.io/projected/4099303a-6f82-4f5d-a962-7ccb1251ca9a-kube-api-access-jp64q\") pod \"payload-pre-processing-e2e-explicit-389dba-859b76bcbc-vv6jx\" (UID: \"4099303a-6f82-4f5d-a962-7ccb1251ca9a\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-389dba-859b76bcbc-vv6jx" Oct 30 01:48:02.819820 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:02.819000 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp64q\" (UniqueName: \"kubernetes.io/projected/4099303a-6f82-4f5d-a962-7ccb1251ca9a-kube-api-access-jp64q\") pod \"payload-pre-processing-e2e-explicit-389dba-859b76bcbc-vv6jx\" (UID: \"4099303a-6f82-4f5d-a962-7ccb1251ca9a\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-389dba-859b76bcbc-vv6jx" Oct 30 01:48:02.819820 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:02.819313 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4099303a-6f82-4f5d-a962-7ccb1251ca9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-389dba-859b76bcbc-vv6jx\" (UID: \"4099303a-6f82-4f5d-a962-7ccb1251ca9a\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-389dba-859b76bcbc-vv6jx" Oct 30 01:48:02.878333 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:02.868530 6572 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 30 01:48:02.885210 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:02.885168 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" podUID="322cef03-4f70-42c8-8571-42c939fba68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 30 01:48:02.888850 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:02.888798 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4099303a-6f82-4f5d-a962-7ccb1251ca9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-389dba-859b76bcbc-vv6jx\" (UID: \"4099303a-6f82-4f5d-a962-7ccb1251ca9a\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-389dba-859b76bcbc-vv6jx" Oct 30 01:48:02.891225 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:02.891183 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp64q\" (UniqueName: \"kubernetes.io/projected/4099303a-6f82-4f5d-a962-7ccb1251ca9a-kube-api-access-jp64q\") pod \"payload-pre-processing-e2e-explicit-389dba-859b76bcbc-vv6jx\" (UID: \"4099303a-6f82-4f5d-a962-7ccb1251ca9a\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-389dba-859b76bcbc-vv6jx" Oct 30 01:48:03.183039 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:03.183000 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-389dba-859b76bcbc-vv6jx" Oct 30 01:48:03.386408 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:03.386326 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:03.570291 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:03.570142 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:03.642000 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:48:03.641962 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4099303a_6f82_4f5d_a962_7ccb1251ca9a.slice/crio-5cf1ced1041ebe5b152c272f999627df5541c0ca48cc78c791ad81466d716bc9 WatchSource:0}: Error finding container 5cf1ced1041ebe5b152c272f999627df5541c0ca48cc78c791ad81466d716bc9: Status 404 returned error can't find the container with id 5cf1ced1041ebe5b152c272f999627df5541c0ca48cc78c791ad81466d716bc9 Oct 30 01:48:03.642186 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:03.642166 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-389dba-859b76bcbc-vv6jx"] Oct 30 01:48:03.765381 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:03.765347 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:03.868224 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:03.868143 6572 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 30 01:48:03.868224 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:03.868202 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" podUID="322cef03-4f70-42c8-8571-42c939fba68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 30 01:48:04.259523 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:04.259482 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-389dba-859b76bcbc-vv6jx" event={"ID":"4099303a-6f82-4f5d-a962-7ccb1251ca9a","Type":"ContainerStarted","Data":"c9970dffc3bf6f3bd575ac7bc1a16dfd684871c2f1e67c446737faf70aa4d6f5"} Oct 30 01:48:04.259997 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:04.259573 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-389dba-859b76bcbc-vv6jx" event={"ID":"4099303a-6f82-4f5d-a962-7ccb1251ca9a","Type":"ContainerStarted","Data":"5cf1ced1041ebe5b152c272f999627df5541c0ca48cc78c791ad81466d716bc9"} Oct 30 01:48:04.259997 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:04.259721 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-389dba-859b76bcbc-vv6jx" Oct 30 01:48:04.303498 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:04.303429 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-389dba-859b76bcbc-vv6jx" podStartSLOduration=2.303416112 podStartE2EDuration="2.303416112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:04.298413115 +0000 UTC m=+1809.874838517" watchObservedRunningTime="2026-10-30 01:48:04.303416112 +0000 UTC m=+1809.879841518" Oct 30 01:48:04.867830 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:04.867793 6572 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 30 01:48:04.868030 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:04.867847 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" podUID="322cef03-4f70-42c8-8571-42c939fba68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 30 01:48:05.332727 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:05.332692 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:05.525361 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:05.525327 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:05.724307 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:05.724214 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:05.867387 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:05.867352 6572 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 30 01:48:05.867564 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:05.867417 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" podUID="322cef03-4f70-42c8-8571-42c939fba68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 30 01:48:05.931722 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:05.931683 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:06.026826 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:06.021858 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 01:48:06.373975 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:06.373885 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:06.570629 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:06.570592 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:06.872130 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:06.872100 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 01:48:06.873362 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:06.873340 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 01:48:07.197092 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:07.197054 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:08.894710 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:08.894596 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:09.145688 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:09.145607 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-264bdd-65d6c7bf46-cq56s" Oct 30 01:48:10.531924 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:10.531875 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:10.743687 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:10.743650 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:12.289370 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:12.289336 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:12.394308 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:12.394268 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-657fba-6d7784b879-n5xjk"] Oct 30 01:48:12.399027 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:12.399001 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-657fba-6d7784b879-n5xjk" Oct 30 01:48:12.416898 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:12.416857 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-657fba-6d7784b879-n5xjk"] Oct 30 01:48:12.422556 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:12.422528 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-657fba\"" Oct 30 01:48:12.435427 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:12.435388 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d336f442-61fa-4ba4-81a6-08eb9f852bad-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-657fba-6d7784b879-n5xjk\" (UID: \"d336f442-61fa-4ba4-81a6-08eb9f852bad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-657fba-6d7784b879-n5xjk" Oct 30 01:48:12.435837 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:12.435804 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6d5t\" (UniqueName: \"kubernetes.io/projected/d336f442-61fa-4ba4-81a6-08eb9f852bad-kube-api-access-z6d5t\") pod \"payload-pre-processing-e2e-shared-b-w4-657fba-6d7784b879-n5xjk\" (UID: \"d336f442-61fa-4ba4-81a6-08eb9f852bad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-657fba-6d7784b879-n5xjk" Oct 30 01:48:12.503224 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:12.502653 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-657fba-dockercfg-t2t2b\"" Oct 30 01:48:12.526019 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:12.525979 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:12.558836 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:12.541655 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d336f442-61fa-4ba4-81a6-08eb9f852bad-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-657fba-6d7784b879-n5xjk\" (UID: \"d336f442-61fa-4ba4-81a6-08eb9f852bad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-657fba-6d7784b879-n5xjk" Oct 30 01:48:12.656337 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:12.656298 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6d5t\" (UniqueName: \"kubernetes.io/projected/d336f442-61fa-4ba4-81a6-08eb9f852bad-kube-api-access-z6d5t\") pod \"payload-pre-processing-e2e-shared-b-w4-657fba-6d7784b879-n5xjk\" (UID: \"d336f442-61fa-4ba4-81a6-08eb9f852bad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-657fba-6d7784b879-n5xjk" Oct 30 01:48:12.656861 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:12.547910 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d336f442-61fa-4ba4-81a6-08eb9f852bad-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-657fba-6d7784b879-n5xjk\" (UID: \"d336f442-61fa-4ba4-81a6-08eb9f852bad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-657fba-6d7784b879-n5xjk" Oct 30 01:48:12.668857 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:12.668762 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-657fba-8485db5fdd-5spds"] Oct 30 01:48:12.673942 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:12.673877 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-657fba-8485db5fdd-5spds"] Oct 30 01:48:12.674149 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:12.673989 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-657fba-8485db5fdd-5spds" Oct 30 01:48:12.692891 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:12.677628 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6d5t\" (UniqueName: \"kubernetes.io/projected/d336f442-61fa-4ba4-81a6-08eb9f852bad-kube-api-access-z6d5t\") pod \"payload-pre-processing-e2e-shared-b-w4-657fba-6d7784b879-n5xjk\" (UID: \"d336f442-61fa-4ba4-81a6-08eb9f852bad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-657fba-6d7784b879-n5xjk" Oct 30 01:48:12.715854 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:12.715816 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-657fba-6d7784b879-n5xjk" Oct 30 01:48:12.756876 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:12.756835 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:12.757874 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:12.757843 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wmv9\" (UniqueName: \"kubernetes.io/projected/8ba16392-25f9-4163-9003-941b83f70c6d-kube-api-access-9wmv9\") pod \"payload-processing-e2e-shared-b-w4-657fba-8485db5fdd-5spds\" (UID: \"8ba16392-25f9-4163-9003-941b83f70c6d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-657fba-8485db5fdd-5spds" Oct 30 01:48:12.758138 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:12.758080 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba16392-25f9-4163-9003-941b83f70c6d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-657fba-8485db5fdd-5spds\" (UID: \"8ba16392-25f9-4163-9003-941b83f70c6d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-657fba-8485db5fdd-5spds" Oct 30 01:48:12.859890 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:12.859849 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wmv9\" (UniqueName: \"kubernetes.io/projected/8ba16392-25f9-4163-9003-941b83f70c6d-kube-api-access-9wmv9\") pod \"payload-processing-e2e-shared-b-w4-657fba-8485db5fdd-5spds\" (UID: \"8ba16392-25f9-4163-9003-941b83f70c6d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-657fba-8485db5fdd-5spds" Oct 30 01:48:12.860100 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:12.859927 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba16392-25f9-4163-9003-941b83f70c6d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-657fba-8485db5fdd-5spds\" (UID: \"8ba16392-25f9-4163-9003-941b83f70c6d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-657fba-8485db5fdd-5spds" Oct 30 01:48:12.860638 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:12.860591 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba16392-25f9-4163-9003-941b83f70c6d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-657fba-8485db5fdd-5spds\" (UID: \"8ba16392-25f9-4163-9003-941b83f70c6d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-657fba-8485db5fdd-5spds" Oct 30 01:48:12.881003 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:12.880966 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wmv9\" (UniqueName: \"kubernetes.io/projected/8ba16392-25f9-4163-9003-941b83f70c6d-kube-api-access-9wmv9\") pod \"payload-processing-e2e-shared-b-w4-657fba-8485db5fdd-5spds\" (UID: \"8ba16392-25f9-4163-9003-941b83f70c6d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-657fba-8485db5fdd-5spds" Oct 30 01:48:13.020882 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:13.020849 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-657fba-8485db5fdd-5spds" Oct 30 01:48:13.059520 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:48:13.059491 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd336f442_61fa_4ba4_81a6_08eb9f852bad.slice/crio-c34efd72763ba771eb282c87b45fd7a38dbd17148e954a9eb191bb92d43d5150 WatchSource:0}: Error finding container c34efd72763ba771eb282c87b45fd7a38dbd17148e954a9eb191bb92d43d5150: Status 404 returned error can't find the container with id c34efd72763ba771eb282c87b45fd7a38dbd17148e954a9eb191bb92d43d5150 Oct 30 01:48:13.061244 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:13.061214 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-657fba-6d7784b879-n5xjk"] Oct 30 01:48:13.331890 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:13.331864 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-657fba-8485db5fdd-5spds"] Oct 30 01:48:13.332245 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:48:13.332178 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba16392_25f9_4163_9003_941b83f70c6d.slice/crio-7b4192c9353441d583e655e72fb78cc0ce0ab6ee3c358c26a900f2b44031816d WatchSource:0}: Error finding container 7b4192c9353441d583e655e72fb78cc0ce0ab6ee3c358c26a900f2b44031816d: Status 404 returned error can't find the container with id 7b4192c9353441d583e655e72fb78cc0ce0ab6ee3c358c26a900f2b44031816d Oct 30 01:48:13.349944 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:13.349907 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-657fba-6d7784b879-n5xjk" event={"ID":"d336f442-61fa-4ba4-81a6-08eb9f852bad","Type":"ContainerStarted","Data":"5b40fad244cd547e94cb3fe03bcbd20baa635a43d5cbfa8d052f70865f2ce164"} Oct 30 01:48:13.350135 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:13.349963 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-657fba-6d7784b879-n5xjk" event={"ID":"d336f442-61fa-4ba4-81a6-08eb9f852bad","Type":"ContainerStarted","Data":"c34efd72763ba771eb282c87b45fd7a38dbd17148e954a9eb191bb92d43d5150"} Oct 30 01:48:13.350135 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:13.350126 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-657fba-6d7784b879-n5xjk" Oct 30 01:48:13.351546 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:13.351523 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-657fba-8485db5fdd-5spds" event={"ID":"8ba16392-25f9-4163-9003-941b83f70c6d","Type":"ContainerStarted","Data":"7b4192c9353441d583e655e72fb78cc0ce0ab6ee3c358c26a900f2b44031816d"} Oct 30 01:48:13.391270 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:13.391219 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-657fba-6d7784b879-n5xjk" podStartSLOduration=1.391201361 podStartE2EDuration="1.391201361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:13.387682471 +0000 UTC m=+1818.964107874" watchObservedRunningTime="2026-10-30 01:48:13.391201361 +0000 UTC m=+1818.967626815" Oct 30 01:48:14.397040 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:14.397003 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-657fba-8485db5fdd-5spds" event={"ID":"8ba16392-25f9-4163-9003-941b83f70c6d","Type":"ContainerStarted","Data":"f5bc9ef526b52f21aa6d007258077fe79157f1221d8ffa7084277b6c029150ce"} Oct 30 01:48:14.397446 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:14.397083 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-657fba-8485db5fdd-5spds" Oct 30 01:48:14.434725 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:14.434643 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-657fba-8485db5fdd-5spds" podStartSLOduration=2.4346248360000002 podStartE2EDuration="2.434624836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:14.434393494 +0000 UTC m=+1820.010818897" watchObservedRunningTime="2026-10-30 01:48:14.434624836 +0000 UTC m=+1820.011050243" Oct 30 01:48:14.486024 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:14.485973 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:14.688774 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:48:14.688394 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:48:15.312044 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:15.312008 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-389dba-859b76bcbc-vv6jx" Oct 30 01:48:16.246165 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:16.246117 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:17.838247 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:17.838213 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:18.020495 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:18.020447 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:19.597818 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:19.597694 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:21.277943 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:21.277905 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:22.829452 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:22.829411 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:23.041711 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:23.041673 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:23.239090 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:23.239037 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:24.400438 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:24.400406 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-657fba-6d7784b879-n5xjk" Oct 30 01:48:24.858479 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:24.858437 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:26.475977 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:26.475946 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-657fba-8485db5fdd-5spds" Oct 30 01:48:26.618990 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:26.618953 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:28.303442 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:28.303406 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:30.124289 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:30.124257 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:31.779025 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:31.778988 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:31.961108 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:31.961069 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:33.607988 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:33.607948 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:35.181613 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:35.181579 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:35.363112 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:35.363073 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:35.552036 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:35.551937 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:37.112209 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:37.112163 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:39.030341 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:39.030301 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:40.955983 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:40.955945 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:42.750094 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:42.750050 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:44.500519 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:44.500486 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:44.687462 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:48:44.687423 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:48:46.357193 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:46.357162 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:46.575840 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:46.575794 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:48.410388 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:48.410349 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:50.028649 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:50.028611 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:51.029792 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:51.029742 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:51.658494 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:51.658458 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:53.485470 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:53.485435 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:55.196519 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:55.196486 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:56.828866 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:56.828831 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:56.876504 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:56.876468 6572 scope.go:122] "RemoveContainer" containerID="fb0c84f36f550373e03817d193f8da2ac1c307e95d9385fbf7de1d4303d0715b" Oct 30 01:48:56.901015 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:56.900985 6572 scope.go:122] "RemoveContainer" containerID="0361785c962212fac6a82f52919e9aec51b7e5407379f13ab6de25e8779413b0" Oct 30 01:48:57.401523 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:57.401487 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:59.180029 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:48:59.179994 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:00.953999 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:00.953957 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:01.137852 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:01.137712 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:02.666913 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:02.666874 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:04.355297 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:04.355259 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:05.669831 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:05.669780 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:06.212491 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:06.212455 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:06.515150 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:06.515074 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l"] Oct 30 01:49:06.521292 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:06.521268 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" Oct 30 01:49:06.535647 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:06.535624 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-657fba-openshift-default-dockercfg-vwt4n\"" Oct 30 01:49:06.561627 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:06.560978 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l"] Oct 30 01:49:06.602967 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:06.602929 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:06.682058 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:06.682025 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-istio-data\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l\" (UID: \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" Oct 30 01:49:06.682447 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:06.682160 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-credential-socket\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l\" (UID: \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" Oct 30 01:49:06.682447 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:06.682311 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-istio-envoy\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l\" (UID: \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" Oct 30 01:49:06.682447 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:06.682334 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-istio-token\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l\" (UID: \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" Oct 30 01:49:06.682447 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:06.682356 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-istio-podinfo\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l\" (UID: \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" Oct 30 01:49:06.682447 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:06.682386 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-istiod-ca-cert\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l\" (UID: \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" Oct 30 01:49:06.682633 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:06.682498 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-workload-socket\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l\" (UID: \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" Oct 30 01:49:06.682633 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:06.682525 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8chm8\" (UniqueName: \"kubernetes.io/projected/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-kube-api-access-8chm8\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l\" (UID: \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" Oct 30 01:49:06.682633 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:06.682562 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-workload-certs\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l\" (UID: \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" Oct 30 01:49:06.784263 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:06.784176 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-istio-data\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l\" (UID: \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" Oct 30 01:49:06.784442 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:06.784284 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-credential-socket\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l\" (UID: \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" Oct 30 01:49:06.784442 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:06.784426 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-istio-envoy\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l\" (UID: \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" Oct 30 01:49:06.784578 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:06.784456 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-istio-token\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l\" (UID: \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" Oct 30 01:49:06.784578 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:06.784493 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-istio-podinfo\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l\" (UID: \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" Oct 30 01:49:06.784578 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:06.784531 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-istiod-ca-cert\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l\" (UID: \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" Oct 30 01:49:06.784762 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:06.784612 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-workload-socket\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l\" (UID: \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" Oct 30 01:49:06.784762 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:06.784638 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8chm8\" (UniqueName: \"kubernetes.io/projected/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-kube-api-access-8chm8\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l\" (UID: \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" Oct 30 01:49:06.784762 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:06.784699 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-workload-certs\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l\" (UID: \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" Oct 30 01:49:06.784762 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:06.784705 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-istio-data\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l\" (UID: \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" Oct 30 01:49:06.784762 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:06.784721 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-credential-socket\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l\" (UID: \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" Oct 30 01:49:06.785113 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:06.785085 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-workload-certs\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l\" (UID: \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" Oct 30 01:49:06.785595 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:06.785547 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-workload-socket\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l\" (UID: \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" Oct 30 01:49:06.785715 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:06.785681 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-istiod-ca-cert\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l\" (UID: \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" Oct 30 01:49:06.787722 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:06.787687 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-istio-podinfo\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l\" (UID: \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" Oct 30 01:49:06.788242 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:06.788209 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-istio-envoy\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l\" (UID: \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" Oct 30 01:49:06.803913 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:06.803889 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8chm8\" (UniqueName: \"kubernetes.io/projected/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-kube-api-access-8chm8\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l\" (UID: \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" Oct 30 01:49:06.804060 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:06.803942 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-istio-token\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l\" (UID: \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" Oct 30 01:49:06.835491 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:06.835460 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" Oct 30 01:49:07.158270 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:49:07.158236 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2afdd4b_323c_41fc_9919_ed0d7cbce70f.slice/crio-bfe82c925666a9c977d4f6afed4b8efed79f79a2bcffb84cc98055c9f084f0f2 WatchSource:0}: Error finding container bfe82c925666a9c977d4f6afed4b8efed79f79a2bcffb84cc98055c9f084f0f2: Status 404 returned error can't find the container with id bfe82c925666a9c977d4f6afed4b8efed79f79a2bcffb84cc98055c9f084f0f2 Oct 30 01:49:07.158890 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:07.158866 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l"] Oct 30 01:49:07.161017 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:07.160983 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:49:07.161123 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:07.161050 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:49:07.161123 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:07.161084 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:49:07.914250 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:07.913952 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:07.998049 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:07.998009 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" event={"ID":"c2afdd4b-323c-41fc-9919-ed0d7cbce70f","Type":"ContainerStarted","Data":"ac429be00f0ab50134a85f48b699af3457cd0985722309a01200c59b8adf90fd"} Oct 30 01:49:07.998240 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:07.998074 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" event={"ID":"c2afdd4b-323c-41fc-9919-ed0d7cbce70f","Type":"ContainerStarted","Data":"bfe82c925666a9c977d4f6afed4b8efed79f79a2bcffb84cc98055c9f084f0f2"} Oct 30 01:49:08.066434 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:08.066371 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" podStartSLOduration=2.06635606 podStartE2EDuration="2.06635606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:49:08.051617413 +0000 UTC m=+1873.628042814" watchObservedRunningTime="2026-10-30 01:49:08.06635606 +0000 UTC m=+1873.642781462" Oct 30 01:49:08.836156 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:08.836120 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" Oct 30 01:49:08.837709 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:08.837682 6572 patch_prober.go:29] interesting pod/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body= Oct 30 01:49:08.837880 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:08.837732 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" podUID="c2afdd4b-323c-41fc-9919-ed0d7cbce70f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" Oct 30 01:49:09.593718 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:09.593675 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:09.836406 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:09.836369 6572 patch_prober.go:29] interesting pod/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body= Oct 30 01:49:09.836594 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:09.836439 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" podUID="c2afdd4b-323c-41fc-9919-ed0d7cbce70f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" Oct 30 01:49:10.836292 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:10.836251 6572 patch_prober.go:29] interesting pod/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body= Oct 30 01:49:10.836717 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:10.836324 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" podUID="c2afdd4b-323c-41fc-9919-ed0d7cbce70f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" Oct 30 01:49:11.274373 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:11.274340 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:11.836412 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:11.836367 6572 patch_prober.go:29] interesting pod/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body= Oct 30 01:49:11.836971 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:11.836432 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" podUID="c2afdd4b-323c-41fc-9919-ed0d7cbce70f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" Oct 30 01:49:12.836737 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:12.836700 6572 patch_prober.go:29] interesting pod/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body= Oct 30 01:49:12.837181 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:12.836787 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" podUID="c2afdd4b-323c-41fc-9919-ed0d7cbce70f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" Oct 30 01:49:13.027345 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:13.027308 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:13.843155 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:13.843101 6572 patch_prober.go:29] interesting pod/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body= Oct 30 01:49:13.843652 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:13.843620 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" podUID="c2afdd4b-323c-41fc-9919-ed0d7cbce70f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" Oct 30 01:49:14.688019 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:49:14.687975 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:49:14.841580 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:14.841495 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" Oct 30 01:49:14.842052 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:14.842021 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" Oct 30 01:49:14.842960 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:14.842934 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" Oct 30 01:49:14.943427 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:14.943388 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:16.683865 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:16.683824 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:18.381778 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:18.379941 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:19.294653 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:19.294598 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:20.078787 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:20.078719 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:21.726731 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:21.726695 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:23.409326 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:23.409282 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:25.195868 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:25.195831 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:26.692897 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:26.692857 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:26.884513 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:26.884476 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:27.962960 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:27.962924 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7"] Oct 30 01:49:27.969969 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:27.969938 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7" Oct 30 01:49:27.986243 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:27.986211 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7"] Oct 30 01:49:27.990636 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:27.990603 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-292018\"/\"openshift-service-ca.crt\"" Oct 30 01:49:27.990846 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:27.990714 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-292018\"/\"auth-test-model-292018-kserve-self-signed-certs\"" Oct 30 01:49:27.997052 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:27.997024 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-292018\"/\"kube-root-ca.crt\"" Oct 30 01:49:27.997052 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:27.997030 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-292018\"/\"default-dockercfg-qpcqt\"" Oct 30 01:49:28.111487 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:28.111447 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:28.123275 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:28.123244 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f89f1d37-12c8-4a6c-a3c0-533234c89a74-dshm\") pod \"auth-test-model-292018-kserve-6fccf6cf7c-mmqr7\" (UID: \"f89f1d37-12c8-4a6c-a3c0-533234c89a74\") " pod="ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7" Oct 30 01:49:28.123456 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:28.123309 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f89f1d37-12c8-4a6c-a3c0-533234c89a74-model-cache\") pod \"auth-test-model-292018-kserve-6fccf6cf7c-mmqr7\" (UID: \"f89f1d37-12c8-4a6c-a3c0-533234c89a74\") " pod="ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7" Oct 30 01:49:28.123529 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:28.123495 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f89f1d37-12c8-4a6c-a3c0-533234c89a74-tmp-dir\") pod \"auth-test-model-292018-kserve-6fccf6cf7c-mmqr7\" (UID: \"f89f1d37-12c8-4a6c-a3c0-533234c89a74\") " pod="ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7" Oct 30 01:49:28.123607 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:28.123548 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f89f1d37-12c8-4a6c-a3c0-533234c89a74-tls-certs\") pod \"auth-test-model-292018-kserve-6fccf6cf7c-mmqr7\" (UID: \"f89f1d37-12c8-4a6c-a3c0-533234c89a74\") " pod="ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7" Oct 30 01:49:28.123667 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:28.123613 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f89f1d37-12c8-4a6c-a3c0-533234c89a74-home\") pod \"auth-test-model-292018-kserve-6fccf6cf7c-mmqr7\" (UID: \"f89f1d37-12c8-4a6c-a3c0-533234c89a74\") " pod="ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7" Oct 30 01:49:28.123828 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:28.123809 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m57cz\" (UniqueName: \"kubernetes.io/projected/f89f1d37-12c8-4a6c-a3c0-533234c89a74-kube-api-access-m57cz\") pod \"auth-test-model-292018-kserve-6fccf6cf7c-mmqr7\" (UID: \"f89f1d37-12c8-4a6c-a3c0-533234c89a74\") " pod="ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7" Oct 30 01:49:28.232166 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:28.232007 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f89f1d37-12c8-4a6c-a3c0-533234c89a74-home\") pod \"auth-test-model-292018-kserve-6fccf6cf7c-mmqr7\" (UID: \"f89f1d37-12c8-4a6c-a3c0-533234c89a74\") " pod="ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7" Oct 30 01:49:28.238900 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:28.238863 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m57cz\" (UniqueName: \"kubernetes.io/projected/f89f1d37-12c8-4a6c-a3c0-533234c89a74-kube-api-access-m57cz\") pod \"auth-test-model-292018-kserve-6fccf6cf7c-mmqr7\" (UID: \"f89f1d37-12c8-4a6c-a3c0-533234c89a74\") " pod="ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7" Oct 30 01:49:28.239466 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:28.239428 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f89f1d37-12c8-4a6c-a3c0-533234c89a74-dshm\") pod \"auth-test-model-292018-kserve-6fccf6cf7c-mmqr7\" (UID: \"f89f1d37-12c8-4a6c-a3c0-533234c89a74\") " pod="ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7" Oct 30 01:49:28.239615 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:28.239531 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f89f1d37-12c8-4a6c-a3c0-533234c89a74-model-cache\") pod \"auth-test-model-292018-kserve-6fccf6cf7c-mmqr7\" (UID: \"f89f1d37-12c8-4a6c-a3c0-533234c89a74\") " pod="ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7" Oct 30 01:49:28.239615 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:28.239601 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f89f1d37-12c8-4a6c-a3c0-533234c89a74-tmp-dir\") pod \"auth-test-model-292018-kserve-6fccf6cf7c-mmqr7\" (UID: \"f89f1d37-12c8-4a6c-a3c0-533234c89a74\") " pod="ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7" Oct 30 01:49:28.239741 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:28.239634 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f89f1d37-12c8-4a6c-a3c0-533234c89a74-tls-certs\") pod \"auth-test-model-292018-kserve-6fccf6cf7c-mmqr7\" (UID: \"f89f1d37-12c8-4a6c-a3c0-533234c89a74\") " pod="ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7" Oct 30 01:49:28.240887 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:28.240863 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f89f1d37-12c8-4a6c-a3c0-533234c89a74-model-cache\") pod \"auth-test-model-292018-kserve-6fccf6cf7c-mmqr7\" (UID: \"f89f1d37-12c8-4a6c-a3c0-533234c89a74\") " pod="ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7" Oct 30 01:49:28.241060 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:28.238330 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f89f1d37-12c8-4a6c-a3c0-533234c89a74-home\") pod \"auth-test-model-292018-kserve-6fccf6cf7c-mmqr7\" (UID: \"f89f1d37-12c8-4a6c-a3c0-533234c89a74\") " pod="ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7" Oct 30 01:49:28.241501 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:28.241472 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f89f1d37-12c8-4a6c-a3c0-533234c89a74-tmp-dir\") pod \"auth-test-model-292018-kserve-6fccf6cf7c-mmqr7\" (UID: \"f89f1d37-12c8-4a6c-a3c0-533234c89a74\") " pod="ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7" Oct 30 01:49:28.277778 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:28.277327 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f89f1d37-12c8-4a6c-a3c0-533234c89a74-tls-certs\") pod \"auth-test-model-292018-kserve-6fccf6cf7c-mmqr7\" (UID: \"f89f1d37-12c8-4a6c-a3c0-533234c89a74\") " pod="ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7" Oct 30 01:49:28.281777 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:28.278150 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m57cz\" (UniqueName: \"kubernetes.io/projected/f89f1d37-12c8-4a6c-a3c0-533234c89a74-kube-api-access-m57cz\") pod \"auth-test-model-292018-kserve-6fccf6cf7c-mmqr7\" (UID: \"f89f1d37-12c8-4a6c-a3c0-533234c89a74\") " pod="ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7" Oct 30 01:49:28.288684 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:28.288649 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f89f1d37-12c8-4a6c-a3c0-533234c89a74-dshm\") pod \"auth-test-model-292018-kserve-6fccf6cf7c-mmqr7\" (UID: \"f89f1d37-12c8-4a6c-a3c0-533234c89a74\") " pod="ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7" Oct 30 01:49:28.293237 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:28.292965 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7" Oct 30 01:49:28.524934 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:28.524620 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:28.619813 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:49:28.619775 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf89f1d37_12c8_4a6c_a3c0_533234c89a74.slice/crio-900f55d8a0e928079f0ac6e2de01da820d9819c38da4a4edb28acaea3c3e2082 WatchSource:0}: Error finding container 900f55d8a0e928079f0ac6e2de01da820d9819c38da4a4edb28acaea3c3e2082: Status 404 returned error can't find the container with id 900f55d8a0e928079f0ac6e2de01da820d9819c38da4a4edb28acaea3c3e2082 Oct 30 01:49:28.619813 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:28.619789 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7"] Oct 30 01:49:29.273531 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:29.273493 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7" event={"ID":"f89f1d37-12c8-4a6c-a3c0-533234c89a74","Type":"ContainerStarted","Data":"ce818bfbab2fc8b99655fc9241577539dfb400827adf9d60e5185c678576226a"} Oct 30 01:49:29.273531 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:29.273541 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7" event={"ID":"f89f1d37-12c8-4a6c-a3c0-533234c89a74","Type":"ContainerStarted","Data":"900f55d8a0e928079f0ac6e2de01da820d9819c38da4a4edb28acaea3c3e2082"} Oct 30 01:49:29.294224 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:29.294185 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7" Oct 30 01:49:29.319301 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:29.319228 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7" podStartSLOduration=2.319197961 podStartE2EDuration="2.319197961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:49:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:49:29.310906505 +0000 UTC m=+1894.887331907" watchObservedRunningTime="2026-10-30 01:49:29.319197961 +0000 UTC m=+1894.895623364" Oct 30 01:49:30.208086 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:30.208047 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:31.918361 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:31.918322 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:33.708401 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:33.708304 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:34.029695 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:34.029592 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-493521fb-zttmx"] Oct 30 01:49:34.041232 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:34.041196 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-493521fb-zttmx"] Oct 30 01:49:34.544697 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:34.544653 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2588b110-cd82-4f31-a36b-4f2fc12876f5" path="/var/lib/kubelet/pods/2588b110-cd82-4f31-a36b-4f2fc12876f5/volumes" Oct 30 01:49:35.404848 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:35.404805 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:35.969366 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:35.969325 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:36.141298 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:36.141265 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-389dba-g7jp7"] Oct 30 01:49:36.146380 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:36.146353 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-389dba-g7jp7" Oct 30 01:49:36.165887 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:36.165851 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-389dba-g7jp7"] Oct 30 01:49:36.171048 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:36.171017 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:49:36.171273 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:36.171085 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:49:36.208805 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:36.208762 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:49:36.208994 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:36.208845 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:49:36.273594 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:36.273507 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-389dba-g7jp7\" (UID: \"d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-389dba-g7jp7" Oct 30 01:49:36.273783 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:36.273657 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p4q6\" (UniqueName: \"kubernetes.io/projected/d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2-kube-api-access-8p4q6\") pod \"maas-api-revoke-keys-e2e-explicit-389dba-g7jp7\" (UID: \"d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-389dba-g7jp7" Oct 30 01:49:36.273837 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:36.273812 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-389dba-g7jp7\" (UID: \"d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-389dba-g7jp7" Oct 30 01:49:36.375612 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:36.375571 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-389dba-g7jp7\" (UID: \"d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-389dba-g7jp7" Oct 30 01:49:36.375859 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:36.375674 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p4q6\" (UniqueName: \"kubernetes.io/projected/d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2-kube-api-access-8p4q6\") pod \"maas-api-revoke-keys-e2e-explicit-389dba-g7jp7\" (UID: \"d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-389dba-g7jp7" Oct 30 01:49:36.375859 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:36.375830 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-389dba-g7jp7\" (UID: \"d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-389dba-g7jp7" Oct 30 01:49:36.376496 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:36.376475 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-389dba-g7jp7\" (UID: \"d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-389dba-g7jp7" Oct 30 01:49:36.378099 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:36.378081 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-389dba-g7jp7\" (UID: \"d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-389dba-g7jp7" Oct 30 01:49:36.395165 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:36.395129 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p4q6\" (UniqueName: \"kubernetes.io/projected/d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2-kube-api-access-8p4q6\") pod \"maas-api-revoke-keys-e2e-explicit-389dba-g7jp7\" (UID: \"d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-389dba-g7jp7" Oct 30 01:49:36.466360 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:36.466318 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-389dba-g7jp7" Oct 30 01:49:36.765581 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:36.765553 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-389dba-g7jp7"] Oct 30 01:49:36.765740 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:49:36.765717 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd32beb8a_f58a_47f0_ba9b_ed6e0ff865e2.slice/crio-af2d27fac6830a432cc62e7b3823dae9238261a5320a734a076ae47042552f07 WatchSource:0}: Error finding container af2d27fac6830a432cc62e7b3823dae9238261a5320a734a076ae47042552f07: Status 404 returned error can't find the container with id af2d27fac6830a432cc62e7b3823dae9238261a5320a734a076ae47042552f07 Oct 30 01:49:37.256032 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:37.255984 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:37.420615 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:37.420579 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-389dba-g7jp7" event={"ID":"d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2","Type":"ContainerStarted","Data":"a68738a7e5773d2d40590a6cdab0a288607fb621a3a8383d2c423e8b8e79ff62"} Oct 30 01:49:37.420793 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:37.420635 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-389dba-g7jp7" event={"ID":"d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2","Type":"ContainerStarted","Data":"af2d27fac6830a432cc62e7b3823dae9238261a5320a734a076ae47042552f07"} Oct 30 01:49:37.462098 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:37.462042 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-389dba-g7jp7" podStartSLOduration=1.462027844 podStartE2EDuration="1.462027844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:49:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:49:37.458379893 +0000 UTC m=+1903.034805294" watchObservedRunningTime="2026-10-30 01:49:37.462027844 +0000 UTC m=+1903.038453245" Oct 30 01:49:38.293781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:38.293721 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7" Oct 30 01:49:38.307155 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:38.307124 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7" Oct 30 01:49:38.432922 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:38.432888 6572 generic.go:356] "Generic (PLEG): container finished" podID="d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2" containerID="a68738a7e5773d2d40590a6cdab0a288607fb621a3a8383d2c423e8b8e79ff62" exitCode=0 Oct 30 01:49:38.433109 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:38.432968 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-389dba-g7jp7" event={"ID":"d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2","Type":"ContainerDied","Data":"a68738a7e5773d2d40590a6cdab0a288607fb621a3a8383d2c423e8b8e79ff62"} Oct 30 01:49:38.445506 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:38.445478 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7" Oct 30 01:49:39.089140 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:39.089107 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:39.626393 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:39.626364 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-389dba-g7jp7" Oct 30 01:49:39.723965 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:39.723873 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2-maas-api-openshift-ca\") pod \"d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2\" (UID: \"d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2\") " Oct 30 01:49:39.723965 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:39.723927 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2-maas-api-xks-ca\") pod \"d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2\" (UID: \"d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2\") " Oct 30 01:49:39.724164 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:39.724084 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2-kube-api-access-8p4q6\" (UniqueName: \"kubernetes.io/projected/d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2-kube-api-access-8p4q6\") pod \"d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2\" (UID: \"d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2\") " Oct 30 01:49:39.724339 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:39.724305 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2-maas-api-openshift-ca" pod "d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2" (UID: "d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:49:39.724680 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:39.724663 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:49:39.726086 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:39.726063 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2-maas-api-xks-ca" pod "d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2" (UID: "d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:49:39.726221 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:39.726201 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2-kube-api-access-8p4q6" pod "d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2" (UID: "d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2"). InnerVolumeSpecName "kube-api-access-8p4q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:49:39.826349 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:39.826311 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:49:39.826349 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:39.826340 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p4q6\" (UniqueName: \"kubernetes.io/projected/d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2-kube-api-access-8p4q6\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:49:40.455859 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:40.455821 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-389dba-g7jp7" Oct 30 01:49:40.456063 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:40.455859 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-389dba-g7jp7" event={"ID":"d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2","Type":"ContainerDied","Data":"af2d27fac6830a432cc62e7b3823dae9238261a5320a734a076ae47042552f07"} Oct 30 01:49:40.456063 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:40.455960 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af2d27fac6830a432cc62e7b3823dae9238261a5320a734a076ae47042552f07" Oct 30 01:49:40.677049 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:40.677018 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:42.318452 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:42.318417 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:44.162533 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:44.162499 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:44.371392 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:44.371350 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc"] Oct 30 01:49:44.372082 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:44.372062 6572 cpu_manager.go:414] "containerMap: removing container" podUID="d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2" containerName="revoke-keys" Oct 30 01:49:44.372082 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:44.372080 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2" containerName="revoke-keys" Oct 30 01:49:44.372275 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:44.372194 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2" containerName="revoke-keys" Oct 30 01:49:44.376248 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:44.376220 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc" Oct 30 01:49:44.395571 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:44.395533 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc"] Oct 30 01:49:44.402046 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:44.401240 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-264bdd\"/\"kube-root-ca.crt\"" Oct 30 01:49:44.402046 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:44.401986 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-264bdd\"/\"openshift-service-ca.crt\"" Oct 30 01:49:44.402818 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:44.402793 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-264bdd\"/\"auth-test-model-264bdd-kserve-self-signed-certs\"" Oct 30 01:49:44.402925 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:44.402862 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-264bdd\"/\"default-dockercfg-hr7k5\"" Oct 30 01:49:44.495073 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:44.494972 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/452989de-dd18-450d-872c-316d00cf3f44-home\") pod \"auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc\" (UID: \"452989de-dd18-450d-872c-316d00cf3f44\") " pod="ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc" Oct 30 01:49:44.495073 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:44.495026 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/452989de-dd18-450d-872c-316d00cf3f44-tls-certs\") pod \"auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc\" (UID: \"452989de-dd18-450d-872c-316d00cf3f44\") " pod="ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc" Oct 30 01:49:44.495310 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:44.495233 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/452989de-dd18-450d-872c-316d00cf3f44-tmp-dir\") pod \"auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc\" (UID: \"452989de-dd18-450d-872c-316d00cf3f44\") " pod="ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc" Oct 30 01:49:44.495434 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:44.495416 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr7vw\" (UniqueName: \"kubernetes.io/projected/452989de-dd18-450d-872c-316d00cf3f44-kube-api-access-rr7vw\") pod \"auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc\" (UID: \"452989de-dd18-450d-872c-316d00cf3f44\") " pod="ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc" Oct 30 01:49:44.495530 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:44.495482 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/452989de-dd18-450d-872c-316d00cf3f44-model-cache\") pod \"auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc\" (UID: \"452989de-dd18-450d-872c-316d00cf3f44\") " pod="ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc" Oct 30 01:49:44.495607 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:44.495526 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/452989de-dd18-450d-872c-316d00cf3f44-dshm\") pod \"auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc\" (UID: \"452989de-dd18-450d-872c-316d00cf3f44\") " pod="ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc" Oct 30 01:49:44.525990 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:44.525939 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:44.596966 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:44.596923 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/452989de-dd18-450d-872c-316d00cf3f44-home\") pod \"auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc\" (UID: \"452989de-dd18-450d-872c-316d00cf3f44\") " pod="ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc" Oct 30 01:49:44.596966 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:44.596967 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/452989de-dd18-450d-872c-316d00cf3f44-tls-certs\") pod \"auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc\" (UID: \"452989de-dd18-450d-872c-316d00cf3f44\") " pod="ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc" Oct 30 01:49:44.597273 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:44.597043 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/452989de-dd18-450d-872c-316d00cf3f44-tmp-dir\") pod \"auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc\" (UID: \"452989de-dd18-450d-872c-316d00cf3f44\") " pod="ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc" Oct 30 01:49:44.597273 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:44.597114 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr7vw\" (UniqueName: \"kubernetes.io/projected/452989de-dd18-450d-872c-316d00cf3f44-kube-api-access-rr7vw\") pod \"auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc\" (UID: \"452989de-dd18-450d-872c-316d00cf3f44\") " pod="ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc" Oct 30 01:49:44.597273 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:44.597147 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/452989de-dd18-450d-872c-316d00cf3f44-model-cache\") pod \"auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc\" (UID: \"452989de-dd18-450d-872c-316d00cf3f44\") " pod="ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc" Oct 30 01:49:44.597273 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:44.597185 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/452989de-dd18-450d-872c-316d00cf3f44-dshm\") pod \"auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc\" (UID: \"452989de-dd18-450d-872c-316d00cf3f44\") " pod="ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc" Oct 30 01:49:44.597574 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:44.597424 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/452989de-dd18-450d-872c-316d00cf3f44-home\") pod \"auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc\" (UID: \"452989de-dd18-450d-872c-316d00cf3f44\") " pod="ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc" Oct 30 01:49:44.597574 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:44.597500 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/452989de-dd18-450d-872c-316d00cf3f44-tmp-dir\") pod \"auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc\" (UID: \"452989de-dd18-450d-872c-316d00cf3f44\") " pod="ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc" Oct 30 01:49:44.597762 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:44.597726 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/452989de-dd18-450d-872c-316d00cf3f44-model-cache\") pod \"auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc\" (UID: \"452989de-dd18-450d-872c-316d00cf3f44\") " pod="ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc" Oct 30 01:49:44.599452 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:44.599434 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/452989de-dd18-450d-872c-316d00cf3f44-dshm\") pod \"auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc\" (UID: \"452989de-dd18-450d-872c-316d00cf3f44\") " pod="ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc" Oct 30 01:49:44.599853 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:44.599827 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/452989de-dd18-450d-872c-316d00cf3f44-tls-certs\") pod \"auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc\" (UID: \"452989de-dd18-450d-872c-316d00cf3f44\") " pod="ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc" Oct 30 01:49:44.635850 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:44.635775 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr7vw\" (UniqueName: \"kubernetes.io/projected/452989de-dd18-450d-872c-316d00cf3f44-kube-api-access-rr7vw\") pod \"auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc\" (UID: \"452989de-dd18-450d-872c-316d00cf3f44\") " pod="ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc" Oct 30 01:49:44.700799 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:44.698049 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc" Oct 30 01:49:44.768930 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:49:44.768888 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:49:45.050131 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:49:45.050098 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod452989de_dd18_450d_872c_316d00cf3f44.slice/crio-7d4b7245b9304db84c66301c28061741d1c9ea5a2b3f6e83a874197592d9de43 WatchSource:0}: Error finding container 7d4b7245b9304db84c66301c28061741d1c9ea5a2b3f6e83a874197592d9de43: Status 404 returned error can't find the container with id 7d4b7245b9304db84c66301c28061741d1c9ea5a2b3f6e83a874197592d9de43 Oct 30 01:49:45.051158 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:45.051129 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc"] Oct 30 01:49:45.510827 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:45.510786 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc" event={"ID":"452989de-dd18-450d-872c-316d00cf3f44","Type":"ContainerStarted","Data":"5e9f59d3929642e70494b1fedfb1efec139fc7b8a1239fff6febdcecf9a98088"} Oct 30 01:49:45.511332 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:45.510845 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc" event={"ID":"452989de-dd18-450d-872c-316d00cf3f44","Type":"ContainerStarted","Data":"7d4b7245b9304db84c66301c28061741d1c9ea5a2b3f6e83a874197592d9de43"} Oct 30 01:49:45.552246 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:45.552194 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc" podStartSLOduration=1.5521797720000001 podStartE2EDuration="1.552179772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:49:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:49:45.548886672 +0000 UTC m=+1911.125312073" watchObservedRunningTime="2026-10-30 01:49:45.552179772 +0000 UTC m=+1911.128605173" Oct 30 01:49:45.700334 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:45.700293 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc" Oct 30 01:49:45.894661 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:45.894577 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:46.578817 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:46.578775 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-389dba-859b76bcbc-vv6jx"] Oct 30 01:49:46.793775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:46.793099 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:46.972675 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:46.972642 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:47.032831 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:47.032794 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-952d1786-6wx7g"] Oct 30 01:49:47.037356 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:47.037329 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-952d1786-6wx7g"] Oct 30 01:49:47.683598 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:47.683556 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:47.899068 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:47.899032 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb"] Oct 30 01:49:47.899288 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:47.899246 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" podUID="c1d4d67f-e18a-4152-be41-e1832bfd78ed" containerName="istio-proxy" containerID="cri-o://be6a218ad2dced4589e2c8870ca85335b324ff7b660c70950ec65701e681be52" gracePeriod=30 Oct 30 01:49:48.028167 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:48.028076 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3612ab3f-p4hql"] Oct 30 01:49:48.032592 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:48.032567 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3612ab3f-p4hql"] Oct 30 01:49:48.178401 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:48.178366 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:48.359537 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:48.359450 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:48.544251 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:48.544219 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="326e903b-0283-438b-b9c6-f463b7f685ae" path="/var/lib/kubelet/pods/326e903b-0283-438b-b9c6-f463b7f685ae/volumes" Oct 30 01:49:48.544713 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:48.544699 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98e0bfb4-2b67-4c67-91ff-72a484a8e201" path="/var/lib/kubelet/pods/98e0bfb4-2b67-4c67-91ff-72a484a8e201/volumes" Oct 30 01:49:49.354173 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:49.354135 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:51.059888 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:51.059855 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55e61c16-9hgtn"] Oct 30 01:49:51.072455 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:51.072413 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55e61c16-9hgtn"] Oct 30 01:49:51.079058 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:51.079006 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:51.579743 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:51.579691 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-389dba-859b76bcbc-vv6jx" podUID="4099303a-6f82-4f5d-a962-7ccb1251ca9a" containerName="payload-pre-processing" containerID="cri-o://c9970dffc3bf6f3bd575ac7bc1a16dfd684871c2f1e67c446737faf70aa4d6f5" gracePeriod=25 Oct 30 01:49:51.869883 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:51.869847 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-389dba-859b76bcbc-vv6jx" Oct 30 01:49:51.921595 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:51.921549 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4099303a-6f82-4f5d-a962-7ccb1251ca9a-kube-api-access-jp64q\" (UniqueName: \"kubernetes.io/projected/4099303a-6f82-4f5d-a962-7ccb1251ca9a-kube-api-access-jp64q\") pod \"4099303a-6f82-4f5d-a962-7ccb1251ca9a\" (UID: \"4099303a-6f82-4f5d-a962-7ccb1251ca9a\") " Oct 30 01:49:51.921932 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:51.921624 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4099303a-6f82-4f5d-a962-7ccb1251ca9a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4099303a-6f82-4f5d-a962-7ccb1251ca9a-plugins-config-volume\") pod \"4099303a-6f82-4f5d-a962-7ccb1251ca9a\" (UID: \"4099303a-6f82-4f5d-a962-7ccb1251ca9a\") " Oct 30 01:49:51.922768 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:51.922723 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4099303a-6f82-4f5d-a962-7ccb1251ca9a-plugins-config-volume" pod "4099303a-6f82-4f5d-a962-7ccb1251ca9a" (UID: "4099303a-6f82-4f5d-a962-7ccb1251ca9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:49:51.931407 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:51.931363 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4099303a-6f82-4f5d-a962-7ccb1251ca9a-kube-api-access-jp64q" pod "4099303a-6f82-4f5d-a962-7ccb1251ca9a" (UID: "4099303a-6f82-4f5d-a962-7ccb1251ca9a"). InnerVolumeSpecName "kube-api-access-jp64q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:49:52.023695 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:52.023661 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp64q\" (UniqueName: \"kubernetes.io/projected/4099303a-6f82-4f5d-a962-7ccb1251ca9a-kube-api-access-jp64q\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:49:52.023695 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:52.023694 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4099303a-6f82-4f5d-a962-7ccb1251ca9a-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:49:52.543662 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:52.543630 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4" path="/var/lib/kubelet/pods/bff214ad-ad9e-4e94-ba24-e43ce7dfa8b4/volumes" Oct 30 01:49:52.576500 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:52.576459 6572 generic.go:356] "Generic (PLEG): container finished" podID="4099303a-6f82-4f5d-a962-7ccb1251ca9a" containerID="c9970dffc3bf6f3bd575ac7bc1a16dfd684871c2f1e67c446737faf70aa4d6f5" exitCode=0 Oct 30 01:49:52.576682 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:52.576517 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-389dba-859b76bcbc-vv6jx" Oct 30 01:49:52.576682 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:52.576540 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-389dba-859b76bcbc-vv6jx" event={"ID":"4099303a-6f82-4f5d-a962-7ccb1251ca9a","Type":"ContainerDied","Data":"c9970dffc3bf6f3bd575ac7bc1a16dfd684871c2f1e67c446737faf70aa4d6f5"} Oct 30 01:49:52.576682 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:52.576599 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-389dba-859b76bcbc-vv6jx" event={"ID":"4099303a-6f82-4f5d-a962-7ccb1251ca9a","Type":"ContainerDied","Data":"5cf1ced1041ebe5b152c272f999627df5541c0ca48cc78c791ad81466d716bc9"} Oct 30 01:49:52.576682 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:52.576627 6572 scope.go:122] "RemoveContainer" containerID="c9970dffc3bf6f3bd575ac7bc1a16dfd684871c2f1e67c446737faf70aa4d6f5" Oct 30 01:49:52.589282 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:52.589256 6572 scope.go:122] "RemoveContainer" containerID="c9970dffc3bf6f3bd575ac7bc1a16dfd684871c2f1e67c446737faf70aa4d6f5" Oct 30 01:49:52.589660 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:49:52.589637 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9970dffc3bf6f3bd575ac7bc1a16dfd684871c2f1e67c446737faf70aa4d6f5\": container with ID starting with c9970dffc3bf6f3bd575ac7bc1a16dfd684871c2f1e67c446737faf70aa4d6f5 not found: ID does not exist" containerID="c9970dffc3bf6f3bd575ac7bc1a16dfd684871c2f1e67c446737faf70aa4d6f5" Oct 30 01:49:52.589738 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:52.589669 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9970dffc3bf6f3bd575ac7bc1a16dfd684871c2f1e67c446737faf70aa4d6f5"} err="failed to get container status \"c9970dffc3bf6f3bd575ac7bc1a16dfd684871c2f1e67c446737faf70aa4d6f5\": rpc error: code = NotFound desc = could not find container \"c9970dffc3bf6f3bd575ac7bc1a16dfd684871c2f1e67c446737faf70aa4d6f5\": container with ID starting with c9970dffc3bf6f3bd575ac7bc1a16dfd684871c2f1e67c446737faf70aa4d6f5 not found: ID does not exist" Oct 30 01:49:52.629624 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:52.629584 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-389dba-859b76bcbc-vv6jx"] Oct 30 01:49:52.634188 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:52.634161 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-389dba-859b76bcbc-vv6jx"] Oct 30 01:49:52.729874 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:52.729825 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:53.186143 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:53.186064 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" Oct 30 01:49:53.241861 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:53.241827 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d4d67f-e18a-4152-be41-e1832bfd78ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d4d67f-e18a-4152-be41-e1832bfd78ed-workload-certs\") pod \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\" (UID: \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\") " Oct 30 01:49:53.242038 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:53.241958 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d4d67f-e18a-4152-be41-e1832bfd78ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d4d67f-e18a-4152-be41-e1832bfd78ed-istio-envoy\") pod \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\" (UID: \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\") " Oct 30 01:49:53.242038 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:53.241999 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d4d67f-e18a-4152-be41-e1832bfd78ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d4d67f-e18a-4152-be41-e1832bfd78ed-credential-socket\") pod \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\" (UID: \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\") " Oct 30 01:49:53.242038 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:53.242030 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1d4d67f-e18a-4152-be41-e1832bfd78ed-istio-token\" (UniqueName: \"kubernetes.io/projected/c1d4d67f-e18a-4152-be41-e1832bfd78ed-istio-token\") pod \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\" (UID: \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\") " Oct 30 01:49:53.242241 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:53.242110 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d4d67f-e18a-4152-be41-e1832bfd78ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d4d67f-e18a-4152-be41-e1832bfd78ed-istio-data\") pod \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\" (UID: \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\") " Oct 30 01:49:53.242241 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:53.242162 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1d4d67f-e18a-4152-be41-e1832bfd78ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d4d67f-e18a-4152-be41-e1832bfd78ed-istio-podinfo\") pod \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\" (UID: \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\") " Oct 30 01:49:53.242241 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:53.242198 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1d4d67f-e18a-4152-be41-e1832bfd78ed-kube-api-access-cbf5x\" (UniqueName: \"kubernetes.io/projected/c1d4d67f-e18a-4152-be41-e1832bfd78ed-kube-api-access-cbf5x\") pod \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\" (UID: \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\") " Oct 30 01:49:53.242456 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:53.242257 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1d4d67f-e18a-4152-be41-e1832bfd78ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d4d67f-e18a-4152-be41-e1832bfd78ed-istiod-ca-cert\") pod \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\" (UID: \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\") " Oct 30 01:49:53.242456 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:53.242286 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d4d67f-e18a-4152-be41-e1832bfd78ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d4d67f-e18a-4152-be41-e1832bfd78ed-workload-socket\") pod \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\" (UID: \"c1d4d67f-e18a-4152-be41-e1832bfd78ed\") " Oct 30 01:49:53.242456 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:53.242407 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d4d67f-e18a-4152-be41-e1832bfd78ed-credential-socket" pod "c1d4d67f-e18a-4152-be41-e1832bfd78ed" (UID: "c1d4d67f-e18a-4152-be41-e1832bfd78ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:49:53.243108 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:53.242895 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d4d67f-e18a-4152-be41-e1832bfd78ed-istio-data" pod "c1d4d67f-e18a-4152-be41-e1832bfd78ed" (UID: "c1d4d67f-e18a-4152-be41-e1832bfd78ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:49:53.243419 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:53.243028 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d4d67f-e18a-4152-be41-e1832bfd78ed-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:49:53.243419 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:53.243380 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d4d67f-e18a-4152-be41-e1832bfd78ed-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:49:53.243419 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:53.243118 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d4d67f-e18a-4152-be41-e1832bfd78ed-workload-socket" pod "c1d4d67f-e18a-4152-be41-e1832bfd78ed" (UID: "c1d4d67f-e18a-4152-be41-e1832bfd78ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:49:53.243419 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:53.243393 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1d4d67f-e18a-4152-be41-e1832bfd78ed-istiod-ca-cert" pod "c1d4d67f-e18a-4152-be41-e1832bfd78ed" (UID: "c1d4d67f-e18a-4152-be41-e1832bfd78ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:49:53.243419 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:53.243289 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d4d67f-e18a-4152-be41-e1832bfd78ed-workload-certs" pod "c1d4d67f-e18a-4152-be41-e1832bfd78ed" (UID: "c1d4d67f-e18a-4152-be41-e1832bfd78ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:49:53.245072 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:53.245037 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d4d67f-e18a-4152-be41-e1832bfd78ed-istio-envoy" pod "c1d4d67f-e18a-4152-be41-e1832bfd78ed" (UID: "c1d4d67f-e18a-4152-be41-e1832bfd78ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:49:53.245436 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:53.245403 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d4d67f-e18a-4152-be41-e1832bfd78ed-istio-token" pod "c1d4d67f-e18a-4152-be41-e1832bfd78ed" (UID: "c1d4d67f-e18a-4152-be41-e1832bfd78ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:49:53.245577 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:53.245428 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d4d67f-e18a-4152-be41-e1832bfd78ed-kube-api-access-cbf5x" pod "c1d4d67f-e18a-4152-be41-e1832bfd78ed" (UID: "c1d4d67f-e18a-4152-be41-e1832bfd78ed"). InnerVolumeSpecName "kube-api-access-cbf5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:49:53.245730 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:53.245705 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1d4d67f-e18a-4152-be41-e1832bfd78ed-istio-podinfo" pod "c1d4d67f-e18a-4152-be41-e1832bfd78ed" (UID: "c1d4d67f-e18a-4152-be41-e1832bfd78ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:49:53.344595 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:53.344556 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d4d67f-e18a-4152-be41-e1832bfd78ed-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:49:53.344595 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:53.344588 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d4d67f-e18a-4152-be41-e1832bfd78ed-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:49:53.344595 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:53.344597 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d4d67f-e18a-4152-be41-e1832bfd78ed-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:49:53.344901 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:53.344608 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbf5x\" (UniqueName: \"kubernetes.io/projected/c1d4d67f-e18a-4152-be41-e1832bfd78ed-kube-api-access-cbf5x\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:49:53.344901 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:53.344619 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d4d67f-e18a-4152-be41-e1832bfd78ed-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:49:53.344901 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:53.344628 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d4d67f-e18a-4152-be41-e1832bfd78ed-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:49:53.344901 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:53.344637 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d4d67f-e18a-4152-be41-e1832bfd78ed-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:49:53.589276 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:53.589173 6572 generic.go:356] "Generic (PLEG): container finished" podID="c1d4d67f-e18a-4152-be41-e1832bfd78ed" containerID="be6a218ad2dced4589e2c8870ca85335b324ff7b660c70950ec65701e681be52" exitCode=0 Oct 30 01:49:53.589276 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:53.589221 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" Oct 30 01:49:53.589276 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:53.589260 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" event={"ID":"c1d4d67f-e18a-4152-be41-e1832bfd78ed","Type":"ContainerDied","Data":"be6a218ad2dced4589e2c8870ca85335b324ff7b660c70950ec65701e681be52"} Oct 30 01:49:53.589945 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:53.589327 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb" event={"ID":"c1d4d67f-e18a-4152-be41-e1832bfd78ed","Type":"ContainerDied","Data":"e9d1ea4d0ad4770670163b524e46e94d377dae48e936b3fdb617a02f044165d2"} Oct 30 01:49:53.589945 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:53.589353 6572 scope.go:122] "RemoveContainer" containerID="be6a218ad2dced4589e2c8870ca85335b324ff7b660c70950ec65701e681be52" Oct 30 01:49:53.602138 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:53.602116 6572 scope.go:122] "RemoveContainer" containerID="be6a218ad2dced4589e2c8870ca85335b324ff7b660c70950ec65701e681be52" Oct 30 01:49:53.602539 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:49:53.602517 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be6a218ad2dced4589e2c8870ca85335b324ff7b660c70950ec65701e681be52\": container with ID starting with be6a218ad2dced4589e2c8870ca85335b324ff7b660c70950ec65701e681be52 not found: ID does not exist" containerID="be6a218ad2dced4589e2c8870ca85335b324ff7b660c70950ec65701e681be52" Oct 30 01:49:53.602602 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:53.602552 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be6a218ad2dced4589e2c8870ca85335b324ff7b660c70950ec65701e681be52"} err="failed to get container status \"be6a218ad2dced4589e2c8870ca85335b324ff7b660c70950ec65701e681be52\": rpc error: code = NotFound desc = could not find container \"be6a218ad2dced4589e2c8870ca85335b324ff7b660c70950ec65701e681be52\": container with ID starting with be6a218ad2dced4589e2c8870ca85335b324ff7b660c70950ec65701e681be52 not found: ID does not exist" Oct 30 01:49:53.644036 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:53.644001 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb"] Oct 30 01:49:53.655550 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:53.655519 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-nsfcb"] Oct 30 01:49:54.473035 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:54.473002 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:54.544520 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:54.544485 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4099303a-6f82-4f5d-a962-7ccb1251ca9a" path="/var/lib/kubelet/pods/4099303a-6f82-4f5d-a962-7ccb1251ca9a/volumes" Oct 30 01:49:54.545120 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:54.545102 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1d4d67f-e18a-4152-be41-e1832bfd78ed" path="/var/lib/kubelet/pods/c1d4d67f-e18a-4152-be41-e1832bfd78ed/volumes" Oct 30 01:49:54.700333 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:54.700298 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc" Oct 30 01:49:54.714061 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:54.714030 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc" Oct 30 01:49:55.622604 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:55.622570 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc" Oct 30 01:49:56.184641 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:56.184601 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:56.943410 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:56.943373 6572 scope.go:122] "RemoveContainer" containerID="2c52c569095a5c908c4a92c928c0caca17ac0870e2967605ba5b7de4b69e8fb7" Oct 30 01:49:56.955884 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:56.955858 6572 scope.go:122] "RemoveContainer" containerID="e01ed8c8a0d4065d9ead85624fa784ef7f202bd22fd246efce25d6146c75216d" Oct 30 01:49:56.967224 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:56.967201 6572 scope.go:122] "RemoveContainer" containerID="8ed4abecda950aa8d35439877ccc030f41033276b85146100db4a9d4674565bd" Oct 30 01:49:57.029074 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:57.029041 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c045c797-2fpp8"] Oct 30 01:49:57.039619 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:57.039581 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c045c797-2fpp8"] Oct 30 01:49:57.767316 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:57.767281 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:58.543936 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:58.543898 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee4f2e25-28ea-41f9-9e33-001c0f482823" path="/var/lib/kubelet/pods/ee4f2e25-28ea-41f9-9e33-001c0f482823/volumes" Oct 30 01:49:59.458166 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:49:59.458130 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:01.027791 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:01.027743 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b80122c5-lc25m"] Oct 30 01:50:01.039549 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:01.039511 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b80122c5-lc25m"] Oct 30 01:50:01.126360 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:01.125841 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:02.543724 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:02.543682 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2810118-aba3-474f-aefb-ce3393aaef31" path="/var/lib/kubelet/pods/c2810118-aba3-474f-aefb-ce3393aaef31/volumes" Oct 30 01:50:02.827875 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:02.827734 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:03.027220 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:03.027185 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9315ff4-f46th"] Oct 30 01:50:03.038445 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:03.038416 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9315ff4-f46th"] Oct 30 01:50:04.511035 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:04.511001 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:04.546640 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:04.546607 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7057658c-88a0-4bce-bb3f-765903a4b7fa" path="/var/lib/kubelet/pods/7057658c-88a0-4bce-bb3f-765903a4b7fa/volumes" Oct 30 01:50:05.775181 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:05.775140 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:06.100319 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:06.100193 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:07.470985 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:07.470951 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:08.264011 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:08.263976 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:08.384476 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:08.384429 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-389dba-g7jp7"] Oct 30 01:50:08.395278 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:08.395237 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-389dba-g7jp7"] Oct 30 01:50:08.466304 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:08.462916 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:08.544231 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:08.544146 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2" path="/var/lib/kubelet/pods/d32beb8a-f58a-47f0-ba9b-ed6e0ff865e2/volumes" Oct 30 01:50:08.650001 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:08.649965 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:08.990014 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:08.989969 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:10.055145 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:10.055093 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5fa3e48-djt6k"] Oct 30 01:50:10.081786 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:10.081574 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5fa3e48-djt6k"] Oct 30 01:50:10.294294 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:10.294253 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:10.655772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:10.655260 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b64f76b0-d2d3-48ea-a283-b901c5ac4b67" path="/var/lib/kubelet/pods/b64f76b0-d2d3-48ea-a283-b901c5ac4b67/volumes" Oct 30 01:50:11.317090 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:11.317055 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:12.097705 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:12.097660 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:13.837997 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:13.837955 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:14.689210 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:50:14.689169 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:50:15.171876 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:15.171837 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:15.359759 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:15.359719 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:15.543615 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:15.543334 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:15.739332 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:15.739290 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:16.167500 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.167465 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:16.495839 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.495760 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn"] Oct 30 01:50:16.496860 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.496835 6572 cpu_manager.go:414] "containerMap: removing container" podUID="c1d4d67f-e18a-4152-be41-e1832bfd78ed" containerName="istio-proxy" Oct 30 01:50:16.496860 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.496860 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1d4d67f-e18a-4152-be41-e1832bfd78ed" containerName="istio-proxy" Oct 30 01:50:16.497014 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.496924 6572 cpu_manager.go:414] "containerMap: removing container" podUID="4099303a-6f82-4f5d-a962-7ccb1251ca9a" containerName="payload-pre-processing" Oct 30 01:50:16.497014 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.496933 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4099303a-6f82-4f5d-a962-7ccb1251ca9a" containerName="payload-pre-processing" Oct 30 01:50:16.497092 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.497079 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1d4d67f-e18a-4152-be41-e1832bfd78ed" containerName="istio-proxy" Oct 30 01:50:16.497134 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.497106 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="4099303a-6f82-4f5d-a962-7ccb1251ca9a" containerName="payload-pre-processing" Oct 30 01:50:16.501597 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.501572 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" Oct 30 01:50:16.520539 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.520512 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn"] Oct 30 01:50:16.525962 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.525937 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-36c949-openshift-default-dockercfg-bsjkw\"" Oct 30 01:50:16.613773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.606878 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3604271e-e5a0-493c-ad79-9ca6e191722a-workload-certs\") pod \"e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn\" (UID: \"3604271e-e5a0-493c-ad79-9ca6e191722a\") " pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" Oct 30 01:50:16.613773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.606987 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3604271e-e5a0-493c-ad79-9ca6e191722a-credential-socket\") pod \"e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn\" (UID: \"3604271e-e5a0-493c-ad79-9ca6e191722a\") " pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" Oct 30 01:50:16.613773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.607026 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3604271e-e5a0-493c-ad79-9ca6e191722a-istio-envoy\") pod \"e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn\" (UID: \"3604271e-e5a0-493c-ad79-9ca6e191722a\") " pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" Oct 30 01:50:16.613773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.607065 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzkgc\" (UniqueName: \"kubernetes.io/projected/3604271e-e5a0-493c-ad79-9ca6e191722a-kube-api-access-xzkgc\") pod \"e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn\" (UID: \"3604271e-e5a0-493c-ad79-9ca6e191722a\") " pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" Oct 30 01:50:16.613773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.607203 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3604271e-e5a0-493c-ad79-9ca6e191722a-istio-data\") pod \"e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn\" (UID: \"3604271e-e5a0-493c-ad79-9ca6e191722a\") " pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" Oct 30 01:50:16.613773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.607281 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3604271e-e5a0-493c-ad79-9ca6e191722a-istio-podinfo\") pod \"e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn\" (UID: \"3604271e-e5a0-493c-ad79-9ca6e191722a\") " pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" Oct 30 01:50:16.613773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.607349 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3604271e-e5a0-493c-ad79-9ca6e191722a-istiod-ca-cert\") pod \"e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn\" (UID: \"3604271e-e5a0-493c-ad79-9ca6e191722a\") " pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" Oct 30 01:50:16.613773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.607395 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3604271e-e5a0-493c-ad79-9ca6e191722a-istio-token\") pod \"e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn\" (UID: \"3604271e-e5a0-493c-ad79-9ca6e191722a\") " pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" Oct 30 01:50:16.613773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.607458 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3604271e-e5a0-493c-ad79-9ca6e191722a-workload-socket\") pod \"e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn\" (UID: \"3604271e-e5a0-493c-ad79-9ca6e191722a\") " pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" Oct 30 01:50:16.659533 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.659497 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:16.716875 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.711106 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3604271e-e5a0-493c-ad79-9ca6e191722a-workload-certs\") pod \"e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn\" (UID: \"3604271e-e5a0-493c-ad79-9ca6e191722a\") " pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" Oct 30 01:50:16.716875 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.711197 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3604271e-e5a0-493c-ad79-9ca6e191722a-credential-socket\") pod \"e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn\" (UID: \"3604271e-e5a0-493c-ad79-9ca6e191722a\") " pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" Oct 30 01:50:16.716875 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.711230 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3604271e-e5a0-493c-ad79-9ca6e191722a-istio-envoy\") pod \"e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn\" (UID: \"3604271e-e5a0-493c-ad79-9ca6e191722a\") " pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" Oct 30 01:50:16.716875 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.711265 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzkgc\" (UniqueName: \"kubernetes.io/projected/3604271e-e5a0-493c-ad79-9ca6e191722a-kube-api-access-xzkgc\") pod \"e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn\" (UID: \"3604271e-e5a0-493c-ad79-9ca6e191722a\") " pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" Oct 30 01:50:16.716875 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.711343 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3604271e-e5a0-493c-ad79-9ca6e191722a-istio-data\") pod \"e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn\" (UID: \"3604271e-e5a0-493c-ad79-9ca6e191722a\") " pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" Oct 30 01:50:16.716875 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.711411 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3604271e-e5a0-493c-ad79-9ca6e191722a-istio-podinfo\") pod \"e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn\" (UID: \"3604271e-e5a0-493c-ad79-9ca6e191722a\") " pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" Oct 30 01:50:16.716875 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.711485 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3604271e-e5a0-493c-ad79-9ca6e191722a-istiod-ca-cert\") pod \"e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn\" (UID: \"3604271e-e5a0-493c-ad79-9ca6e191722a\") " pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" Oct 30 01:50:16.716875 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.711541 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3604271e-e5a0-493c-ad79-9ca6e191722a-istio-token\") pod \"e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn\" (UID: \"3604271e-e5a0-493c-ad79-9ca6e191722a\") " pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" Oct 30 01:50:16.716875 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.711597 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3604271e-e5a0-493c-ad79-9ca6e191722a-workload-socket\") pod \"e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn\" (UID: \"3604271e-e5a0-493c-ad79-9ca6e191722a\") " pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" Oct 30 01:50:16.716875 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.712454 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3604271e-e5a0-493c-ad79-9ca6e191722a-workload-certs\") pod \"e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn\" (UID: \"3604271e-e5a0-493c-ad79-9ca6e191722a\") " pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" Oct 30 01:50:16.716875 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.712700 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3604271e-e5a0-493c-ad79-9ca6e191722a-credential-socket\") pod \"e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn\" (UID: \"3604271e-e5a0-493c-ad79-9ca6e191722a\") " pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" Oct 30 01:50:16.732573 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.732527 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3604271e-e5a0-493c-ad79-9ca6e191722a-istio-data\") pod \"e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn\" (UID: \"3604271e-e5a0-493c-ad79-9ca6e191722a\") " pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" Oct 30 01:50:16.734803 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.734407 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3604271e-e5a0-493c-ad79-9ca6e191722a-istiod-ca-cert\") pod \"e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn\" (UID: \"3604271e-e5a0-493c-ad79-9ca6e191722a\") " pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" Oct 30 01:50:16.755855 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.745808 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3604271e-e5a0-493c-ad79-9ca6e191722a-workload-socket\") pod \"e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn\" (UID: \"3604271e-e5a0-493c-ad79-9ca6e191722a\") " pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" Oct 30 01:50:16.765782 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.761448 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3604271e-e5a0-493c-ad79-9ca6e191722a-istio-podinfo\") pod \"e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn\" (UID: \"3604271e-e5a0-493c-ad79-9ca6e191722a\") " pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" Oct 30 01:50:16.766140 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.766114 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3604271e-e5a0-493c-ad79-9ca6e191722a-istio-token\") pod \"e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn\" (UID: \"3604271e-e5a0-493c-ad79-9ca6e191722a\") " pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" Oct 30 01:50:16.766615 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.766565 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3604271e-e5a0-493c-ad79-9ca6e191722a-istio-envoy\") pod \"e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn\" (UID: \"3604271e-e5a0-493c-ad79-9ca6e191722a\") " pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" Oct 30 01:50:16.782770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.776239 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzkgc\" (UniqueName: \"kubernetes.io/projected/3604271e-e5a0-493c-ad79-9ca6e191722a-kube-api-access-xzkgc\") pod \"e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn\" (UID: \"3604271e-e5a0-493c-ad79-9ca6e191722a\") " pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" Oct 30 01:50:16.818370 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.818337 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" Oct 30 01:50:16.873997 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:16.873954 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:17.148832 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:50:17.148800 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3604271e_e5a0_493c_ad79_9ca6e191722a.slice/crio-76f8838b39d4a3346f719030e47f9ebb18fc5d136ed8fc89675ccc7b7d3620a9 WatchSource:0}: Error finding container 76f8838b39d4a3346f719030e47f9ebb18fc5d136ed8fc89675ccc7b7d3620a9: Status 404 returned error can't find the container with id 76f8838b39d4a3346f719030e47f9ebb18fc5d136ed8fc89675ccc7b7d3620a9 Oct 30 01:50:17.148992 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:17.148931 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn"] Oct 30 01:50:17.151722 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:17.151685 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:50:17.151954 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:17.151777 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:50:17.151954 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:17.151812 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:50:17.519018 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:17.511774 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:17.879435 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:17.879338 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" event={"ID":"3604271e-e5a0-493c-ad79-9ca6e191722a","Type":"ContainerStarted","Data":"71f157789a6d98f1ca0b584d0bedc85d2a8add0cf84e586424b013c6b5966a8c"} Oct 30 01:50:17.879435 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:17.879402 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" event={"ID":"3604271e-e5a0-493c-ad79-9ca6e191722a","Type":"ContainerStarted","Data":"76f8838b39d4a3346f719030e47f9ebb18fc5d136ed8fc89675ccc7b7d3620a9"} Oct 30 01:50:17.936408 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:17.936353 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" podStartSLOduration=1.936337932 podStartE2EDuration="1.936337932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:50:17.932062264 +0000 UTC m=+1943.508487666" watchObservedRunningTime="2026-10-30 01:50:17.936337932 +0000 UTC m=+1943.512763334" Oct 30 01:50:18.369853 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:18.369809 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:18.624347 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:18.624256 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:18.818995 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:18.818960 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" Oct 30 01:50:18.824673 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:18.824644 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" Oct 30 01:50:18.824909 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:18.824739 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:18.888892 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:18.888786 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" Oct 30 01:50:18.889841 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:18.889820 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" Oct 30 01:50:19.308262 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:19.308227 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:20.730010 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:20.729974 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:21.030853 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:21.030707 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-513ef70e-zwvrc"] Oct 30 01:50:21.041659 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:21.041625 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-513ef70e-zwvrc"] Oct 30 01:50:21.118142 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:21.118109 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:22.554888 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:22.554840 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0169ab93-f50e-4660-b20c-72ccde5254c9" path="/var/lib/kubelet/pods/0169ab93-f50e-4660-b20c-72ccde5254c9/volumes" Oct 30 01:50:22.813492 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:22.813396 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:24.027572 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:24.027534 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5559857-wv4vq"] Oct 30 01:50:24.038762 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:24.038723 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5559857-wv4vq"] Oct 30 01:50:24.414924 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:24.414833 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:24.545931 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:24.545894 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd21026f-4167-49df-ace8-dbc4d9060fc0" path="/var/lib/kubelet/pods/bd21026f-4167-49df-ace8-dbc4d9060fc0/volumes" Oct 30 01:50:25.028385 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:25.028346 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cb08419-d8vwm"] Oct 30 01:50:25.039967 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:25.039937 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cb08419-d8vwm"] Oct 30 01:50:26.186532 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:26.186484 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:26.543994 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:26.543892 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df" path="/var/lib/kubelet/pods/6d9184c1-673f-4ea3-baf0-d5dd0c5eb2df/volumes" Oct 30 01:50:27.904535 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:27.904500 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:29.603731 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:29.603682 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:31.194464 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:31.194430 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:33.063897 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:33.063851 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:34.772341 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:34.772306 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:36.471301 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:36.471223 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:38.132013 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:38.131978 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:39.027146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:39.027109 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebfaa245-25g2p"] Oct 30 01:50:39.039444 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:39.039401 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebfaa245-25g2p"] Oct 30 01:50:39.961701 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:39.961655 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:40.543869 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:40.543826 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f2871c4-f508-4186-8e8c-1f0cfaae81fb" path="/var/lib/kubelet/pods/8f2871c4-f508-4186-8e8c-1f0cfaae81fb/volumes" Oct 30 01:50:41.770454 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:41.770418 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:43.505139 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:43.505097 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:44.689421 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:50:44.689375 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:50:45.275163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:45.274581 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:45.694316 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:45.694279 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:45.762419 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:45.762383 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn"] Oct 30 01:50:45.762933 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:45.762902 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" podUID="3604271e-e5a0-493c-ad79-9ca6e191722a" containerName="istio-proxy" containerID="cri-o://71f157789a6d98f1ca0b584d0bedc85d2a8add0cf84e586424b013c6b5966a8c" gracePeriod=30 Oct 30 01:50:45.882995 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:45.882939 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:46.206410 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:46.206369 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:46.480717 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:46.480619 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:47.052429 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:47.052377 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:48.729914 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:48.729879 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:48.891203 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:48.891167 6572 patch_prober.go:29] interesting pod/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:50:48.891395 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:48.891227 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" podUID="3604271e-e5a0-493c-ad79-9ca6e191722a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:50:49.029468 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:49.029372 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7c26448-th5p8"] Oct 30 01:50:49.041324 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:49.041290 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7c26448-th5p8"] Oct 30 01:50:50.486989 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:50.486951 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:50.543471 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:50.543437 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8e92603-8f43-4815-bc95-1b88ab8dd06c" path="/var/lib/kubelet/pods/c8e92603-8f43-4815-bc95-1b88ab8dd06c/volumes" Oct 30 01:50:51.043794 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.043689 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" Oct 30 01:50:51.162503 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.162469 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3604271e-e5a0-493c-ad79-9ca6e191722a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3604271e-e5a0-493c-ad79-9ca6e191722a-credential-socket\") pod \"3604271e-e5a0-493c-ad79-9ca6e191722a\" (UID: \"3604271e-e5a0-493c-ad79-9ca6e191722a\") " Oct 30 01:50:51.162704 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.162531 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3604271e-e5a0-493c-ad79-9ca6e191722a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3604271e-e5a0-493c-ad79-9ca6e191722a-workload-socket\") pod \"3604271e-e5a0-493c-ad79-9ca6e191722a\" (UID: \"3604271e-e5a0-493c-ad79-9ca6e191722a\") " Oct 30 01:50:51.162704 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.162571 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3604271e-e5a0-493c-ad79-9ca6e191722a-kube-api-access-xzkgc\" (UniqueName: \"kubernetes.io/projected/3604271e-e5a0-493c-ad79-9ca6e191722a-kube-api-access-xzkgc\") pod \"3604271e-e5a0-493c-ad79-9ca6e191722a\" (UID: \"3604271e-e5a0-493c-ad79-9ca6e191722a\") " Oct 30 01:50:51.162704 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.162613 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3604271e-e5a0-493c-ad79-9ca6e191722a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3604271e-e5a0-493c-ad79-9ca6e191722a-istio-data\") pod \"3604271e-e5a0-493c-ad79-9ca6e191722a\" (UID: \"3604271e-e5a0-493c-ad79-9ca6e191722a\") " Oct 30 01:50:51.162704 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.162648 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3604271e-e5a0-493c-ad79-9ca6e191722a-istio-token\" (UniqueName: \"kubernetes.io/projected/3604271e-e5a0-493c-ad79-9ca6e191722a-istio-token\") pod \"3604271e-e5a0-493c-ad79-9ca6e191722a\" (UID: \"3604271e-e5a0-493c-ad79-9ca6e191722a\") " Oct 30 01:50:51.162998 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.162712 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3604271e-e5a0-493c-ad79-9ca6e191722a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3604271e-e5a0-493c-ad79-9ca6e191722a-istiod-ca-cert\") pod \"3604271e-e5a0-493c-ad79-9ca6e191722a\" (UID: \"3604271e-e5a0-493c-ad79-9ca6e191722a\") " Oct 30 01:50:51.162998 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.162793 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3604271e-e5a0-493c-ad79-9ca6e191722a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3604271e-e5a0-493c-ad79-9ca6e191722a-istio-envoy\") pod \"3604271e-e5a0-493c-ad79-9ca6e191722a\" (UID: \"3604271e-e5a0-493c-ad79-9ca6e191722a\") " Oct 30 01:50:51.162998 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.162818 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3604271e-e5a0-493c-ad79-9ca6e191722a-credential-socket" pod "3604271e-e5a0-493c-ad79-9ca6e191722a" (UID: "3604271e-e5a0-493c-ad79-9ca6e191722a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:50:51.162998 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.162828 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3604271e-e5a0-493c-ad79-9ca6e191722a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3604271e-e5a0-493c-ad79-9ca6e191722a-istio-podinfo\") pod \"3604271e-e5a0-493c-ad79-9ca6e191722a\" (UID: \"3604271e-e5a0-493c-ad79-9ca6e191722a\") " Oct 30 01:50:51.162998 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.162860 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3604271e-e5a0-493c-ad79-9ca6e191722a-workload-socket" pod "3604271e-e5a0-493c-ad79-9ca6e191722a" (UID: "3604271e-e5a0-493c-ad79-9ca6e191722a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:50:51.162998 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.162993 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3604271e-e5a0-493c-ad79-9ca6e191722a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3604271e-e5a0-493c-ad79-9ca6e191722a-workload-certs\") pod \"3604271e-e5a0-493c-ad79-9ca6e191722a\" (UID: \"3604271e-e5a0-493c-ad79-9ca6e191722a\") " Oct 30 01:50:51.163346 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.163251 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3604271e-e5a0-493c-ad79-9ca6e191722a-istio-data" pod "3604271e-e5a0-493c-ad79-9ca6e191722a" (UID: "3604271e-e5a0-493c-ad79-9ca6e191722a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:50:51.163346 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.163275 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3604271e-e5a0-493c-ad79-9ca6e191722a-istiod-ca-cert" pod "3604271e-e5a0-493c-ad79-9ca6e191722a" (UID: "3604271e-e5a0-493c-ad79-9ca6e191722a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:50:51.163471 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.163457 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3604271e-e5a0-493c-ad79-9ca6e191722a-workload-certs" pod "3604271e-e5a0-493c-ad79-9ca6e191722a" (UID: "3604271e-e5a0-493c-ad79-9ca6e191722a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:50:51.163846 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.163816 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3604271e-e5a0-493c-ad79-9ca6e191722a-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:50:51.163961 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.163848 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3604271e-e5a0-493c-ad79-9ca6e191722a-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:50:51.163961 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.163863 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3604271e-e5a0-493c-ad79-9ca6e191722a-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:50:51.163961 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.163877 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3604271e-e5a0-493c-ad79-9ca6e191722a-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:50:51.163961 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.163890 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3604271e-e5a0-493c-ad79-9ca6e191722a-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:50:51.165182 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.165150 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3604271e-e5a0-493c-ad79-9ca6e191722a-istio-token" pod "3604271e-e5a0-493c-ad79-9ca6e191722a" (UID: "3604271e-e5a0-493c-ad79-9ca6e191722a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:50:51.165280 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.165246 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3604271e-e5a0-493c-ad79-9ca6e191722a-istio-podinfo" pod "3604271e-e5a0-493c-ad79-9ca6e191722a" (UID: "3604271e-e5a0-493c-ad79-9ca6e191722a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:50:51.165473 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.165447 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3604271e-e5a0-493c-ad79-9ca6e191722a-istio-envoy" pod "3604271e-e5a0-493c-ad79-9ca6e191722a" (UID: "3604271e-e5a0-493c-ad79-9ca6e191722a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:50:51.165543 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.165455 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3604271e-e5a0-493c-ad79-9ca6e191722a-kube-api-access-xzkgc" pod "3604271e-e5a0-493c-ad79-9ca6e191722a" (UID: "3604271e-e5a0-493c-ad79-9ca6e191722a"). InnerVolumeSpecName "kube-api-access-xzkgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:50:51.254037 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.253995 6572 generic.go:356] "Generic (PLEG): container finished" podID="3604271e-e5a0-493c-ad79-9ca6e191722a" containerID="71f157789a6d98f1ca0b584d0bedc85d2a8add0cf84e586424b013c6b5966a8c" exitCode=0 Oct 30 01:50:51.254257 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.254046 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" Oct 30 01:50:51.254257 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.254077 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" event={"ID":"3604271e-e5a0-493c-ad79-9ca6e191722a","Type":"ContainerDied","Data":"71f157789a6d98f1ca0b584d0bedc85d2a8add0cf84e586424b013c6b5966a8c"} Oct 30 01:50:51.254257 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.254128 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn" event={"ID":"3604271e-e5a0-493c-ad79-9ca6e191722a","Type":"ContainerDied","Data":"76f8838b39d4a3346f719030e47f9ebb18fc5d136ed8fc89675ccc7b7d3620a9"} Oct 30 01:50:51.254257 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.254150 6572 scope.go:122] "RemoveContainer" containerID="71f157789a6d98f1ca0b584d0bedc85d2a8add0cf84e586424b013c6b5966a8c" Oct 30 01:50:51.264708 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.264683 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3604271e-e5a0-493c-ad79-9ca6e191722a-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:50:51.264708 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.264708 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3604271e-e5a0-493c-ad79-9ca6e191722a-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:50:51.264902 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.264718 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3604271e-e5a0-493c-ad79-9ca6e191722a-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:50:51.264902 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.264727 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzkgc\" (UniqueName: \"kubernetes.io/projected/3604271e-e5a0-493c-ad79-9ca6e191722a-kube-api-access-xzkgc\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:50:51.266915 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.266895 6572 scope.go:122] "RemoveContainer" containerID="71f157789a6d98f1ca0b584d0bedc85d2a8add0cf84e586424b013c6b5966a8c" Oct 30 01:50:51.267234 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:50:51.267216 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71f157789a6d98f1ca0b584d0bedc85d2a8add0cf84e586424b013c6b5966a8c\": container with ID starting with 71f157789a6d98f1ca0b584d0bedc85d2a8add0cf84e586424b013c6b5966a8c not found: ID does not exist" containerID="71f157789a6d98f1ca0b584d0bedc85d2a8add0cf84e586424b013c6b5966a8c" Oct 30 01:50:51.267311 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.267242 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71f157789a6d98f1ca0b584d0bedc85d2a8add0cf84e586424b013c6b5966a8c"} err="failed to get container status \"71f157789a6d98f1ca0b584d0bedc85d2a8add0cf84e586424b013c6b5966a8c\": rpc error: code = NotFound desc = could not find container \"71f157789a6d98f1ca0b584d0bedc85d2a8add0cf84e586424b013c6b5966a8c\": container with ID starting with 71f157789a6d98f1ca0b584d0bedc85d2a8add0cf84e586424b013c6b5966a8c not found: ID does not exist" Oct 30 01:50:51.307816 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.307685 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn"] Oct 30 01:50:51.314066 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.314034 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-36c949-openshift-default-5f5fbdf64f-dl8vn"] Oct 30 01:50:51.384807 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:51.384713 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:52.104873 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:52.104832 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:52.543858 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:52.543823 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3604271e-e5a0-493c-ad79-9ca6e191722a" path="/var/lib/kubelet/pods/3604271e-e5a0-493c-ad79-9ca6e191722a/volumes" Oct 30 01:50:53.028805 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:53.028771 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ca28f14-26xcn"] Oct 30 01:50:53.040199 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:53.040164 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ca28f14-26xcn"] Oct 30 01:50:53.796925 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:53.795895 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:54.544920 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:54.544882 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08ce0dba-99e6-475e-92e5-01bc7a9514c9" path="/var/lib/kubelet/pods/08ce0dba-99e6-475e-92e5-01bc7a9514c9/volumes" Oct 30 01:50:55.475550 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:55.475510 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:57.046794 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:57.046759 6572 scope.go:122] "RemoveContainer" containerID="221c8472b869a0eb6e9d39f2b10058bcd3af51fa14fd59a5a81540eacb8a155f" Oct 30 01:50:57.060542 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:57.060507 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:57.072717 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:57.072691 6572 scope.go:122] "RemoveContainer" containerID="85be00953de5b97ae327f898d5cb13716763ff1955ad5749d08ff7d20abef4c8" Oct 30 01:50:57.085618 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:57.085588 6572 scope.go:122] "RemoveContainer" containerID="af87b55d416f5f2d80311daac615ce2a4a1825340993bbc995fdfdcf76f9103b" Oct 30 01:50:57.100292 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:57.100268 6572 scope.go:122] "RemoveContainer" containerID="d11af34c1dc087ec9ffe66ef4ea007a7f6b06791e2806e54a6a55fcfc2155eeb" Oct 30 01:50:57.112451 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:57.112422 6572 scope.go:122] "RemoveContainer" containerID="c13db6414e2222b842933236b47de9a4cf6b2716d4caf57b7bfcd5c6c9c606bd" Oct 30 01:50:57.126095 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:57.126069 6572 scope.go:122] "RemoveContainer" containerID="ec732daa0b1aa2efa1b32ec1597c5839de3c67dc2955245967b828c062cac3f3" Oct 30 01:50:57.137404 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:57.137363 6572 scope.go:122] "RemoveContainer" containerID="5f4abeaf206b5f2738cbe38e41ef24f2dae0682a08e380ecc80a0f293ccc225e" Oct 30 01:50:57.148959 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:57.148934 6572 scope.go:122] "RemoveContainer" containerID="3d1ba0386b1328db6b9d3583deb6e3c70fe2b6ceee978f6a58e50ee0a54dbf20" Oct 30 01:50:57.160485 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:57.160457 6572 scope.go:122] "RemoveContainer" containerID="26f0ed90857f308c581a77c72be6a788fd8df45b92d96c9fa6935ecf51a207bd" Oct 30 01:50:58.647514 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:50:58.647478 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:00.028614 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:00.028578 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfcbcb00-5lwws"] Oct 30 01:51:00.040035 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:00.040006 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfcbcb00-5lwws"] Oct 30 01:51:00.310727 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:00.310293 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:00.544374 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:00.544333 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beb287c9-9698-483d-b3f3-b749388f0eb5" path="/var/lib/kubelet/pods/beb287c9-9698-483d-b3f3-b749388f0eb5/volumes" Oct 30 01:51:02.120240 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:02.120201 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:03.883327 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:03.883233 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:05.449055 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:05.449021 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:05.630662 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:05.630626 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:05.886768 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:05.886658 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:07.029254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:07.029216 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eced6ac6-xp8qt"] Oct 30 01:51:07.036315 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:07.036278 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eced6ac6-xp8qt"] Oct 30 01:51:07.173601 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:07.173553 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:08.544110 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:08.544067 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4525db7-9596-46b0-bfff-95c6933db9a9" path="/var/lib/kubelet/pods/f4525db7-9596-46b0-bfff-95c6933db9a9/volumes" Oct 30 01:51:08.761253 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:08.761212 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:10.504834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:10.504794 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:12.027786 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:12.027740 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33da9fb8-8sxpg"] Oct 30 01:51:12.039154 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:12.039109 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33da9fb8-8sxpg"] Oct 30 01:51:12.393934 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:12.393816 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:12.547296 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:12.546963 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c227494-ff4c-4c0a-8170-8bce3a7b0f1e" path="/var/lib/kubelet/pods/0c227494-ff4c-4c0a-8170-8bce3a7b0f1e/volumes" Oct 30 01:51:13.029784 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:13.029732 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9657f785-lw8pf"] Oct 30 01:51:13.041336 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:13.041299 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9657f785-lw8pf"] Oct 30 01:51:14.271358 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:14.271297 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:14.544698 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:14.544617 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b04e00c-7278-4024-9352-71bea03a9b7e" path="/var/lib/kubelet/pods/7b04e00c-7278-4024-9352-71bea03a9b7e/volumes" Oct 30 01:51:14.689982 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:51:14.689940 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:51:15.891539 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:15.891496 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:17.495467 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:17.495434 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:19.303111 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:19.303074 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:20.991627 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:20.991568 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:22.560860 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:22.560815 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5f5417b-d69pm"] Oct 30 01:51:22.561496 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:22.561479 6572 cpu_manager.go:414] "containerMap: removing container" podUID="3604271e-e5a0-493c-ad79-9ca6e191722a" containerName="istio-proxy" Oct 30 01:51:22.561545 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:22.561497 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3604271e-e5a0-493c-ad79-9ca6e191722a" containerName="istio-proxy" Oct 30 01:51:22.561650 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:22.561639 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="3604271e-e5a0-493c-ad79-9ca6e191722a" containerName="istio-proxy" Oct 30 01:51:22.565667 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:22.565641 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5f5417b-d69pm" Oct 30 01:51:22.590046 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:22.590004 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:51:22.598624 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:22.598586 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:51:22.598624 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:22.598596 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:51:22.599008 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:22.598625 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:51:22.603821 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:22.603787 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5f5417b-d69pm"] Oct 30 01:51:22.664364 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:22.664329 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e7f967-443d-44d1-8f82-3ccdd3aecff8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5f5417b-d69pm\" (UID: \"65e7f967-443d-44d1-8f82-3ccdd3aecff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5f5417b-d69pm" Oct 30 01:51:22.664607 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:22.664440 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e7f967-443d-44d1-8f82-3ccdd3aecff8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5f5417b-d69pm\" (UID: \"65e7f967-443d-44d1-8f82-3ccdd3aecff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5f5417b-d69pm" Oct 30 01:51:22.664607 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:22.664530 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7f5d\" (UniqueName: \"kubernetes.io/projected/65e7f967-443d-44d1-8f82-3ccdd3aecff8-kube-api-access-s7f5d\") pod \"maas-api-revoke-sub-d5f5417b-d69pm\" (UID: \"65e7f967-443d-44d1-8f82-3ccdd3aecff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5f5417b-d69pm" Oct 30 01:51:22.733577 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:22.733540 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:22.766620 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:22.766583 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e7f967-443d-44d1-8f82-3ccdd3aecff8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5f5417b-d69pm\" (UID: \"65e7f967-443d-44d1-8f82-3ccdd3aecff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5f5417b-d69pm" Oct 30 01:51:22.766847 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:22.766674 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e7f967-443d-44d1-8f82-3ccdd3aecff8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5f5417b-d69pm\" (UID: \"65e7f967-443d-44d1-8f82-3ccdd3aecff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5f5417b-d69pm" Oct 30 01:51:22.766847 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:22.766737 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7f5d\" (UniqueName: \"kubernetes.io/projected/65e7f967-443d-44d1-8f82-3ccdd3aecff8-kube-api-access-s7f5d\") pod \"maas-api-revoke-sub-d5f5417b-d69pm\" (UID: \"65e7f967-443d-44d1-8f82-3ccdd3aecff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5f5417b-d69pm" Oct 30 01:51:22.767347 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:22.767321 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e7f967-443d-44d1-8f82-3ccdd3aecff8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5f5417b-d69pm\" (UID: \"65e7f967-443d-44d1-8f82-3ccdd3aecff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5f5417b-d69pm" Oct 30 01:51:22.769346 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:22.769322 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e7f967-443d-44d1-8f82-3ccdd3aecff8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5f5417b-d69pm\" (UID: \"65e7f967-443d-44d1-8f82-3ccdd3aecff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5f5417b-d69pm" Oct 30 01:51:22.792612 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:22.792580 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7f5d\" (UniqueName: \"kubernetes.io/projected/65e7f967-443d-44d1-8f82-3ccdd3aecff8-kube-api-access-s7f5d\") pod \"maas-api-revoke-sub-d5f5417b-d69pm\" (UID: \"65e7f967-443d-44d1-8f82-3ccdd3aecff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5f5417b-d69pm" Oct 30 01:51:22.880498 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:22.880411 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5f5417b-d69pm" Oct 30 01:51:23.036123 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:23.036079 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68e4f632-pbv87"] Oct 30 01:51:23.040227 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:23.040196 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68e4f632-pbv87"] Oct 30 01:51:23.181786 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:23.181738 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5f5417b-d69pm"] Oct 30 01:51:23.182671 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:51:23.182646 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65e7f967_443d_44d1_8f82_3ccdd3aecff8.slice/crio-3b2034bddf73eb1fd638a14f74d770c28b44b25794cbb485c71c955e2fe68d74 WatchSource:0}: Error finding container 3b2034bddf73eb1fd638a14f74d770c28b44b25794cbb485c71c955e2fe68d74: Status 404 returned error can't find the container with id 3b2034bddf73eb1fd638a14f74d770c28b44b25794cbb485c71c955e2fe68d74 Oct 30 01:51:23.553848 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:23.553796 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5f5417b-d69pm" event={"ID":"65e7f967-443d-44d1-8f82-3ccdd3aecff8","Type":"ContainerStarted","Data":"c6574c63fe35ea428f30eb8b46912a7652f072b6fee155ca2f5f80e5850e9077"} Oct 30 01:51:23.554101 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:23.553869 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5f5417b-d69pm" event={"ID":"65e7f967-443d-44d1-8f82-3ccdd3aecff8","Type":"ContainerStarted","Data":"3b2034bddf73eb1fd638a14f74d770c28b44b25794cbb485c71c955e2fe68d74"} Oct 30 01:51:23.605241 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:23.605189 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5f5417b-d69pm" podStartSLOduration=1.605173759 podStartE2EDuration="1.605173759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:51:23.601905393 +0000 UTC m=+2009.178330807" watchObservedRunningTime="2026-10-30 01:51:23.605173759 +0000 UTC m=+2009.181599161" Oct 30 01:51:24.550356 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:24.546103 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e1d5968-a0a2-4e66-885d-da648cffb3f7" path="/var/lib/kubelet/pods/6e1d5968-a0a2-4e66-885d-da648cffb3f7/volumes" Oct 30 01:51:24.586699 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:24.586597 6572 generic.go:356] "Generic (PLEG): container finished" podID="65e7f967-443d-44d1-8f82-3ccdd3aecff8" containerID="c6574c63fe35ea428f30eb8b46912a7652f072b6fee155ca2f5f80e5850e9077" exitCode=0 Oct 30 01:51:24.586901 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:24.586689 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5f5417b-d69pm" event={"ID":"65e7f967-443d-44d1-8f82-3ccdd3aecff8","Type":"ContainerDied","Data":"c6574c63fe35ea428f30eb8b46912a7652f072b6fee155ca2f5f80e5850e9077"} Oct 30 01:51:24.590901 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:24.590867 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:25.766777 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:25.766734 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5f5417b-d69pm" Oct 30 01:51:25.812075 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:25.812037 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65e7f967-443d-44d1-8f82-3ccdd3aecff8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e7f967-443d-44d1-8f82-3ccdd3aecff8-maas-api-xks-ca\") pod \"65e7f967-443d-44d1-8f82-3ccdd3aecff8\" (UID: \"65e7f967-443d-44d1-8f82-3ccdd3aecff8\") " Oct 30 01:51:25.812264 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:25.812221 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65e7f967-443d-44d1-8f82-3ccdd3aecff8-kube-api-access-s7f5d\" (UniqueName: \"kubernetes.io/projected/65e7f967-443d-44d1-8f82-3ccdd3aecff8-kube-api-access-s7f5d\") pod \"65e7f967-443d-44d1-8f82-3ccdd3aecff8\" (UID: \"65e7f967-443d-44d1-8f82-3ccdd3aecff8\") " Oct 30 01:51:25.812419 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:25.812385 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65e7f967-443d-44d1-8f82-3ccdd3aecff8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e7f967-443d-44d1-8f82-3ccdd3aecff8-maas-api-openshift-ca\") pod \"65e7f967-443d-44d1-8f82-3ccdd3aecff8\" (UID: \"65e7f967-443d-44d1-8f82-3ccdd3aecff8\") " Oct 30 01:51:25.812935 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:25.812905 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65e7f967-443d-44d1-8f82-3ccdd3aecff8-maas-api-openshift-ca" pod "65e7f967-443d-44d1-8f82-3ccdd3aecff8" (UID: "65e7f967-443d-44d1-8f82-3ccdd3aecff8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:51:25.813091 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:25.813066 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e7f967-443d-44d1-8f82-3ccdd3aecff8-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:51:25.814489 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:25.814465 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65e7f967-443d-44d1-8f82-3ccdd3aecff8-kube-api-access-s7f5d" pod "65e7f967-443d-44d1-8f82-3ccdd3aecff8" (UID: "65e7f967-443d-44d1-8f82-3ccdd3aecff8"). InnerVolumeSpecName "kube-api-access-s7f5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:51:25.814612 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:25.814593 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65e7f967-443d-44d1-8f82-3ccdd3aecff8-maas-api-xks-ca" pod "65e7f967-443d-44d1-8f82-3ccdd3aecff8" (UID: "65e7f967-443d-44d1-8f82-3ccdd3aecff8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:51:25.914019 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:25.913983 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e7f967-443d-44d1-8f82-3ccdd3aecff8-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:51:25.914019 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:25.914016 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7f5d\" (UniqueName: \"kubernetes.io/projected/65e7f967-443d-44d1-8f82-3ccdd3aecff8-kube-api-access-s7f5d\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:51:26.030829 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:26.030788 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c8739dd-74v8c"] Oct 30 01:51:26.034566 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:26.034538 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c8739dd-74v8c"] Oct 30 01:51:26.156031 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:26.155935 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:26.544149 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:26.544109 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5cf289d-5797-422d-93a3-6904f21eb4c8" path="/var/lib/kubelet/pods/c5cf289d-5797-422d-93a3-6904f21eb4c8/volumes" Oct 30 01:51:26.606117 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:26.606088 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5f5417b-d69pm" Oct 30 01:51:26.606117 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:26.606104 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5f5417b-d69pm" event={"ID":"65e7f967-443d-44d1-8f82-3ccdd3aecff8","Type":"ContainerDied","Data":"3b2034bddf73eb1fd638a14f74d770c28b44b25794cbb485c71c955e2fe68d74"} Oct 30 01:51:26.606364 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:26.606145 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b2034bddf73eb1fd638a14f74d770c28b44b25794cbb485c71c955e2fe68d74" Oct 30 01:51:27.613332 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:27.613298 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:28.590332 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:28.590292 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:29.179665 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:29.179633 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:30.779539 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:30.779499 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:31.051680 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.051596 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:31.405628 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.405540 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv"] Oct 30 01:51:31.406634 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.406612 6572 cpu_manager.go:414] "containerMap: removing container" podUID="65e7f967-443d-44d1-8f82-3ccdd3aecff8" containerName="revoke-keys" Oct 30 01:51:31.406780 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.406637 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65e7f967-443d-44d1-8f82-3ccdd3aecff8" containerName="revoke-keys" Oct 30 01:51:31.406843 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.406815 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="65e7f967-443d-44d1-8f82-3ccdd3aecff8" containerName="revoke-keys" Oct 30 01:51:31.411107 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.411079 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 01:51:31.433071 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.433039 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv"] Oct 30 01:51:31.436212 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.436179 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-5fb950-openshift-default-dockercfg-x9lq7\"" Oct 30 01:51:31.505215 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.505176 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/223ead86-cf13-4c05-9655-893ffc0b9907-istio-data\") pod \"e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv\" (UID: \"223ead86-cf13-4c05-9655-893ffc0b9907\") " pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 01:51:31.505378 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.505257 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/223ead86-cf13-4c05-9655-893ffc0b9907-credential-socket\") pod \"e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv\" (UID: \"223ead86-cf13-4c05-9655-893ffc0b9907\") " pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 01:51:31.505378 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.505317 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/223ead86-cf13-4c05-9655-893ffc0b9907-workload-certs\") pod \"e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv\" (UID: \"223ead86-cf13-4c05-9655-893ffc0b9907\") " pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 01:51:31.505378 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.505351 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/223ead86-cf13-4c05-9655-893ffc0b9907-istio-token\") pod \"e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv\" (UID: \"223ead86-cf13-4c05-9655-893ffc0b9907\") " pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 01:51:31.505521 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.505410 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/223ead86-cf13-4c05-9655-893ffc0b9907-istiod-ca-cert\") pod \"e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv\" (UID: \"223ead86-cf13-4c05-9655-893ffc0b9907\") " pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 01:51:31.505521 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.505497 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/223ead86-cf13-4c05-9655-893ffc0b9907-istio-envoy\") pod \"e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv\" (UID: \"223ead86-cf13-4c05-9655-893ffc0b9907\") " pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 01:51:31.505592 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.505562 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/223ead86-cf13-4c05-9655-893ffc0b9907-workload-socket\") pod \"e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv\" (UID: \"223ead86-cf13-4c05-9655-893ffc0b9907\") " pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 01:51:31.505670 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.505646 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/223ead86-cf13-4c05-9655-893ffc0b9907-istio-podinfo\") pod \"e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv\" (UID: \"223ead86-cf13-4c05-9655-893ffc0b9907\") " pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 01:51:31.505725 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.505708 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvcw4\" (UniqueName: \"kubernetes.io/projected/223ead86-cf13-4c05-9655-893ffc0b9907-kube-api-access-qvcw4\") pod \"e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv\" (UID: \"223ead86-cf13-4c05-9655-893ffc0b9907\") " pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 01:51:31.538810 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.538742 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:31.608816 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.608667 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/223ead86-cf13-4c05-9655-893ffc0b9907-credential-socket\") pod \"e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv\" (UID: \"223ead86-cf13-4c05-9655-893ffc0b9907\") " pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 01:51:31.608816 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.608735 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/223ead86-cf13-4c05-9655-893ffc0b9907-workload-certs\") pod \"e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv\" (UID: \"223ead86-cf13-4c05-9655-893ffc0b9907\") " pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 01:51:31.608816 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.608792 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/223ead86-cf13-4c05-9655-893ffc0b9907-istio-token\") pod \"e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv\" (UID: \"223ead86-cf13-4c05-9655-893ffc0b9907\") " pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 01:51:31.609269 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.608870 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/223ead86-cf13-4c05-9655-893ffc0b9907-istiod-ca-cert\") pod \"e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv\" (UID: \"223ead86-cf13-4c05-9655-893ffc0b9907\") " pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 01:51:31.609269 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.608901 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/223ead86-cf13-4c05-9655-893ffc0b9907-istio-envoy\") pod \"e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv\" (UID: \"223ead86-cf13-4c05-9655-893ffc0b9907\") " pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 01:51:31.609269 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.609030 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/223ead86-cf13-4c05-9655-893ffc0b9907-workload-socket\") pod \"e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv\" (UID: \"223ead86-cf13-4c05-9655-893ffc0b9907\") " pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 01:51:31.609269 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.609091 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/223ead86-cf13-4c05-9655-893ffc0b9907-istio-podinfo\") pod \"e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv\" (UID: \"223ead86-cf13-4c05-9655-893ffc0b9907\") " pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 01:51:31.609269 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.609149 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvcw4\" (UniqueName: \"kubernetes.io/projected/223ead86-cf13-4c05-9655-893ffc0b9907-kube-api-access-qvcw4\") pod \"e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv\" (UID: \"223ead86-cf13-4c05-9655-893ffc0b9907\") " pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 01:51:31.609269 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.609246 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/223ead86-cf13-4c05-9655-893ffc0b9907-istio-data\") pod \"e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv\" (UID: \"223ead86-cf13-4c05-9655-893ffc0b9907\") " pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 01:51:31.629770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.609736 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/223ead86-cf13-4c05-9655-893ffc0b9907-istio-data\") pod \"e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv\" (UID: \"223ead86-cf13-4c05-9655-893ffc0b9907\") " pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 01:51:31.629770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.610067 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/223ead86-cf13-4c05-9655-893ffc0b9907-credential-socket\") pod \"e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv\" (UID: \"223ead86-cf13-4c05-9655-893ffc0b9907\") " pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 01:51:31.629770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.610297 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/223ead86-cf13-4c05-9655-893ffc0b9907-workload-certs\") pod \"e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv\" (UID: \"223ead86-cf13-4c05-9655-893ffc0b9907\") " pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 01:51:31.629770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.611280 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/223ead86-cf13-4c05-9655-893ffc0b9907-istiod-ca-cert\") pod \"e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv\" (UID: \"223ead86-cf13-4c05-9655-893ffc0b9907\") " pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 01:51:31.629770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.611885 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/223ead86-cf13-4c05-9655-893ffc0b9907-workload-socket\") pod \"e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv\" (UID: \"223ead86-cf13-4c05-9655-893ffc0b9907\") " pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 01:51:31.630463 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.630424 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/223ead86-cf13-4c05-9655-893ffc0b9907-istio-podinfo\") pod \"e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv\" (UID: \"223ead86-cf13-4c05-9655-893ffc0b9907\") " pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 01:51:31.633780 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.631021 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/223ead86-cf13-4c05-9655-893ffc0b9907-istio-envoy\") pod \"e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv\" (UID: \"223ead86-cf13-4c05-9655-893ffc0b9907\") " pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 01:51:31.644869 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.644837 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/223ead86-cf13-4c05-9655-893ffc0b9907-istio-token\") pod \"e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv\" (UID: \"223ead86-cf13-4c05-9655-893ffc0b9907\") " pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 01:51:31.665824 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.665791 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvcw4\" (UniqueName: \"kubernetes.io/projected/223ead86-cf13-4c05-9655-893ffc0b9907-kube-api-access-qvcw4\") pod \"e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv\" (UID: \"223ead86-cf13-4c05-9655-893ffc0b9907\") " pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 01:51:31.721760 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.721706 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:31.748434 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:31.748394 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 01:51:32.085923 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:32.085887 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv"] Oct 30 01:51:32.086555 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:51:32.086531 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod223ead86_cf13_4c05_9655_893ffc0b9907.slice/crio-2ceae912aa7728a7e0f8a4b8490f75f749fc939dd28a00d2bc537196e8f4ff8f WatchSource:0}: Error finding container 2ceae912aa7728a7e0f8a4b8490f75f749fc939dd28a00d2bc537196e8f4ff8f: Status 404 returned error can't find the container with id 2ceae912aa7728a7e0f8a4b8490f75f749fc939dd28a00d2bc537196e8f4ff8f Oct 30 01:51:32.089281 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:32.089243 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:51:32.089390 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:32.089312 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:51:32.089390 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:32.089343 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:51:32.326911 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:32.326657 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:32.739974 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:32.739934 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" event={"ID":"223ead86-cf13-4c05-9655-893ffc0b9907","Type":"ContainerStarted","Data":"03fce063a6260553f3b174cfb950a99452488a18c1f8906d3b4262e1c4de7878"} Oct 30 01:51:32.740189 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:32.739999 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" event={"ID":"223ead86-cf13-4c05-9655-893ffc0b9907","Type":"ContainerStarted","Data":"2ceae912aa7728a7e0f8a4b8490f75f749fc939dd28a00d2bc537196e8f4ff8f"} Oct 30 01:51:32.749415 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:32.749374 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 01:51:32.793985 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:32.793927 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" podStartSLOduration=1.793907838 podStartE2EDuration="1.793907838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:51:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:51:32.791346975 +0000 UTC m=+2018.367772376" watchObservedRunningTime="2026-10-30 01:51:32.793907838 +0000 UTC m=+2018.370333239" Oct 30 01:51:32.880402 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:32.880352 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ee021ae-l7zzf"] Oct 30 01:51:32.885482 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:32.885450 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee021ae-l7zzf" Oct 30 01:51:32.900247 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:32.900185 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:51:32.904209 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:32.901089 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:51:32.904209 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:32.901229 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:51:32.904209 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:32.901311 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:51:32.904601 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:32.904316 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ee021ae-l7zzf"] Oct 30 01:51:33.044942 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:33.044834 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68078396-b940-4d83-ac04-39f7ca2c535c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ee021ae-l7zzf\" (UID: \"68078396-b940-4d83-ac04-39f7ca2c535c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee021ae-l7zzf" Oct 30 01:51:33.044942 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:33.044913 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqrg4\" (UniqueName: \"kubernetes.io/projected/68078396-b940-4d83-ac04-39f7ca2c535c-kube-api-access-pqrg4\") pod \"maas-api-revoke-sub-3ee021ae-l7zzf\" (UID: \"68078396-b940-4d83-ac04-39f7ca2c535c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee021ae-l7zzf" Oct 30 01:51:33.045162 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:33.045016 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68078396-b940-4d83-ac04-39f7ca2c535c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ee021ae-l7zzf\" (UID: \"68078396-b940-4d83-ac04-39f7ca2c535c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee021ae-l7zzf" Oct 30 01:51:33.146168 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:33.146121 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68078396-b940-4d83-ac04-39f7ca2c535c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ee021ae-l7zzf\" (UID: \"68078396-b940-4d83-ac04-39f7ca2c535c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee021ae-l7zzf" Oct 30 01:51:33.146666 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:33.146368 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68078396-b940-4d83-ac04-39f7ca2c535c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ee021ae-l7zzf\" (UID: \"68078396-b940-4d83-ac04-39f7ca2c535c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee021ae-l7zzf" Oct 30 01:51:33.146666 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:33.146433 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqrg4\" (UniqueName: \"kubernetes.io/projected/68078396-b940-4d83-ac04-39f7ca2c535c-kube-api-access-pqrg4\") pod \"maas-api-revoke-sub-3ee021ae-l7zzf\" (UID: \"68078396-b940-4d83-ac04-39f7ca2c535c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee021ae-l7zzf" Oct 30 01:51:33.146977 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:33.146957 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68078396-b940-4d83-ac04-39f7ca2c535c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ee021ae-l7zzf\" (UID: \"68078396-b940-4d83-ac04-39f7ca2c535c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee021ae-l7zzf" Oct 30 01:51:33.149179 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:33.149152 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68078396-b940-4d83-ac04-39f7ca2c535c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ee021ae-l7zzf\" (UID: \"68078396-b940-4d83-ac04-39f7ca2c535c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee021ae-l7zzf" Oct 30 01:51:33.167471 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:33.167431 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqrg4\" (UniqueName: \"kubernetes.io/projected/68078396-b940-4d83-ac04-39f7ca2c535c-kube-api-access-pqrg4\") pod \"maas-api-revoke-sub-3ee021ae-l7zzf\" (UID: \"68078396-b940-4d83-ac04-39f7ca2c535c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee021ae-l7zzf" Oct 30 01:51:33.211524 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:33.211483 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee021ae-l7zzf" Oct 30 01:51:33.517272 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:33.517241 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ee021ae-l7zzf"] Oct 30 01:51:33.518857 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:51:33.518819 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68078396_b940_4d83_ac04_39f7ca2c535c.slice/crio-da670b628c4ccc87e952d4503d82cdf9fa178f2fa05d170eb5c67696b054e6c5 WatchSource:0}: Error finding container da670b628c4ccc87e952d4503d82cdf9fa178f2fa05d170eb5c67696b054e6c5: Status 404 returned error can't find the container with id da670b628c4ccc87e952d4503d82cdf9fa178f2fa05d170eb5c67696b054e6c5 Oct 30 01:51:33.750498 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:33.750463 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee021ae-l7zzf" event={"ID":"68078396-b940-4d83-ac04-39f7ca2c535c","Type":"ContainerStarted","Data":"da670b628c4ccc87e952d4503d82cdf9fa178f2fa05d170eb5c67696b054e6c5"} Oct 30 01:51:33.755766 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:33.755717 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 01:51:33.876517 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:33.875073 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:34.761373 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:34.761286 6572 generic.go:356] "Generic (PLEG): container finished" podID="68078396-b940-4d83-ac04-39f7ca2c535c" containerID="545269faf33fc7f6b600e8611fa80a2e7bd8882493c52418dda3537fa9381c10" exitCode=0 Oct 30 01:51:34.761373 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:34.761356 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee021ae-l7zzf" event={"ID":"68078396-b940-4d83-ac04-39f7ca2c535c","Type":"ContainerDied","Data":"545269faf33fc7f6b600e8611fa80a2e7bd8882493c52418dda3537fa9381c10"} Oct 30 01:51:34.761918 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:34.761664 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 01:51:34.762830 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:34.762814 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 01:51:35.197281 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:35.197247 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:35.462728 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:35.462639 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:35.696457 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:35.696418 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:35.934355 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:35.934324 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee021ae-l7zzf" Oct 30 01:51:35.993268 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:35.993238 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68078396-b940-4d83-ac04-39f7ca2c535c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68078396-b940-4d83-ac04-39f7ca2c535c-maas-api-openshift-ca\") pod \"68078396-b940-4d83-ac04-39f7ca2c535c\" (UID: \"68078396-b940-4d83-ac04-39f7ca2c535c\") " Oct 30 01:51:35.993449 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:35.993284 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68078396-b940-4d83-ac04-39f7ca2c535c-kube-api-access-pqrg4\" (UniqueName: \"kubernetes.io/projected/68078396-b940-4d83-ac04-39f7ca2c535c-kube-api-access-pqrg4\") pod \"68078396-b940-4d83-ac04-39f7ca2c535c\" (UID: \"68078396-b940-4d83-ac04-39f7ca2c535c\") " Oct 30 01:51:35.993674 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:35.993651 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68078396-b940-4d83-ac04-39f7ca2c535c-maas-api-openshift-ca" pod "68078396-b940-4d83-ac04-39f7ca2c535c" (UID: "68078396-b940-4d83-ac04-39f7ca2c535c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:51:35.993946 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:35.993925 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68078396-b940-4d83-ac04-39f7ca2c535c-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:51:35.995961 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:35.995935 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68078396-b940-4d83-ac04-39f7ca2c535c-kube-api-access-pqrg4" pod "68078396-b940-4d83-ac04-39f7ca2c535c" (UID: "68078396-b940-4d83-ac04-39f7ca2c535c"). InnerVolumeSpecName "kube-api-access-pqrg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:51:36.029056 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:36.028943 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd559747-z2t4k"] Oct 30 01:51:36.033517 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:36.033481 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd559747-z2t4k"] Oct 30 01:51:36.095020 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:36.094985 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68078396-b940-4d83-ac04-39f7ca2c535c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68078396-b940-4d83-ac04-39f7ca2c535c-maas-api-xks-ca\") pod \"68078396-b940-4d83-ac04-39f7ca2c535c\" (UID: \"68078396-b940-4d83-ac04-39f7ca2c535c\") " Oct 30 01:51:36.095482 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:36.095460 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqrg4\" (UniqueName: \"kubernetes.io/projected/68078396-b940-4d83-ac04-39f7ca2c535c-kube-api-access-pqrg4\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:51:36.097006 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:36.096984 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68078396-b940-4d83-ac04-39f7ca2c535c-maas-api-xks-ca" pod "68078396-b940-4d83-ac04-39f7ca2c535c" (UID: "68078396-b940-4d83-ac04-39f7ca2c535c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:51:36.196685 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:36.196645 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68078396-b940-4d83-ac04-39f7ca2c535c-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:51:36.543943 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:36.543911 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f64ef38-8649-4f43-a27e-2ad93b5fc9dd" path="/var/lib/kubelet/pods/0f64ef38-8649-4f43-a27e-2ad93b5fc9dd/volumes" Oct 30 01:51:36.781592 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:36.781565 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee021ae-l7zzf" Oct 30 01:51:36.781776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:36.781598 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee021ae-l7zzf" event={"ID":"68078396-b940-4d83-ac04-39f7ca2c535c","Type":"ContainerDied","Data":"da670b628c4ccc87e952d4503d82cdf9fa178f2fa05d170eb5c67696b054e6c5"} Oct 30 01:51:36.781776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:36.781653 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da670b628c4ccc87e952d4503d82cdf9fa178f2fa05d170eb5c67696b054e6c5" Oct 30 01:51:36.821590 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:36.821497 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:36.906572 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:36.906536 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5fb950-79c94597b7-jh8xt"] Oct 30 01:51:36.907250 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:36.907233 6572 cpu_manager.go:414] "containerMap: removing container" podUID="68078396-b940-4d83-ac04-39f7ca2c535c" containerName="revoke-keys" Oct 30 01:51:36.907311 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:36.907251 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68078396-b940-4d83-ac04-39f7ca2c535c" containerName="revoke-keys" Oct 30 01:51:36.907397 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:36.907387 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="68078396-b940-4d83-ac04-39f7ca2c535c" containerName="revoke-keys" Oct 30 01:51:36.911277 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:36.911256 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5fb950-79c94597b7-jh8xt" Oct 30 01:51:36.927493 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:36.927456 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-5fb950\"" Oct 30 01:51:36.932519 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:36.932483 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5fb950-79c94597b7-jh8xt"] Oct 30 01:51:37.021771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:37.007144 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-5fb950-dockercfg-7g5v6\"" Oct 30 01:51:37.021771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:37.012298 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/895641ba-c480-4aa8-81d6-a831b4fc93e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-5fb950-79c94597b7-jh8xt\" (UID: \"895641ba-c480-4aa8-81d6-a831b4fc93e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5fb950-79c94597b7-jh8xt" Oct 30 01:51:37.021771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:37.012422 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr5bt\" (UniqueName: \"kubernetes.io/projected/895641ba-c480-4aa8-81d6-a831b4fc93e9-kube-api-access-pr5bt\") pod \"payload-pre-processing-e2e-shared-a-w3-5fb950-79c94597b7-jh8xt\" (UID: \"895641ba-c480-4aa8-81d6-a831b4fc93e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5fb950-79c94597b7-jh8xt" Oct 30 01:51:37.042770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:37.035323 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:37.061804 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:37.056890 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5fb950-7479858b6d-sn7h4"] Oct 30 01:51:37.071887 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:37.071814 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5fb950-7479858b6d-sn7h4"] Oct 30 01:51:37.074003 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:37.073976 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5fb950-7479858b6d-sn7h4" Oct 30 01:51:37.153209 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:37.147846 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/895641ba-c480-4aa8-81d6-a831b4fc93e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-5fb950-79c94597b7-jh8xt\" (UID: \"895641ba-c480-4aa8-81d6-a831b4fc93e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5fb950-79c94597b7-jh8xt" Oct 30 01:51:37.153209 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:37.148047 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr5bt\" (UniqueName: \"kubernetes.io/projected/895641ba-c480-4aa8-81d6-a831b4fc93e9-kube-api-access-pr5bt\") pod \"payload-pre-processing-e2e-shared-a-w3-5fb950-79c94597b7-jh8xt\" (UID: \"895641ba-c480-4aa8-81d6-a831b4fc93e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5fb950-79c94597b7-jh8xt" Oct 30 01:51:37.153209 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:37.150353 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/895641ba-c480-4aa8-81d6-a831b4fc93e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-5fb950-79c94597b7-jh8xt\" (UID: \"895641ba-c480-4aa8-81d6-a831b4fc93e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5fb950-79c94597b7-jh8xt" Oct 30 01:51:37.201652 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:37.198013 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr5bt\" (UniqueName: \"kubernetes.io/projected/895641ba-c480-4aa8-81d6-a831b4fc93e9-kube-api-access-pr5bt\") pod \"payload-pre-processing-e2e-shared-a-w3-5fb950-79c94597b7-jh8xt\" (UID: \"895641ba-c480-4aa8-81d6-a831b4fc93e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5fb950-79c94597b7-jh8xt" Oct 30 01:51:37.226210 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:37.226175 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5fb950-79c94597b7-jh8xt" Oct 30 01:51:37.250301 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:37.250266 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj2w6\" (UniqueName: \"kubernetes.io/projected/6f409677-36e8-4781-8208-841f96df1888-kube-api-access-mj2w6\") pod \"payload-processing-e2e-shared-a-w3-5fb950-7479858b6d-sn7h4\" (UID: \"6f409677-36e8-4781-8208-841f96df1888\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5fb950-7479858b6d-sn7h4" Oct 30 01:51:37.250615 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:37.250592 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f409677-36e8-4781-8208-841f96df1888-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-5fb950-7479858b6d-sn7h4\" (UID: \"6f409677-36e8-4781-8208-841f96df1888\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5fb950-7479858b6d-sn7h4" Oct 30 01:51:37.352310 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:37.352215 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj2w6\" (UniqueName: \"kubernetes.io/projected/6f409677-36e8-4781-8208-841f96df1888-kube-api-access-mj2w6\") pod \"payload-processing-e2e-shared-a-w3-5fb950-7479858b6d-sn7h4\" (UID: \"6f409677-36e8-4781-8208-841f96df1888\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5fb950-7479858b6d-sn7h4" Oct 30 01:51:37.352474 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:37.352370 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f409677-36e8-4781-8208-841f96df1888-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-5fb950-7479858b6d-sn7h4\" (UID: \"6f409677-36e8-4781-8208-841f96df1888\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5fb950-7479858b6d-sn7h4" Oct 30 01:51:37.353177 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:37.353151 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f409677-36e8-4781-8208-841f96df1888-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-5fb950-7479858b6d-sn7h4\" (UID: \"6f409677-36e8-4781-8208-841f96df1888\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5fb950-7479858b6d-sn7h4" Oct 30 01:51:37.372005 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:37.371974 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj2w6\" (UniqueName: \"kubernetes.io/projected/6f409677-36e8-4781-8208-841f96df1888-kube-api-access-mj2w6\") pod \"payload-processing-e2e-shared-a-w3-5fb950-7479858b6d-sn7h4\" (UID: \"6f409677-36e8-4781-8208-841f96df1888\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5fb950-7479858b6d-sn7h4" Oct 30 01:51:37.453801 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:37.447539 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:37.484927 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:37.484895 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5fb950-7479858b6d-sn7h4" Oct 30 01:51:37.532055 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:37.531951 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5fb950-79c94597b7-jh8xt"] Oct 30 01:51:37.536207 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:51:37.536171 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod895641ba_c480_4aa8_81d6_a831b4fc93e9.slice/crio-2a9d1e177f24f2faccbc4aa96b88eac605baa146c4c225fb444d3dedca875a2d WatchSource:0}: Error finding container 2a9d1e177f24f2faccbc4aa96b88eac605baa146c4c225fb444d3dedca875a2d: Status 404 returned error can't find the container with id 2a9d1e177f24f2faccbc4aa96b88eac605baa146c4c225fb444d3dedca875a2d Oct 30 01:51:37.792520 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:37.792472 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5fb950-79c94597b7-jh8xt" event={"ID":"895641ba-c480-4aa8-81d6-a831b4fc93e9","Type":"ContainerStarted","Data":"d069e3ca5fce8e59a31ff3239ce06b54e75611f7f8f691a4ae86c897e20026ad"} Oct 30 01:51:37.792718 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:37.792548 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5fb950-79c94597b7-jh8xt" event={"ID":"895641ba-c480-4aa8-81d6-a831b4fc93e9","Type":"ContainerStarted","Data":"2a9d1e177f24f2faccbc4aa96b88eac605baa146c4c225fb444d3dedca875a2d"} Oct 30 01:51:37.792718 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:37.792576 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5fb950-79c94597b7-jh8xt" Oct 30 01:51:37.796107 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:37.796081 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5fb950-7479858b6d-sn7h4"] Oct 30 01:51:37.798330 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:51:37.798304 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f409677_36e8_4781_8208_841f96df1888.slice/crio-5c0607ebfe29c553d9cad664ed420fddea318cd1a08c897ecfadc9a8f4e109ce WatchSource:0}: Error finding container 5c0607ebfe29c553d9cad664ed420fddea318cd1a08c897ecfadc9a8f4e109ce: Status 404 returned error can't find the container with id 5c0607ebfe29c553d9cad664ed420fddea318cd1a08c897ecfadc9a8f4e109ce Oct 30 01:51:37.832136 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:37.832077 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5fb950-79c94597b7-jh8xt" podStartSLOduration=1.8320597570000001 podStartE2EDuration="1.832059757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:51:37.83130578 +0000 UTC m=+2023.407731181" watchObservedRunningTime="2026-10-30 01:51:37.832059757 +0000 UTC m=+2023.408485160" Oct 30 01:51:38.817704 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:38.817660 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5fb950-7479858b6d-sn7h4" event={"ID":"6f409677-36e8-4781-8208-841f96df1888","Type":"ContainerStarted","Data":"7e757d727cd6000faa0fe250f37455fb72dbf40d8e95e7b8e366fa561ea62bb2"} Oct 30 01:51:38.818328 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:38.817734 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5fb950-7479858b6d-sn7h4" event={"ID":"6f409677-36e8-4781-8208-841f96df1888","Type":"ContainerStarted","Data":"5c0607ebfe29c553d9cad664ed420fddea318cd1a08c897ecfadc9a8f4e109ce"} Oct 30 01:51:38.818328 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:38.817828 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5fb950-7479858b6d-sn7h4" Oct 30 01:51:38.859532 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:38.859470 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5fb950-7479858b6d-sn7h4" podStartSLOduration=2.859454841 podStartE2EDuration="2.859454841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:51:38.856636993 +0000 UTC m=+2024.433062395" watchObservedRunningTime="2026-10-30 01:51:38.859454841 +0000 UTC m=+2024.435880237" Oct 30 01:51:39.102301 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:39.102214 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:39.289443 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:39.289401 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:40.638395 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:40.638346 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:42.187160 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:42.187122 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:43.036598 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:43.036565 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07406684-6pkdb"] Oct 30 01:51:43.048255 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:43.048220 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07406684-6pkdb"] Oct 30 01:51:43.811128 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:43.811086 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:44.545968 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:44.545927 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93e155d3-0a3e-49bf-8951-193b216ec87c" path="/var/lib/kubelet/pods/93e155d3-0a3e-49bf-8951-193b216ec87c/volumes" Oct 30 01:51:44.689104 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:51:44.689063 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:51:45.136783 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:45.134337 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:45.504643 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:45.504606 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:47.139803 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:47.139766 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:48.702948 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:48.702904 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:48.821118 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:48.821089 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5fb950-79c94597b7-jh8xt" Oct 30 01:51:49.829413 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:49.829383 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5fb950-7479858b6d-sn7h4" Oct 30 01:51:50.267388 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:50.267351 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:51.769200 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:51.769164 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:53.029058 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:53.029022 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cc3986b-q5gw2"] Oct 30 01:51:53.040009 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:53.039981 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cc3986b-q5gw2"] Oct 30 01:51:53.251198 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:53.251163 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:54.547845 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:54.547809 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db6add3a-0d1b-462f-bee9-09d022cd41ab" path="/var/lib/kubelet/pods/db6add3a-0d1b-462f-bee9-09d022cd41ab/volumes" Oct 30 01:51:54.770006 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:54.769946 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:56.349800 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:56.349358 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:57.265621 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:57.265585 6572 scope.go:122] "RemoveContainer" containerID="2c9e598321b8a25383fd3353dc908dc21cdeede678c0588f0b7e9d053b2f73cf" Oct 30 01:51:57.277162 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:57.277137 6572 scope.go:122] "RemoveContainer" containerID="334781df94aba36cadabc85b3b842584da92a13cbb19d1c43552617a031d8b69" Oct 30 01:51:57.287941 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:57.287916 6572 scope.go:122] "RemoveContainer" containerID="f3d1435ebaada96efd09dfebabeefeb5d8b744f063cfa6198693124942c05fb9" Oct 30 01:51:57.304143 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:57.304075 6572 scope.go:122] "RemoveContainer" containerID="b84175e6d38b2c997cfb4448761b1ef455d3df28876af3d5d48d1908c6e075d8" Oct 30 01:51:57.316887 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:57.316820 6572 scope.go:122] "RemoveContainer" containerID="8bb2193200f14fddc22587bcc06c0c4a48ee78dd4ea90a1b682069c6700f4a68" Oct 30 01:51:57.327555 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:57.327531 6572 scope.go:122] "RemoveContainer" containerID="b910cad64710bdef0c68bacd8d4ed1ce4f5fe2e1d44639df0f9b2c5f4925b64a" Oct 30 01:51:57.338633 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:57.338606 6572 scope.go:122] "RemoveContainer" containerID="4bcf8f538c6bed420f0f63babb43032e36bb6dc2bc8dab980d8169dfe5ef7466" Oct 30 01:51:57.349204 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:57.349183 6572 scope.go:122] "RemoveContainer" containerID="84b96e42b94e8028f8ea8dec77368ba88ffb039b95f844aad402ad355905a608" Oct 30 01:51:57.359793 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:57.359768 6572 scope.go:122] "RemoveContainer" containerID="c0183ebdcb58f614dbc508fbd8e3f9a2f058f52f940564f18f6c021cd04fef4a" Oct 30 01:51:57.370185 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:57.370161 6572 scope.go:122] "RemoveContainer" containerID="616f1dc1812df540ed67d900985414e07756855907f12993b9433ba76f5f8ee9" Oct 30 01:51:57.381357 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:57.381334 6572 scope.go:122] "RemoveContainer" containerID="b438b6300ca0023bed380fc8bf30bc742a5d159fa1b9f7ec144aaea4943456f9" Oct 30 01:51:57.917263 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:57.917224 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:58.115036 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:58.114993 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:59.349781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:51:59.349260 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:00.874597 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:00.874562 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:01.459693 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:01.459660 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:03.078560 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:03.078519 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:04.590355 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:04.586974 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:06.017240 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:06.017202 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:06.219147 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:06.219102 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:07.991609 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:07.991573 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:09.026585 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:09.026548 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc614d4b-hcvf8"] Oct 30 01:52:09.038016 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:09.037984 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc614d4b-hcvf8"] Oct 30 01:52:09.609118 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:09.609083 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:10.205359 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:10.205324 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2561142-xl9mk"] Oct 30 01:52:10.209854 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:10.209829 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2561142-xl9mk" Oct 30 01:52:10.228700 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:10.228668 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2561142-xl9mk"] Oct 30 01:52:10.234815 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:10.234783 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:52:10.234815 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:10.234786 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:52:10.236331 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:10.236302 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:52:10.236331 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:10.236306 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:52:10.333919 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:10.333871 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2561142-xl9mk\" (UID: \"ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2561142-xl9mk" Oct 30 01:52:10.334125 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:10.333927 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btqxv\" (UniqueName: \"kubernetes.io/projected/ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11-kube-api-access-btqxv\") pod \"maas-api-revoke-sub-d2561142-xl9mk\" (UID: \"ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2561142-xl9mk" Oct 30 01:52:10.334125 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:10.334051 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2561142-xl9mk\" (UID: \"ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2561142-xl9mk" Oct 30 01:52:10.435201 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:10.435157 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2561142-xl9mk\" (UID: \"ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2561142-xl9mk" Oct 30 01:52:10.435201 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:10.435207 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btqxv\" (UniqueName: \"kubernetes.io/projected/ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11-kube-api-access-btqxv\") pod \"maas-api-revoke-sub-d2561142-xl9mk\" (UID: \"ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2561142-xl9mk" Oct 30 01:52:10.435476 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:10.435260 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2561142-xl9mk\" (UID: \"ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2561142-xl9mk" Oct 30 01:52:10.436049 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:10.436025 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2561142-xl9mk\" (UID: \"ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2561142-xl9mk" Oct 30 01:52:10.437781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:10.437730 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2561142-xl9mk\" (UID: \"ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2561142-xl9mk" Oct 30 01:52:10.454371 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:10.454343 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btqxv\" (UniqueName: \"kubernetes.io/projected/ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11-kube-api-access-btqxv\") pod \"maas-api-revoke-sub-d2561142-xl9mk\" (UID: \"ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2561142-xl9mk" Oct 30 01:52:10.525263 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:10.525175 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2561142-xl9mk" Oct 30 01:52:10.551992 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:10.548864 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10cff785-4b67-4c00-9ae3-135985d56904" path="/var/lib/kubelet/pods/10cff785-4b67-4c00-9ae3-135985d56904/volumes" Oct 30 01:52:10.830783 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:10.830729 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2561142-xl9mk"] Oct 30 01:52:10.832112 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:52:10.832068 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad76cbaf_e21f_4ab4_bbcf_c04996fa8a11.slice/crio-77bc8ef3ecd77436d5cdcd99a5bf0a150b6ee389840ca98a9233e0b93b7fb87c WatchSource:0}: Error finding container 77bc8ef3ecd77436d5cdcd99a5bf0a150b6ee389840ca98a9233e0b93b7fb87c: Status 404 returned error can't find the container with id 77bc8ef3ecd77436d5cdcd99a5bf0a150b6ee389840ca98a9233e0b93b7fb87c Oct 30 01:52:11.136431 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:11.136397 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2561142-xl9mk" event={"ID":"ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11","Type":"ContainerStarted","Data":"b09de6cb4d593027d42fdcc87a63ac592a74b3fbd52c30fd3f20ce49cbeaf2ed"} Oct 30 01:52:11.136592 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:11.136461 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2561142-xl9mk" event={"ID":"ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11","Type":"ContainerStarted","Data":"77bc8ef3ecd77436d5cdcd99a5bf0a150b6ee389840ca98a9233e0b93b7fb87c"} Oct 30 01:52:11.195915 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:11.195855 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2561142-xl9mk" podStartSLOduration=1.195839712 podStartE2EDuration="1.195839712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:11.195198459 +0000 UTC m=+2056.771623867" watchObservedRunningTime="2026-10-30 01:52:11.195839712 +0000 UTC m=+2056.772265146" Oct 30 01:52:11.209459 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:11.209423 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:12.148663 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:12.148566 6572 generic.go:356] "Generic (PLEG): container finished" podID="ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11" containerID="b09de6cb4d593027d42fdcc87a63ac592a74b3fbd52c30fd3f20ce49cbeaf2ed" exitCode=0 Oct 30 01:52:12.148899 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:12.148656 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2561142-xl9mk" event={"ID":"ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11","Type":"ContainerDied","Data":"b09de6cb4d593027d42fdcc87a63ac592a74b3fbd52c30fd3f20ce49cbeaf2ed"} Oct 30 01:52:12.330738 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:12.330693 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:12.804353 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:12.804307 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:12.986661 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:12.986627 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:13.227599 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:13.227564 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:13.349631 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:13.349602 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2561142-xl9mk" Oct 30 01:52:13.388301 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:13.387546 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11-kube-api-access-btqxv\" (UniqueName: \"kubernetes.io/projected/ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11-kube-api-access-btqxv\") pod \"ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11\" (UID: \"ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11\") " Oct 30 01:52:13.388301 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:13.387637 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11-maas-api-openshift-ca\") pod \"ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11\" (UID: \"ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11\") " Oct 30 01:52:13.388301 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:13.387882 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11-maas-api-xks-ca\") pod \"ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11\" (UID: \"ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11\") " Oct 30 01:52:13.388301 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:13.388136 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11-maas-api-openshift-ca" pod "ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11" (UID: "ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:52:13.388923 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:13.388483 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:52:13.391460 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:13.391413 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11-kube-api-access-btqxv" pod "ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11" (UID: "ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11"). InnerVolumeSpecName "kube-api-access-btqxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:13.391965 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:13.391941 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11-maas-api-xks-ca" pod "ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11" (UID: "ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:52:13.490261 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:13.490179 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:52:13.490261 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:13.490206 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btqxv\" (UniqueName: \"kubernetes.io/projected/ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11-kube-api-access-btqxv\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:52:14.183147 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:14.183116 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2561142-xl9mk" Oct 30 01:52:14.183147 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:14.183140 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2561142-xl9mk" event={"ID":"ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11","Type":"ContainerDied","Data":"77bc8ef3ecd77436d5cdcd99a5bf0a150b6ee389840ca98a9233e0b93b7fb87c"} Oct 30 01:52:14.183394 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:14.183187 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77bc8ef3ecd77436d5cdcd99a5bf0a150b6ee389840ca98a9233e0b93b7fb87c" Oct 30 01:52:14.572724 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:14.572680 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:14.693234 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:52:14.693196 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:52:16.206734 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:16.206679 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:16.408221 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:16.408178 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:17.874023 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:17.873981 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:18.151823 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:18.137163 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:19.521471 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:19.521427 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:20.534724 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:20.534687 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe25dd41-ftbx9"] Oct 30 01:52:20.535377 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:20.535362 6572 cpu_manager.go:414] "containerMap: removing container" podUID="ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11" containerName="revoke-keys" Oct 30 01:52:20.535424 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:20.535378 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11" containerName="revoke-keys" Oct 30 01:52:20.535524 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:20.535516 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11" containerName="revoke-keys" Oct 30 01:52:20.539430 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:20.539399 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe25dd41-ftbx9" Oct 30 01:52:20.557276 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:20.557247 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe25dd41-ftbx9"] Oct 30 01:52:20.564043 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:20.564004 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:52:20.564043 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:20.564018 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:52:20.564454 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:20.564428 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:52:20.565471 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:20.565451 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:52:20.581397 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:20.581357 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bb94b2d-4e93-47f8-a33c-f51f56b32222-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe25dd41-ftbx9\" (UID: \"9bb94b2d-4e93-47f8-a33c-f51f56b32222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe25dd41-ftbx9" Oct 30 01:52:20.581606 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:20.581574 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhhsm\" (UniqueName: \"kubernetes.io/projected/9bb94b2d-4e93-47f8-a33c-f51f56b32222-kube-api-access-rhhsm\") pod \"maas-api-revoke-sub-fe25dd41-ftbx9\" (UID: \"9bb94b2d-4e93-47f8-a33c-f51f56b32222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe25dd41-ftbx9" Oct 30 01:52:20.581705 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:20.581689 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bb94b2d-4e93-47f8-a33c-f51f56b32222-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe25dd41-ftbx9\" (UID: \"9bb94b2d-4e93-47f8-a33c-f51f56b32222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe25dd41-ftbx9" Oct 30 01:52:20.683123 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:20.683084 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhhsm\" (UniqueName: \"kubernetes.io/projected/9bb94b2d-4e93-47f8-a33c-f51f56b32222-kube-api-access-rhhsm\") pod \"maas-api-revoke-sub-fe25dd41-ftbx9\" (UID: \"9bb94b2d-4e93-47f8-a33c-f51f56b32222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe25dd41-ftbx9" Oct 30 01:52:20.683309 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:20.683170 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bb94b2d-4e93-47f8-a33c-f51f56b32222-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe25dd41-ftbx9\" (UID: \"9bb94b2d-4e93-47f8-a33c-f51f56b32222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe25dd41-ftbx9" Oct 30 01:52:20.683309 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:20.683294 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bb94b2d-4e93-47f8-a33c-f51f56b32222-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe25dd41-ftbx9\" (UID: \"9bb94b2d-4e93-47f8-a33c-f51f56b32222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe25dd41-ftbx9" Oct 30 01:52:20.684068 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:20.684046 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bb94b2d-4e93-47f8-a33c-f51f56b32222-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe25dd41-ftbx9\" (UID: \"9bb94b2d-4e93-47f8-a33c-f51f56b32222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe25dd41-ftbx9" Oct 30 01:52:20.685612 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:20.685587 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bb94b2d-4e93-47f8-a33c-f51f56b32222-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe25dd41-ftbx9\" (UID: \"9bb94b2d-4e93-47f8-a33c-f51f56b32222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe25dd41-ftbx9" Oct 30 01:52:20.702355 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:20.702317 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhhsm\" (UniqueName: \"kubernetes.io/projected/9bb94b2d-4e93-47f8-a33c-f51f56b32222-kube-api-access-rhhsm\") pod \"maas-api-revoke-sub-fe25dd41-ftbx9\" (UID: \"9bb94b2d-4e93-47f8-a33c-f51f56b32222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe25dd41-ftbx9" Oct 30 01:52:20.856659 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:20.856575 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe25dd41-ftbx9" Oct 30 01:52:21.166807 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:21.166384 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:21.205732 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:21.205705 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe25dd41-ftbx9"] Oct 30 01:52:21.208271 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:52:21.208239 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb94b2d_4e93_47f8_a33c_f51f56b32222.slice/crio-b89e9c2a2e8b1e8907f1c57e468351db3ce3da02baa3e45ab38001981ac6b96b WatchSource:0}: Error finding container b89e9c2a2e8b1e8907f1c57e468351db3ce3da02baa3e45ab38001981ac6b96b: Status 404 returned error can't find the container with id b89e9c2a2e8b1e8907f1c57e468351db3ce3da02baa3e45ab38001981ac6b96b Oct 30 01:52:21.298280 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:21.298233 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe25dd41-ftbx9" event={"ID":"9bb94b2d-4e93-47f8-a33c-f51f56b32222","Type":"ContainerStarted","Data":"b89e9c2a2e8b1e8907f1c57e468351db3ce3da02baa3e45ab38001981ac6b96b"} Oct 30 01:52:22.308025 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:22.307986 6572 generic.go:356] "Generic (PLEG): container finished" podID="9bb94b2d-4e93-47f8-a33c-f51f56b32222" containerID="9c69b978275ac4ffbe40cf489ca075e8855b01ef7417af6f68bb8ab4923dcf96" exitCode=0 Oct 30 01:52:22.308442 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:22.308061 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe25dd41-ftbx9" event={"ID":"9bb94b2d-4e93-47f8-a33c-f51f56b32222","Type":"ContainerDied","Data":"9c69b978275ac4ffbe40cf489ca075e8855b01ef7417af6f68bb8ab4923dcf96"} Oct 30 01:52:22.795812 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:22.795769 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:23.486691 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:23.486660 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe25dd41-ftbx9" Oct 30 01:52:23.527369 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:23.527332 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bb94b2d-4e93-47f8-a33c-f51f56b32222-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bb94b2d-4e93-47f8-a33c-f51f56b32222-maas-api-openshift-ca\") pod \"9bb94b2d-4e93-47f8-a33c-f51f56b32222\" (UID: \"9bb94b2d-4e93-47f8-a33c-f51f56b32222\") " Oct 30 01:52:23.527612 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:23.527592 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bb94b2d-4e93-47f8-a33c-f51f56b32222-kube-api-access-rhhsm\" (UniqueName: \"kubernetes.io/projected/9bb94b2d-4e93-47f8-a33c-f51f56b32222-kube-api-access-rhhsm\") pod \"9bb94b2d-4e93-47f8-a33c-f51f56b32222\" (UID: \"9bb94b2d-4e93-47f8-a33c-f51f56b32222\") " Oct 30 01:52:23.527829 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:23.527809 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bb94b2d-4e93-47f8-a33c-f51f56b32222-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bb94b2d-4e93-47f8-a33c-f51f56b32222-maas-api-xks-ca\") pod \"9bb94b2d-4e93-47f8-a33c-f51f56b32222\" (UID: \"9bb94b2d-4e93-47f8-a33c-f51f56b32222\") " Oct 30 01:52:23.527904 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:23.527838 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bb94b2d-4e93-47f8-a33c-f51f56b32222-maas-api-openshift-ca" pod "9bb94b2d-4e93-47f8-a33c-f51f56b32222" (UID: "9bb94b2d-4e93-47f8-a33c-f51f56b32222"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:52:23.528426 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:23.528404 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bb94b2d-4e93-47f8-a33c-f51f56b32222-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:52:23.530253 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:23.530226 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bb94b2d-4e93-47f8-a33c-f51f56b32222-maas-api-xks-ca" pod "9bb94b2d-4e93-47f8-a33c-f51f56b32222" (UID: "9bb94b2d-4e93-47f8-a33c-f51f56b32222"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:52:23.530471 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:23.530444 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb94b2d-4e93-47f8-a33c-f51f56b32222-kube-api-access-rhhsm" pod "9bb94b2d-4e93-47f8-a33c-f51f56b32222" (UID: "9bb94b2d-4e93-47f8-a33c-f51f56b32222"). InnerVolumeSpecName "kube-api-access-rhhsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:23.630584 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:23.630495 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhhsm\" (UniqueName: \"kubernetes.io/projected/9bb94b2d-4e93-47f8-a33c-f51f56b32222-kube-api-access-rhhsm\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:52:23.630584 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:23.630530 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bb94b2d-4e93-47f8-a33c-f51f56b32222-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:52:24.297925 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:24.297885 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:24.326394 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:24.326366 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe25dd41-ftbx9" Oct 30 01:52:24.326552 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:24.326393 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe25dd41-ftbx9" event={"ID":"9bb94b2d-4e93-47f8-a33c-f51f56b32222","Type":"ContainerDied","Data":"b89e9c2a2e8b1e8907f1c57e468351db3ce3da02baa3e45ab38001981ac6b96b"} Oct 30 01:52:24.326552 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:24.326436 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b89e9c2a2e8b1e8907f1c57e468351db3ce3da02baa3e45ab38001981ac6b96b" Oct 30 01:52:25.915715 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:25.915680 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:26.144167 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:26.144125 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:26.585206 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:26.585168 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:27.095807 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:27.095010 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:27.390471 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:27.390384 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:27.605218 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:27.605174 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:28.299370 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:28.299339 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl"] Oct 30 01:52:28.299837 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:28.299544 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" podUID="e7422bb8-dea0-4478-b1c8-05d1f1df1dcb" containerName="istio-proxy" containerID="cri-o://83597389fbfc1de2fabbee107ee6deda551155cde09f7936df199a20c6428634" gracePeriod=30 Oct 30 01:52:29.201779 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:29.195310 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:30.567339 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:30.567304 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:30.766064 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:30.766024 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:31.253127 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:31.253082 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:32.189284 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:32.189251 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:32.388172 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:32.388135 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:32.945235 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:32.945203 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:33.141740 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:33.141697 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:33.572925 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:33.572878 6572 generic.go:356] "Generic (PLEG): container finished" podID="e7422bb8-dea0-4478-b1c8-05d1f1df1dcb" containerID="83597389fbfc1de2fabbee107ee6deda551155cde09f7936df199a20c6428634" exitCode=0 Oct 30 01:52:33.573298 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:33.572947 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" event={"ID":"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb","Type":"ContainerDied","Data":"83597389fbfc1de2fabbee107ee6deda551155cde09f7936df199a20c6428634"} Oct 30 01:52:33.673196 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:33.673166 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" Oct 30 01:52:33.748918 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:33.748879 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-istio-data\") pod \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\" (UID: \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\") " Oct 30 01:52:33.749130 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:33.748987 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-kube-api-access-zdpw5\" (UniqueName: \"kubernetes.io/projected/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-kube-api-access-zdpw5\") pod \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\" (UID: \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\") " Oct 30 01:52:33.749130 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:33.749028 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-istio-token\" (UniqueName: \"kubernetes.io/projected/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-istio-token\") pod \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\" (UID: \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\") " Oct 30 01:52:33.749130 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:33.749051 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-istiod-ca-cert\") pod \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\" (UID: \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\") " Oct 30 01:52:33.749130 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:33.749074 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-istio-envoy\") pod \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\" (UID: \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\") " Oct 30 01:52:33.749130 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:33.749125 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-istio-podinfo\") pod \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\" (UID: \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\") " Oct 30 01:52:33.749360 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:33.749158 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-credential-socket\") pod \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\" (UID: \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\") " Oct 30 01:52:33.749360 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:33.749200 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-workload-certs\") pod \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\" (UID: \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\") " Oct 30 01:52:33.749360 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:33.749226 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-workload-socket\") pod \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\" (UID: \"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb\") " Oct 30 01:52:33.750065 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:33.750033 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-workload-socket" pod "e7422bb8-dea0-4478-b1c8-05d1f1df1dcb" (UID: "e7422bb8-dea0-4478-b1c8-05d1f1df1dcb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:33.750186 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:33.750155 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-istiod-ca-cert" pod "e7422bb8-dea0-4478-b1c8-05d1f1df1dcb" (UID: "e7422bb8-dea0-4478-b1c8-05d1f1df1dcb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:52:33.750441 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:33.750418 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-istio-data" pod "e7422bb8-dea0-4478-b1c8-05d1f1df1dcb" (UID: "e7422bb8-dea0-4478-b1c8-05d1f1df1dcb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:33.751278 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:33.751252 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-workload-certs" pod "e7422bb8-dea0-4478-b1c8-05d1f1df1dcb" (UID: "e7422bb8-dea0-4478-b1c8-05d1f1df1dcb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:33.751400 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:33.751337 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-credential-socket" pod "e7422bb8-dea0-4478-b1c8-05d1f1df1dcb" (UID: "e7422bb8-dea0-4478-b1c8-05d1f1df1dcb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:33.752770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:33.752733 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-istio-envoy" pod "e7422bb8-dea0-4478-b1c8-05d1f1df1dcb" (UID: "e7422bb8-dea0-4478-b1c8-05d1f1df1dcb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:33.753016 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:33.752988 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-kube-api-access-zdpw5" pod "e7422bb8-dea0-4478-b1c8-05d1f1df1dcb" (UID: "e7422bb8-dea0-4478-b1c8-05d1f1df1dcb"). InnerVolumeSpecName "kube-api-access-zdpw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:33.753866 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:33.753838 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-istio-podinfo" pod "e7422bb8-dea0-4478-b1c8-05d1f1df1dcb" (UID: "e7422bb8-dea0-4478-b1c8-05d1f1df1dcb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:52:33.754638 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:33.754609 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-istio-token" pod "e7422bb8-dea0-4478-b1c8-05d1f1df1dcb" (UID: "e7422bb8-dea0-4478-b1c8-05d1f1df1dcb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:33.851542 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:33.851427 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:52:33.851542 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:33.851455 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:52:33.851542 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:33.851465 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:52:33.851542 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:33.851475 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:52:33.851542 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:33.851484 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:52:33.851542 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:33.851494 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:52:33.851542 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:33.851504 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdpw5\" (UniqueName: \"kubernetes.io/projected/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-kube-api-access-zdpw5\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:52:33.851542 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:33.851513 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:52:33.851542 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:33.851521 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:52:34.573090 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:34.572654 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:34.584804 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:34.584773 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" Oct 30 01:52:34.584804 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:34.584786 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" event={"ID":"e7422bb8-dea0-4478-b1c8-05d1f1df1dcb","Type":"ContainerDied","Data":"6747cb5448e33cd4e279c88635198c2706803b76f9f78423b596916626a2b475"} Oct 30 01:52:34.584964 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:34.584871 6572 scope.go:122] "RemoveContainer" containerID="83597389fbfc1de2fabbee107ee6deda551155cde09f7936df199a20c6428634" Oct 30 01:52:34.655012 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:34.654974 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl"] Oct 30 01:52:34.658449 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:34.658423 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl"] Oct 30 01:52:34.667062 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:34.667033 6572 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:52:34.667205 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:34.667091 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-s92zl" podUID="e7422bb8-dea0-4478-b1c8-05d1f1df1dcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:52:36.111375 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:36.111341 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:36.543716 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:36.543681 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7422bb8-dea0-4478-b1c8-05d1f1df1dcb" path="/var/lib/kubelet/pods/e7422bb8-dea0-4478-b1c8-05d1f1df1dcb/volumes" Oct 30 01:52:37.597265 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:37.597206 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:39.092283 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:39.092238 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:40.540840 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:40.540796 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:40.746742 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:40.746704 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:41.659501 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:41.659469 6572 generic.go:356] "Generic (PLEG): container finished" podID="223ead86-cf13-4c05-9655-893ffc0b9907" containerID="03fce063a6260553f3b174cfb950a99452488a18c1f8906d3b4262e1c4de7878" exitCode=0 Oct 30 01:52:41.659985 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:41.659537 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" event={"ID":"223ead86-cf13-4c05-9655-893ffc0b9907","Type":"ContainerDied","Data":"03fce063a6260553f3b174cfb950a99452488a18c1f8906d3b4262e1c4de7878"} Oct 30 01:52:41.660043 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:41.659986 6572 scope.go:122] "RemoveContainer" containerID="03fce063a6260553f3b174cfb950a99452488a18c1f8906d3b4262e1c4de7878" Oct 30 01:52:41.661718 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:41.661684 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:52:41.661875 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:41.661792 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:52:41.661875 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:41.661845 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:52:42.118772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:42.117780 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:42.669439 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:42.669404 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" event={"ID":"223ead86-cf13-4c05-9655-893ffc0b9907","Type":"ContainerStarted","Data":"904eed12435e699e4bdab08f527f0c4a450cf2611c545b27b9ce2ebad112bb3e"} Oct 30 01:52:42.750024 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:42.749975 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 01:52:42.750237 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:42.750210 6572 patch_prober.go:29] interesting pod/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Oct 30 01:52:42.750322 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:42.750269 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" podUID="223ead86-cf13-4c05-9655-893ffc0b9907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Oct 30 01:52:42.884068 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:42.884032 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:43.645780 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:43.645724 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:43.750205 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:43.750169 6572 patch_prober.go:29] interesting pod/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Oct 30 01:52:43.750619 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:43.750218 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" podUID="223ead86-cf13-4c05-9655-893ffc0b9907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Oct 30 01:52:44.688456 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:52:44.688425 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:52:44.749861 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:44.749813 6572 patch_prober.go:29] interesting pod/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Oct 30 01:52:44.750073 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:44.749888 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" podUID="223ead86-cf13-4c05-9655-893ffc0b9907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Oct 30 01:52:45.198044 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:45.198008 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:45.749392 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:45.749357 6572 patch_prober.go:29] interesting pod/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Oct 30 01:52:45.749587 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:45.749407 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" podUID="223ead86-cf13-4c05-9655-893ffc0b9907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Oct 30 01:52:46.750425 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:46.750384 6572 patch_prober.go:29] interesting pod/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Oct 30 01:52:46.750936 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:46.750454 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" podUID="223ead86-cf13-4c05-9655-893ffc0b9907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Oct 30 01:52:46.826770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:46.807952 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:47.749437 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:47.749401 6572 patch_prober.go:29] interesting pod/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Oct 30 01:52:47.749642 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:47.749467 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" podUID="223ead86-cf13-4c05-9655-893ffc0b9907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Oct 30 01:52:48.312125 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:48.312088 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:48.635935 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:48.635845 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:48.755286 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:48.755245 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 01:52:48.755685 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:48.755662 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 01:52:48.756608 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:48.756588 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 01:52:49.875994 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:49.875955 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:50.237991 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:50.237955 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:50.785771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:50.785718 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:50.984303 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:50.984261 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:51.576729 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:51.576687 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:53.057403 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:53.057354 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:54.756624 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:54.756588 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:55.076574 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:55.076477 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:56.421121 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:56.421066 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:56.632557 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:56.632516 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:56.920062 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:56.920028 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:57.483308 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:57.483276 6572 scope.go:122] "RemoveContainer" containerID="17ae3616f188bae0e6a9937faa7e60eb266fc7c672c10f0fbf20be36e19f2b9e" Oct 30 01:52:57.576057 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:57.575157 6572 scope.go:122] "RemoveContainer" containerID="1a6268881b7083bbc83fe25724523d1ecf4e0a41598f4bb55c8671371361e859" Oct 30 01:52:57.594333 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:57.594308 6572 scope.go:122] "RemoveContainer" containerID="fe8ebf8f1297e302554d4a2f43292e70eda611fe93d77b2a9cc2927df9ce3d21" Oct 30 01:52:57.606066 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:57.606043 6572 scope.go:122] "RemoveContainer" containerID="e70f26ded9b2648143e8a4bd32980443d1322656c0e2ecf05cb6a336c318c207" Oct 30 01:52:57.617269 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:57.617243 6572 scope.go:122] "RemoveContainer" containerID="d6daf7d3fe4414366717d7023ab628b8112d32d19ba4bd5853cf3a7b438e9869" Oct 30 01:52:57.628642 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:57.628620 6572 scope.go:122] "RemoveContainer" containerID="833e496ef6a45e6b2d937b988270eda26c2ce9bf0719f206bfbbb88fcd9ffa9b" Oct 30 01:52:57.639106 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:57.639086 6572 scope.go:122] "RemoveContainer" containerID="0d73ab548463a4ef265192ef2957aa13eea38050a8e501154f1336c41fd4c0fc" Oct 30 01:52:57.649219 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:57.649199 6572 scope.go:122] "RemoveContainer" containerID="61cafa0ca98efd48ff866256aa1b3f80bdffa318324da945568c503a7092df4e" Oct 30 01:52:57.661630 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:57.661603 6572 scope.go:122] "RemoveContainer" containerID="c582eb740d2e16dde0a35ede24ae9408572442fc9bafb6ceeaa484bd54389c51" Oct 30 01:52:58.251771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:58.249178 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:59.913043 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:52:59.912992 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:01.534519 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:01.534480 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:02.837489 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:02.837445 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38e55eea-42q97"] Oct 30 01:53:02.838324 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:02.838300 6572 cpu_manager.go:414] "containerMap: removing container" podUID="9bb94b2d-4e93-47f8-a33c-f51f56b32222" containerName="revoke-keys" Oct 30 01:53:02.838443 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:02.838325 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bb94b2d-4e93-47f8-a33c-f51f56b32222" containerName="revoke-keys" Oct 30 01:53:02.838443 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:02.838399 6572 cpu_manager.go:414] "containerMap: removing container" podUID="e7422bb8-dea0-4478-b1c8-05d1f1df1dcb" containerName="istio-proxy" Oct 30 01:53:02.838443 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:02.838408 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7422bb8-dea0-4478-b1c8-05d1f1df1dcb" containerName="istio-proxy" Oct 30 01:53:02.838612 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:02.838592 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bb94b2d-4e93-47f8-a33c-f51f56b32222" containerName="revoke-keys" Oct 30 01:53:02.838663 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:02.838617 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7422bb8-dea0-4478-b1c8-05d1f1df1dcb" containerName="istio-proxy" Oct 30 01:53:02.842994 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:02.842971 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e55eea-42q97" Oct 30 01:53:02.859678 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:02.859651 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38e55eea-42q97"] Oct 30 01:53:02.873776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:02.865688 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:53:02.873776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:02.865713 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:53:02.873776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:02.865826 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:53:02.873776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:02.866504 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29n5v\" (UniqueName: \"kubernetes.io/projected/496115bd-ea68-46f9-811d-40204f4ee403-kube-api-access-29n5v\") pod \"maas-api-revoke-sub-38e55eea-42q97\" (UID: \"496115bd-ea68-46f9-811d-40204f4ee403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e55eea-42q97" Oct 30 01:53:02.873776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:02.866542 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/496115bd-ea68-46f9-811d-40204f4ee403-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38e55eea-42q97\" (UID: \"496115bd-ea68-46f9-811d-40204f4ee403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e55eea-42q97" Oct 30 01:53:02.873776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:02.866926 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/496115bd-ea68-46f9-811d-40204f4ee403-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38e55eea-42q97\" (UID: \"496115bd-ea68-46f9-811d-40204f4ee403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e55eea-42q97" Oct 30 01:53:02.873776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:02.868859 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:53:02.968503 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:02.968473 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/496115bd-ea68-46f9-811d-40204f4ee403-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38e55eea-42q97\" (UID: \"496115bd-ea68-46f9-811d-40204f4ee403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e55eea-42q97" Oct 30 01:53:02.968714 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:02.968586 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29n5v\" (UniqueName: \"kubernetes.io/projected/496115bd-ea68-46f9-811d-40204f4ee403-kube-api-access-29n5v\") pod \"maas-api-revoke-sub-38e55eea-42q97\" (UID: \"496115bd-ea68-46f9-811d-40204f4ee403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e55eea-42q97" Oct 30 01:53:02.968714 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:02.968618 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/496115bd-ea68-46f9-811d-40204f4ee403-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38e55eea-42q97\" (UID: \"496115bd-ea68-46f9-811d-40204f4ee403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e55eea-42q97" Oct 30 01:53:02.969336 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:02.969315 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/496115bd-ea68-46f9-811d-40204f4ee403-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38e55eea-42q97\" (UID: \"496115bd-ea68-46f9-811d-40204f4ee403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e55eea-42q97" Oct 30 01:53:02.970981 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:02.970962 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/496115bd-ea68-46f9-811d-40204f4ee403-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38e55eea-42q97\" (UID: \"496115bd-ea68-46f9-811d-40204f4ee403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e55eea-42q97" Oct 30 01:53:02.987378 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:02.987355 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29n5v\" (UniqueName: \"kubernetes.io/projected/496115bd-ea68-46f9-811d-40204f4ee403-kube-api-access-29n5v\") pod \"maas-api-revoke-sub-38e55eea-42q97\" (UID: \"496115bd-ea68-46f9-811d-40204f4ee403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e55eea-42q97" Oct 30 01:53:03.157325 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:03.157249 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e55eea-42q97" Oct 30 01:53:03.226816 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:03.226777 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:03.509035 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:03.509004 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38e55eea-42q97"] Oct 30 01:53:03.511143 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:53:03.511118 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod496115bd_ea68_46f9_811d_40204f4ee403.slice/crio-15abaa7ce2622080efed5ed8b8b555d1bc0ba2a276c172a9b26c073fd7f90cce WatchSource:0}: Error finding container 15abaa7ce2622080efed5ed8b8b555d1bc0ba2a276c172a9b26c073fd7f90cce: Status 404 returned error can't find the container with id 15abaa7ce2622080efed5ed8b8b555d1bc0ba2a276c172a9b26c073fd7f90cce Oct 30 01:53:04.031147 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:04.031103 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e55eea-42q97" event={"ID":"496115bd-ea68-46f9-811d-40204f4ee403","Type":"ContainerStarted","Data":"e3533a68fa2b05877137216ac0c005ca03fc36112581c1213c9633090a3f45d8"} Oct 30 01:53:04.031637 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:04.031170 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e55eea-42q97" event={"ID":"496115bd-ea68-46f9-811d-40204f4ee403","Type":"ContainerStarted","Data":"15abaa7ce2622080efed5ed8b8b555d1bc0ba2a276c172a9b26c073fd7f90cce"} Oct 30 01:53:04.080456 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:04.080354 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e55eea-42q97" podStartSLOduration=2.080335858 podStartE2EDuration="2.080335858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:04.067557226 +0000 UTC m=+2109.643982627" watchObservedRunningTime="2026-10-30 01:53:04.080335858 +0000 UTC m=+2109.656761260" Oct 30 01:53:04.925347 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:04.925299 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:05.039900 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:05.039863 6572 generic.go:356] "Generic (PLEG): container finished" podID="496115bd-ea68-46f9-811d-40204f4ee403" containerID="e3533a68fa2b05877137216ac0c005ca03fc36112581c1213c9633090a3f45d8" exitCode=0 Oct 30 01:53:05.040372 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:05.039947 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e55eea-42q97" event={"ID":"496115bd-ea68-46f9-811d-40204f4ee403","Type":"ContainerDied","Data":"e3533a68fa2b05877137216ac0c005ca03fc36112581c1213c9633090a3f45d8"} Oct 30 01:53:05.542818 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:05.542788 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-523e05e3-h5j7x"] Oct 30 01:53:05.546114 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:05.546096 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523e05e3-h5j7x" Oct 30 01:53:05.567590 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:05.567563 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-523e05e3-h5j7x"] Oct 30 01:53:05.623724 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:05.623689 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cba074e7-47f7-440b-bf10-db550a616970-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-523e05e3-h5j7x\" (UID: \"cba074e7-47f7-440b-bf10-db550a616970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523e05e3-h5j7x" Oct 30 01:53:05.623937 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:05.623914 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2vtt\" (UniqueName: \"kubernetes.io/projected/cba074e7-47f7-440b-bf10-db550a616970-kube-api-access-m2vtt\") pod \"maas-api-revoke-sub-523e05e3-h5j7x\" (UID: \"cba074e7-47f7-440b-bf10-db550a616970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523e05e3-h5j7x" Oct 30 01:53:05.624051 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:05.624031 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cba074e7-47f7-440b-bf10-db550a616970-maas-api-xks-ca\") pod \"maas-api-revoke-sub-523e05e3-h5j7x\" (UID: \"cba074e7-47f7-440b-bf10-db550a616970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523e05e3-h5j7x" Oct 30 01:53:05.725509 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:05.725466 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cba074e7-47f7-440b-bf10-db550a616970-maas-api-xks-ca\") pod \"maas-api-revoke-sub-523e05e3-h5j7x\" (UID: \"cba074e7-47f7-440b-bf10-db550a616970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523e05e3-h5j7x" Oct 30 01:53:05.725732 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:05.725556 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cba074e7-47f7-440b-bf10-db550a616970-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-523e05e3-h5j7x\" (UID: \"cba074e7-47f7-440b-bf10-db550a616970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523e05e3-h5j7x" Oct 30 01:53:05.725732 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:05.725684 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2vtt\" (UniqueName: \"kubernetes.io/projected/cba074e7-47f7-440b-bf10-db550a616970-kube-api-access-m2vtt\") pod \"maas-api-revoke-sub-523e05e3-h5j7x\" (UID: \"cba074e7-47f7-440b-bf10-db550a616970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523e05e3-h5j7x" Oct 30 01:53:05.726348 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:05.726326 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cba074e7-47f7-440b-bf10-db550a616970-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-523e05e3-h5j7x\" (UID: \"cba074e7-47f7-440b-bf10-db550a616970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523e05e3-h5j7x" Oct 30 01:53:05.728252 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:05.728218 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cba074e7-47f7-440b-bf10-db550a616970-maas-api-xks-ca\") pod \"maas-api-revoke-sub-523e05e3-h5j7x\" (UID: \"cba074e7-47f7-440b-bf10-db550a616970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523e05e3-h5j7x" Oct 30 01:53:05.743891 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:05.743866 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2vtt\" (UniqueName: \"kubernetes.io/projected/cba074e7-47f7-440b-bf10-db550a616970-kube-api-access-m2vtt\") pod \"maas-api-revoke-sub-523e05e3-h5j7x\" (UID: \"cba074e7-47f7-440b-bf10-db550a616970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523e05e3-h5j7x" Oct 30 01:53:05.859917 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:05.859829 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523e05e3-h5j7x" Oct 30 01:53:06.120940 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:06.120577 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:06.165741 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:06.165711 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-523e05e3-h5j7x"] Oct 30 01:53:06.169257 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:53:06.169227 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcba074e7_47f7_440b_bf10_db550a616970.slice/crio-45213a16b67c1f642566e1d67e1d8b7c4cc06d76b92678a584ff88b7c98453f4 WatchSource:0}: Error finding container 45213a16b67c1f642566e1d67e1d8b7c4cc06d76b92678a584ff88b7c98453f4: Status 404 returned error can't find the container with id 45213a16b67c1f642566e1d67e1d8b7c4cc06d76b92678a584ff88b7c98453f4 Oct 30 01:53:06.188214 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:06.188188 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e55eea-42q97" Oct 30 01:53:06.230552 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:06.230525 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/496115bd-ea68-46f9-811d-40204f4ee403-kube-api-access-29n5v\" (UniqueName: \"kubernetes.io/projected/496115bd-ea68-46f9-811d-40204f4ee403-kube-api-access-29n5v\") pod \"496115bd-ea68-46f9-811d-40204f4ee403\" (UID: \"496115bd-ea68-46f9-811d-40204f4ee403\") " Oct 30 01:53:06.230708 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:06.230626 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/496115bd-ea68-46f9-811d-40204f4ee403-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/496115bd-ea68-46f9-811d-40204f4ee403-maas-api-openshift-ca\") pod \"496115bd-ea68-46f9-811d-40204f4ee403\" (UID: \"496115bd-ea68-46f9-811d-40204f4ee403\") " Oct 30 01:53:06.230844 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:06.230721 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/496115bd-ea68-46f9-811d-40204f4ee403-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/496115bd-ea68-46f9-811d-40204f4ee403-maas-api-xks-ca\") pod \"496115bd-ea68-46f9-811d-40204f4ee403\" (UID: \"496115bd-ea68-46f9-811d-40204f4ee403\") " Oct 30 01:53:06.231345 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:06.231050 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/496115bd-ea68-46f9-811d-40204f4ee403-maas-api-openshift-ca" pod "496115bd-ea68-46f9-811d-40204f4ee403" (UID: "496115bd-ea68-46f9-811d-40204f4ee403"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:53:06.231464 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:06.231433 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/496115bd-ea68-46f9-811d-40204f4ee403-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:06.235239 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:06.233384 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/496115bd-ea68-46f9-811d-40204f4ee403-maas-api-xks-ca" pod "496115bd-ea68-46f9-811d-40204f4ee403" (UID: "496115bd-ea68-46f9-811d-40204f4ee403"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:53:06.235239 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:06.233694 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/496115bd-ea68-46f9-811d-40204f4ee403-kube-api-access-29n5v" pod "496115bd-ea68-46f9-811d-40204f4ee403" (UID: "496115bd-ea68-46f9-811d-40204f4ee403"). InnerVolumeSpecName "kube-api-access-29n5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:53:06.332390 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:06.332359 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29n5v\" (UniqueName: \"kubernetes.io/projected/496115bd-ea68-46f9-811d-40204f4ee403-kube-api-access-29n5v\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:06.332390 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:06.332389 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/496115bd-ea68-46f9-811d-40204f4ee403-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:06.577467 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:06.574457 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:07.055811 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:07.055777 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523e05e3-h5j7x" event={"ID":"cba074e7-47f7-440b-bf10-db550a616970","Type":"ContainerStarted","Data":"4ebf6f2755c92bf9bff4017d44017db4c4d0eff92259a295a3bc09e8ddc6c5db"} Oct 30 01:53:07.056011 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:07.055832 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523e05e3-h5j7x" event={"ID":"cba074e7-47f7-440b-bf10-db550a616970","Type":"ContainerStarted","Data":"45213a16b67c1f642566e1d67e1d8b7c4cc06d76b92678a584ff88b7c98453f4"} Oct 30 01:53:07.057412 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:07.057390 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e55eea-42q97" Oct 30 01:53:07.057525 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:07.057384 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38e55eea-42q97" event={"ID":"496115bd-ea68-46f9-811d-40204f4ee403","Type":"ContainerDied","Data":"15abaa7ce2622080efed5ed8b8b555d1bc0ba2a276c172a9b26c073fd7f90cce"} Oct 30 01:53:07.057525 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:07.057512 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15abaa7ce2622080efed5ed8b8b555d1bc0ba2a276c172a9b26c073fd7f90cce" Oct 30 01:53:07.096606 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:07.096561 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523e05e3-h5j7x" podStartSLOduration=2.096546962 podStartE2EDuration="2.096546962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:07.092322151 +0000 UTC m=+2112.668747553" watchObservedRunningTime="2026-10-30 01:53:07.096546962 +0000 UTC m=+2112.672972363" Oct 30 01:53:08.089900 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:08.089077 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:08.100261 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:08.100227 6572 generic.go:356] "Generic (PLEG): container finished" podID="cba074e7-47f7-440b-bf10-db550a616970" containerID="4ebf6f2755c92bf9bff4017d44017db4c4d0eff92259a295a3bc09e8ddc6c5db" exitCode=0 Oct 30 01:53:08.100450 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:08.100305 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523e05e3-h5j7x" event={"ID":"cba074e7-47f7-440b-bf10-db550a616970","Type":"ContainerDied","Data":"4ebf6f2755c92bf9bff4017d44017db4c4d0eff92259a295a3bc09e8ddc6c5db"} Oct 30 01:53:09.286290 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:09.286253 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523e05e3-h5j7x" Oct 30 01:53:09.381368 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:09.381265 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cba074e7-47f7-440b-bf10-db550a616970-kube-api-access-m2vtt\" (UniqueName: \"kubernetes.io/projected/cba074e7-47f7-440b-bf10-db550a616970-kube-api-access-m2vtt\") pod \"cba074e7-47f7-440b-bf10-db550a616970\" (UID: \"cba074e7-47f7-440b-bf10-db550a616970\") " Oct 30 01:53:09.381571 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:09.381422 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cba074e7-47f7-440b-bf10-db550a616970-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cba074e7-47f7-440b-bf10-db550a616970-maas-api-openshift-ca\") pod \"cba074e7-47f7-440b-bf10-db550a616970\" (UID: \"cba074e7-47f7-440b-bf10-db550a616970\") " Oct 30 01:53:09.381571 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:09.381456 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cba074e7-47f7-440b-bf10-db550a616970-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cba074e7-47f7-440b-bf10-db550a616970-maas-api-xks-ca\") pod \"cba074e7-47f7-440b-bf10-db550a616970\" (UID: \"cba074e7-47f7-440b-bf10-db550a616970\") " Oct 30 01:53:09.381927 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:09.381901 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cba074e7-47f7-440b-bf10-db550a616970-maas-api-openshift-ca" pod "cba074e7-47f7-440b-bf10-db550a616970" (UID: "cba074e7-47f7-440b-bf10-db550a616970"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:53:09.382273 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:09.382250 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cba074e7-47f7-440b-bf10-db550a616970-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:09.384256 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:09.384222 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cba074e7-47f7-440b-bf10-db550a616970-maas-api-xks-ca" pod "cba074e7-47f7-440b-bf10-db550a616970" (UID: "cba074e7-47f7-440b-bf10-db550a616970"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:53:09.384424 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:09.384401 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cba074e7-47f7-440b-bf10-db550a616970-kube-api-access-m2vtt" pod "cba074e7-47f7-440b-bf10-db550a616970" (UID: "cba074e7-47f7-440b-bf10-db550a616970"). InnerVolumeSpecName "kube-api-access-m2vtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:53:09.492769 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:09.490875 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2vtt\" (UniqueName: \"kubernetes.io/projected/cba074e7-47f7-440b-bf10-db550a616970-kube-api-access-m2vtt\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:09.492769 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:09.490922 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cba074e7-47f7-440b-bf10-db550a616970-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:09.751546 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:09.751503 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:10.117796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:10.117661 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523e05e3-h5j7x" event={"ID":"cba074e7-47f7-440b-bf10-db550a616970","Type":"ContainerDied","Data":"45213a16b67c1f642566e1d67e1d8b7c4cc06d76b92678a584ff88b7c98453f4"} Oct 30 01:53:10.117796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:10.117719 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45213a16b67c1f642566e1d67e1d8b7c4cc06d76b92678a584ff88b7c98453f4" Oct 30 01:53:10.117796 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:10.117670 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523e05e3-h5j7x" Oct 30 01:53:11.483091 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:11.483055 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:12.530099 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:12.530059 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:13.278607 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:13.278557 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb95e69a-rdmmv"] Oct 30 01:53:13.280061 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:13.280030 6572 cpu_manager.go:414] "containerMap: removing container" podUID="cba074e7-47f7-440b-bf10-db550a616970" containerName="revoke-keys" Oct 30 01:53:13.280248 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:13.280233 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cba074e7-47f7-440b-bf10-db550a616970" containerName="revoke-keys" Oct 30 01:53:13.280348 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:13.280337 6572 cpu_manager.go:414] "containerMap: removing container" podUID="496115bd-ea68-46f9-811d-40204f4ee403" containerName="revoke-keys" Oct 30 01:53:13.280432 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:13.280422 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="496115bd-ea68-46f9-811d-40204f4ee403" containerName="revoke-keys" Oct 30 01:53:13.280730 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:13.280703 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="cba074e7-47f7-440b-bf10-db550a616970" containerName="revoke-keys" Oct 30 01:53:13.280912 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:13.280735 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="496115bd-ea68-46f9-811d-40204f4ee403" containerName="revoke-keys" Oct 30 01:53:13.285256 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:13.285231 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95e69a-rdmmv" Oct 30 01:53:13.298207 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:13.293831 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb95e69a-rdmmv"] Oct 30 01:53:13.298633 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:13.298417 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:13.312456 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:13.312076 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:53:13.312456 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:13.312231 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:53:13.342814 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:13.342763 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:53:13.343125 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:13.342830 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:53:13.364286 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:13.364248 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cde2113-52f8-40d6-a1ee-fb94a2d497a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb95e69a-rdmmv\" (UID: \"3cde2113-52f8-40d6-a1ee-fb94a2d497a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95e69a-rdmmv" Oct 30 01:53:13.364462 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:13.364313 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cde2113-52f8-40d6-a1ee-fb94a2d497a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb95e69a-rdmmv\" (UID: \"3cde2113-52f8-40d6-a1ee-fb94a2d497a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95e69a-rdmmv" Oct 30 01:53:13.364462 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:13.364415 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms9kp\" (UniqueName: \"kubernetes.io/projected/3cde2113-52f8-40d6-a1ee-fb94a2d497a2-kube-api-access-ms9kp\") pod \"maas-api-revoke-sub-bb95e69a-rdmmv\" (UID: \"3cde2113-52f8-40d6-a1ee-fb94a2d497a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95e69a-rdmmv" Oct 30 01:53:13.466381 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:13.466343 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cde2113-52f8-40d6-a1ee-fb94a2d497a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb95e69a-rdmmv\" (UID: \"3cde2113-52f8-40d6-a1ee-fb94a2d497a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95e69a-rdmmv" Oct 30 01:53:13.466579 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:13.466397 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cde2113-52f8-40d6-a1ee-fb94a2d497a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb95e69a-rdmmv\" (UID: \"3cde2113-52f8-40d6-a1ee-fb94a2d497a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95e69a-rdmmv" Oct 30 01:53:13.466579 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:13.466437 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms9kp\" (UniqueName: \"kubernetes.io/projected/3cde2113-52f8-40d6-a1ee-fb94a2d497a2-kube-api-access-ms9kp\") pod \"maas-api-revoke-sub-bb95e69a-rdmmv\" (UID: \"3cde2113-52f8-40d6-a1ee-fb94a2d497a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95e69a-rdmmv" Oct 30 01:53:13.467143 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:13.467118 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cde2113-52f8-40d6-a1ee-fb94a2d497a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb95e69a-rdmmv\" (UID: \"3cde2113-52f8-40d6-a1ee-fb94a2d497a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95e69a-rdmmv" Oct 30 01:53:13.468848 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:13.468819 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cde2113-52f8-40d6-a1ee-fb94a2d497a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb95e69a-rdmmv\" (UID: \"3cde2113-52f8-40d6-a1ee-fb94a2d497a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95e69a-rdmmv" Oct 30 01:53:13.485659 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:13.485629 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms9kp\" (UniqueName: \"kubernetes.io/projected/3cde2113-52f8-40d6-a1ee-fb94a2d497a2-kube-api-access-ms9kp\") pod \"maas-api-revoke-sub-bb95e69a-rdmmv\" (UID: \"3cde2113-52f8-40d6-a1ee-fb94a2d497a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95e69a-rdmmv" Oct 30 01:53:13.617114 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:13.617017 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95e69a-rdmmv" Oct 30 01:53:13.889284 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:13.889208 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:13.921331 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:13.921292 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb95e69a-rdmmv"] Oct 30 01:53:13.922033 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:53:13.922002 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cde2113_52f8_40d6_a1ee_fb94a2d497a2.slice/crio-214111c56d9ac33707d9c71c4294e3872a6e490c9d785ecdd22342ef9beda59d WatchSource:0}: Error finding container 214111c56d9ac33707d9c71c4294e3872a6e490c9d785ecdd22342ef9beda59d: Status 404 returned error can't find the container with id 214111c56d9ac33707d9c71c4294e3872a6e490c9d785ecdd22342ef9beda59d Oct 30 01:53:14.151046 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:14.150939 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95e69a-rdmmv" event={"ID":"3cde2113-52f8-40d6-a1ee-fb94a2d497a2","Type":"ContainerStarted","Data":"214111c56d9ac33707d9c71c4294e3872a6e490c9d785ecdd22342ef9beda59d"} Oct 30 01:53:14.449812 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:14.441500 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:14.626567 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:14.626448 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:14.690813 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:53:14.690777 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:53:15.161500 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:15.161463 6572 generic.go:356] "Generic (PLEG): container finished" podID="3cde2113-52f8-40d6-a1ee-fb94a2d497a2" containerID="f1578462e840b980d2d3160338344aeab13c6381881ba13b47dfe1e6d8c3581b" exitCode=0 Oct 30 01:53:15.161686 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:15.161548 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95e69a-rdmmv" event={"ID":"3cde2113-52f8-40d6-a1ee-fb94a2d497a2","Type":"ContainerDied","Data":"f1578462e840b980d2d3160338344aeab13c6381881ba13b47dfe1e6d8c3581b"} Oct 30 01:53:15.234232 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:15.233485 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:15.968428 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:15.968386 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f579307-5p8ng"] Oct 30 01:53:15.972854 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:15.972830 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f579307-5p8ng" Oct 30 01:53:16.008292 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:16.008257 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f579307-5p8ng"] Oct 30 01:53:16.038163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:16.038132 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgwb8\" (UniqueName: \"kubernetes.io/projected/928c4da4-c1a1-4f68-9852-d1e80408a421-kube-api-access-rgwb8\") pod \"maas-api-revoke-sub-1f579307-5p8ng\" (UID: \"928c4da4-c1a1-4f68-9852-d1e80408a421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f579307-5p8ng" Oct 30 01:53:16.038447 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:16.038353 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/928c4da4-c1a1-4f68-9852-d1e80408a421-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f579307-5p8ng\" (UID: \"928c4da4-c1a1-4f68-9852-d1e80408a421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f579307-5p8ng" Oct 30 01:53:16.038447 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:16.038386 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/928c4da4-c1a1-4f68-9852-d1e80408a421-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f579307-5p8ng\" (UID: \"928c4da4-c1a1-4f68-9852-d1e80408a421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f579307-5p8ng" Oct 30 01:53:16.139634 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:16.139598 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/928c4da4-c1a1-4f68-9852-d1e80408a421-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f579307-5p8ng\" (UID: \"928c4da4-c1a1-4f68-9852-d1e80408a421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f579307-5p8ng" Oct 30 01:53:16.139634 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:16.139640 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/928c4da4-c1a1-4f68-9852-d1e80408a421-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f579307-5p8ng\" (UID: \"928c4da4-c1a1-4f68-9852-d1e80408a421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f579307-5p8ng" Oct 30 01:53:16.139941 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:16.139797 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgwb8\" (UniqueName: \"kubernetes.io/projected/928c4da4-c1a1-4f68-9852-d1e80408a421-kube-api-access-rgwb8\") pod \"maas-api-revoke-sub-1f579307-5p8ng\" (UID: \"928c4da4-c1a1-4f68-9852-d1e80408a421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f579307-5p8ng" Oct 30 01:53:16.140526 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:16.140503 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/928c4da4-c1a1-4f68-9852-d1e80408a421-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f579307-5p8ng\" (UID: \"928c4da4-c1a1-4f68-9852-d1e80408a421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f579307-5p8ng" Oct 30 01:53:16.142013 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:16.141989 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/928c4da4-c1a1-4f68-9852-d1e80408a421-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f579307-5p8ng\" (UID: \"928c4da4-c1a1-4f68-9852-d1e80408a421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f579307-5p8ng" Oct 30 01:53:16.158686 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:16.158662 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgwb8\" (UniqueName: \"kubernetes.io/projected/928c4da4-c1a1-4f68-9852-d1e80408a421-kube-api-access-rgwb8\") pod \"maas-api-revoke-sub-1f579307-5p8ng\" (UID: \"928c4da4-c1a1-4f68-9852-d1e80408a421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f579307-5p8ng" Oct 30 01:53:16.296229 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:16.296193 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f579307-5p8ng" Oct 30 01:53:16.345675 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:16.345644 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95e69a-rdmmv" Oct 30 01:53:16.458918 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:16.458885 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cde2113-52f8-40d6-a1ee-fb94a2d497a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cde2113-52f8-40d6-a1ee-fb94a2d497a2-maas-api-xks-ca\") pod \"3cde2113-52f8-40d6-a1ee-fb94a2d497a2\" (UID: \"3cde2113-52f8-40d6-a1ee-fb94a2d497a2\") " Oct 30 01:53:16.459101 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:16.458986 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cde2113-52f8-40d6-a1ee-fb94a2d497a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cde2113-52f8-40d6-a1ee-fb94a2d497a2-maas-api-openshift-ca\") pod \"3cde2113-52f8-40d6-a1ee-fb94a2d497a2\" (UID: \"3cde2113-52f8-40d6-a1ee-fb94a2d497a2\") " Oct 30 01:53:16.459150 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:16.459115 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cde2113-52f8-40d6-a1ee-fb94a2d497a2-kube-api-access-ms9kp\" (UniqueName: \"kubernetes.io/projected/3cde2113-52f8-40d6-a1ee-fb94a2d497a2-kube-api-access-ms9kp\") pod \"3cde2113-52f8-40d6-a1ee-fb94a2d497a2\" (UID: \"3cde2113-52f8-40d6-a1ee-fb94a2d497a2\") " Oct 30 01:53:16.459397 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:16.459370 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cde2113-52f8-40d6-a1ee-fb94a2d497a2-maas-api-openshift-ca" pod "3cde2113-52f8-40d6-a1ee-fb94a2d497a2" (UID: "3cde2113-52f8-40d6-a1ee-fb94a2d497a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:53:16.459542 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:16.459527 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cde2113-52f8-40d6-a1ee-fb94a2d497a2-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:16.461156 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:16.461132 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cde2113-52f8-40d6-a1ee-fb94a2d497a2-maas-api-xks-ca" pod "3cde2113-52f8-40d6-a1ee-fb94a2d497a2" (UID: "3cde2113-52f8-40d6-a1ee-fb94a2d497a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:53:16.461249 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:16.461157 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cde2113-52f8-40d6-a1ee-fb94a2d497a2-kube-api-access-ms9kp" pod "3cde2113-52f8-40d6-a1ee-fb94a2d497a2" (UID: "3cde2113-52f8-40d6-a1ee-fb94a2d497a2"). InnerVolumeSpecName "kube-api-access-ms9kp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:53:16.560881 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:16.560843 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms9kp\" (UniqueName: \"kubernetes.io/projected/3cde2113-52f8-40d6-a1ee-fb94a2d497a2-kube-api-access-ms9kp\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:16.560881 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:16.560883 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cde2113-52f8-40d6-a1ee-fb94a2d497a2-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:16.596347 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:16.596316 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f579307-5p8ng"] Oct 30 01:53:16.596486 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:53:16.596361 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod928c4da4_c1a1_4f68_9852_d1e80408a421.slice/crio-bd06750c86b371fd0544eca1f7f3d38595d8e910eb019097ad9150ef42dcab6f WatchSource:0}: Error finding container bd06750c86b371fd0544eca1f7f3d38595d8e910eb019097ad9150ef42dcab6f: Status 404 returned error can't find the container with id bd06750c86b371fd0544eca1f7f3d38595d8e910eb019097ad9150ef42dcab6f Oct 30 01:53:16.983767 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:16.977696 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:17.182686 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:17.182653 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95e69a-rdmmv" Oct 30 01:53:17.182905 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:17.182682 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb95e69a-rdmmv" event={"ID":"3cde2113-52f8-40d6-a1ee-fb94a2d497a2","Type":"ContainerDied","Data":"214111c56d9ac33707d9c71c4294e3872a6e490c9d785ecdd22342ef9beda59d"} Oct 30 01:53:17.182905 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:17.182737 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="214111c56d9ac33707d9c71c4294e3872a6e490c9d785ecdd22342ef9beda59d" Oct 30 01:53:17.184775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:17.184733 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f579307-5p8ng" event={"ID":"928c4da4-c1a1-4f68-9852-d1e80408a421","Type":"ContainerStarted","Data":"4d458581ea8ab8b57265dd4da2742c6b22e53ed55d18537df49cce0130c296d1"} Oct 30 01:53:17.184946 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:17.184797 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f579307-5p8ng" event={"ID":"928c4da4-c1a1-4f68-9852-d1e80408a421","Type":"ContainerStarted","Data":"bd06750c86b371fd0544eca1f7f3d38595d8e910eb019097ad9150ef42dcab6f"} Oct 30 01:53:17.225370 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:17.225306 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f579307-5p8ng" podStartSLOduration=2.225286152 podStartE2EDuration="2.225286152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:17.223447448 +0000 UTC m=+2122.799872851" watchObservedRunningTime="2026-10-30 01:53:17.225286152 +0000 UTC m=+2122.801711555" Oct 30 01:53:18.043309 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:18.043274 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:18.193778 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:18.193726 6572 generic.go:356] "Generic (PLEG): container finished" podID="928c4da4-c1a1-4f68-9852-d1e80408a421" containerID="4d458581ea8ab8b57265dd4da2742c6b22e53ed55d18537df49cce0130c296d1" exitCode=0 Oct 30 01:53:18.193963 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:18.193818 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f579307-5p8ng" event={"ID":"928c4da4-c1a1-4f68-9852-d1e80408a421","Type":"ContainerDied","Data":"4d458581ea8ab8b57265dd4da2742c6b22e53ed55d18537df49cce0130c296d1"} Oct 30 01:53:18.247737 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:18.247706 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:18.681565 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:18.681513 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:19.384807 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:19.384778 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f579307-5p8ng" Oct 30 01:53:19.405844 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:19.405809 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/928c4da4-c1a1-4f68-9852-d1e80408a421-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/928c4da4-c1a1-4f68-9852-d1e80408a421-maas-api-xks-ca\") pod \"928c4da4-c1a1-4f68-9852-d1e80408a421\" (UID: \"928c4da4-c1a1-4f68-9852-d1e80408a421\") " Oct 30 01:53:19.406042 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:19.405954 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/928c4da4-c1a1-4f68-9852-d1e80408a421-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/928c4da4-c1a1-4f68-9852-d1e80408a421-maas-api-openshift-ca\") pod \"928c4da4-c1a1-4f68-9852-d1e80408a421\" (UID: \"928c4da4-c1a1-4f68-9852-d1e80408a421\") " Oct 30 01:53:19.406162 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:19.406143 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/928c4da4-c1a1-4f68-9852-d1e80408a421-kube-api-access-rgwb8\" (UniqueName: \"kubernetes.io/projected/928c4da4-c1a1-4f68-9852-d1e80408a421-kube-api-access-rgwb8\") pod \"928c4da4-c1a1-4f68-9852-d1e80408a421\" (UID: \"928c4da4-c1a1-4f68-9852-d1e80408a421\") " Oct 30 01:53:19.406400 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:19.406370 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/928c4da4-c1a1-4f68-9852-d1e80408a421-maas-api-openshift-ca" pod "928c4da4-c1a1-4f68-9852-d1e80408a421" (UID: "928c4da4-c1a1-4f68-9852-d1e80408a421"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:53:19.406888 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:19.406867 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/928c4da4-c1a1-4f68-9852-d1e80408a421-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:19.408188 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:19.408161 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/928c4da4-c1a1-4f68-9852-d1e80408a421-maas-api-xks-ca" pod "928c4da4-c1a1-4f68-9852-d1e80408a421" (UID: "928c4da4-c1a1-4f68-9852-d1e80408a421"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:53:19.408188 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:19.408173 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/928c4da4-c1a1-4f68-9852-d1e80408a421-kube-api-access-rgwb8" pod "928c4da4-c1a1-4f68-9852-d1e80408a421" (UID: "928c4da4-c1a1-4f68-9852-d1e80408a421"). InnerVolumeSpecName "kube-api-access-rgwb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:53:19.508561 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:19.508463 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/928c4da4-c1a1-4f68-9852-d1e80408a421-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:19.508561 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:19.508492 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgwb8\" (UniqueName: \"kubernetes.io/projected/928c4da4-c1a1-4f68-9852-d1e80408a421-kube-api-access-rgwb8\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:19.686513 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:19.686477 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:19.906882 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:19.906782 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:20.217691 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:20.217656 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f579307-5p8ng" Oct 30 01:53:20.217920 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:20.217663 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f579307-5p8ng" event={"ID":"928c4da4-c1a1-4f68-9852-d1e80408a421","Type":"ContainerDied","Data":"bd06750c86b371fd0544eca1f7f3d38595d8e910eb019097ad9150ef42dcab6f"} Oct 30 01:53:20.217920 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:20.217850 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd06750c86b371fd0544eca1f7f3d38595d8e910eb019097ad9150ef42dcab6f" Oct 30 01:53:20.497834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:20.497399 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:21.145974 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:21.145936 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:21.342532 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:21.342493 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:21.719417 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:21.719372 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79"] Oct 30 01:53:21.719995 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:21.719609 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" podUID="8b24c1c0-b61b-43fc-84b0-994045057bc6" containerName="istio-proxy" containerID="cri-o://68d9b90790271e8bf890d5d8e4ceffe0bf9459fd275aeacd20de216ac22d9920" gracePeriod=30 Oct 30 01:53:22.211736 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:22.211697 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:23.844185 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:23.844147 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:24.031812 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:24.031742 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:24.236493 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:24.236461 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:25.681279 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:25.681203 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:25.874767 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:25.874709 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:26.990637 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:26.990575 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" Oct 30 01:53:27.121311 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:27.121272 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b24c1c0-b61b-43fc-84b0-994045057bc6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b24c1c0-b61b-43fc-84b0-994045057bc6-istio-data\") pod \"8b24c1c0-b61b-43fc-84b0-994045057bc6\" (UID: \"8b24c1c0-b61b-43fc-84b0-994045057bc6\") " Oct 30 01:53:27.121484 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:27.121328 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b24c1c0-b61b-43fc-84b0-994045057bc6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b24c1c0-b61b-43fc-84b0-994045057bc6-istio-podinfo\") pod \"8b24c1c0-b61b-43fc-84b0-994045057bc6\" (UID: \"8b24c1c0-b61b-43fc-84b0-994045057bc6\") " Oct 30 01:53:27.121484 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:27.121370 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b24c1c0-b61b-43fc-84b0-994045057bc6-istio-token\" (UniqueName: \"kubernetes.io/projected/8b24c1c0-b61b-43fc-84b0-994045057bc6-istio-token\") pod \"8b24c1c0-b61b-43fc-84b0-994045057bc6\" (UID: \"8b24c1c0-b61b-43fc-84b0-994045057bc6\") " Oct 30 01:53:27.121484 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:27.121417 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b24c1c0-b61b-43fc-84b0-994045057bc6-kube-api-access-4q9l9\" (UniqueName: \"kubernetes.io/projected/8b24c1c0-b61b-43fc-84b0-994045057bc6-kube-api-access-4q9l9\") pod \"8b24c1c0-b61b-43fc-84b0-994045057bc6\" (UID: \"8b24c1c0-b61b-43fc-84b0-994045057bc6\") " Oct 30 01:53:27.121484 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:27.121457 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b24c1c0-b61b-43fc-84b0-994045057bc6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b24c1c0-b61b-43fc-84b0-994045057bc6-credential-socket\") pod \"8b24c1c0-b61b-43fc-84b0-994045057bc6\" (UID: \"8b24c1c0-b61b-43fc-84b0-994045057bc6\") " Oct 30 01:53:27.121698 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:27.121503 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b24c1c0-b61b-43fc-84b0-994045057bc6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b24c1c0-b61b-43fc-84b0-994045057bc6-workload-certs\") pod \"8b24c1c0-b61b-43fc-84b0-994045057bc6\" (UID: \"8b24c1c0-b61b-43fc-84b0-994045057bc6\") " Oct 30 01:53:27.121698 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:27.121543 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b24c1c0-b61b-43fc-84b0-994045057bc6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b24c1c0-b61b-43fc-84b0-994045057bc6-istio-envoy\") pod \"8b24c1c0-b61b-43fc-84b0-994045057bc6\" (UID: \"8b24c1c0-b61b-43fc-84b0-994045057bc6\") " Oct 30 01:53:27.121698 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:27.121591 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b24c1c0-b61b-43fc-84b0-994045057bc6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b24c1c0-b61b-43fc-84b0-994045057bc6-workload-socket\") pod \"8b24c1c0-b61b-43fc-84b0-994045057bc6\" (UID: \"8b24c1c0-b61b-43fc-84b0-994045057bc6\") " Oct 30 01:53:27.121698 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:27.121630 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b24c1c0-b61b-43fc-84b0-994045057bc6-istio-data" pod "8b24c1c0-b61b-43fc-84b0-994045057bc6" (UID: "8b24c1c0-b61b-43fc-84b0-994045057bc6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:27.121959 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:27.121782 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b24c1c0-b61b-43fc-84b0-994045057bc6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b24c1c0-b61b-43fc-84b0-994045057bc6-istiod-ca-cert\") pod \"8b24c1c0-b61b-43fc-84b0-994045057bc6\" (UID: \"8b24c1c0-b61b-43fc-84b0-994045057bc6\") " Oct 30 01:53:27.121959 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:27.121881 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b24c1c0-b61b-43fc-84b0-994045057bc6-credential-socket" pod "8b24c1c0-b61b-43fc-84b0-994045057bc6" (UID: "8b24c1c0-b61b-43fc-84b0-994045057bc6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:27.121959 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:27.121904 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b24c1c0-b61b-43fc-84b0-994045057bc6-workload-certs" pod "8b24c1c0-b61b-43fc-84b0-994045057bc6" (UID: "8b24c1c0-b61b-43fc-84b0-994045057bc6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:27.122149 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:27.122104 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b24c1c0-b61b-43fc-84b0-994045057bc6-workload-socket" pod "8b24c1c0-b61b-43fc-84b0-994045057bc6" (UID: "8b24c1c0-b61b-43fc-84b0-994045057bc6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:27.122281 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:27.122258 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b24c1c0-b61b-43fc-84b0-994045057bc6-istiod-ca-cert" pod "8b24c1c0-b61b-43fc-84b0-994045057bc6" (UID: "8b24c1c0-b61b-43fc-84b0-994045057bc6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:53:27.122600 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:27.122577 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b24c1c0-b61b-43fc-84b0-994045057bc6-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:27.122685 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:27.122606 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b24c1c0-b61b-43fc-84b0-994045057bc6-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:27.122685 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:27.122622 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b24c1c0-b61b-43fc-84b0-994045057bc6-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:27.122685 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:27.122640 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b24c1c0-b61b-43fc-84b0-994045057bc6-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:27.122685 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:27.122654 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b24c1c0-b61b-43fc-84b0-994045057bc6-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:27.123915 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:27.123887 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b24c1c0-b61b-43fc-84b0-994045057bc6-istio-envoy" pod "8b24c1c0-b61b-43fc-84b0-994045057bc6" (UID: "8b24c1c0-b61b-43fc-84b0-994045057bc6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:27.124034 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:27.123919 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b24c1c0-b61b-43fc-84b0-994045057bc6-istio-podinfo" pod "8b24c1c0-b61b-43fc-84b0-994045057bc6" (UID: "8b24c1c0-b61b-43fc-84b0-994045057bc6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:53:27.124034 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:27.123938 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b24c1c0-b61b-43fc-84b0-994045057bc6-istio-token" pod "8b24c1c0-b61b-43fc-84b0-994045057bc6" (UID: "8b24c1c0-b61b-43fc-84b0-994045057bc6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:53:27.124034 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:27.123991 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b24c1c0-b61b-43fc-84b0-994045057bc6-kube-api-access-4q9l9" pod "8b24c1c0-b61b-43fc-84b0-994045057bc6" (UID: "8b24c1c0-b61b-43fc-84b0-994045057bc6"). InnerVolumeSpecName "kube-api-access-4q9l9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:53:27.223982 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:27.223945 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b24c1c0-b61b-43fc-84b0-994045057bc6-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:27.223982 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:27.223970 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b24c1c0-b61b-43fc-84b0-994045057bc6-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:27.223982 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:27.223979 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b24c1c0-b61b-43fc-84b0-994045057bc6-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:27.223982 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:27.223989 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q9l9\" (UniqueName: \"kubernetes.io/projected/8b24c1c0-b61b-43fc-84b0-994045057bc6-kube-api-access-4q9l9\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:27.276235 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:27.276146 6572 generic.go:356] "Generic (PLEG): container finished" podID="8b24c1c0-b61b-43fc-84b0-994045057bc6" containerID="68d9b90790271e8bf890d5d8e4ceffe0bf9459fd275aeacd20de216ac22d9920" exitCode=0 Oct 30 01:53:27.276235 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:27.276205 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" Oct 30 01:53:27.276439 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:27.276205 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" event={"ID":"8b24c1c0-b61b-43fc-84b0-994045057bc6","Type":"ContainerDied","Data":"68d9b90790271e8bf890d5d8e4ceffe0bf9459fd275aeacd20de216ac22d9920"} Oct 30 01:53:27.276439 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:27.276339 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79" event={"ID":"8b24c1c0-b61b-43fc-84b0-994045057bc6","Type":"ContainerDied","Data":"87856ddba85bd3c72ef4c4697f262598d3c0a44843288f748ba30db71d5a0cab"} Oct 30 01:53:27.276439 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:27.276366 6572 scope.go:122] "RemoveContainer" containerID="68d9b90790271e8bf890d5d8e4ceffe0bf9459fd275aeacd20de216ac22d9920" Oct 30 01:53:27.284808 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:27.284767 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:27.292501 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:27.292473 6572 scope.go:122] "RemoveContainer" containerID="68d9b90790271e8bf890d5d8e4ceffe0bf9459fd275aeacd20de216ac22d9920" Oct 30 01:53:27.293132 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:53:27.293108 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68d9b90790271e8bf890d5d8e4ceffe0bf9459fd275aeacd20de216ac22d9920\": container with ID starting with 68d9b90790271e8bf890d5d8e4ceffe0bf9459fd275aeacd20de216ac22d9920 not found: ID does not exist" containerID="68d9b90790271e8bf890d5d8e4ceffe0bf9459fd275aeacd20de216ac22d9920" Oct 30 01:53:27.293236 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:27.293143 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68d9b90790271e8bf890d5d8e4ceffe0bf9459fd275aeacd20de216ac22d9920"} err="failed to get container status \"68d9b90790271e8bf890d5d8e4ceffe0bf9459fd275aeacd20de216ac22d9920\": rpc error: code = NotFound desc = could not find container \"68d9b90790271e8bf890d5d8e4ceffe0bf9459fd275aeacd20de216ac22d9920\": container with ID starting with 68d9b90790271e8bf890d5d8e4ceffe0bf9459fd275aeacd20de216ac22d9920 not found: ID does not exist" Oct 30 01:53:27.341438 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:27.341384 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79"] Oct 30 01:53:27.346611 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:27.346583 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2pk79"] Oct 30 01:53:28.274421 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:28.274388 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8"] Oct 30 01:53:28.274950 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:28.274665 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" podUID="00464688-4494-49d1-b56d-0b7baa8ca9c5" containerName="istio-proxy" containerID="cri-o://a4490045fc342f05c8dad5df2e0024ea64832221014d10845b9d0e70724e662c" gracePeriod=30 Oct 30 01:53:28.578893 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:28.578804 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b24c1c0-b61b-43fc-84b0-994045057bc6" path="/var/lib/kubelet/pods/8b24c1c0-b61b-43fc-84b0-994045057bc6/volumes" Oct 30 01:53:28.829249 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:28.829138 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:30.622093 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:30.621933 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:30.637310 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:30.637266 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7"] Oct 30 01:53:30.863730 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:30.863697 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:31.938660 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:31.938620 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:32.293944 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:32.293665 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:33.555963 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:33.555895 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" Oct 30 01:53:33.675290 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:33.675251 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00464688-4494-49d1-b56d-0b7baa8ca9c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00464688-4494-49d1-b56d-0b7baa8ca9c5-istio-podinfo\") pod \"00464688-4494-49d1-b56d-0b7baa8ca9c5\" (UID: \"00464688-4494-49d1-b56d-0b7baa8ca9c5\") " Oct 30 01:53:33.675487 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:33.675310 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00464688-4494-49d1-b56d-0b7baa8ca9c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00464688-4494-49d1-b56d-0b7baa8ca9c5-istiod-ca-cert\") pod \"00464688-4494-49d1-b56d-0b7baa8ca9c5\" (UID: \"00464688-4494-49d1-b56d-0b7baa8ca9c5\") " Oct 30 01:53:33.675487 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:33.675364 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00464688-4494-49d1-b56d-0b7baa8ca9c5-istio-token\" (UniqueName: \"kubernetes.io/projected/00464688-4494-49d1-b56d-0b7baa8ca9c5-istio-token\") pod \"00464688-4494-49d1-b56d-0b7baa8ca9c5\" (UID: \"00464688-4494-49d1-b56d-0b7baa8ca9c5\") " Oct 30 01:53:33.675487 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:33.675429 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00464688-4494-49d1-b56d-0b7baa8ca9c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00464688-4494-49d1-b56d-0b7baa8ca9c5-workload-socket\") pod \"00464688-4494-49d1-b56d-0b7baa8ca9c5\" (UID: \"00464688-4494-49d1-b56d-0b7baa8ca9c5\") " Oct 30 01:53:33.675656 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:33.675515 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00464688-4494-49d1-b56d-0b7baa8ca9c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00464688-4494-49d1-b56d-0b7baa8ca9c5-workload-certs\") pod \"00464688-4494-49d1-b56d-0b7baa8ca9c5\" (UID: \"00464688-4494-49d1-b56d-0b7baa8ca9c5\") " Oct 30 01:53:33.675656 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:33.675588 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00464688-4494-49d1-b56d-0b7baa8ca9c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00464688-4494-49d1-b56d-0b7baa8ca9c5-credential-socket\") pod \"00464688-4494-49d1-b56d-0b7baa8ca9c5\" (UID: \"00464688-4494-49d1-b56d-0b7baa8ca9c5\") " Oct 30 01:53:33.675656 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:33.675622 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00464688-4494-49d1-b56d-0b7baa8ca9c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00464688-4494-49d1-b56d-0b7baa8ca9c5-istio-envoy\") pod \"00464688-4494-49d1-b56d-0b7baa8ca9c5\" (UID: \"00464688-4494-49d1-b56d-0b7baa8ca9c5\") " Oct 30 01:53:33.675851 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:33.675688 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00464688-4494-49d1-b56d-0b7baa8ca9c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00464688-4494-49d1-b56d-0b7baa8ca9c5-istio-data\") pod \"00464688-4494-49d1-b56d-0b7baa8ca9c5\" (UID: \"00464688-4494-49d1-b56d-0b7baa8ca9c5\") " Oct 30 01:53:33.675851 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:33.675721 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00464688-4494-49d1-b56d-0b7baa8ca9c5-kube-api-access-kkl9q\" (UniqueName: \"kubernetes.io/projected/00464688-4494-49d1-b56d-0b7baa8ca9c5-kube-api-access-kkl9q\") pod \"00464688-4494-49d1-b56d-0b7baa8ca9c5\" (UID: \"00464688-4494-49d1-b56d-0b7baa8ca9c5\") " Oct 30 01:53:33.675851 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:33.675814 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00464688-4494-49d1-b56d-0b7baa8ca9c5-workload-socket" pod "00464688-4494-49d1-b56d-0b7baa8ca9c5" (UID: "00464688-4494-49d1-b56d-0b7baa8ca9c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:33.675998 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:33.675888 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00464688-4494-49d1-b56d-0b7baa8ca9c5-workload-certs" pod "00464688-4494-49d1-b56d-0b7baa8ca9c5" (UID: "00464688-4494-49d1-b56d-0b7baa8ca9c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:33.675998 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:33.675911 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00464688-4494-49d1-b56d-0b7baa8ca9c5-credential-socket" pod "00464688-4494-49d1-b56d-0b7baa8ca9c5" (UID: "00464688-4494-49d1-b56d-0b7baa8ca9c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:33.675998 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:33.675948 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00464688-4494-49d1-b56d-0b7baa8ca9c5-istiod-ca-cert" pod "00464688-4494-49d1-b56d-0b7baa8ca9c5" (UID: "00464688-4494-49d1-b56d-0b7baa8ca9c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:53:33.676150 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:33.676096 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00464688-4494-49d1-b56d-0b7baa8ca9c5-istio-data" pod "00464688-4494-49d1-b56d-0b7baa8ca9c5" (UID: "00464688-4494-49d1-b56d-0b7baa8ca9c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:33.676377 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:33.676354 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00464688-4494-49d1-b56d-0b7baa8ca9c5-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:33.676377 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:33.676378 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00464688-4494-49d1-b56d-0b7baa8ca9c5-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:33.676590 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:33.676393 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00464688-4494-49d1-b56d-0b7baa8ca9c5-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:33.676590 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:33.676406 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00464688-4494-49d1-b56d-0b7baa8ca9c5-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:33.676590 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:33.676418 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00464688-4494-49d1-b56d-0b7baa8ca9c5-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:33.678266 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:33.678236 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00464688-4494-49d1-b56d-0b7baa8ca9c5-istio-token" pod "00464688-4494-49d1-b56d-0b7baa8ca9c5" (UID: "00464688-4494-49d1-b56d-0b7baa8ca9c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:53:33.678390 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:33.678327 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00464688-4494-49d1-b56d-0b7baa8ca9c5-kube-api-access-kkl9q" pod "00464688-4494-49d1-b56d-0b7baa8ca9c5" (UID: "00464688-4494-49d1-b56d-0b7baa8ca9c5"). InnerVolumeSpecName "kube-api-access-kkl9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:53:33.678390 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:33.678371 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00464688-4494-49d1-b56d-0b7baa8ca9c5-istio-envoy" pod "00464688-4494-49d1-b56d-0b7baa8ca9c5" (UID: "00464688-4494-49d1-b56d-0b7baa8ca9c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:33.678507 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:33.678435 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00464688-4494-49d1-b56d-0b7baa8ca9c5-istio-podinfo" pod "00464688-4494-49d1-b56d-0b7baa8ca9c5" (UID: "00464688-4494-49d1-b56d-0b7baa8ca9c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:53:33.778125 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:33.778089 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00464688-4494-49d1-b56d-0b7baa8ca9c5-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:33.778125 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:33.778122 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00464688-4494-49d1-b56d-0b7baa8ca9c5-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:33.778125 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:33.778134 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00464688-4494-49d1-b56d-0b7baa8ca9c5-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:33.778368 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:33.778146 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkl9q\" (UniqueName: \"kubernetes.io/projected/00464688-4494-49d1-b56d-0b7baa8ca9c5-kube-api-access-kkl9q\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:34.034610 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:34.034571 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:34.346603 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:34.346514 6572 generic.go:356] "Generic (PLEG): container finished" podID="00464688-4494-49d1-b56d-0b7baa8ca9c5" containerID="a4490045fc342f05c8dad5df2e0024ea64832221014d10845b9d0e70724e662c" exitCode=0 Oct 30 01:53:34.346603 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:34.346571 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" Oct 30 01:53:34.346825 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:34.346610 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" event={"ID":"00464688-4494-49d1-b56d-0b7baa8ca9c5","Type":"ContainerDied","Data":"a4490045fc342f05c8dad5df2e0024ea64832221014d10845b9d0e70724e662c"} Oct 30 01:53:34.346825 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:34.346709 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8" event={"ID":"00464688-4494-49d1-b56d-0b7baa8ca9c5","Type":"ContainerDied","Data":"ab46a6b24c2d29e0dd42fcf7dcb25c6231844966a45d5844f07d116c643bf8ab"} Oct 30 01:53:34.346914 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:34.346798 6572 scope.go:122] "RemoveContainer" containerID="a4490045fc342f05c8dad5df2e0024ea64832221014d10845b9d0e70724e662c" Oct 30 01:53:34.359304 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:34.359284 6572 scope.go:122] "RemoveContainer" containerID="a4490045fc342f05c8dad5df2e0024ea64832221014d10845b9d0e70724e662c" Oct 30 01:53:34.359626 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:53:34.359606 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4490045fc342f05c8dad5df2e0024ea64832221014d10845b9d0e70724e662c\": container with ID starting with a4490045fc342f05c8dad5df2e0024ea64832221014d10845b9d0e70724e662c not found: ID does not exist" containerID="a4490045fc342f05c8dad5df2e0024ea64832221014d10845b9d0e70724e662c" Oct 30 01:53:34.359702 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:34.359635 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4490045fc342f05c8dad5df2e0024ea64832221014d10845b9d0e70724e662c"} err="failed to get container status \"a4490045fc342f05c8dad5df2e0024ea64832221014d10845b9d0e70724e662c\": rpc error: code = NotFound desc = could not find container \"a4490045fc342f05c8dad5df2e0024ea64832221014d10845b9d0e70724e662c\": container with ID starting with a4490045fc342f05c8dad5df2e0024ea64832221014d10845b9d0e70724e662c not found: ID does not exist" Oct 30 01:53:34.414802 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:34.414764 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8"] Oct 30 01:53:34.423501 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:34.423460 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-69pt8"] Oct 30 01:53:34.495163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:34.495127 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:34.543953 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:34.543919 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00464688-4494-49d1-b56d-0b7baa8ca9c5" path="/var/lib/kubelet/pods/00464688-4494-49d1-b56d-0b7baa8ca9c5/volumes" Oct 30 01:53:35.053548 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:35.049406 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:35.670032 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:35.669996 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r"] Oct 30 01:53:35.670240 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:35.670218 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" podUID="f1174e25-63b1-4693-841f-72732c52e068" containerName="istio-proxy" containerID="cri-o://9e7cb755c1b61d34c51778f691ffb09b0fb8116ea606ec53d6b96207b1df620a" gracePeriod=30 Oct 30 01:53:35.744494 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:35.744464 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:35.941561 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:35.941467 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:37.587186 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:37.587146 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:39.342411 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:39.342372 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc"] Oct 30 01:53:39.362536 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:39.362493 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:39.576192 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:39.576159 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:40.960565 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:40.960465 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" Oct 30 01:53:41.150782 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.146406 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1174e25-63b1-4693-841f-72732c52e068-kube-api-access-tvf2j\" (UniqueName: \"kubernetes.io/projected/f1174e25-63b1-4693-841f-72732c52e068-kube-api-access-tvf2j\") pod \"f1174e25-63b1-4693-841f-72732c52e068\" (UID: \"f1174e25-63b1-4693-841f-72732c52e068\") " Oct 30 01:53:41.150782 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.146504 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1174e25-63b1-4693-841f-72732c52e068-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1174e25-63b1-4693-841f-72732c52e068-istio-envoy\") pod \"f1174e25-63b1-4693-841f-72732c52e068\" (UID: \"f1174e25-63b1-4693-841f-72732c52e068\") " Oct 30 01:53:41.150782 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.146561 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1174e25-63b1-4693-841f-72732c52e068-istio-token\" (UniqueName: \"kubernetes.io/projected/f1174e25-63b1-4693-841f-72732c52e068-istio-token\") pod \"f1174e25-63b1-4693-841f-72732c52e068\" (UID: \"f1174e25-63b1-4693-841f-72732c52e068\") " Oct 30 01:53:41.150782 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.146625 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1174e25-63b1-4693-841f-72732c52e068-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1174e25-63b1-4693-841f-72732c52e068-workload-certs\") pod \"f1174e25-63b1-4693-841f-72732c52e068\" (UID: \"f1174e25-63b1-4693-841f-72732c52e068\") " Oct 30 01:53:41.150782 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.146739 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1174e25-63b1-4693-841f-72732c52e068-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1174e25-63b1-4693-841f-72732c52e068-istio-podinfo\") pod \"f1174e25-63b1-4693-841f-72732c52e068\" (UID: \"f1174e25-63b1-4693-841f-72732c52e068\") " Oct 30 01:53:41.150782 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.146835 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1174e25-63b1-4693-841f-72732c52e068-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1174e25-63b1-4693-841f-72732c52e068-istiod-ca-cert\") pod \"f1174e25-63b1-4693-841f-72732c52e068\" (UID: \"f1174e25-63b1-4693-841f-72732c52e068\") " Oct 30 01:53:41.150782 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.146905 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1174e25-63b1-4693-841f-72732c52e068-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1174e25-63b1-4693-841f-72732c52e068-workload-socket\") pod \"f1174e25-63b1-4693-841f-72732c52e068\" (UID: \"f1174e25-63b1-4693-841f-72732c52e068\") " Oct 30 01:53:41.150782 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.146938 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1174e25-63b1-4693-841f-72732c52e068-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1174e25-63b1-4693-841f-72732c52e068-credential-socket\") pod \"f1174e25-63b1-4693-841f-72732c52e068\" (UID: \"f1174e25-63b1-4693-841f-72732c52e068\") " Oct 30 01:53:41.150782 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.147063 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1174e25-63b1-4693-841f-72732c52e068-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1174e25-63b1-4693-841f-72732c52e068-istio-data\") pod \"f1174e25-63b1-4693-841f-72732c52e068\" (UID: \"f1174e25-63b1-4693-841f-72732c52e068\") " Oct 30 01:53:41.150782 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.148133 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1174e25-63b1-4693-841f-72732c52e068-istio-data" pod "f1174e25-63b1-4693-841f-72732c52e068" (UID: "f1174e25-63b1-4693-841f-72732c52e068"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:41.162716 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.162374 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1174e25-63b1-4693-841f-72732c52e068-kube-api-access-tvf2j" pod "f1174e25-63b1-4693-841f-72732c52e068" (UID: "f1174e25-63b1-4693-841f-72732c52e068"). InnerVolumeSpecName "kube-api-access-tvf2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:53:41.179493 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.178862 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1174e25-63b1-4693-841f-72732c52e068-credential-socket" pod "f1174e25-63b1-4693-841f-72732c52e068" (UID: "f1174e25-63b1-4693-841f-72732c52e068"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:41.179493 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.179258 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1174e25-63b1-4693-841f-72732c52e068-workload-socket" pod "f1174e25-63b1-4693-841f-72732c52e068" (UID: "f1174e25-63b1-4693-841f-72732c52e068"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:41.180408 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.180281 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1174e25-63b1-4693-841f-72732c52e068-istiod-ca-cert" pod "f1174e25-63b1-4693-841f-72732c52e068" (UID: "f1174e25-63b1-4693-841f-72732c52e068"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:53:41.181552 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.180621 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1174e25-63b1-4693-841f-72732c52e068-workload-certs" pod "f1174e25-63b1-4693-841f-72732c52e068" (UID: "f1174e25-63b1-4693-841f-72732c52e068"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:41.182675 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.182544 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:41.182829 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.182683 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1174e25-63b1-4693-841f-72732c52e068-istio-podinfo" pod "f1174e25-63b1-4693-841f-72732c52e068" (UID: "f1174e25-63b1-4693-841f-72732c52e068"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:53:41.183197 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.183161 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1174e25-63b1-4693-841f-72732c52e068-istio-token" pod "f1174e25-63b1-4693-841f-72732c52e068" (UID: "f1174e25-63b1-4693-841f-72732c52e068"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:53:41.183611 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.183586 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1174e25-63b1-4693-841f-72732c52e068-istio-envoy" pod "f1174e25-63b1-4693-841f-72732c52e068" (UID: "f1174e25-63b1-4693-841f-72732c52e068"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:41.249965 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.249727 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1174e25-63b1-4693-841f-72732c52e068-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:41.249965 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.249778 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvf2j\" (UniqueName: \"kubernetes.io/projected/f1174e25-63b1-4693-841f-72732c52e068-kube-api-access-tvf2j\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:41.249965 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.249795 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1174e25-63b1-4693-841f-72732c52e068-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:41.249965 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.249809 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1174e25-63b1-4693-841f-72732c52e068-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:41.249965 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.249823 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1174e25-63b1-4693-841f-72732c52e068-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:41.249965 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.249835 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1174e25-63b1-4693-841f-72732c52e068-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:41.249965 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.249849 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1174e25-63b1-4693-841f-72732c52e068-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:41.249965 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.249864 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1174e25-63b1-4693-841f-72732c52e068-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:41.249965 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.249877 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1174e25-63b1-4693-841f-72732c52e068-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:41.309770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.309593 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d44413-c6c7b8c75-qqdbt"] Oct 30 01:53:41.313773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.311116 6572 cpu_manager.go:414] "containerMap: removing container" podUID="3cde2113-52f8-40d6-a1ee-fb94a2d497a2" containerName="revoke-keys" Oct 30 01:53:41.313773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.311143 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cde2113-52f8-40d6-a1ee-fb94a2d497a2" containerName="revoke-keys" Oct 30 01:53:41.313773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.311228 6572 cpu_manager.go:414] "containerMap: removing container" podUID="00464688-4494-49d1-b56d-0b7baa8ca9c5" containerName="istio-proxy" Oct 30 01:53:41.313773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.311241 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00464688-4494-49d1-b56d-0b7baa8ca9c5" containerName="istio-proxy" Oct 30 01:53:41.313773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.311264 6572 cpu_manager.go:414] "containerMap: removing container" podUID="928c4da4-c1a1-4f68-9852-d1e80408a421" containerName="revoke-keys" Oct 30 01:53:41.313773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.311273 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="928c4da4-c1a1-4f68-9852-d1e80408a421" containerName="revoke-keys" Oct 30 01:53:41.313773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.311312 6572 cpu_manager.go:414] "containerMap: removing container" podUID="f1174e25-63b1-4693-841f-72732c52e068" containerName="istio-proxy" Oct 30 01:53:41.313773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.311325 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1174e25-63b1-4693-841f-72732c52e068" containerName="istio-proxy" Oct 30 01:53:41.313773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.311346 6572 cpu_manager.go:414] "containerMap: removing container" podUID="8b24c1c0-b61b-43fc-84b0-994045057bc6" containerName="istio-proxy" Oct 30 01:53:41.313773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.311353 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b24c1c0-b61b-43fc-84b0-994045057bc6" containerName="istio-proxy" Oct 30 01:53:41.313773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.311648 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cde2113-52f8-40d6-a1ee-fb94a2d497a2" containerName="revoke-keys" Oct 30 01:53:41.313773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.311679 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="00464688-4494-49d1-b56d-0b7baa8ca9c5" containerName="istio-proxy" Oct 30 01:53:41.313773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.311695 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="928c4da4-c1a1-4f68-9852-d1e80408a421" containerName="revoke-keys" Oct 30 01:53:41.313773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.311707 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1174e25-63b1-4693-841f-72732c52e068" containerName="istio-proxy" Oct 30 01:53:41.313773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.311722 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b24c1c0-b61b-43fc-84b0-994045057bc6" containerName="istio-proxy" Oct 30 01:53:41.323815 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.323729 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d44413-c6c7b8c75-qqdbt" Oct 30 01:53:41.341331 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.341301 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-d44413\"" Oct 30 01:53:41.342695 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.342662 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d44413-c6c7b8c75-qqdbt"] Oct 30 01:53:41.347857 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.347825 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d44413-86f78b95f6-v547l"] Oct 30 01:53:41.353132 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.353063 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d44413-86f78b95f6-v547l" Oct 30 01:53:41.379341 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.379296 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d44413-86f78b95f6-v547l"] Oct 30 01:53:41.426813 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.426184 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:41.438668 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.438635 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-d44413-dockercfg-kgkbj\"" Oct 30 01:53:41.463778 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.458630 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdsqp\" (UniqueName: \"kubernetes.io/projected/1e5005f7-fccd-4447-b09d-dd991c1c7733-kube-api-access-bdsqp\") pod \"payload-processing-e2e-extmod-a-d44413-86f78b95f6-v547l\" (UID: \"1e5005f7-fccd-4447-b09d-dd991c1c7733\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d44413-86f78b95f6-v547l" Oct 30 01:53:41.463778 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.458792 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e5005f7-fccd-4447-b09d-dd991c1c7733-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d44413-86f78b95f6-v547l\" (UID: \"1e5005f7-fccd-4447-b09d-dd991c1c7733\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d44413-86f78b95f6-v547l" Oct 30 01:53:41.463778 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.459039 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfztl\" (UniqueName: \"kubernetes.io/projected/96c5b66f-d98b-491a-93ae-4b2587cf9809-kube-api-access-pfztl\") pod \"payload-pre-processing-e2e-extmod-a-d44413-c6c7b8c75-qqdbt\" (UID: \"96c5b66f-d98b-491a-93ae-4b2587cf9809\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d44413-c6c7b8c75-qqdbt" Oct 30 01:53:41.463778 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.459274 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96c5b66f-d98b-491a-93ae-4b2587cf9809-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d44413-c6c7b8c75-qqdbt\" (UID: \"96c5b66f-d98b-491a-93ae-4b2587cf9809\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d44413-c6c7b8c75-qqdbt" Oct 30 01:53:41.481104 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.481061 6572 generic.go:356] "Generic (PLEG): container finished" podID="f1174e25-63b1-4693-841f-72732c52e068" containerID="9e7cb755c1b61d34c51778f691ffb09b0fb8116ea606ec53d6b96207b1df620a" exitCode=0 Oct 30 01:53:41.481414 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.481379 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" event={"ID":"f1174e25-63b1-4693-841f-72732c52e068","Type":"ContainerDied","Data":"9e7cb755c1b61d34c51778f691ffb09b0fb8116ea606ec53d6b96207b1df620a"} Oct 30 01:53:41.481521 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.481438 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" event={"ID":"f1174e25-63b1-4693-841f-72732c52e068","Type":"ContainerDied","Data":"b94a4125174b3aa3071b720772980041fc077d7b5008c2d6b6f324817c08a47f"} Oct 30 01:53:41.481521 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.481473 6572 scope.go:122] "RemoveContainer" containerID="9e7cb755c1b61d34c51778f691ffb09b0fb8116ea606ec53d6b96207b1df620a" Oct 30 01:53:41.481680 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.481664 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r" Oct 30 01:53:41.573674 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.560965 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e5005f7-fccd-4447-b09d-dd991c1c7733-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d44413-86f78b95f6-v547l\" (UID: \"1e5005f7-fccd-4447-b09d-dd991c1c7733\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d44413-86f78b95f6-v547l" Oct 30 01:53:41.575213 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.562580 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e5005f7-fccd-4447-b09d-dd991c1c7733-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d44413-86f78b95f6-v547l\" (UID: \"1e5005f7-fccd-4447-b09d-dd991c1c7733\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d44413-86f78b95f6-v547l" Oct 30 01:53:41.576071 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.575903 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfztl\" (UniqueName: \"kubernetes.io/projected/96c5b66f-d98b-491a-93ae-4b2587cf9809-kube-api-access-pfztl\") pod \"payload-pre-processing-e2e-extmod-a-d44413-c6c7b8c75-qqdbt\" (UID: \"96c5b66f-d98b-491a-93ae-4b2587cf9809\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d44413-c6c7b8c75-qqdbt" Oct 30 01:53:41.576573 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.576206 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96c5b66f-d98b-491a-93ae-4b2587cf9809-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d44413-c6c7b8c75-qqdbt\" (UID: \"96c5b66f-d98b-491a-93ae-4b2587cf9809\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d44413-c6c7b8c75-qqdbt" Oct 30 01:53:41.576573 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.576307 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdsqp\" (UniqueName: \"kubernetes.io/projected/1e5005f7-fccd-4447-b09d-dd991c1c7733-kube-api-access-bdsqp\") pod \"payload-processing-e2e-extmod-a-d44413-86f78b95f6-v547l\" (UID: \"1e5005f7-fccd-4447-b09d-dd991c1c7733\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d44413-86f78b95f6-v547l" Oct 30 01:53:41.577764 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.577716 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96c5b66f-d98b-491a-93ae-4b2587cf9809-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d44413-c6c7b8c75-qqdbt\" (UID: \"96c5b66f-d98b-491a-93ae-4b2587cf9809\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d44413-c6c7b8c75-qqdbt" Oct 30 01:53:41.596849 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.596818 6572 scope.go:122] "RemoveContainer" containerID="9e7cb755c1b61d34c51778f691ffb09b0fb8116ea606ec53d6b96207b1df620a" Oct 30 01:53:41.607767 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:53:41.598917 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e7cb755c1b61d34c51778f691ffb09b0fb8116ea606ec53d6b96207b1df620a\": container with ID starting with 9e7cb755c1b61d34c51778f691ffb09b0fb8116ea606ec53d6b96207b1df620a not found: ID does not exist" containerID="9e7cb755c1b61d34c51778f691ffb09b0fb8116ea606ec53d6b96207b1df620a" Oct 30 01:53:41.607767 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.598959 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e7cb755c1b61d34c51778f691ffb09b0fb8116ea606ec53d6b96207b1df620a"} err="failed to get container status \"9e7cb755c1b61d34c51778f691ffb09b0fb8116ea606ec53d6b96207b1df620a\": rpc error: code = NotFound desc = could not find container \"9e7cb755c1b61d34c51778f691ffb09b0fb8116ea606ec53d6b96207b1df620a\": container with ID starting with 9e7cb755c1b61d34c51778f691ffb09b0fb8116ea606ec53d6b96207b1df620a not found: ID does not exist" Oct 30 01:53:41.607767 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.603061 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfztl\" (UniqueName: \"kubernetes.io/projected/96c5b66f-d98b-491a-93ae-4b2587cf9809-kube-api-access-pfztl\") pod \"payload-pre-processing-e2e-extmod-a-d44413-c6c7b8c75-qqdbt\" (UID: \"96c5b66f-d98b-491a-93ae-4b2587cf9809\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d44413-c6c7b8c75-qqdbt" Oct 30 01:53:41.607767 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.606644 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdsqp\" (UniqueName: \"kubernetes.io/projected/1e5005f7-fccd-4447-b09d-dd991c1c7733-kube-api-access-bdsqp\") pod \"payload-processing-e2e-extmod-a-d44413-86f78b95f6-v547l\" (UID: \"1e5005f7-fccd-4447-b09d-dd991c1c7733\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d44413-86f78b95f6-v547l" Oct 30 01:53:41.644783 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.644469 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d44413-c6c7b8c75-qqdbt" Oct 30 01:53:41.650199 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.647653 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r"] Oct 30 01:53:41.650956 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.650806 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-wz85r"] Oct 30 01:53:41.682082 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.682041 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d44413-86f78b95f6-v547l" Oct 30 01:53:41.978458 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:41.977875 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:42.035134 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:42.035110 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d44413-c6c7b8c75-qqdbt"] Oct 30 01:53:42.051642 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:42.051614 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d44413-86f78b95f6-v547l"] Oct 30 01:53:42.052180 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:53:42.052155 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e5005f7_fccd_4447_b09d_dd991c1c7733.slice/crio-b71fc06e6f0cae5522805ea2c9a0217c7ba1248fdb2df7f33b907642790a7989 WatchSource:0}: Error finding container b71fc06e6f0cae5522805ea2c9a0217c7ba1248fdb2df7f33b907642790a7989: Status 404 returned error can't find the container with id b71fc06e6f0cae5522805ea2c9a0217c7ba1248fdb2df7f33b907642790a7989 Oct 30 01:53:42.493413 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:42.493374 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d44413-c6c7b8c75-qqdbt" event={"ID":"96c5b66f-d98b-491a-93ae-4b2587cf9809","Type":"ContainerStarted","Data":"e8b9af652baa34b52f5e9747ad0e65dcdec8d32cab773c5e99812ce57d60bf6b"} Oct 30 01:53:42.493599 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:42.493432 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d44413-c6c7b8c75-qqdbt" event={"ID":"96c5b66f-d98b-491a-93ae-4b2587cf9809","Type":"ContainerStarted","Data":"2315d26c5b358f32204ad93a3118e1c58e260c5d5eee8cb025202a4c3eee43ed"} Oct 30 01:53:42.493599 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:42.493459 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d44413-c6c7b8c75-qqdbt" Oct 30 01:53:42.495574 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:42.495545 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d44413-86f78b95f6-v547l" event={"ID":"1e5005f7-fccd-4447-b09d-dd991c1c7733","Type":"ContainerStarted","Data":"512472424c4eda1866a70f9230e096f63b90958173aa5510108308a64ef6f812"} Oct 30 01:53:42.495784 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:42.495589 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d44413-86f78b95f6-v547l" event={"ID":"1e5005f7-fccd-4447-b09d-dd991c1c7733","Type":"ContainerStarted","Data":"b71fc06e6f0cae5522805ea2c9a0217c7ba1248fdb2df7f33b907642790a7989"} Oct 30 01:53:42.495784 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:42.495632 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-d44413-86f78b95f6-v547l" Oct 30 01:53:42.543611 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:42.543531 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d44413-c6c7b8c75-qqdbt" podStartSLOduration=1.5435112119999999 podStartE2EDuration="1.543511212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:42.533961613 +0000 UTC m=+2148.110387016" watchObservedRunningTime="2026-10-30 01:53:42.543511212 +0000 UTC m=+2148.119936615" Oct 30 01:53:42.547297 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:42.547260 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1174e25-63b1-4693-841f-72732c52e068" path="/var/lib/kubelet/pods/f1174e25-63b1-4693-841f-72732c52e068/volumes" Oct 30 01:53:42.583148 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:42.583074 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-d44413-86f78b95f6-v547l" podStartSLOduration=1.583055698 podStartE2EDuration="1.583055698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:42.578272263 +0000 UTC m=+2148.154697666" watchObservedRunningTime="2026-10-30 01:53:42.583055698 +0000 UTC m=+2148.159481099" Oct 30 01:53:42.601519 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:42.601486 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:42.912760 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:42.912722 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps"] Oct 30 01:53:42.918168 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:42.918137 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:53:42.945117 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:42.943889 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps"] Oct 30 01:53:42.945711 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:42.945683 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-0835dd-openshift-default-dockercfg-qjnh4\"" Oct 30 01:53:43.001391 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:43.001354 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a6c761-0adf-4fcf-b4a5-1c8401957993-workload-socket\") pod \"e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps\" (UID: \"57a6c761-0adf-4fcf-b4a5-1c8401957993\") " pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:53:43.001872 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:43.001398 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57a6c761-0adf-4fcf-b4a5-1c8401957993-istio-token\") pod \"e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps\" (UID: \"57a6c761-0adf-4fcf-b4a5-1c8401957993\") " pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:53:43.001872 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:43.001547 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57a6c761-0adf-4fcf-b4a5-1c8401957993-istio-data\") pod \"e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps\" (UID: \"57a6c761-0adf-4fcf-b4a5-1c8401957993\") " pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:53:43.001872 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:43.001664 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zss8r\" (UniqueName: \"kubernetes.io/projected/57a6c761-0adf-4fcf-b4a5-1c8401957993-kube-api-access-zss8r\") pod \"e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps\" (UID: \"57a6c761-0adf-4fcf-b4a5-1c8401957993\") " pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:53:43.001872 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:43.001717 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57a6c761-0adf-4fcf-b4a5-1c8401957993-istio-podinfo\") pod \"e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps\" (UID: \"57a6c761-0adf-4fcf-b4a5-1c8401957993\") " pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:53:43.001872 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:43.001777 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a6c761-0adf-4fcf-b4a5-1c8401957993-credential-socket\") pod \"e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps\" (UID: \"57a6c761-0adf-4fcf-b4a5-1c8401957993\") " pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:53:43.002152 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:43.001914 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57a6c761-0adf-4fcf-b4a5-1c8401957993-workload-certs\") pod \"e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps\" (UID: \"57a6c761-0adf-4fcf-b4a5-1c8401957993\") " pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:53:43.002152 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:43.001982 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57a6c761-0adf-4fcf-b4a5-1c8401957993-istiod-ca-cert\") pod \"e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps\" (UID: \"57a6c761-0adf-4fcf-b4a5-1c8401957993\") " pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:53:43.002152 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:43.002102 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57a6c761-0adf-4fcf-b4a5-1c8401957993-istio-envoy\") pod \"e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps\" (UID: \"57a6c761-0adf-4fcf-b4a5-1c8401957993\") " pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:53:43.089836 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:43.084385 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:43.117815 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:43.115534 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57a6c761-0adf-4fcf-b4a5-1c8401957993-istio-envoy\") pod \"e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps\" (UID: \"57a6c761-0adf-4fcf-b4a5-1c8401957993\") " pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:53:43.117815 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:43.115615 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a6c761-0adf-4fcf-b4a5-1c8401957993-workload-socket\") pod \"e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps\" (UID: \"57a6c761-0adf-4fcf-b4a5-1c8401957993\") " pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:53:43.117815 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:43.115655 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57a6c761-0adf-4fcf-b4a5-1c8401957993-istio-token\") pod \"e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps\" (UID: \"57a6c761-0adf-4fcf-b4a5-1c8401957993\") " pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:53:43.117815 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:43.115802 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57a6c761-0adf-4fcf-b4a5-1c8401957993-istio-data\") pod \"e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps\" (UID: \"57a6c761-0adf-4fcf-b4a5-1c8401957993\") " pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:53:43.117815 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:43.116099 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zss8r\" (UniqueName: \"kubernetes.io/projected/57a6c761-0adf-4fcf-b4a5-1c8401957993-kube-api-access-zss8r\") pod \"e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps\" (UID: \"57a6c761-0adf-4fcf-b4a5-1c8401957993\") " pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:53:43.117815 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:43.116148 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57a6c761-0adf-4fcf-b4a5-1c8401957993-istio-podinfo\") pod \"e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps\" (UID: \"57a6c761-0adf-4fcf-b4a5-1c8401957993\") " pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:53:43.117815 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:43.116228 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a6c761-0adf-4fcf-b4a5-1c8401957993-credential-socket\") pod \"e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps\" (UID: \"57a6c761-0adf-4fcf-b4a5-1c8401957993\") " pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:53:43.117815 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:43.116381 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57a6c761-0adf-4fcf-b4a5-1c8401957993-workload-certs\") pod \"e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps\" (UID: \"57a6c761-0adf-4fcf-b4a5-1c8401957993\") " pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:53:43.117815 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:43.116478 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57a6c761-0adf-4fcf-b4a5-1c8401957993-istiod-ca-cert\") pod \"e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps\" (UID: \"57a6c761-0adf-4fcf-b4a5-1c8401957993\") " pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:53:43.117815 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:43.117429 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57a6c761-0adf-4fcf-b4a5-1c8401957993-istiod-ca-cert\") pod \"e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps\" (UID: \"57a6c761-0adf-4fcf-b4a5-1c8401957993\") " pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:53:43.118509 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:43.118194 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a6c761-0adf-4fcf-b4a5-1c8401957993-credential-socket\") pod \"e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps\" (UID: \"57a6c761-0adf-4fcf-b4a5-1c8401957993\") " pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:53:43.118509 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:43.118461 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a6c761-0adf-4fcf-b4a5-1c8401957993-workload-socket\") pod \"e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps\" (UID: \"57a6c761-0adf-4fcf-b4a5-1c8401957993\") " pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:53:43.121785 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:43.119064 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57a6c761-0adf-4fcf-b4a5-1c8401957993-istio-data\") pod \"e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps\" (UID: \"57a6c761-0adf-4fcf-b4a5-1c8401957993\") " pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:53:43.121785 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:43.119444 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57a6c761-0adf-4fcf-b4a5-1c8401957993-workload-certs\") pod \"e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps\" (UID: \"57a6c761-0adf-4fcf-b4a5-1c8401957993\") " pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:53:43.143814 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:43.140080 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57a6c761-0adf-4fcf-b4a5-1c8401957993-istio-token\") pod \"e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps\" (UID: \"57a6c761-0adf-4fcf-b4a5-1c8401957993\") " pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:53:43.147775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:43.145206 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57a6c761-0adf-4fcf-b4a5-1c8401957993-istio-envoy\") pod \"e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps\" (UID: \"57a6c761-0adf-4fcf-b4a5-1c8401957993\") " pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:53:43.158809 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:43.152452 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zss8r\" (UniqueName: \"kubernetes.io/projected/57a6c761-0adf-4fcf-b4a5-1c8401957993-kube-api-access-zss8r\") pod \"e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps\" (UID: \"57a6c761-0adf-4fcf-b4a5-1c8401957993\") " pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:53:43.158809 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:43.153517 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57a6c761-0adf-4fcf-b4a5-1c8401957993-istio-podinfo\") pod \"e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps\" (UID: \"57a6c761-0adf-4fcf-b4a5-1c8401957993\") " pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:53:43.246901 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:43.246254 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:53:43.288735 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:43.288699 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:43.569781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:43.569646 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:43.593473 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:43.593437 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps"] Oct 30 01:53:43.593921 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:53:43.593893 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57a6c761_0adf_4fcf_b4a5_1c8401957993.slice/crio-5a21cc359980029aa617bc742bddc6a6b2d24af9d938fd585ceda04e6626148e WatchSource:0}: Error finding container 5a21cc359980029aa617bc742bddc6a6b2d24af9d938fd585ceda04e6626148e: Status 404 returned error can't find the container with id 5a21cc359980029aa617bc742bddc6a6b2d24af9d938fd585ceda04e6626148e Oct 30 01:53:43.597005 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:43.596966 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:53:43.597133 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:43.597055 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:53:43.597133 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:43.597105 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:53:44.262967 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:44.262931 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:44.516264 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:44.516169 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" event={"ID":"57a6c761-0adf-4fcf-b4a5-1c8401957993","Type":"ContainerStarted","Data":"78a007e88a5b645bb18d9960a40bf7aef53d9b5ea63dd8ce0f599e527d84860d"} Oct 30 01:53:44.516264 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:44.516261 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" event={"ID":"57a6c761-0adf-4fcf-b4a5-1c8401957993","Type":"ContainerStarted","Data":"5a21cc359980029aa617bc742bddc6a6b2d24af9d938fd585ceda04e6626148e"} Oct 30 01:53:44.632387 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:44.629844 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" podStartSLOduration=2.629821829 podStartE2EDuration="2.629821829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:44.582679192 +0000 UTC m=+2150.159104619" watchObservedRunningTime="2026-10-30 01:53:44.629821829 +0000 UTC m=+2150.206247232" Oct 30 01:53:44.774853 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:53:44.770936 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:53:44.849723 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:44.849676 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:45.247534 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:45.247500 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:53:45.253880 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:45.253849 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:53:45.325363 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:45.325327 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:45.515044 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:45.514945 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:45.524254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:45.524221 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:53:45.525503 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:45.525481 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:53:45.661859 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:45.661816 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7" podUID="f89f1d37-12c8-4a6c-a3c0-533234c89a74" containerName="main" containerID="cri-o://ce818bfbab2fc8b99655fc9241577539dfb400827adf9d60e5185c678576226a" gracePeriod=45 Oct 30 01:53:45.984613 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:45.984585 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7" Oct 30 01:53:46.092148 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:46.092063 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f89f1d37-12c8-4a6c-a3c0-533234c89a74-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f89f1d37-12c8-4a6c-a3c0-533234c89a74-dshm\") pod \"f89f1d37-12c8-4a6c-a3c0-533234c89a74\" (UID: \"f89f1d37-12c8-4a6c-a3c0-533234c89a74\") " Oct 30 01:53:46.092148 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:46.092106 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f89f1d37-12c8-4a6c-a3c0-533234c89a74-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f89f1d37-12c8-4a6c-a3c0-533234c89a74-model-cache\") pod \"f89f1d37-12c8-4a6c-a3c0-533234c89a74\" (UID: \"f89f1d37-12c8-4a6c-a3c0-533234c89a74\") " Oct 30 01:53:46.092148 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:46.092143 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f89f1d37-12c8-4a6c-a3c0-533234c89a74-home\" (UniqueName: \"kubernetes.io/empty-dir/f89f1d37-12c8-4a6c-a3c0-533234c89a74-home\") pod \"f89f1d37-12c8-4a6c-a3c0-533234c89a74\" (UID: \"f89f1d37-12c8-4a6c-a3c0-533234c89a74\") " Oct 30 01:53:46.092878 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:46.092194 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f89f1d37-12c8-4a6c-a3c0-533234c89a74-kube-api-access-m57cz\" (UniqueName: \"kubernetes.io/projected/f89f1d37-12c8-4a6c-a3c0-533234c89a74-kube-api-access-m57cz\") pod \"f89f1d37-12c8-4a6c-a3c0-533234c89a74\" (UID: \"f89f1d37-12c8-4a6c-a3c0-533234c89a74\") " Oct 30 01:53:46.092878 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:46.092220 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f89f1d37-12c8-4a6c-a3c0-533234c89a74-tls-certs\" (UniqueName: \"kubernetes.io/secret/f89f1d37-12c8-4a6c-a3c0-533234c89a74-tls-certs\") pod \"f89f1d37-12c8-4a6c-a3c0-533234c89a74\" (UID: \"f89f1d37-12c8-4a6c-a3c0-533234c89a74\") " Oct 30 01:53:46.092878 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:46.092253 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f89f1d37-12c8-4a6c-a3c0-533234c89a74-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f89f1d37-12c8-4a6c-a3c0-533234c89a74-tmp-dir\") pod \"f89f1d37-12c8-4a6c-a3c0-533234c89a74\" (UID: \"f89f1d37-12c8-4a6c-a3c0-533234c89a74\") " Oct 30 01:53:46.092878 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:46.092467 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f89f1d37-12c8-4a6c-a3c0-533234c89a74-model-cache" pod "f89f1d37-12c8-4a6c-a3c0-533234c89a74" (UID: "f89f1d37-12c8-4a6c-a3c0-533234c89a74"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:46.092878 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:46.092480 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f89f1d37-12c8-4a6c-a3c0-533234c89a74-home" pod "f89f1d37-12c8-4a6c-a3c0-533234c89a74" (UID: "f89f1d37-12c8-4a6c-a3c0-533234c89a74"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:46.092878 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:46.092646 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f89f1d37-12c8-4a6c-a3c0-533234c89a74-tmp-dir" pod "f89f1d37-12c8-4a6c-a3c0-533234c89a74" (UID: "f89f1d37-12c8-4a6c-a3c0-533234c89a74"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:46.093603 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:46.093005 6572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f89f1d37-12c8-4a6c-a3c0-533234c89a74-tmp-dir\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:46.093603 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:46.093031 6572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f89f1d37-12c8-4a6c-a3c0-533234c89a74-model-cache\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:46.093603 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:46.093047 6572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f89f1d37-12c8-4a6c-a3c0-533234c89a74-home\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:46.094577 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:46.094545 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f89f1d37-12c8-4a6c-a3c0-533234c89a74-dshm" pod "f89f1d37-12c8-4a6c-a3c0-533234c89a74" (UID: "f89f1d37-12c8-4a6c-a3c0-533234c89a74"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:46.094684 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:46.094663 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f89f1d37-12c8-4a6c-a3c0-533234c89a74-kube-api-access-m57cz" pod "f89f1d37-12c8-4a6c-a3c0-533234c89a74" (UID: "f89f1d37-12c8-4a6c-a3c0-533234c89a74"). InnerVolumeSpecName "kube-api-access-m57cz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:53:46.094730 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:46.094691 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f89f1d37-12c8-4a6c-a3c0-533234c89a74-tls-certs" pod "f89f1d37-12c8-4a6c-a3c0-533234c89a74" (UID: "f89f1d37-12c8-4a6c-a3c0-533234c89a74"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:53:46.194742 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:46.194699 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m57cz\" (UniqueName: \"kubernetes.io/projected/f89f1d37-12c8-4a6c-a3c0-533234c89a74-kube-api-access-m57cz\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:46.194742 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:46.194738 6572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f89f1d37-12c8-4a6c-a3c0-533234c89a74-tls-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:46.194742 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:46.194761 6572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f89f1d37-12c8-4a6c-a3c0-533234c89a74-dshm\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:46.533539 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:46.533500 6572 generic.go:356] "Generic (PLEG): container finished" podID="f89f1d37-12c8-4a6c-a3c0-533234c89a74" containerID="ce818bfbab2fc8b99655fc9241577539dfb400827adf9d60e5185c678576226a" exitCode=0 Oct 30 01:53:46.534118 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:46.533561 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7" Oct 30 01:53:46.534118 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:46.533584 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7" event={"ID":"f89f1d37-12c8-4a6c-a3c0-533234c89a74","Type":"ContainerDied","Data":"ce818bfbab2fc8b99655fc9241577539dfb400827adf9d60e5185c678576226a"} Oct 30 01:53:46.534118 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:46.533631 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7" event={"ID":"f89f1d37-12c8-4a6c-a3c0-533234c89a74","Type":"ContainerDied","Data":"900f55d8a0e928079f0ac6e2de01da820d9819c38da4a4edb28acaea3c3e2082"} Oct 30 01:53:46.534118 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:46.533650 6572 scope.go:122] "RemoveContainer" containerID="ce818bfbab2fc8b99655fc9241577539dfb400827adf9d60e5185c678576226a" Oct 30 01:53:46.549392 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:46.549366 6572 scope.go:122] "RemoveContainer" containerID="ce818bfbab2fc8b99655fc9241577539dfb400827adf9d60e5185c678576226a" Oct 30 01:53:46.549825 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:53:46.549798 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce818bfbab2fc8b99655fc9241577539dfb400827adf9d60e5185c678576226a\": container with ID starting with ce818bfbab2fc8b99655fc9241577539dfb400827adf9d60e5185c678576226a not found: ID does not exist" containerID="ce818bfbab2fc8b99655fc9241577539dfb400827adf9d60e5185c678576226a" Oct 30 01:53:46.549942 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:46.549835 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce818bfbab2fc8b99655fc9241577539dfb400827adf9d60e5185c678576226a"} err="failed to get container status \"ce818bfbab2fc8b99655fc9241577539dfb400827adf9d60e5185c678576226a\": rpc error: code = NotFound desc = could not find container \"ce818bfbab2fc8b99655fc9241577539dfb400827adf9d60e5185c678576226a\": container with ID starting with ce818bfbab2fc8b99655fc9241577539dfb400827adf9d60e5185c678576226a not found: ID does not exist" Oct 30 01:53:46.588028 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:46.587975 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7"] Oct 30 01:53:46.599743 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:46.599704 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-292018/auth-test-model-292018-kserve-6fccf6cf7c-mmqr7"] Oct 30 01:53:47.276712 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:47.276680 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:47.520323 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:47.520285 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:48.147043 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:48.147006 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:48.347658 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:48.347622 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:48.543617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:48.543572 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f89f1d37-12c8-4a6c-a3c0-533234c89a74" path="/var/lib/kubelet/pods/f89f1d37-12c8-4a6c-a3c0-533234c89a74/volumes" Oct 30 01:53:48.961655 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:48.961623 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:48.988889 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:48.988850 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-0835dd-df6f4cccb-mglq6"] Oct 30 01:53:48.989521 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:48.989504 6572 cpu_manager.go:414] "containerMap: removing container" podUID="f89f1d37-12c8-4a6c-a3c0-533234c89a74" containerName="main" Oct 30 01:53:48.989521 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:48.989520 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f89f1d37-12c8-4a6c-a3c0-533234c89a74" containerName="main" Oct 30 01:53:48.989693 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:48.989682 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="f89f1d37-12c8-4a6c-a3c0-533234c89a74" containerName="main" Oct 30 01:53:48.993690 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:48.993665 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-0835dd-df6f4cccb-mglq6" Oct 30 01:53:49.014039 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:49.014006 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-0835dd\"" Oct 30 01:53:49.039665 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:49.039623 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-0835dd-df6f4cccb-mglq6"] Oct 30 01:53:49.164835 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:49.164650 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f336af73-f4bf-4e67-94ed-0131538293f8-plugins-config-volume\") pod \"payload-processing-e2e-autores-0835dd-df6f4cccb-mglq6\" (UID: \"f336af73-f4bf-4e67-94ed-0131538293f8\") " pod="openshift-ingress/payload-processing-e2e-autores-0835dd-df6f4cccb-mglq6" Oct 30 01:53:49.165362 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:49.164872 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7czm\" (UniqueName: \"kubernetes.io/projected/f336af73-f4bf-4e67-94ed-0131538293f8-kube-api-access-v7czm\") pod \"payload-processing-e2e-autores-0835dd-df6f4cccb-mglq6\" (UID: \"f336af73-f4bf-4e67-94ed-0131538293f8\") " pod="openshift-ingress/payload-processing-e2e-autores-0835dd-df6f4cccb-mglq6" Oct 30 01:53:49.183601 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:49.183568 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:49.276866 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:49.273641 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7czm\" (UniqueName: \"kubernetes.io/projected/f336af73-f4bf-4e67-94ed-0131538293f8-kube-api-access-v7czm\") pod \"payload-processing-e2e-autores-0835dd-df6f4cccb-mglq6\" (UID: \"f336af73-f4bf-4e67-94ed-0131538293f8\") " pod="openshift-ingress/payload-processing-e2e-autores-0835dd-df6f4cccb-mglq6" Oct 30 01:53:49.276866 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:49.273836 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f336af73-f4bf-4e67-94ed-0131538293f8-plugins-config-volume\") pod \"payload-processing-e2e-autores-0835dd-df6f4cccb-mglq6\" (UID: \"f336af73-f4bf-4e67-94ed-0131538293f8\") " pod="openshift-ingress/payload-processing-e2e-autores-0835dd-df6f4cccb-mglq6" Oct 30 01:53:49.276866 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:49.274917 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f336af73-f4bf-4e67-94ed-0131538293f8-plugins-config-volume\") pod \"payload-processing-e2e-autores-0835dd-df6f4cccb-mglq6\" (UID: \"f336af73-f4bf-4e67-94ed-0131538293f8\") " pod="openshift-ingress/payload-processing-e2e-autores-0835dd-df6f4cccb-mglq6" Oct 30 01:53:49.297778 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:49.297696 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7czm\" (UniqueName: \"kubernetes.io/projected/f336af73-f4bf-4e67-94ed-0131538293f8-kube-api-access-v7czm\") pod \"payload-processing-e2e-autores-0835dd-df6f4cccb-mglq6\" (UID: \"f336af73-f4bf-4e67-94ed-0131538293f8\") " pod="openshift-ingress/payload-processing-e2e-autores-0835dd-df6f4cccb-mglq6" Oct 30 01:53:49.311993 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:49.309526 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-0835dd-df6f4cccb-mglq6" Oct 30 01:53:49.509157 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:49.509127 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:49.678379 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:53:49.678335 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf336af73_f4bf_4e67_94ed_0131538293f8.slice/crio-f7a308b5b40880ebda02b3d4138576e43b0b1dcc8347286efd148d4556cf5497 WatchSource:0}: Error finding container f7a308b5b40880ebda02b3d4138576e43b0b1dcc8347286efd148d4556cf5497: Status 404 returned error can't find the container with id f7a308b5b40880ebda02b3d4138576e43b0b1dcc8347286efd148d4556cf5497 Oct 30 01:53:49.678793 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:49.678764 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-0835dd-df6f4cccb-mglq6"] Oct 30 01:53:49.702111 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:49.702072 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:50.574638 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:50.574590 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-0835dd-df6f4cccb-mglq6" event={"ID":"f336af73-f4bf-4e67-94ed-0131538293f8","Type":"ContainerStarted","Data":"c40317a8191c4da96ec8019fe18d93025f65641de43918d53c4d43f49effeb4f"} Oct 30 01:53:50.575148 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:50.574663 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-0835dd-df6f4cccb-mglq6" event={"ID":"f336af73-f4bf-4e67-94ed-0131538293f8","Type":"ContainerStarted","Data":"f7a308b5b40880ebda02b3d4138576e43b0b1dcc8347286efd148d4556cf5497"} Oct 30 01:53:50.576456 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:50.576420 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-0835dd-df6f4cccb-mglq6" Oct 30 01:53:50.618863 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:50.618789 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-0835dd-df6f4cccb-mglq6" podStartSLOduration=2.61876399 podStartE2EDuration="2.61876399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:50.612292951 +0000 UTC m=+2156.188718365" watchObservedRunningTime="2026-10-30 01:53:50.61876399 +0000 UTC m=+2156.195189393" Oct 30 01:53:51.296936 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:51.296894 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:51.369695 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:51.369640 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2eae39-7989c5cb95-k9lsj"] Oct 30 01:53:51.377717 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:51.377679 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2eae39-7989c5cb95-k9lsj" Oct 30 01:53:51.399480 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:51.396200 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2eae39-7989c5cb95-k9lsj"] Oct 30 01:53:51.519771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:51.516831 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4ngj\" (UniqueName: \"kubernetes.io/projected/5387b465-e2a2-44dd-aa7e-42a8447a7639-kube-api-access-p4ngj\") pod \"payload-pre-processing-e2e-extmod-a-2eae39-7989c5cb95-k9lsj\" (UID: \"5387b465-e2a2-44dd-aa7e-42a8447a7639\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2eae39-7989c5cb95-k9lsj" Oct 30 01:53:51.519771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:51.517069 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5387b465-e2a2-44dd-aa7e-42a8447a7639-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2eae39-7989c5cb95-k9lsj\" (UID: \"5387b465-e2a2-44dd-aa7e-42a8447a7639\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2eae39-7989c5cb95-k9lsj" Oct 30 01:53:51.630706 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:51.630575 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-2eae39-dockercfg-6br8x\"" Oct 30 01:53:51.631284 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:51.630741 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-2eae39\"" Oct 30 01:53:51.652789 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:51.632208 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:51.652789 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:51.635202 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4ngj\" (UniqueName: \"kubernetes.io/projected/5387b465-e2a2-44dd-aa7e-42a8447a7639-kube-api-access-p4ngj\") pod \"payload-pre-processing-e2e-extmod-a-2eae39-7989c5cb95-k9lsj\" (UID: \"5387b465-e2a2-44dd-aa7e-42a8447a7639\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2eae39-7989c5cb95-k9lsj" Oct 30 01:53:51.652789 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:51.635592 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5387b465-e2a2-44dd-aa7e-42a8447a7639-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2eae39-7989c5cb95-k9lsj\" (UID: \"5387b465-e2a2-44dd-aa7e-42a8447a7639\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2eae39-7989c5cb95-k9lsj" Oct 30 01:53:51.652789 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:51.637295 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5387b465-e2a2-44dd-aa7e-42a8447a7639-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2eae39-7989c5cb95-k9lsj\" (UID: \"5387b465-e2a2-44dd-aa7e-42a8447a7639\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2eae39-7989c5cb95-k9lsj" Oct 30 01:53:51.670798 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:51.670766 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4ngj\" (UniqueName: \"kubernetes.io/projected/5387b465-e2a2-44dd-aa7e-42a8447a7639-kube-api-access-p4ngj\") pod \"payload-pre-processing-e2e-extmod-a-2eae39-7989c5cb95-k9lsj\" (UID: \"5387b465-e2a2-44dd-aa7e-42a8447a7639\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2eae39-7989c5cb95-k9lsj" Oct 30 01:53:51.700602 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:51.700557 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2eae39-7989c5cb95-k9lsj" Oct 30 01:53:51.811393 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:51.811354 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:52.011257 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:52.011224 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2eae39-7989c5cb95-k9lsj"] Oct 30 01:53:52.011474 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:53:52.011453 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5387b465_e2a2_44dd_aa7e_42a8447a7639.slice/crio-c46a04c80f945818c3fee74268d17bef98bb4102d90641d652cc97057b3ed359 WatchSource:0}: Error finding container c46a04c80f945818c3fee74268d17bef98bb4102d90641d652cc97057b3ed359: Status 404 returned error can't find the container with id c46a04c80f945818c3fee74268d17bef98bb4102d90641d652cc97057b3ed359 Oct 30 01:53:52.663713 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:52.663656 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2eae39-7989c5cb95-k9lsj" event={"ID":"5387b465-e2a2-44dd-aa7e-42a8447a7639","Type":"ContainerStarted","Data":"4bb34ab80172d72178911912feabdedc1d842d78e58fe1568c10ea20adac234b"} Oct 30 01:53:52.664245 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:52.663730 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2eae39-7989c5cb95-k9lsj" event={"ID":"5387b465-e2a2-44dd-aa7e-42a8447a7639","Type":"ContainerStarted","Data":"c46a04c80f945818c3fee74268d17bef98bb4102d90641d652cc97057b3ed359"} Oct 30 01:53:52.664245 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:52.663887 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2eae39-7989c5cb95-k9lsj" Oct 30 01:53:52.702853 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:52.702785 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2eae39-7989c5cb95-k9lsj" podStartSLOduration=1.702766131 podStartE2EDuration="1.702766131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:52.699692809 +0000 UTC m=+2158.276118214" watchObservedRunningTime="2026-10-30 01:53:52.702766131 +0000 UTC m=+2158.279191535" Oct 30 01:53:53.297368 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:53.297331 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:53.509249 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:53.509219 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d44413-c6c7b8c75-qqdbt" Oct 30 01:53:53.509429 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:53.509380 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-d44413-86f78b95f6-v547l" Oct 30 01:53:54.365633 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:54.365572 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc" podUID="452989de-dd18-450d-872c-316d00cf3f44" containerName="main" containerID="cri-o://5e9f59d3929642e70494b1fedfb1efec139fc7b8a1239fff6febdcecf9a98088" gracePeriod=45 Oct 30 01:53:54.681920 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:54.681863 6572 generic.go:356] "Generic (PLEG): container finished" podID="452989de-dd18-450d-872c-316d00cf3f44" containerID="5e9f59d3929642e70494b1fedfb1efec139fc7b8a1239fff6febdcecf9a98088" exitCode=0 Oct 30 01:53:54.682090 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:54.681939 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc" event={"ID":"452989de-dd18-450d-872c-316d00cf3f44","Type":"ContainerDied","Data":"5e9f59d3929642e70494b1fedfb1efec139fc7b8a1239fff6febdcecf9a98088"} Oct 30 01:53:54.682090 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:54.681989 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc" event={"ID":"452989de-dd18-450d-872c-316d00cf3f44","Type":"ContainerDied","Data":"7d4b7245b9304db84c66301c28061741d1c9ea5a2b3f6e83a874197592d9de43"} Oct 30 01:53:54.682090 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:54.682002 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d4b7245b9304db84c66301c28061741d1c9ea5a2b3f6e83a874197592d9de43" Oct 30 01:53:54.686308 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:54.686278 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc" Oct 30 01:53:54.783275 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:54.783221 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/452989de-dd18-450d-872c-316d00cf3f44-tls-certs\" (UniqueName: \"kubernetes.io/secret/452989de-dd18-450d-872c-316d00cf3f44-tls-certs\") pod \"452989de-dd18-450d-872c-316d00cf3f44\" (UID: \"452989de-dd18-450d-872c-316d00cf3f44\") " Oct 30 01:53:54.783483 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:54.783302 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/452989de-dd18-450d-872c-316d00cf3f44-kube-api-access-rr7vw\" (UniqueName: \"kubernetes.io/projected/452989de-dd18-450d-872c-316d00cf3f44-kube-api-access-rr7vw\") pod \"452989de-dd18-450d-872c-316d00cf3f44\" (UID: \"452989de-dd18-450d-872c-316d00cf3f44\") " Oct 30 01:53:54.783483 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:54.783332 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/452989de-dd18-450d-872c-316d00cf3f44-dshm\" (UniqueName: \"kubernetes.io/empty-dir/452989de-dd18-450d-872c-316d00cf3f44-dshm\") pod \"452989de-dd18-450d-872c-316d00cf3f44\" (UID: \"452989de-dd18-450d-872c-316d00cf3f44\") " Oct 30 01:53:54.783483 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:54.783360 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/452989de-dd18-450d-872c-316d00cf3f44-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/452989de-dd18-450d-872c-316d00cf3f44-tmp-dir\") pod \"452989de-dd18-450d-872c-316d00cf3f44\" (UID: \"452989de-dd18-450d-872c-316d00cf3f44\") " Oct 30 01:53:54.783483 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:54.783433 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/452989de-dd18-450d-872c-316d00cf3f44-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/452989de-dd18-450d-872c-316d00cf3f44-model-cache\") pod \"452989de-dd18-450d-872c-316d00cf3f44\" (UID: \"452989de-dd18-450d-872c-316d00cf3f44\") " Oct 30 01:53:54.783483 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:54.783467 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/452989de-dd18-450d-872c-316d00cf3f44-home\" (UniqueName: \"kubernetes.io/empty-dir/452989de-dd18-450d-872c-316d00cf3f44-home\") pod \"452989de-dd18-450d-872c-316d00cf3f44\" (UID: \"452989de-dd18-450d-872c-316d00cf3f44\") " Oct 30 01:53:54.783880 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:54.783840 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452989de-dd18-450d-872c-316d00cf3f44-tmp-dir" pod "452989de-dd18-450d-872c-316d00cf3f44" (UID: "452989de-dd18-450d-872c-316d00cf3f44"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:54.784013 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:54.783917 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452989de-dd18-450d-872c-316d00cf3f44-home" pod "452989de-dd18-450d-872c-316d00cf3f44" (UID: "452989de-dd18-450d-872c-316d00cf3f44"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:54.784096 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:54.784022 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452989de-dd18-450d-872c-316d00cf3f44-model-cache" pod "452989de-dd18-450d-872c-316d00cf3f44" (UID: "452989de-dd18-450d-872c-316d00cf3f44"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:54.784164 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:54.784112 6572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/452989de-dd18-450d-872c-316d00cf3f44-tmp-dir\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:54.784164 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:54.784132 6572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/452989de-dd18-450d-872c-316d00cf3f44-model-cache\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:54.784164 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:54.784142 6572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/452989de-dd18-450d-872c-316d00cf3f44-home\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:54.785891 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:54.785858 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/452989de-dd18-450d-872c-316d00cf3f44-tls-certs" pod "452989de-dd18-450d-872c-316d00cf3f44" (UID: "452989de-dd18-450d-872c-316d00cf3f44"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:53:54.786010 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:54.785916 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452989de-dd18-450d-872c-316d00cf3f44-dshm" pod "452989de-dd18-450d-872c-316d00cf3f44" (UID: "452989de-dd18-450d-872c-316d00cf3f44"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:54.786010 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:54.785957 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/452989de-dd18-450d-872c-316d00cf3f44-kube-api-access-rr7vw" pod "452989de-dd18-450d-872c-316d00cf3f44" (UID: "452989de-dd18-450d-872c-316d00cf3f44"). InnerVolumeSpecName "kube-api-access-rr7vw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:53:54.885767 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:54.885720 6572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/452989de-dd18-450d-872c-316d00cf3f44-tls-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:54.885978 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:54.885802 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr7vw\" (UniqueName: \"kubernetes.io/projected/452989de-dd18-450d-872c-316d00cf3f44-kube-api-access-rr7vw\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:54.885978 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:54.885815 6572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/452989de-dd18-450d-872c-316d00cf3f44-dshm\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:53:55.040959 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:55.040919 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:55.612281 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:55.612232 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc" podUID="452989de-dd18-450d-872c-316d00cf3f44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.243:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:55.690187 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:55.690148 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc" Oct 30 01:53:55.743191 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:55.743152 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc"] Oct 30 01:53:55.753652 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:55.753613 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-264bdd/auth-test-model-264bdd-kserve-5fdc59f7bf-hssrc"] Oct 30 01:53:56.544640 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:56.544604 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="452989de-dd18-450d-872c-316d00cf3f44" path="/var/lib/kubelet/pods/452989de-dd18-450d-872c-316d00cf3f44/volumes" Oct 30 01:53:56.839517 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:56.839418 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:57.744899 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:57.744864 6572 scope.go:122] "RemoveContainer" containerID="ca423b3b178354a751abc2ca88e6db907cd24ca68c3dd913def338870d1c046b" Oct 30 01:53:57.755740 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:57.755717 6572 scope.go:122] "RemoveContainer" containerID="15f75a99e926a11445878ef36f2c81e0051c5076536c0c708494cb95ec58475f" Oct 30 01:53:58.385978 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:58.385945 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:59.848111 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:53:59.848070 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:01.233035 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:01.233003 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:01.416884 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:01.416845 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:02.614045 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:02.614009 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:02.667061 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:02.667033 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-0835dd-df6f4cccb-mglq6" Oct 30 01:54:02.921015 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:02.920980 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:02.948885 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:02.948858 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh"] Oct 30 01:54:02.949517 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:02.949503 6572 cpu_manager.go:414] "containerMap: removing container" podUID="452989de-dd18-450d-872c-316d00cf3f44" containerName="main" Oct 30 01:54:02.949592 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:02.949518 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="452989de-dd18-450d-872c-316d00cf3f44" containerName="main" Oct 30 01:54:02.949678 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:02.949663 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="452989de-dd18-450d-872c-316d00cf3f44" containerName="main" Oct 30 01:54:02.955574 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:02.955548 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 01:54:02.975836 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:02.971698 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-f1edaf-openshift-default-dockercfg-9s4vn\"" Oct 30 01:54:02.979425 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:02.979397 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh"] Oct 30 01:54:03.149475 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:03.149425 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e0c074-1c61-435e-b086-0c06657d2ab9-workload-socket\") pod \"e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh\" (UID: \"22e0c074-1c61-435e-b086-0c06657d2ab9\") " pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 01:54:03.152043 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:03.152011 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22e0c074-1c61-435e-b086-0c06657d2ab9-istio-envoy\") pod \"e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh\" (UID: \"22e0c074-1c61-435e-b086-0c06657d2ab9\") " pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 01:54:03.152291 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:03.152271 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22e0c074-1c61-435e-b086-0c06657d2ab9-istio-podinfo\") pod \"e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh\" (UID: \"22e0c074-1c61-435e-b086-0c06657d2ab9\") " pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 01:54:03.152423 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:03.152406 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e0c074-1c61-435e-b086-0c06657d2ab9-credential-socket\") pod \"e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh\" (UID: \"22e0c074-1c61-435e-b086-0c06657d2ab9\") " pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 01:54:03.152582 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:03.152565 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22e0c074-1c61-435e-b086-0c06657d2ab9-istio-token\") pod \"e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh\" (UID: \"22e0c074-1c61-435e-b086-0c06657d2ab9\") " pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 01:54:03.152797 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:03.152772 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7nh2\" (UniqueName: \"kubernetes.io/projected/22e0c074-1c61-435e-b086-0c06657d2ab9-kube-api-access-w7nh2\") pod \"e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh\" (UID: \"22e0c074-1c61-435e-b086-0c06657d2ab9\") " pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 01:54:03.152984 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:03.152966 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22e0c074-1c61-435e-b086-0c06657d2ab9-istiod-ca-cert\") pod \"e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh\" (UID: \"22e0c074-1c61-435e-b086-0c06657d2ab9\") " pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 01:54:03.153139 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:03.153124 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22e0c074-1c61-435e-b086-0c06657d2ab9-istio-data\") pod \"e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh\" (UID: \"22e0c074-1c61-435e-b086-0c06657d2ab9\") " pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 01:54:03.153340 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:03.153322 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22e0c074-1c61-435e-b086-0c06657d2ab9-workload-certs\") pod \"e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh\" (UID: \"22e0c074-1c61-435e-b086-0c06657d2ab9\") " pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 01:54:03.153540 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:03.151522 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:03.254527 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:03.254437 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22e0c074-1c61-435e-b086-0c06657d2ab9-workload-certs\") pod \"e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh\" (UID: \"22e0c074-1c61-435e-b086-0c06657d2ab9\") " pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 01:54:03.254527 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:03.254520 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e0c074-1c61-435e-b086-0c06657d2ab9-workload-socket\") pod \"e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh\" (UID: \"22e0c074-1c61-435e-b086-0c06657d2ab9\") " pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 01:54:03.254819 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:03.254551 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22e0c074-1c61-435e-b086-0c06657d2ab9-istio-envoy\") pod \"e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh\" (UID: \"22e0c074-1c61-435e-b086-0c06657d2ab9\") " pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 01:54:03.254819 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:03.254596 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22e0c074-1c61-435e-b086-0c06657d2ab9-istio-podinfo\") pod \"e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh\" (UID: \"22e0c074-1c61-435e-b086-0c06657d2ab9\") " pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 01:54:03.254819 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:03.254632 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e0c074-1c61-435e-b086-0c06657d2ab9-credential-socket\") pod \"e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh\" (UID: \"22e0c074-1c61-435e-b086-0c06657d2ab9\") " pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 01:54:03.254819 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:03.254689 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22e0c074-1c61-435e-b086-0c06657d2ab9-istio-token\") pod \"e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh\" (UID: \"22e0c074-1c61-435e-b086-0c06657d2ab9\") " pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 01:54:03.255062 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:03.254817 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7nh2\" (UniqueName: \"kubernetes.io/projected/22e0c074-1c61-435e-b086-0c06657d2ab9-kube-api-access-w7nh2\") pod \"e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh\" (UID: \"22e0c074-1c61-435e-b086-0c06657d2ab9\") " pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 01:54:03.255062 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:03.254907 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22e0c074-1c61-435e-b086-0c06657d2ab9-istiod-ca-cert\") pod \"e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh\" (UID: \"22e0c074-1c61-435e-b086-0c06657d2ab9\") " pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 01:54:03.255062 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:03.254965 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22e0c074-1c61-435e-b086-0c06657d2ab9-workload-certs\") pod \"e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh\" (UID: \"22e0c074-1c61-435e-b086-0c06657d2ab9\") " pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 01:54:03.255062 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:03.254975 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22e0c074-1c61-435e-b086-0c06657d2ab9-istio-data\") pod \"e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh\" (UID: \"22e0c074-1c61-435e-b086-0c06657d2ab9\") " pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 01:54:03.255062 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:03.255011 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e0c074-1c61-435e-b086-0c06657d2ab9-workload-socket\") pod \"e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh\" (UID: \"22e0c074-1c61-435e-b086-0c06657d2ab9\") " pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 01:54:03.255344 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:03.255291 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22e0c074-1c61-435e-b086-0c06657d2ab9-istio-data\") pod \"e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh\" (UID: \"22e0c074-1c61-435e-b086-0c06657d2ab9\") " pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 01:54:03.255344 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:03.255323 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e0c074-1c61-435e-b086-0c06657d2ab9-credential-socket\") pod \"e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh\" (UID: \"22e0c074-1c61-435e-b086-0c06657d2ab9\") " pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 01:54:03.255619 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:03.255596 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22e0c074-1c61-435e-b086-0c06657d2ab9-istiod-ca-cert\") pod \"e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh\" (UID: \"22e0c074-1c61-435e-b086-0c06657d2ab9\") " pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 01:54:03.257047 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:03.257028 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22e0c074-1c61-435e-b086-0c06657d2ab9-istio-envoy\") pod \"e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh\" (UID: \"22e0c074-1c61-435e-b086-0c06657d2ab9\") " pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 01:54:03.257409 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:03.257388 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22e0c074-1c61-435e-b086-0c06657d2ab9-istio-podinfo\") pod \"e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh\" (UID: \"22e0c074-1c61-435e-b086-0c06657d2ab9\") " pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 01:54:03.285765 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:03.285719 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22e0c074-1c61-435e-b086-0c06657d2ab9-istio-token\") pod \"e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh\" (UID: \"22e0c074-1c61-435e-b086-0c06657d2ab9\") " pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 01:54:03.289535 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:03.289506 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7nh2\" (UniqueName: \"kubernetes.io/projected/22e0c074-1c61-435e-b086-0c06657d2ab9-kube-api-access-w7nh2\") pod \"e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh\" (UID: \"22e0c074-1c61-435e-b086-0c06657d2ab9\") " pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 01:54:03.457189 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:03.457098 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:03.584361 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:03.584276 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 01:54:03.674067 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:03.674043 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2eae39-7989c5cb95-k9lsj" Oct 30 01:54:03.903544 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:03.903513 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh"] Oct 30 01:54:03.904641 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:54:03.904617 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22e0c074_1c61_435e_b086_0c06657d2ab9.slice/crio-06dcee2519e219fd4c58718f6204a075a1d47930701a55583af3d35751cbcc6d WatchSource:0}: Error finding container 06dcee2519e219fd4c58718f6204a075a1d47930701a55583af3d35751cbcc6d: Status 404 returned error can't find the container with id 06dcee2519e219fd4c58718f6204a075a1d47930701a55583af3d35751cbcc6d Oct 30 01:54:03.908515 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:03.908479 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:54:03.908604 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:03.908547 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:54:03.908604 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:03.908577 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:54:04.479501 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:04.479465 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:04.673087 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:04.673050 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:04.764861 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:04.764755 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" event={"ID":"22e0c074-1c61-435e-b086-0c06657d2ab9","Type":"ContainerStarted","Data":"a61525cca28467846d8814fad0e50b1779a391008b10a753031c8f7c52162699"} Oct 30 01:54:04.764861 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:04.764829 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" event={"ID":"22e0c074-1c61-435e-b086-0c06657d2ab9","Type":"ContainerStarted","Data":"06dcee2519e219fd4c58718f6204a075a1d47930701a55583af3d35751cbcc6d"} Oct 30 01:54:04.821729 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:04.821674 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" podStartSLOduration=2.821657583 podStartE2EDuration="2.821657583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:54:04.820868648 +0000 UTC m=+2170.397294050" watchObservedRunningTime="2026-10-30 01:54:04.821657583 +0000 UTC m=+2170.398082985" Oct 30 01:54:05.585019 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:05.584982 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 01:54:05.591163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:05.591128 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 01:54:05.772818 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:05.772783 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 01:54:05.773965 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:05.773945 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 01:54:06.195305 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:06.195267 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:06.392826 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:06.392739 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:07.340864 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:07.340817 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:07.542425 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:07.542387 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:07.924887 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:07.924852 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:08.984332 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:08.984300 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:09.067414 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:09.067377 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f1edaf-55c447f959-nh7w8"] Oct 30 01:54:09.072093 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:09.072072 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1edaf-55c447f959-nh7w8" Oct 30 01:54:09.089278 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:09.089248 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f1edaf-55c447f959-nh7w8"] Oct 30 01:54:09.095663 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:09.095636 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-f1edaf-dockercfg-5nrcq\"" Oct 30 01:54:09.095663 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:09.095651 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-f1edaf\"" Oct 30 01:54:09.161787 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:09.148413 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvpkl\" (UniqueName: \"kubernetes.io/projected/f730cd49-e419-4643-bbaa-c5e7d7ab82eb-kube-api-access-pvpkl\") pod \"payload-pre-processing-e2e-extmod-b-f1edaf-55c447f959-nh7w8\" (UID: \"f730cd49-e419-4643-bbaa-c5e7d7ab82eb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1edaf-55c447f959-nh7w8" Oct 30 01:54:09.161787 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:09.148511 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f730cd49-e419-4643-bbaa-c5e7d7ab82eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-f1edaf-55c447f959-nh7w8\" (UID: \"f730cd49-e419-4643-bbaa-c5e7d7ab82eb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1edaf-55c447f959-nh7w8" Oct 30 01:54:09.247334 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:09.247245 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:09.276153 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:09.276117 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvpkl\" (UniqueName: \"kubernetes.io/projected/f730cd49-e419-4643-bbaa-c5e7d7ab82eb-kube-api-access-pvpkl\") pod \"payload-pre-processing-e2e-extmod-b-f1edaf-55c447f959-nh7w8\" (UID: \"f730cd49-e419-4643-bbaa-c5e7d7ab82eb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1edaf-55c447f959-nh7w8" Oct 30 01:54:09.289959 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:09.289924 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f730cd49-e419-4643-bbaa-c5e7d7ab82eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-f1edaf-55c447f959-nh7w8\" (UID: \"f730cd49-e419-4643-bbaa-c5e7d7ab82eb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1edaf-55c447f959-nh7w8" Oct 30 01:54:09.292032 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:09.290910 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f730cd49-e419-4643-bbaa-c5e7d7ab82eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-f1edaf-55c447f959-nh7w8\" (UID: \"f730cd49-e419-4643-bbaa-c5e7d7ab82eb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1edaf-55c447f959-nh7w8" Oct 30 01:54:09.309460 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:09.309433 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvpkl\" (UniqueName: \"kubernetes.io/projected/f730cd49-e419-4643-bbaa-c5e7d7ab82eb-kube-api-access-pvpkl\") pod \"payload-pre-processing-e2e-extmod-b-f1edaf-55c447f959-nh7w8\" (UID: \"f730cd49-e419-4643-bbaa-c5e7d7ab82eb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1edaf-55c447f959-nh7w8" Oct 30 01:54:09.387946 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:09.387905 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1edaf-55c447f959-nh7w8" Oct 30 01:54:09.664076 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:09.664032 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:09.699905 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:09.699877 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f1edaf-55c447f959-nh7w8"] Oct 30 01:54:09.700156 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:54:09.700129 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf730cd49_e419_4643_bbaa_c5e7d7ab82eb.slice/crio-d7e1546974e0a0d5b7302293c3798df14982ef778cd6d9cbc28cf0c58ac494fa WatchSource:0}: Error finding container d7e1546974e0a0d5b7302293c3798df14982ef778cd6d9cbc28cf0c58ac494fa: Status 404 returned error can't find the container with id d7e1546974e0a0d5b7302293c3798df14982ef778cd6d9cbc28cf0c58ac494fa Oct 30 01:54:09.807056 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:09.807019 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1edaf-55c447f959-nh7w8" event={"ID":"f730cd49-e419-4643-bbaa-c5e7d7ab82eb","Type":"ContainerStarted","Data":"55566dbaf540987bdcd8feab8f1669a9896888c89dfe17575dd7be9de4a6c940"} Oct 30 01:54:09.807245 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:09.807081 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1edaf-55c447f959-nh7w8" event={"ID":"f730cd49-e419-4643-bbaa-c5e7d7ab82eb","Type":"ContainerStarted","Data":"d7e1546974e0a0d5b7302293c3798df14982ef778cd6d9cbc28cf0c58ac494fa"} Oct 30 01:54:09.807339 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:09.807247 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1edaf-55c447f959-nh7w8" Oct 30 01:54:09.847156 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:09.847096 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1edaf-55c447f959-nh7w8" podStartSLOduration=0.847077006 podStartE2EDuration="847.077006ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:54:09.845341711 +0000 UTC m=+2175.421767113" watchObservedRunningTime="2026-10-30 01:54:09.847077006 +0000 UTC m=+2175.423502408" Oct 30 01:54:11.390768 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:11.390287 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:12.857937 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:12.857890 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:13.056035 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:13.056002 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:14.575668 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:14.575629 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:14.688660 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:54:14.688625 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:54:16.065069 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:16.065023 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:17.841762 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:17.841641 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:19.600049 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:19.600011 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:20.820186 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:20.820154 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1edaf-55c447f959-nh7w8" Oct 30 01:54:21.202513 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:21.202471 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:21.389445 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:21.389386 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:21.596533 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:21.596436 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:23.004911 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:23.004865 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:23.233865 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:23.233829 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:24.876176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:24.876134 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:26.556789 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:26.556710 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:28.242216 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:28.242168 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:29.685897 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:29.685861 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:30.113337 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:30.113240 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:30.881161 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:30.881129 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:31.211182 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:31.211148 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:31.525075 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:31.524973 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:31.791634 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:31.791540 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:32.006559 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:32.006516 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:32.481630 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:32.481588 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:32.753830 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:32.752897 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:33.803776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:33.796430 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:33.945034 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:33.944988 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr"] Oct 30 01:54:33.950888 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:33.950862 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr" Oct 30 01:54:33.966379 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:33.966348 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2e618f\"/\"agc-cred-ca46afa9ce58\"" Oct 30 01:54:33.972192 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:33.972151 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr"] Oct 30 01:54:33.977310 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:33.977277 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2e618f\"/\"openshift-service-ca.crt\"" Oct 30 01:54:33.977563 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:33.977397 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2e618f\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2e618f\"" Oct 30 01:54:33.978107 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:33.978081 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2e618f\"/\"routing-overlay\"" Oct 30 01:54:33.978282 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:33.978112 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2e618f\"/\"kube-root-ca.crt\"" Oct 30 01:54:33.978282 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:33.978231 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2e618f\"/\"payload-processing-external-model-e2e--66567ea0-dockercfg-x5mqg\"" Oct 30 01:54:34.006360 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:34.006321 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:34.137832 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:34.117900 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-228vh\" (UniqueName: \"kubernetes.io/projected/14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6-kube-api-access-228vh\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr\" (UID: \"14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr" Oct 30 01:54:34.138205 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:34.138169 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr\" (UID: \"14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr" Oct 30 01:54:34.138483 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:34.138456 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr\" (UID: \"14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr" Oct 30 01:54:34.138683 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:34.138663 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr\" (UID: \"14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr" Oct 30 01:54:34.240798 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:34.240722 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-228vh\" (UniqueName: \"kubernetes.io/projected/14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6-kube-api-access-228vh\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr\" (UID: \"14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr" Oct 30 01:54:34.240798 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:34.240811 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr\" (UID: \"14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr" Oct 30 01:54:34.241064 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:34.240851 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr\" (UID: \"14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr" Oct 30 01:54:34.241064 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:34.240883 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr\" (UID: \"14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr" Oct 30 01:54:34.241593 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:34.241570 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr\" (UID: \"14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr" Oct 30 01:54:34.241711 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:34.241660 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr\" (UID: \"14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr" Oct 30 01:54:34.243381 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:34.243358 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr\" (UID: \"14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr" Oct 30 01:54:34.260474 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:34.260444 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-228vh\" (UniqueName: \"kubernetes.io/projected/14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6-kube-api-access-228vh\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr\" (UID: \"14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr" Oct 30 01:54:34.266773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:34.266726 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr" Oct 30 01:54:34.580888 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:34.580856 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr"] Oct 30 01:54:34.581541 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:54:34.581516 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14cf9ea1_29bc_4908_b97d_e4f6df1ebcf6.slice/crio-9d7fa3d85fa23090c460505af646bbffb9f4a6b2838c1d0cdcff91b11523fe7a WatchSource:0}: Error finding container 9d7fa3d85fa23090c460505af646bbffb9f4a6b2838c1d0cdcff91b11523fe7a: Status 404 returned error can't find the container with id 9d7fa3d85fa23090c460505af646bbffb9f4a6b2838c1d0cdcff91b11523fe7a Oct 30 01:54:35.133590 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:35.133553 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr" event={"ID":"14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6","Type":"ContainerStarted","Data":"74e57d2eafe1e4466c4bb97fd69061e386683e10b4197852f5594ff827d04429"} Oct 30 01:54:35.134054 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:35.133612 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr" event={"ID":"14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6","Type":"ContainerStarted","Data":"9d7fa3d85fa23090c460505af646bbffb9f4a6b2838c1d0cdcff91b11523fe7a"} Oct 30 01:54:35.134054 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:35.133646 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr" Oct 30 01:54:35.176687 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:35.176623 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr" podStartSLOduration=2.176607808 podStartE2EDuration="2.176607808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:54:35.172091513 +0000 UTC m=+2200.748516984" watchObservedRunningTime="2026-10-30 01:54:35.176607808 +0000 UTC m=+2200.753033211" Oct 30 01:54:35.612378 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:35.612334 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:37.197389 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:37.197349 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:38.681282 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:38.681249 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:40.236101 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:40.236066 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:41.681592 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:41.681554 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:43.206764 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:43.192521 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:43.396276 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:43.396230 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:43.697924 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:43.697885 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:44.689126 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:54:44.689086 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:54:44.983870 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:44.983768 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:46.146192 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:46.146149 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr" Oct 30 01:54:46.574687 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:46.574650 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:48.311888 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:48.311847 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:50.023490 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:50.023444 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:51.609594 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:51.609553 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:53.260587 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:53.260552 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:55.011203 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:55.011145 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:56.039320 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:56.039280 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:56.455065 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:56.455024 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:57.426474 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:57.426438 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:57.767925 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:57.767829 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns"] Oct 30 01:54:57.781272 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:57.781239 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 01:54:57.800696 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:57.800663 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns"] Oct 30 01:54:57.805805 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:57.805719 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-093d23-openshift-default-dockercfg-kkfl4\"" Oct 30 01:54:57.861227 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:57.861194 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d64c9fe-26e0-455a-9200-cbc6c4af615f-istio-envoy\") pod \"e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns\" (UID: \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\") " pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 01:54:57.861227 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:57.861239 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d64c9fe-26e0-455a-9200-cbc6c4af615f-istio-data\") pod \"e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns\" (UID: \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\") " pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 01:54:57.861512 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:57.861334 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d64c9fe-26e0-455a-9200-cbc6c4af615f-workload-certs\") pod \"e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns\" (UID: \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\") " pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 01:54:57.861512 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:57.861475 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d64c9fe-26e0-455a-9200-cbc6c4af615f-istio-podinfo\") pod \"e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns\" (UID: \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\") " pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 01:54:57.861629 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:57.861518 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d64c9fe-26e0-455a-9200-cbc6c4af615f-workload-socket\") pod \"e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns\" (UID: \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\") " pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 01:54:57.861629 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:57.861592 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d64c9fe-26e0-455a-9200-cbc6c4af615f-credential-socket\") pod \"e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns\" (UID: \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\") " pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 01:54:57.861737 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:57.861629 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d64c9fe-26e0-455a-9200-cbc6c4af615f-istio-token\") pod \"e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns\" (UID: \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\") " pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 01:54:57.861737 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:57.861660 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d64c9fe-26e0-455a-9200-cbc6c4af615f-istiod-ca-cert\") pod \"e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns\" (UID: \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\") " pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 01:54:57.861737 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:57.861720 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qxh6\" (UniqueName: \"kubernetes.io/projected/6d64c9fe-26e0-455a-9200-cbc6c4af615f-kube-api-access-5qxh6\") pod \"e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns\" (UID: \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\") " pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 01:54:57.914177 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:57.914138 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:57.963005 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:57.962965 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d64c9fe-26e0-455a-9200-cbc6c4af615f-istio-podinfo\") pod \"e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns\" (UID: \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\") " pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 01:54:57.963224 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:57.963023 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d64c9fe-26e0-455a-9200-cbc6c4af615f-workload-socket\") pod \"e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns\" (UID: \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\") " pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 01:54:57.963224 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:57.963087 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d64c9fe-26e0-455a-9200-cbc6c4af615f-credential-socket\") pod \"e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns\" (UID: \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\") " pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 01:54:57.963224 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:57.963160 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d64c9fe-26e0-455a-9200-cbc6c4af615f-istio-token\") pod \"e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns\" (UID: \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\") " pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 01:54:57.963224 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:57.963204 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d64c9fe-26e0-455a-9200-cbc6c4af615f-istiod-ca-cert\") pod \"e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns\" (UID: \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\") " pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 01:54:57.963478 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:57.963258 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qxh6\" (UniqueName: \"kubernetes.io/projected/6d64c9fe-26e0-455a-9200-cbc6c4af615f-kube-api-access-5qxh6\") pod \"e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns\" (UID: \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\") " pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 01:54:57.963478 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:57.963317 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d64c9fe-26e0-455a-9200-cbc6c4af615f-istio-envoy\") pod \"e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns\" (UID: \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\") " pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 01:54:57.963478 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:57.963370 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d64c9fe-26e0-455a-9200-cbc6c4af615f-istio-data\") pod \"e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns\" (UID: \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\") " pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 01:54:57.963478 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:57.963443 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d64c9fe-26e0-455a-9200-cbc6c4af615f-workload-socket\") pod \"e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns\" (UID: \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\") " pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 01:54:57.963478 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:57.963458 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d64c9fe-26e0-455a-9200-cbc6c4af615f-workload-certs\") pod \"e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns\" (UID: \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\") " pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 01:54:57.963874 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:57.963692 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d64c9fe-26e0-455a-9200-cbc6c4af615f-credential-socket\") pod \"e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns\" (UID: \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\") " pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 01:54:57.963874 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:57.963765 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d64c9fe-26e0-455a-9200-cbc6c4af615f-workload-certs\") pod \"e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns\" (UID: \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\") " pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 01:54:57.964198 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:57.964177 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d64c9fe-26e0-455a-9200-cbc6c4af615f-istio-data\") pod \"e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns\" (UID: \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\") " pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 01:54:57.964293 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:57.964216 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d64c9fe-26e0-455a-9200-cbc6c4af615f-istiod-ca-cert\") pod \"e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns\" (UID: \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\") " pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 01:54:57.966028 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:57.965999 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d64c9fe-26e0-455a-9200-cbc6c4af615f-istio-envoy\") pod \"e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns\" (UID: \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\") " pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 01:54:57.966148 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:57.966092 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d64c9fe-26e0-455a-9200-cbc6c4af615f-istio-podinfo\") pod \"e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns\" (UID: \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\") " pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 01:54:57.982950 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:57.982910 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d64c9fe-26e0-455a-9200-cbc6c4af615f-istio-token\") pod \"e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns\" (UID: \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\") " pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 01:54:57.983115 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:57.982994 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qxh6\" (UniqueName: \"kubernetes.io/projected/6d64c9fe-26e0-455a-9200-cbc6c4af615f-kube-api-access-5qxh6\") pod \"e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns\" (UID: \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\") " pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 01:54:58.109849 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:58.104887 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 01:54:58.128805 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:58.126281 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:58.568000 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:58.567969 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns"] Oct 30 01:54:58.568504 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:54:58.568479 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d64c9fe_26e0_455a_9200_cbc6c4af615f.slice/crio-6897c9f32b2496fc1e9bc98ef49f5342fde920581d433aad66b38cadd484d9e0 WatchSource:0}: Error finding container 6897c9f32b2496fc1e9bc98ef49f5342fde920581d433aad66b38cadd484d9e0: Status 404 returned error can't find the container with id 6897c9f32b2496fc1e9bc98ef49f5342fde920581d433aad66b38cadd484d9e0 Oct 30 01:54:58.571467 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:58.571431 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:54:58.571569 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:58.571519 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:54:58.571569 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:58.571553 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:54:59.472730 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:59.472689 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" event={"ID":"6d64c9fe-26e0-455a-9200-cbc6c4af615f","Type":"ContainerStarted","Data":"a4cee2ef6ff23279e71e847d8abc9eb923541ae7755b1444569447768ddaa202"} Oct 30 01:54:59.472964 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:59.472774 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" event={"ID":"6d64c9fe-26e0-455a-9200-cbc6c4af615f","Type":"ContainerStarted","Data":"6897c9f32b2496fc1e9bc98ef49f5342fde920581d433aad66b38cadd484d9e0"} Oct 30 01:54:59.528227 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:59.528164 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" podStartSLOduration=2.5281468350000003 podStartE2EDuration="2.528146835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:54:59.526529366 +0000 UTC m=+2225.102954767" watchObservedRunningTime="2026-10-30 01:54:59.528146835 +0000 UTC m=+2225.104572238" Oct 30 01:54:59.817762 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:54:59.817658 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:00.105823 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:00.105711 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 01:55:00.112351 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:00.112322 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 01:55:00.481052 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:00.481014 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 01:55:00.482136 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:00.482113 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 01:55:02.575739 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:02.575709 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:02.761247 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:02.761212 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:03.583562 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:03.583529 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:03.874923 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:03.874834 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:04.870602 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:04.870567 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:05.094440 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:05.094401 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:05.176504 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:05.176464 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-093d23-6d8dccf8c9-428qh"] Oct 30 01:55:05.182313 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:05.182285 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-093d23-6d8dccf8c9-428qh" Oct 30 01:55:05.198698 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:05.198668 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-093d23-6d8dccf8c9-428qh"] Oct 30 01:55:05.250253 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:05.250220 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-093d23\"" Oct 30 01:55:05.250253 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:05.250234 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-093d23-dockercfg-hn79p\"" Oct 30 01:55:05.284564 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:05.284529 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z92l\" (UniqueName: \"kubernetes.io/projected/cd6ba6df-cc18-462b-a9f5-d34f1e4591fb-kube-api-access-7z92l\") pod \"payload-pre-processing-e2e-extmod-b-093d23-6d8dccf8c9-428qh\" (UID: \"cd6ba6df-cc18-462b-a9f5-d34f1e4591fb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-093d23-6d8dccf8c9-428qh" Oct 30 01:55:05.284861 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:05.284835 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd6ba6df-cc18-462b-a9f5-d34f1e4591fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-093d23-6d8dccf8c9-428qh\" (UID: \"cd6ba6df-cc18-462b-a9f5-d34f1e4591fb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-093d23-6d8dccf8c9-428qh" Oct 30 01:55:05.285994 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:05.285970 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:05.400776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:05.400084 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z92l\" (UniqueName: \"kubernetes.io/projected/cd6ba6df-cc18-462b-a9f5-d34f1e4591fb-kube-api-access-7z92l\") pod \"payload-pre-processing-e2e-extmod-b-093d23-6d8dccf8c9-428qh\" (UID: \"cd6ba6df-cc18-462b-a9f5-d34f1e4591fb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-093d23-6d8dccf8c9-428qh" Oct 30 01:55:05.400776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:05.400371 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd6ba6df-cc18-462b-a9f5-d34f1e4591fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-093d23-6d8dccf8c9-428qh\" (UID: \"cd6ba6df-cc18-462b-a9f5-d34f1e4591fb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-093d23-6d8dccf8c9-428qh" Oct 30 01:55:05.416779 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:05.401604 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd6ba6df-cc18-462b-a9f5-d34f1e4591fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-093d23-6d8dccf8c9-428qh\" (UID: \"cd6ba6df-cc18-462b-a9f5-d34f1e4591fb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-093d23-6d8dccf8c9-428qh" Oct 30 01:55:05.445922 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:05.445819 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z92l\" (UniqueName: \"kubernetes.io/projected/cd6ba6df-cc18-462b-a9f5-d34f1e4591fb-kube-api-access-7z92l\") pod \"payload-pre-processing-e2e-extmod-b-093d23-6d8dccf8c9-428qh\" (UID: \"cd6ba6df-cc18-462b-a9f5-d34f1e4591fb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-093d23-6d8dccf8c9-428qh" Oct 30 01:55:05.481781 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:05.480204 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:05.511072 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:05.511036 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-093d23-6d8dccf8c9-428qh" Oct 30 01:55:05.937441 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:55:05.937409 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd6ba6df_cc18_462b_a9f5_d34f1e4591fb.slice/crio-c8892552528a5fd104351e0977db4cc7cfe2f9f590fddfeaeead4cd5d9575bc1 WatchSource:0}: Error finding container c8892552528a5fd104351e0977db4cc7cfe2f9f590fddfeaeead4cd5d9575bc1: Status 404 returned error can't find the container with id c8892552528a5fd104351e0977db4cc7cfe2f9f590fddfeaeead4cd5d9575bc1 Oct 30 01:55:05.938262 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:05.938202 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-093d23-6d8dccf8c9-428qh"] Oct 30 01:55:06.171814 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:06.171778 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:06.378173 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:06.378077 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:06.649486 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:06.648685 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:06.653070 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:06.653038 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-093d23-6d8dccf8c9-428qh" event={"ID":"cd6ba6df-cc18-462b-a9f5-d34f1e4591fb","Type":"ContainerStarted","Data":"b36df82b679949861a63ecdccb59610809ce1c5a2c12f11a7ca89a3337329712"} Oct 30 01:55:06.653239 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:06.653098 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-093d23-6d8dccf8c9-428qh" event={"ID":"cd6ba6df-cc18-462b-a9f5-d34f1e4591fb","Type":"ContainerStarted","Data":"c8892552528a5fd104351e0977db4cc7cfe2f9f590fddfeaeead4cd5d9575bc1"} Oct 30 01:55:06.653335 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:06.653319 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-093d23-6d8dccf8c9-428qh" Oct 30 01:55:06.696910 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:06.696857 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-093d23-6d8dccf8c9-428qh" podStartSLOduration=1.696840227 podStartE2EDuration="1.696840227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:55:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:55:06.696456144 +0000 UTC m=+2232.272881546" watchObservedRunningTime="2026-10-30 01:55:06.696840227 +0000 UTC m=+2232.273265630" Oct 30 01:55:06.941854 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:06.941668 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:07.205415 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:07.205325 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:07.421601 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:07.421557 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:08.910700 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:08.910620 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:10.412777 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:10.412099 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:10.619782 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:10.618398 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:10.823599 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:10.823486 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:12.466817 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:12.466348 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:12.685537 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:12.685505 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:12.736995 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:12.736961 6572 generic.go:356] "Generic (PLEG): container finished" podID="57a6c761-0adf-4fcf-b4a5-1c8401957993" containerID="78a007e88a5b645bb18d9960a40bf7aef53d9b5ea63dd8ce0f599e527d84860d" exitCode=0 Oct 30 01:55:12.737211 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:12.737132 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" event={"ID":"57a6c761-0adf-4fcf-b4a5-1c8401957993","Type":"ContainerDied","Data":"78a007e88a5b645bb18d9960a40bf7aef53d9b5ea63dd8ce0f599e527d84860d"} Oct 30 01:55:12.738243 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:12.738220 6572 scope.go:122] "RemoveContainer" containerID="78a007e88a5b645bb18d9960a40bf7aef53d9b5ea63dd8ce0f599e527d84860d" Oct 30 01:55:12.740536 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:12.740502 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:55:12.740644 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:12.740594 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:55:12.740644 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:12.740637 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:55:13.077679 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:13.044830 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-cd8173-8445cbb698-7wtm5"] Oct 30 01:55:13.077679 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:13.055770 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:13.084504 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:13.083632 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-cd8173-8445cbb698-7wtm5"] Oct 30 01:55:13.084504 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:13.083822 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-cd8173-8445cbb698-7wtm5" Oct 30 01:55:13.102334 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:13.101487 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-cd8173-dockercfg-n8js8\"" Oct 30 01:55:13.102334 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:13.101516 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-cd8173\"" Oct 30 01:55:13.169389 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:13.169352 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/909b1459-46ea-4c3c-bc2e-1fb29527200a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-cd8173-8445cbb698-7wtm5\" (UID: \"909b1459-46ea-4c3c-bc2e-1fb29527200a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-cd8173-8445cbb698-7wtm5" Oct 30 01:55:13.169611 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:13.169591 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hdpg\" (UniqueName: \"kubernetes.io/projected/909b1459-46ea-4c3c-bc2e-1fb29527200a-kube-api-access-6hdpg\") pod \"payload-processing-e2e-extmod-b-cd8173-8445cbb698-7wtm5\" (UID: \"909b1459-46ea-4c3c-bc2e-1fb29527200a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-cd8173-8445cbb698-7wtm5" Oct 30 01:55:13.270540 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:13.270510 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hdpg\" (UniqueName: \"kubernetes.io/projected/909b1459-46ea-4c3c-bc2e-1fb29527200a-kube-api-access-6hdpg\") pod \"payload-processing-e2e-extmod-b-cd8173-8445cbb698-7wtm5\" (UID: \"909b1459-46ea-4c3c-bc2e-1fb29527200a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-cd8173-8445cbb698-7wtm5" Oct 30 01:55:13.270722 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:13.270645 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/909b1459-46ea-4c3c-bc2e-1fb29527200a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-cd8173-8445cbb698-7wtm5\" (UID: \"909b1459-46ea-4c3c-bc2e-1fb29527200a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-cd8173-8445cbb698-7wtm5" Oct 30 01:55:13.271616 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:13.271591 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/909b1459-46ea-4c3c-bc2e-1fb29527200a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-cd8173-8445cbb698-7wtm5\" (UID: \"909b1459-46ea-4c3c-bc2e-1fb29527200a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-cd8173-8445cbb698-7wtm5" Oct 30 01:55:13.290734 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:13.290706 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hdpg\" (UniqueName: \"kubernetes.io/projected/909b1459-46ea-4c3c-bc2e-1fb29527200a-kube-api-access-6hdpg\") pod \"payload-processing-e2e-extmod-b-cd8173-8445cbb698-7wtm5\" (UID: \"909b1459-46ea-4c3c-bc2e-1fb29527200a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-cd8173-8445cbb698-7wtm5" Oct 30 01:55:13.407212 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:13.407114 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-cd8173-8445cbb698-7wtm5" Oct 30 01:55:13.739859 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:13.738241 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-cd8173-8445cbb698-7wtm5"] Oct 30 01:55:13.740450 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:55:13.740419 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod909b1459_46ea_4c3c_bc2e_1fb29527200a.slice/crio-1ff3c9b50c63b15341ecb78f3c27a1db8bd9fbaf866b9b1318d1e477c1d9b5ff WatchSource:0}: Error finding container 1ff3c9b50c63b15341ecb78f3c27a1db8bd9fbaf866b9b1318d1e477c1d9b5ff: Status 404 returned error can't find the container with id 1ff3c9b50c63b15341ecb78f3c27a1db8bd9fbaf866b9b1318d1e477c1d9b5ff Oct 30 01:55:13.750556 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:13.750524 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" event={"ID":"57a6c761-0adf-4fcf-b4a5-1c8401957993","Type":"ContainerStarted","Data":"7df91a2698213ba57e7ac907e2d2f52db95052cbbe079e62732ecf33e09b57be"} Oct 30 01:55:14.258799 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:14.255392 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:14.258799 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:14.255922 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:55:14.261139 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:14.261060 6572 patch_prober.go:29] interesting pod/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body= Oct 30 01:55:14.261405 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:14.261331 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" podUID="57a6c761-0adf-4fcf-b4a5-1c8401957993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" Oct 30 01:55:14.690068 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:55:14.690029 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:55:14.776586 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:14.776532 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-cd8173-8445cbb698-7wtm5" event={"ID":"909b1459-46ea-4c3c-bc2e-1fb29527200a","Type":"ContainerStarted","Data":"62ea3c1253f7e97159102d95e5784501cc79b28778bed95989c3ae698979a14a"} Oct 30 01:55:14.777042 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:14.777019 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-cd8173-8445cbb698-7wtm5" Oct 30 01:55:14.777119 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:14.777073 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-cd8173-8445cbb698-7wtm5" event={"ID":"909b1459-46ea-4c3c-bc2e-1fb29527200a","Type":"ContainerStarted","Data":"1ff3c9b50c63b15341ecb78f3c27a1db8bd9fbaf866b9b1318d1e477c1d9b5ff"} Oct 30 01:55:15.247436 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:15.247397 6572 patch_prober.go:29] interesting pod/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body= Oct 30 01:55:15.247634 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:15.247463 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" podUID="57a6c761-0adf-4fcf-b4a5-1c8401957993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" Oct 30 01:55:15.529146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:15.524882 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:55:15.728976 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:15.727914 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:16.247385 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:16.247346 6572 patch_prober.go:29] interesting pod/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body= Oct 30 01:55:16.247813 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:16.247434 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" podUID="57a6c761-0adf-4fcf-b4a5-1c8401957993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" Oct 30 01:55:17.197238 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:17.197043 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:17.247148 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:17.247110 6572 patch_prober.go:29] interesting pod/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body= Oct 30 01:55:17.247342 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:17.247177 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" podUID="57a6c761-0adf-4fcf-b4a5-1c8401957993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" Oct 30 01:55:17.665499 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:17.665470 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-093d23-6d8dccf8c9-428qh" Oct 30 01:55:17.790390 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:17.706517 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-cd8173-8445cbb698-7wtm5" podStartSLOduration=5.706499799 podStartE2EDuration="5.706499799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:55:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:55:14.836556088 +0000 UTC m=+2240.412981494" watchObservedRunningTime="2026-10-30 01:55:17.706499799 +0000 UTC m=+2243.282925203" Oct 30 01:55:18.247355 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:18.247316 6572 patch_prober.go:29] interesting pod/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body= Oct 30 01:55:18.247560 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:18.247378 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" podUID="57a6c761-0adf-4fcf-b4a5-1c8401957993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" Oct 30 01:55:18.826819 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:18.821405 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:18.848109 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:18.848077 6572 generic.go:356] "Generic (PLEG): container finished" podID="22e0c074-1c61-435e-b086-0c06657d2ab9" containerID="a61525cca28467846d8814fad0e50b1779a391008b10a753031c8f7c52162699" exitCode=0 Oct 30 01:55:18.848307 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:18.848121 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" event={"ID":"22e0c074-1c61-435e-b086-0c06657d2ab9","Type":"ContainerDied","Data":"a61525cca28467846d8814fad0e50b1779a391008b10a753031c8f7c52162699"} Oct 30 01:55:18.848625 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:18.848605 6572 scope.go:122] "RemoveContainer" containerID="a61525cca28467846d8814fad0e50b1779a391008b10a753031c8f7c52162699" Oct 30 01:55:18.850790 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:18.850735 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:55:18.850887 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:18.850847 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:55:18.850931 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:18.850899 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:55:19.324615 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:19.323825 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:55:19.325893 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:19.325865 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:55:19.858266 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:19.858233 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" event={"ID":"22e0c074-1c61-435e-b086-0c06657d2ab9","Type":"ContainerStarted","Data":"3de5bdb45184edb4b47cf4041bfbcf0ba9cb5ee42b0383d6f481fa51c99b312d"} Oct 30 01:55:20.295181 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:20.295133 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:20.585249 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:20.585168 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 01:55:20.586579 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:20.586556 6572 patch_prober.go:29] interesting pod/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body= Oct 30 01:55:20.586645 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:20.586610 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" podUID="22e0c074-1c61-435e-b086-0c06657d2ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" Oct 30 01:55:20.773087 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:20.773048 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 01:55:21.584810 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:21.584733 6572 patch_prober.go:29] interesting pod/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body= Oct 30 01:55:21.585233 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:21.584840 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" podUID="22e0c074-1c61-435e-b086-0c06657d2ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" Oct 30 01:55:21.794108 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:21.794072 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:22.584828 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:22.584794 6572 patch_prober.go:29] interesting pod/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body= Oct 30 01:55:22.585256 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:22.584843 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" podUID="22e0c074-1c61-435e-b086-0c06657d2ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" Oct 30 01:55:23.348723 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:23.348448 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:23.585505 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:23.585469 6572 patch_prober.go:29] interesting pod/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body= Oct 30 01:55:23.585987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:23.585521 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" podUID="22e0c074-1c61-435e-b086-0c06657d2ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" Oct 30 01:55:24.043775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:24.037359 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:24.584858 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:24.584824 6572 patch_prober.go:29] interesting pod/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body= Oct 30 01:55:24.585057 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:24.584878 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" podUID="22e0c074-1c61-435e-b086-0c06657d2ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" Oct 30 01:55:25.138171 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:25.138135 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:25.357196 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:25.357157 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:25.566868 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:25.566837 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:25.789804 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:25.789736 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-cd8173-8445cbb698-7wtm5" Oct 30 01:55:25.835178 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:25.835093 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 01:55:25.837804 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:25.837779 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 01:55:25.887099 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:25.887057 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:26.094833 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:26.094717 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:26.648915 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:26.648878 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:26.845486 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:26.845335 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:27.079690 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:27.079658 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:27.958633 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:27.958593 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:28.137106 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:28.137065 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9"] Oct 30 01:55:28.140713 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:28.140689 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9" Oct 30 01:55:28.168631 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:28.168597 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1edaf\"/\"routing-overlay\"" Oct 30 01:55:28.168907 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:28.168622 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1edaf\"/\"kube-root-ca.crt\"" Oct 30 01:55:28.169019 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:28.168674 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1edaf\"/\"payload-processing-external-model-plugins-e2e-extmod-b-f1edaf\"" Oct 30 01:55:28.169384 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:28.168736 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1edaf\"/\"openshift-service-ca.crt\"" Oct 30 01:55:28.169384 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:28.169263 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f1edaf\"/\"payload-processing-external-model-e2e--a558def5-dockercfg-7chfl\"" Oct 30 01:55:28.169384 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:28.169353 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f1edaf\"/\"agc-cred-c798e484e2b4\"" Oct 30 01:55:28.176838 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:28.176451 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9"] Oct 30 01:55:28.273481 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:28.269225 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18bbd0d9-bc71-4f59-bc75-7a94156bca1a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9\" (UID: \"18bbd0d9-bc71-4f59-bc75-7a94156bca1a\") " pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9" Oct 30 01:55:28.273481 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:28.269362 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/18bbd0d9-bc71-4f59-bc75-7a94156bca1a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9\" (UID: \"18bbd0d9-bc71-4f59-bc75-7a94156bca1a\") " pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9" Oct 30 01:55:28.273481 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:28.269457 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/18bbd0d9-bc71-4f59-bc75-7a94156bca1a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9\" (UID: \"18bbd0d9-bc71-4f59-bc75-7a94156bca1a\") " pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9" Oct 30 01:55:28.273481 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:28.269492 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4r99\" (UniqueName: \"kubernetes.io/projected/18bbd0d9-bc71-4f59-bc75-7a94156bca1a-kube-api-access-s4r99\") pod \"payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9\" (UID: \"18bbd0d9-bc71-4f59-bc75-7a94156bca1a\") " pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9" Oct 30 01:55:28.370733 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:28.370701 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18bbd0d9-bc71-4f59-bc75-7a94156bca1a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9\" (UID: \"18bbd0d9-bc71-4f59-bc75-7a94156bca1a\") " pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9" Oct 30 01:55:28.371115 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:28.371095 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/18bbd0d9-bc71-4f59-bc75-7a94156bca1a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9\" (UID: \"18bbd0d9-bc71-4f59-bc75-7a94156bca1a\") " pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9" Oct 30 01:55:28.372563 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:28.372036 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/18bbd0d9-bc71-4f59-bc75-7a94156bca1a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9\" (UID: \"18bbd0d9-bc71-4f59-bc75-7a94156bca1a\") " pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9" Oct 30 01:55:28.372909 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:28.372460 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18bbd0d9-bc71-4f59-bc75-7a94156bca1a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9\" (UID: \"18bbd0d9-bc71-4f59-bc75-7a94156bca1a\") " pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9" Oct 30 01:55:28.373078 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:28.372555 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/18bbd0d9-bc71-4f59-bc75-7a94156bca1a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9\" (UID: \"18bbd0d9-bc71-4f59-bc75-7a94156bca1a\") " pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9" Oct 30 01:55:28.373291 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:28.373250 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4r99\" (UniqueName: \"kubernetes.io/projected/18bbd0d9-bc71-4f59-bc75-7a94156bca1a-kube-api-access-s4r99\") pod \"payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9\" (UID: \"18bbd0d9-bc71-4f59-bc75-7a94156bca1a\") " pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9" Oct 30 01:55:28.400275 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:28.400233 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/18bbd0d9-bc71-4f59-bc75-7a94156bca1a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9\" (UID: \"18bbd0d9-bc71-4f59-bc75-7a94156bca1a\") " pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9" Oct 30 01:55:28.415553 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:28.415517 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4r99\" (UniqueName: \"kubernetes.io/projected/18bbd0d9-bc71-4f59-bc75-7a94156bca1a-kube-api-access-s4r99\") pod \"payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9\" (UID: \"18bbd0d9-bc71-4f59-bc75-7a94156bca1a\") " pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9" Oct 30 01:55:28.461265 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:28.461230 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9" Oct 30 01:55:28.790084 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:28.784267 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9"] Oct 30 01:55:28.810701 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:28.810559 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:28.952065 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:28.952025 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9" event={"ID":"18bbd0d9-bc71-4f59-bc75-7a94156bca1a","Type":"ContainerStarted","Data":"5f53e04575bbddb672b1eae4b8b9abe7818166bbf2d979cd52e81f7a54f109e1"} Oct 30 01:55:28.952288 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:28.952088 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9" event={"ID":"18bbd0d9-bc71-4f59-bc75-7a94156bca1a","Type":"ContainerStarted","Data":"4aa8ddd1652cd8d05496a87501c4e1c93c693277ea83369b680c683a76e59c75"} Oct 30 01:55:28.952288 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:28.952115 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9" Oct 30 01:55:28.995354 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:28.995299 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9" podStartSLOduration=0.995285166 podStartE2EDuration="995.285166ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:55:28.991604834 +0000 UTC m=+2254.568030238" watchObservedRunningTime="2026-10-30 01:55:28.995285166 +0000 UTC m=+2254.571710569" Oct 30 01:55:30.009934 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:30.009901 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:30.209241 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:30.209194 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:30.702112 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:30.702070 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:30.902605 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:30.902570 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:31.443495 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:31.443450 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6033f115-d5rgn"] Oct 30 01:55:31.447601 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:31.447577 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6033f115-d5rgn" Oct 30 01:55:31.463988 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:31.463962 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6033f115-d5rgn"] Oct 30 01:55:31.469638 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:31.469607 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:55:31.469804 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:31.469612 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:55:31.469804 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:31.469653 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:55:31.469804 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:31.469653 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:55:31.523888 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:31.523851 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx9sm\" (UniqueName: \"kubernetes.io/projected/a63de485-78a7-4de4-b777-d2f3b085de13-kube-api-access-kx9sm\") pod \"maas-api-revoke-sub-6033f115-d5rgn\" (UID: \"a63de485-78a7-4de4-b777-d2f3b085de13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6033f115-d5rgn" Oct 30 01:55:31.524069 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:31.523910 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a63de485-78a7-4de4-b777-d2f3b085de13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6033f115-d5rgn\" (UID: \"a63de485-78a7-4de4-b777-d2f3b085de13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6033f115-d5rgn" Oct 30 01:55:31.524069 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:31.524031 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a63de485-78a7-4de4-b777-d2f3b085de13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6033f115-d5rgn\" (UID: \"a63de485-78a7-4de4-b777-d2f3b085de13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6033f115-d5rgn" Oct 30 01:55:31.624990 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:31.624957 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx9sm\" (UniqueName: \"kubernetes.io/projected/a63de485-78a7-4de4-b777-d2f3b085de13-kube-api-access-kx9sm\") pod \"maas-api-revoke-sub-6033f115-d5rgn\" (UID: \"a63de485-78a7-4de4-b777-d2f3b085de13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6033f115-d5rgn" Oct 30 01:55:31.625214 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:31.625032 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a63de485-78a7-4de4-b777-d2f3b085de13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6033f115-d5rgn\" (UID: \"a63de485-78a7-4de4-b777-d2f3b085de13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6033f115-d5rgn" Oct 30 01:55:31.625214 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:31.625146 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a63de485-78a7-4de4-b777-d2f3b085de13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6033f115-d5rgn\" (UID: \"a63de485-78a7-4de4-b777-d2f3b085de13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6033f115-d5rgn" Oct 30 01:55:31.625808 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:31.625783 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a63de485-78a7-4de4-b777-d2f3b085de13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6033f115-d5rgn\" (UID: \"a63de485-78a7-4de4-b777-d2f3b085de13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6033f115-d5rgn" Oct 30 01:55:31.627418 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:31.627399 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a63de485-78a7-4de4-b777-d2f3b085de13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6033f115-d5rgn\" (UID: \"a63de485-78a7-4de4-b777-d2f3b085de13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6033f115-d5rgn" Oct 30 01:55:31.643713 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:31.643682 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx9sm\" (UniqueName: \"kubernetes.io/projected/a63de485-78a7-4de4-b777-d2f3b085de13-kube-api-access-kx9sm\") pod \"maas-api-revoke-sub-6033f115-d5rgn\" (UID: \"a63de485-78a7-4de4-b777-d2f3b085de13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6033f115-d5rgn" Oct 30 01:55:31.763342 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:31.763254 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6033f115-d5rgn" Oct 30 01:55:32.035838 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:32.032121 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:32.097221 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:32.097186 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6033f115-d5rgn"] Oct 30 01:55:32.098334 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:55:32.098307 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda63de485_78a7_4de4_b777_d2f3b085de13.slice/crio-721d3a13fc57da1778c32f086560bbbe675975b6241a66369604fb1b74b4fd68 WatchSource:0}: Error finding container 721d3a13fc57da1778c32f086560bbbe675975b6241a66369604fb1b74b4fd68: Status 404 returned error can't find the container with id 721d3a13fc57da1778c32f086560bbbe675975b6241a66369604fb1b74b4fd68 Oct 30 01:55:32.211190 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:32.211157 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0835dd-mlt62"] Oct 30 01:55:32.215344 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:32.215319 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0835dd-mlt62" Oct 30 01:55:32.236906 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:32.236870 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0835dd-mlt62"] Oct 30 01:55:32.269435 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:32.269397 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/097b8384-7e8b-4acd-b44f-b9e2dc07628c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-0835dd-mlt62\" (UID: \"097b8384-7e8b-4acd-b44f-b9e2dc07628c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0835dd-mlt62" Oct 30 01:55:32.269647 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:32.269554 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szpg5\" (UniqueName: \"kubernetes.io/projected/097b8384-7e8b-4acd-b44f-b9e2dc07628c-kube-api-access-szpg5\") pod \"maas-api-revoke-keys-e2e-autores-0835dd-mlt62\" (UID: \"097b8384-7e8b-4acd-b44f-b9e2dc07628c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0835dd-mlt62" Oct 30 01:55:32.269724 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:32.269650 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/097b8384-7e8b-4acd-b44f-b9e2dc07628c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-0835dd-mlt62\" (UID: \"097b8384-7e8b-4acd-b44f-b9e2dc07628c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0835dd-mlt62" Oct 30 01:55:32.396823 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:32.396519 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/097b8384-7e8b-4acd-b44f-b9e2dc07628c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-0835dd-mlt62\" (UID: \"097b8384-7e8b-4acd-b44f-b9e2dc07628c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0835dd-mlt62" Oct 30 01:55:32.396823 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:32.396650 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szpg5\" (UniqueName: \"kubernetes.io/projected/097b8384-7e8b-4acd-b44f-b9e2dc07628c-kube-api-access-szpg5\") pod \"maas-api-revoke-keys-e2e-autores-0835dd-mlt62\" (UID: \"097b8384-7e8b-4acd-b44f-b9e2dc07628c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0835dd-mlt62" Oct 30 01:55:32.396823 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:32.396707 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/097b8384-7e8b-4acd-b44f-b9e2dc07628c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-0835dd-mlt62\" (UID: \"097b8384-7e8b-4acd-b44f-b9e2dc07628c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0835dd-mlt62" Oct 30 01:55:32.399890 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:32.398576 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/097b8384-7e8b-4acd-b44f-b9e2dc07628c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-0835dd-mlt62\" (UID: \"097b8384-7e8b-4acd-b44f-b9e2dc07628c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0835dd-mlt62" Oct 30 01:55:32.403678 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:32.403630 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/097b8384-7e8b-4acd-b44f-b9e2dc07628c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-0835dd-mlt62\" (UID: \"097b8384-7e8b-4acd-b44f-b9e2dc07628c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0835dd-mlt62" Oct 30 01:55:32.410998 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:32.410886 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:32.418666 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:32.418629 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szpg5\" (UniqueName: \"kubernetes.io/projected/097b8384-7e8b-4acd-b44f-b9e2dc07628c-kube-api-access-szpg5\") pod \"maas-api-revoke-keys-e2e-autores-0835dd-mlt62\" (UID: \"097b8384-7e8b-4acd-b44f-b9e2dc07628c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0835dd-mlt62" Oct 30 01:55:32.528706 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:32.528676 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0835dd-mlt62" Oct 30 01:55:32.832700 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:32.832669 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0835dd-mlt62"] Oct 30 01:55:32.833322 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:55:32.833284 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod097b8384_7e8b_4acd_b44f_b9e2dc07628c.slice/crio-0334b17cc244cb3326f5b6b6dd442f4f8c338976fafb0a4030e9e676fe8eab34 WatchSource:0}: Error finding container 0334b17cc244cb3326f5b6b6dd442f4f8c338976fafb0a4030e9e676fe8eab34: Status 404 returned error can't find the container with id 0334b17cc244cb3326f5b6b6dd442f4f8c338976fafb0a4030e9e676fe8eab34 Oct 30 01:55:32.989616 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:32.989574 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6033f115-d5rgn" event={"ID":"a63de485-78a7-4de4-b777-d2f3b085de13","Type":"ContainerStarted","Data":"26deb63e4e9da60299a5441ba926eed4619104eadd8938a96a65cde4ab737be9"} Oct 30 01:55:32.989844 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:32.989635 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6033f115-d5rgn" event={"ID":"a63de485-78a7-4de4-b777-d2f3b085de13","Type":"ContainerStarted","Data":"721d3a13fc57da1778c32f086560bbbe675975b6241a66369604fb1b74b4fd68"} Oct 30 01:55:32.990983 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:32.990958 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0835dd-mlt62" event={"ID":"097b8384-7e8b-4acd-b44f-b9e2dc07628c","Type":"ContainerStarted","Data":"0334b17cc244cb3326f5b6b6dd442f4f8c338976fafb0a4030e9e676fe8eab34"} Oct 30 01:55:33.031481 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:33.031423 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6033f115-d5rgn" podStartSLOduration=2.03140736 podStartE2EDuration="2.03140736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:55:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:55:33.026827472 +0000 UTC m=+2258.603252878" watchObservedRunningTime="2026-10-30 01:55:33.03140736 +0000 UTC m=+2258.607832761" Oct 30 01:55:34.000497 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:34.000462 6572 generic.go:356] "Generic (PLEG): container finished" podID="a63de485-78a7-4de4-b777-d2f3b085de13" containerID="26deb63e4e9da60299a5441ba926eed4619104eadd8938a96a65cde4ab737be9" exitCode=0 Oct 30 01:55:34.001114 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:34.000543 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6033f115-d5rgn" event={"ID":"a63de485-78a7-4de4-b777-d2f3b085de13","Type":"ContainerDied","Data":"26deb63e4e9da60299a5441ba926eed4619104eadd8938a96a65cde4ab737be9"} Oct 30 01:55:34.002419 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:34.002397 6572 generic.go:356] "Generic (PLEG): container finished" podID="097b8384-7e8b-4acd-b44f-b9e2dc07628c" containerID="2b6217efdd0c833fbea16563ef24e7ce09ac5bded090de2375fef42897c71705" exitCode=0 Oct 30 01:55:34.002539 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:34.002479 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0835dd-mlt62" event={"ID":"097b8384-7e8b-4acd-b44f-b9e2dc07628c","Type":"ContainerDied","Data":"2b6217efdd0c833fbea16563ef24e7ce09ac5bded090de2375fef42897c71705"} Oct 30 01:55:34.068995 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:34.068958 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:35.232720 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:35.232688 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6033f115-d5rgn" Oct 30 01:55:35.236845 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:35.236821 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0835dd-mlt62" Oct 30 01:55:35.339043 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:35.338935 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a63de485-78a7-4de4-b777-d2f3b085de13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a63de485-78a7-4de4-b777-d2f3b085de13-maas-api-openshift-ca\") pod \"a63de485-78a7-4de4-b777-d2f3b085de13\" (UID: \"a63de485-78a7-4de4-b777-d2f3b085de13\") " Oct 30 01:55:35.339043 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:35.339003 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/097b8384-7e8b-4acd-b44f-b9e2dc07628c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/097b8384-7e8b-4acd-b44f-b9e2dc07628c-maas-api-xks-ca\") pod \"097b8384-7e8b-4acd-b44f-b9e2dc07628c\" (UID: \"097b8384-7e8b-4acd-b44f-b9e2dc07628c\") " Oct 30 01:55:35.339316 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:35.339195 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/097b8384-7e8b-4acd-b44f-b9e2dc07628c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/097b8384-7e8b-4acd-b44f-b9e2dc07628c-maas-api-openshift-ca\") pod \"097b8384-7e8b-4acd-b44f-b9e2dc07628c\" (UID: \"097b8384-7e8b-4acd-b44f-b9e2dc07628c\") " Oct 30 01:55:35.339380 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:35.339336 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/097b8384-7e8b-4acd-b44f-b9e2dc07628c-kube-api-access-szpg5\" (UniqueName: \"kubernetes.io/projected/097b8384-7e8b-4acd-b44f-b9e2dc07628c-kube-api-access-szpg5\") pod \"097b8384-7e8b-4acd-b44f-b9e2dc07628c\" (UID: \"097b8384-7e8b-4acd-b44f-b9e2dc07628c\") " Oct 30 01:55:35.339380 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:35.339349 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a63de485-78a7-4de4-b777-d2f3b085de13-maas-api-openshift-ca" pod "a63de485-78a7-4de4-b777-d2f3b085de13" (UID: "a63de485-78a7-4de4-b777-d2f3b085de13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:55:35.339507 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:35.339399 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a63de485-78a7-4de4-b777-d2f3b085de13-kube-api-access-kx9sm\" (UniqueName: \"kubernetes.io/projected/a63de485-78a7-4de4-b777-d2f3b085de13-kube-api-access-kx9sm\") pod \"a63de485-78a7-4de4-b777-d2f3b085de13\" (UID: \"a63de485-78a7-4de4-b777-d2f3b085de13\") " Oct 30 01:55:35.339507 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:35.339475 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a63de485-78a7-4de4-b777-d2f3b085de13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a63de485-78a7-4de4-b777-d2f3b085de13-maas-api-xks-ca\") pod \"a63de485-78a7-4de4-b777-d2f3b085de13\" (UID: \"a63de485-78a7-4de4-b777-d2f3b085de13\") " Oct 30 01:55:35.339987 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:35.339858 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/097b8384-7e8b-4acd-b44f-b9e2dc07628c-maas-api-openshift-ca" pod "097b8384-7e8b-4acd-b44f-b9e2dc07628c" (UID: "097b8384-7e8b-4acd-b44f-b9e2dc07628c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:55:35.340195 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:35.340175 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a63de485-78a7-4de4-b777-d2f3b085de13-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:55:35.340276 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:35.340202 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/097b8384-7e8b-4acd-b44f-b9e2dc07628c-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:55:35.341945 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:35.341916 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/097b8384-7e8b-4acd-b44f-b9e2dc07628c-maas-api-xks-ca" pod "097b8384-7e8b-4acd-b44f-b9e2dc07628c" (UID: "097b8384-7e8b-4acd-b44f-b9e2dc07628c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:55:35.342133 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:35.342076 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/097b8384-7e8b-4acd-b44f-b9e2dc07628c-kube-api-access-szpg5" pod "097b8384-7e8b-4acd-b44f-b9e2dc07628c" (UID: "097b8384-7e8b-4acd-b44f-b9e2dc07628c"). InnerVolumeSpecName "kube-api-access-szpg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:55:35.342133 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:35.342118 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a63de485-78a7-4de4-b777-d2f3b085de13-maas-api-xks-ca" pod "a63de485-78a7-4de4-b777-d2f3b085de13" (UID: "a63de485-78a7-4de4-b777-d2f3b085de13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:55:35.342519 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:35.342492 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a63de485-78a7-4de4-b777-d2f3b085de13-kube-api-access-kx9sm" pod "a63de485-78a7-4de4-b777-d2f3b085de13" (UID: "a63de485-78a7-4de4-b777-d2f3b085de13"). InnerVolumeSpecName "kube-api-access-kx9sm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:55:35.442092 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:35.442053 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx9sm\" (UniqueName: \"kubernetes.io/projected/a63de485-78a7-4de4-b777-d2f3b085de13-kube-api-access-kx9sm\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:55:35.442092 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:35.442086 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a63de485-78a7-4de4-b777-d2f3b085de13-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:55:35.442092 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:35.442096 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/097b8384-7e8b-4acd-b44f-b9e2dc07628c-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:55:35.442092 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:35.442105 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szpg5\" (UniqueName: \"kubernetes.io/projected/097b8384-7e8b-4acd-b44f-b9e2dc07628c-kube-api-access-szpg5\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:55:35.621912 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:35.621819 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:36.020728 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:36.020692 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6033f115-d5rgn" Oct 30 01:55:36.020942 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:36.020695 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6033f115-d5rgn" event={"ID":"a63de485-78a7-4de4-b777-d2f3b085de13","Type":"ContainerDied","Data":"721d3a13fc57da1778c32f086560bbbe675975b6241a66369604fb1b74b4fd68"} Oct 30 01:55:36.020942 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:36.020847 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="721d3a13fc57da1778c32f086560bbbe675975b6241a66369604fb1b74b4fd68" Oct 30 01:55:36.022258 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:36.022228 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0835dd-mlt62" Oct 30 01:55:36.022359 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:36.022235 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0835dd-mlt62" event={"ID":"097b8384-7e8b-4acd-b44f-b9e2dc07628c","Type":"ContainerDied","Data":"0334b17cc244cb3326f5b6b6dd442f4f8c338976fafb0a4030e9e676fe8eab34"} Oct 30 01:55:36.022359 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:36.022341 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0334b17cc244cb3326f5b6b6dd442f4f8c338976fafb0a4030e9e676fe8eab34" Oct 30 01:55:37.119253 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:37.117662 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:38.687769 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:38.687051 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:39.964604 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:39.964572 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9" Oct 30 01:55:40.256154 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:40.256062 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:40.434235 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:40.434184 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:41.872426 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:41.872366 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:42.624484 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:42.624447 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-0835dd-df6f4cccb-mglq6"] Oct 30 01:55:42.840314 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:42.840277 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:43.062768 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:43.049411 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:43.464880 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:43.464840 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr"] Oct 30 01:55:43.540321 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:43.540286 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:43.725063 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:43.724970 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:43.916014 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:43.915964 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:44.688231 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:55:44.688194 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:55:45.382853 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:45.382810 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:46.931832 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:46.931788 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:47.625640 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:47.625594 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-0835dd-df6f4cccb-mglq6" podUID="f336af73-f4bf-4e67-94ed-0131538293f8" containerName="payload-processing" containerID="cri-o://c40317a8191c4da96ec8019fe18d93025f65641de43918d53c4d43f49effeb4f" gracePeriod=25 Oct 30 01:55:47.928675 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:47.928650 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-0835dd-df6f4cccb-mglq6" Oct 30 01:55:47.953207 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:47.953178 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f336af73-f4bf-4e67-94ed-0131538293f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f336af73-f4bf-4e67-94ed-0131538293f8-plugins-config-volume\") pod \"f336af73-f4bf-4e67-94ed-0131538293f8\" (UID: \"f336af73-f4bf-4e67-94ed-0131538293f8\") " Oct 30 01:55:47.953552 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:47.953288 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f336af73-f4bf-4e67-94ed-0131538293f8-kube-api-access-v7czm\" (UniqueName: \"kubernetes.io/projected/f336af73-f4bf-4e67-94ed-0131538293f8-kube-api-access-v7czm\") pod \"f336af73-f4bf-4e67-94ed-0131538293f8\" (UID: \"f336af73-f4bf-4e67-94ed-0131538293f8\") " Oct 30 01:55:47.953629 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:47.953605 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f336af73-f4bf-4e67-94ed-0131538293f8-plugins-config-volume" pod "f336af73-f4bf-4e67-94ed-0131538293f8" (UID: "f336af73-f4bf-4e67-94ed-0131538293f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:55:47.954000 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:47.953979 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f336af73-f4bf-4e67-94ed-0131538293f8-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:55:47.955650 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:47.955623 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f336af73-f4bf-4e67-94ed-0131538293f8-kube-api-access-v7czm" pod "f336af73-f4bf-4e67-94ed-0131538293f8" (UID: "f336af73-f4bf-4e67-94ed-0131538293f8"). InnerVolumeSpecName "kube-api-access-v7czm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:55:48.055002 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:48.054961 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7czm\" (UniqueName: \"kubernetes.io/projected/f336af73-f4bf-4e67-94ed-0131538293f8-kube-api-access-v7czm\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:55:48.174861 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:48.174819 6572 generic.go:356] "Generic (PLEG): container finished" podID="f336af73-f4bf-4e67-94ed-0131538293f8" containerID="c40317a8191c4da96ec8019fe18d93025f65641de43918d53c4d43f49effeb4f" exitCode=0 Oct 30 01:55:48.175045 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:48.174871 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-0835dd-df6f4cccb-mglq6" Oct 30 01:55:48.175045 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:48.174913 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-0835dd-df6f4cccb-mglq6" event={"ID":"f336af73-f4bf-4e67-94ed-0131538293f8","Type":"ContainerDied","Data":"c40317a8191c4da96ec8019fe18d93025f65641de43918d53c4d43f49effeb4f"} Oct 30 01:55:48.175045 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:48.174972 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-0835dd-df6f4cccb-mglq6" event={"ID":"f336af73-f4bf-4e67-94ed-0131538293f8","Type":"ContainerDied","Data":"f7a308b5b40880ebda02b3d4138576e43b0b1dcc8347286efd148d4556cf5497"} Oct 30 01:55:48.175045 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:48.175001 6572 scope.go:122] "RemoveContainer" containerID="c40317a8191c4da96ec8019fe18d93025f65641de43918d53c4d43f49effeb4f" Oct 30 01:55:48.186961 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:48.186940 6572 scope.go:122] "RemoveContainer" containerID="c40317a8191c4da96ec8019fe18d93025f65641de43918d53c4d43f49effeb4f" Oct 30 01:55:48.187253 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:55:48.187230 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c40317a8191c4da96ec8019fe18d93025f65641de43918d53c4d43f49effeb4f\": container with ID starting with c40317a8191c4da96ec8019fe18d93025f65641de43918d53c4d43f49effeb4f not found: ID does not exist" containerID="c40317a8191c4da96ec8019fe18d93025f65641de43918d53c4d43f49effeb4f" Oct 30 01:55:48.187301 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:48.187262 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c40317a8191c4da96ec8019fe18d93025f65641de43918d53c4d43f49effeb4f"} err="failed to get container status \"c40317a8191c4da96ec8019fe18d93025f65641de43918d53c4d43f49effeb4f\": rpc error: code = NotFound desc = could not find container \"c40317a8191c4da96ec8019fe18d93025f65641de43918d53c4d43f49effeb4f\": container with ID starting with c40317a8191c4da96ec8019fe18d93025f65641de43918d53c4d43f49effeb4f not found: ID does not exist" Oct 30 01:55:48.230868 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:48.227021 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-0835dd-df6f4cccb-mglq6"] Oct 30 01:55:48.241576 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:48.241541 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-0835dd-df6f4cccb-mglq6"] Oct 30 01:55:48.468432 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:48.468283 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr" podUID="14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6" containerName="payload-processing" containerID="cri-o://74e57d2eafe1e4466c4bb97fd69061e386683e10b4197852f5594ff827d04429" gracePeriod=25 Oct 30 01:55:48.511244 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:48.511214 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:48.543986 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:48.543952 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f336af73-f4bf-4e67-94ed-0131538293f8" path="/var/lib/kubelet/pods/f336af73-f4bf-4e67-94ed-0131538293f8/volumes" Oct 30 01:55:48.785491 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:48.785460 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr" Oct 30 01:55:48.866253 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:48.866210 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6-kube-api-access-228vh\" (UniqueName: \"kubernetes.io/projected/14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6-kube-api-access-228vh\") pod \"14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6\" (UID: \"14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6\") " Oct 30 01:55:48.866507 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:48.866360 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6-provider-credentials\") pod \"14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6\" (UID: \"14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6\") " Oct 30 01:55:48.866507 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:48.866440 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6-plugins-config-volume\") pod \"14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6\" (UID: \"14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6\") " Oct 30 01:55:48.866708 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:48.866599 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6-routing-overlay-volume\") pod \"14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6\" (UID: \"14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6\") " Oct 30 01:55:48.866965 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:48.866936 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6-plugins-config-volume" pod "14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6" (UID: "14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:55:48.867065 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:48.866946 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6-routing-overlay-volume" pod "14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6" (UID: "14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:55:48.867307 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:48.867287 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:55:48.867307 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:48.867307 6572 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6-routing-overlay-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:55:48.868888 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:48.868863 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6-kube-api-access-228vh" pod "14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6" (UID: "14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6"). InnerVolumeSpecName "kube-api-access-228vh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:55:48.869113 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:48.869090 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6-provider-credentials" pod "14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6" (UID: "14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:55:48.969197 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:48.969163 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-228vh\" (UniqueName: \"kubernetes.io/projected/14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6-kube-api-access-228vh\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:55:48.969197 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:48.969191 6572 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6-provider-credentials\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:55:49.185115 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:49.185077 6572 generic.go:356] "Generic (PLEG): container finished" podID="14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6" containerID="74e57d2eafe1e4466c4bb97fd69061e386683e10b4197852f5594ff827d04429" exitCode=0 Oct 30 01:55:49.185287 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:49.185242 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr" event={"ID":"14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6","Type":"ContainerDied","Data":"74e57d2eafe1e4466c4bb97fd69061e386683e10b4197852f5594ff827d04429"} Oct 30 01:55:49.185287 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:49.185280 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr" event={"ID":"14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6","Type":"ContainerDied","Data":"9d7fa3d85fa23090c460505af646bbffb9f4a6b2838c1d0cdcff91b11523fe7a"} Oct 30 01:55:49.185396 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:49.185300 6572 scope.go:122] "RemoveContainer" containerID="74e57d2eafe1e4466c4bb97fd69061e386683e10b4197852f5594ff827d04429" Oct 30 01:55:49.185439 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:49.185411 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr" Oct 30 01:55:49.200449 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:49.200430 6572 scope.go:122] "RemoveContainer" containerID="74e57d2eafe1e4466c4bb97fd69061e386683e10b4197852f5594ff827d04429" Oct 30 01:55:49.200683 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:55:49.200663 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74e57d2eafe1e4466c4bb97fd69061e386683e10b4197852f5594ff827d04429\": container with ID starting with 74e57d2eafe1e4466c4bb97fd69061e386683e10b4197852f5594ff827d04429 not found: ID does not exist" containerID="74e57d2eafe1e4466c4bb97fd69061e386683e10b4197852f5594ff827d04429" Oct 30 01:55:49.200817 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:49.200690 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74e57d2eafe1e4466c4bb97fd69061e386683e10b4197852f5594ff827d04429"} err="failed to get container status \"74e57d2eafe1e4466c4bb97fd69061e386683e10b4197852f5594ff827d04429\": rpc error: code = NotFound desc = could not find container \"74e57d2eafe1e4466c4bb97fd69061e386683e10b4197852f5594ff827d04429\": container with ID starting with 74e57d2eafe1e4466c4bb97fd69061e386683e10b4197852f5594ff827d04429 not found: ID does not exist" Oct 30 01:55:49.238579 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:49.238549 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr"] Oct 30 01:55:49.242791 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:49.242769 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-7468zs4fr"] Oct 30 01:55:49.828323 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:49.828288 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:50.133631 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:50.133553 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:50.404185 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:50.404090 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:50.544299 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:50.544265 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6" path="/var/lib/kubelet/pods/14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6/volumes" Oct 30 01:55:51.832411 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:51.832372 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:53.379772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:53.378847 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:54.956098 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:54.956063 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:56.269891 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:56.269860 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:56.468155 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:56.468118 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:56.657195 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:56.657098 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:56.876530 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:56.876491 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:57.095774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:57.095326 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:57.829216 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:57.829183 6572 scope.go:122] "RemoveContainer" containerID="5e9f59d3929642e70494b1fedfb1efec139fc7b8a1239fff6febdcecf9a98088" Oct 30 01:55:57.843548 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:57.843522 6572 scope.go:122] "RemoveContainer" containerID="a68738a7e5773d2d40590a6cdab0a288607fb621a3a8383d2c423e8b8e79ff62" Oct 30 01:55:57.873533 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:57.873496 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:58.051601 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:58.051557 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst"] Oct 30 01:55:58.052287 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:58.052262 6572 cpu_manager.go:414] "containerMap: removing container" podUID="14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6" containerName="payload-processing" Oct 30 01:55:58.052287 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:58.052283 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6" containerName="payload-processing" Oct 30 01:55:58.052476 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:58.052300 6572 cpu_manager.go:414] "containerMap: removing container" podUID="f336af73-f4bf-4e67-94ed-0131538293f8" containerName="payload-processing" Oct 30 01:55:58.052476 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:58.052309 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f336af73-f4bf-4e67-94ed-0131538293f8" containerName="payload-processing" Oct 30 01:55:58.052476 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:58.052412 6572 cpu_manager.go:414] "containerMap: removing container" podUID="097b8384-7e8b-4acd-b44f-b9e2dc07628c" containerName="revoke-keys" Oct 30 01:55:58.052476 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:58.052423 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="097b8384-7e8b-4acd-b44f-b9e2dc07628c" containerName="revoke-keys" Oct 30 01:55:58.052476 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:58.052438 6572 cpu_manager.go:414] "containerMap: removing container" podUID="a63de485-78a7-4de4-b777-d2f3b085de13" containerName="revoke-keys" Oct 30 01:55:58.052476 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:58.052446 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a63de485-78a7-4de4-b777-d2f3b085de13" containerName="revoke-keys" Oct 30 01:55:58.052690 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:58.052658 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="097b8384-7e8b-4acd-b44f-b9e2dc07628c" containerName="revoke-keys" Oct 30 01:55:58.052690 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:58.052670 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="a63de485-78a7-4de4-b777-d2f3b085de13" containerName="revoke-keys" Oct 30 01:55:58.052690 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:58.052687 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="14cf9ea1-29bc-4908-b97d-e4f6df1ebcf6" containerName="payload-processing" Oct 30 01:55:58.052872 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:58.052699 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="f336af73-f4bf-4e67-94ed-0131538293f8" containerName="payload-processing" Oct 30 01:55:58.058816 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:58.058792 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst" Oct 30 01:55:58.079336 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:58.079271 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst"] Oct 30 01:55:58.085983 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:58.085955 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-76a19e\"/\"kube-root-ca.crt\"" Oct 30 01:55:58.086137 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:58.086038 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-76a19e\"/\"payload-processing-external-model-e2e--70e20018-dockercfg-x2px9\"" Oct 30 01:55:58.086137 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:58.086107 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-76a19e\"/\"openshift-service-ca.crt\"" Oct 30 01:55:58.086700 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:58.086677 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-76a19e\"/\"e2e-cmp-legacy-7c99da-provider-cred\"" Oct 30 01:55:58.086873 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:58.086767 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-76a19e\"/\"routing-overlay\"" Oct 30 01:55:58.086873 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:58.086814 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-76a19e\"/\"payload-processing-external-model-plugins-e2e-extmod-a-76a19e\"" Oct 30 01:55:58.235679 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:58.235036 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc1f69a6-3a72-4be6-b33e-c3b87abffcde-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst\" (UID: \"cc1f69a6-3a72-4be6-b33e-c3b87abffcde\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst" Oct 30 01:55:58.235679 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:58.235122 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc1f69a6-3a72-4be6-b33e-c3b87abffcde-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst\" (UID: \"cc1f69a6-3a72-4be6-b33e-c3b87abffcde\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst" Oct 30 01:55:58.235679 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:58.235229 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfl7n\" (UniqueName: \"kubernetes.io/projected/cc1f69a6-3a72-4be6-b33e-c3b87abffcde-kube-api-access-dfl7n\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst\" (UID: \"cc1f69a6-3a72-4be6-b33e-c3b87abffcde\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst" Oct 30 01:55:58.235679 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:58.235326 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc1f69a6-3a72-4be6-b33e-c3b87abffcde-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst\" (UID: \"cc1f69a6-3a72-4be6-b33e-c3b87abffcde\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst" Oct 30 01:55:58.291777 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:58.283165 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:58.337373 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:58.337277 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc1f69a6-3a72-4be6-b33e-c3b87abffcde-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst\" (UID: \"cc1f69a6-3a72-4be6-b33e-c3b87abffcde\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst" Oct 30 01:55:58.337671 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:58.337623 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc1f69a6-3a72-4be6-b33e-c3b87abffcde-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst\" (UID: \"cc1f69a6-3a72-4be6-b33e-c3b87abffcde\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst" Oct 30 01:55:58.337926 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:58.337897 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfl7n\" (UniqueName: \"kubernetes.io/projected/cc1f69a6-3a72-4be6-b33e-c3b87abffcde-kube-api-access-dfl7n\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst\" (UID: \"cc1f69a6-3a72-4be6-b33e-c3b87abffcde\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst" Oct 30 01:55:58.338119 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:58.338084 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc1f69a6-3a72-4be6-b33e-c3b87abffcde-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst\" (UID: \"cc1f69a6-3a72-4be6-b33e-c3b87abffcde\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst" Oct 30 01:55:58.339428 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:58.339391 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc1f69a6-3a72-4be6-b33e-c3b87abffcde-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst\" (UID: \"cc1f69a6-3a72-4be6-b33e-c3b87abffcde\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst" Oct 30 01:55:58.340292 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:58.340272 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc1f69a6-3a72-4be6-b33e-c3b87abffcde-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst\" (UID: \"cc1f69a6-3a72-4be6-b33e-c3b87abffcde\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst" Oct 30 01:55:58.344395 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:58.344333 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc1f69a6-3a72-4be6-b33e-c3b87abffcde-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst\" (UID: \"cc1f69a6-3a72-4be6-b33e-c3b87abffcde\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst" Oct 30 01:55:58.378773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:58.378689 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfl7n\" (UniqueName: \"kubernetes.io/projected/cc1f69a6-3a72-4be6-b33e-c3b87abffcde-kube-api-access-dfl7n\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst\" (UID: \"cc1f69a6-3a72-4be6-b33e-c3b87abffcde\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst" Oct 30 01:55:58.674101 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:58.674061 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst" Oct 30 01:55:58.986686 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:55:58.986655 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc1f69a6_3a72_4be6_b33e_c3b87abffcde.slice/crio-3ac505a8d9fbf67b6398015ebc16a85cc657246b002afe90cd12d3b2652ca1b4 WatchSource:0}: Error finding container 3ac505a8d9fbf67b6398015ebc16a85cc657246b002afe90cd12d3b2652ca1b4: Status 404 returned error can't find the container with id 3ac505a8d9fbf67b6398015ebc16a85cc657246b002afe90cd12d3b2652ca1b4 Oct 30 01:55:58.987022 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:58.986770 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst"] Oct 30 01:55:59.318848 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:59.316495 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst" event={"ID":"cc1f69a6-3a72-4be6-b33e-c3b87abffcde","Type":"ContainerStarted","Data":"e0e51a8b3113f9dbe9c8be6dc0302cffaa00d2a21f18f821c1d422693c4350b0"} Oct 30 01:55:59.318848 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:59.316566 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst" event={"ID":"cc1f69a6-3a72-4be6-b33e-c3b87abffcde","Type":"ContainerStarted","Data":"3ac505a8d9fbf67b6398015ebc16a85cc657246b002afe90cd12d3b2652ca1b4"} Oct 30 01:55:59.318848 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:59.318696 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst" Oct 30 01:55:59.377847 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:59.375037 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst" podStartSLOduration=1.375017156 podStartE2EDuration="1.375017156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:55:59.367169066 +0000 UTC m=+2284.943594479" watchObservedRunningTime="2026-10-30 01:55:59.375017156 +0000 UTC m=+2284.951442559" Oct 30 01:55:59.966516 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:55:59.966456 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:01.492259 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:01.492218 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:02.991286 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:02.991256 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:04.444428 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:04.444392 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:06.013714 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:06.013676 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:06.464553 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:06.464520 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:07.475168 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:07.475060 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:09.064075 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:09.064036 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:10.429718 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:10.429688 6572 generic.go:356] "Generic (PLEG): container finished" podID="6d64c9fe-26e0-455a-9200-cbc6c4af615f" containerID="a4cee2ef6ff23279e71e847d8abc9eb923541ae7755b1444569447768ddaa202" exitCode=0 Oct 30 01:56:10.430165 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:10.429770 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" event={"ID":"6d64c9fe-26e0-455a-9200-cbc6c4af615f","Type":"ContainerDied","Data":"a4cee2ef6ff23279e71e847d8abc9eb923541ae7755b1444569447768ddaa202"} Oct 30 01:56:10.430165 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:10.430159 6572 scope.go:122] "RemoveContainer" containerID="a4cee2ef6ff23279e71e847d8abc9eb923541ae7755b1444569447768ddaa202" Oct 30 01:56:10.431911 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:10.431880 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:56:10.432007 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:10.431955 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:56:10.432007 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:10.431992 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:56:10.694638 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:10.694549 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:11.358791 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:11.358759 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst" Oct 30 01:56:11.451961 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:11.451918 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" event={"ID":"6d64c9fe-26e0-455a-9200-cbc6c4af615f","Type":"ContainerStarted","Data":"5bf477f73c3ee2494fc39442aa1b0aa6bf8b7074497e57d7b237f0c4213c6dcc"} Oct 30 01:56:12.127771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:12.126811 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 01:56:12.128129 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:12.128106 6572 patch_prober.go:29] interesting pod/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body= Oct 30 01:56:12.128319 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:12.128291 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" podUID="6d64c9fe-26e0-455a-9200-cbc6c4af615f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" Oct 30 01:56:12.179227 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:12.179193 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:13.120775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:13.106007 6572 patch_prober.go:29] interesting pod/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body= Oct 30 01:56:13.120775 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:13.106063 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" podUID="6d64c9fe-26e0-455a-9200-cbc6c4af615f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" Oct 30 01:56:13.670995 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:13.670195 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:14.114826 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:14.111087 6572 patch_prober.go:29] interesting pod/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body= Oct 30 01:56:14.114826 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:14.111148 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" podUID="6d64c9fe-26e0-455a-9200-cbc6c4af615f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" Oct 30 01:56:14.689566 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:56:14.689523 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:56:15.105733 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:15.105693 6572 patch_prober.go:29] interesting pod/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body= Oct 30 01:56:15.105960 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:15.105781 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" podUID="6d64c9fe-26e0-455a-9200-cbc6c4af615f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" Oct 30 01:56:15.203039 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:15.202996 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:15.481034 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:15.480947 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 01:56:16.105664 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:16.105625 6572 patch_prober.go:29] interesting pod/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body= Oct 30 01:56:16.106207 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:16.105684 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" podUID="6d64c9fe-26e0-455a-9200-cbc6c4af615f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" Oct 30 01:56:16.799834 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:16.799799 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:17.111276 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:17.111198 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 01:56:17.112359 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:17.112339 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 01:56:18.125179 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:18.125144 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:18.686194 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:18.686157 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:18.736392 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:18.736361 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0835dd-mlt62"] Oct 30 01:56:18.740211 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:18.740191 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0835dd-mlt62"] Oct 30 01:56:18.871120 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:18.871084 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:19.323951 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:19.323918 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:20.580219 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:20.577706 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="097b8384-7e8b-4acd-b44f-b9e2dc07628c" path="/var/lib/kubelet/pods/097b8384-7e8b-4acd-b44f-b9e2dc07628c/volumes" Oct 30 01:56:20.597726 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:20.597679 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:22.117482 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:22.117433 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:23.574801 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:23.574726 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:23.942368 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:23.942332 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:25.155566 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:25.155532 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:25.678913 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:25.678873 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:25.869437 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:25.869396 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:26.061529 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:26.061429 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5f5417b-d69pm"] Oct 30 01:56:26.065338 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:26.065304 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:26.175105 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:26.175051 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5f5417b-d69pm"] Oct 30 01:56:26.254731 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:26.254692 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps"] Oct 30 01:56:26.254972 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:26.254937 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" podUID="57a6c761-0adf-4fcf-b4a5-1c8401957993" containerName="istio-proxy" containerID="cri-o://7df91a2698213ba57e7ac907e2d2f52db95052cbbe079e62732ecf33e09b57be" gracePeriod=30 Oct 30 01:56:26.356828 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:26.355340 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:26.545016 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:26.544975 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65e7f967-443d-44d1-8f82-3ccdd3aecff8" path="/var/lib/kubelet/pods/65e7f967-443d-44d1-8f82-3ccdd3aecff8/volumes" Oct 30 01:56:26.594437 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:26.594390 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:26.824790 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:26.824730 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:27.305430 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:27.305378 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:27.936146 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:27.935330 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:28.295826 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:28.295158 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:28.845788 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:28.845376 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:29.048616 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:29.048581 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:29.244360 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:29.244329 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:30.596387 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:30.596333 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:31.534039 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.533982 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:56:31.619637 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.619607 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57a6c761-0adf-4fcf-b4a5-1c8401957993-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a6c761-0adf-4fcf-b4a5-1c8401957993-credential-socket\") pod \"57a6c761-0adf-4fcf-b4a5-1c8401957993\" (UID: \"57a6c761-0adf-4fcf-b4a5-1c8401957993\") " Oct 30 01:56:31.620114 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.619712 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57a6c761-0adf-4fcf-b4a5-1c8401957993-kube-api-access-zss8r\" (UniqueName: \"kubernetes.io/projected/57a6c761-0adf-4fcf-b4a5-1c8401957993-kube-api-access-zss8r\") pod \"57a6c761-0adf-4fcf-b4a5-1c8401957993\" (UID: \"57a6c761-0adf-4fcf-b4a5-1c8401957993\") " Oct 30 01:56:31.620114 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.619805 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57a6c761-0adf-4fcf-b4a5-1c8401957993-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57a6c761-0adf-4fcf-b4a5-1c8401957993-istio-envoy\") pod \"57a6c761-0adf-4fcf-b4a5-1c8401957993\" (UID: \"57a6c761-0adf-4fcf-b4a5-1c8401957993\") " Oct 30 01:56:31.620114 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.619905 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57a6c761-0adf-4fcf-b4a5-1c8401957993-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57a6c761-0adf-4fcf-b4a5-1c8401957993-istiod-ca-cert\") pod \"57a6c761-0adf-4fcf-b4a5-1c8401957993\" (UID: \"57a6c761-0adf-4fcf-b4a5-1c8401957993\") " Oct 30 01:56:31.620114 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.619940 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57a6c761-0adf-4fcf-b4a5-1c8401957993-credential-socket" pod "57a6c761-0adf-4fcf-b4a5-1c8401957993" (UID: "57a6c761-0adf-4fcf-b4a5-1c8401957993"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:56:31.620114 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.619951 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57a6c761-0adf-4fcf-b4a5-1c8401957993-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a6c761-0adf-4fcf-b4a5-1c8401957993-workload-socket\") pod \"57a6c761-0adf-4fcf-b4a5-1c8401957993\" (UID: \"57a6c761-0adf-4fcf-b4a5-1c8401957993\") " Oct 30 01:56:31.620114 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.619986 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57a6c761-0adf-4fcf-b4a5-1c8401957993-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57a6c761-0adf-4fcf-b4a5-1c8401957993-istio-podinfo\") pod \"57a6c761-0adf-4fcf-b4a5-1c8401957993\" (UID: \"57a6c761-0adf-4fcf-b4a5-1c8401957993\") " Oct 30 01:56:31.620114 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.620012 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57a6c761-0adf-4fcf-b4a5-1c8401957993-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57a6c761-0adf-4fcf-b4a5-1c8401957993-workload-certs\") pod \"57a6c761-0adf-4fcf-b4a5-1c8401957993\" (UID: \"57a6c761-0adf-4fcf-b4a5-1c8401957993\") " Oct 30 01:56:31.620114 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.620055 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57a6c761-0adf-4fcf-b4a5-1c8401957993-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57a6c761-0adf-4fcf-b4a5-1c8401957993-istio-data\") pod \"57a6c761-0adf-4fcf-b4a5-1c8401957993\" (UID: \"57a6c761-0adf-4fcf-b4a5-1c8401957993\") " Oct 30 01:56:31.620114 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.620107 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57a6c761-0adf-4fcf-b4a5-1c8401957993-istio-token\" (UniqueName: \"kubernetes.io/projected/57a6c761-0adf-4fcf-b4a5-1c8401957993-istio-token\") pod \"57a6c761-0adf-4fcf-b4a5-1c8401957993\" (UID: \"57a6c761-0adf-4fcf-b4a5-1c8401957993\") " Oct 30 01:56:31.620669 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.620396 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57a6c761-0adf-4fcf-b4a5-1c8401957993-workload-socket" pod "57a6c761-0adf-4fcf-b4a5-1c8401957993" (UID: "57a6c761-0adf-4fcf-b4a5-1c8401957993"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:56:31.620669 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.620638 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57a6c761-0adf-4fcf-b4a5-1c8401957993-istiod-ca-cert" pod "57a6c761-0adf-4fcf-b4a5-1c8401957993" (UID: "57a6c761-0adf-4fcf-b4a5-1c8401957993"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:56:31.620871 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.620776 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57a6c761-0adf-4fcf-b4a5-1c8401957993-istio-data" pod "57a6c761-0adf-4fcf-b4a5-1c8401957993" (UID: "57a6c761-0adf-4fcf-b4a5-1c8401957993"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:56:31.620871 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.620848 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57a6c761-0adf-4fcf-b4a5-1c8401957993-workload-certs" pod "57a6c761-0adf-4fcf-b4a5-1c8401957993" (UID: "57a6c761-0adf-4fcf-b4a5-1c8401957993"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:56:31.621096 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.621072 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57a6c761-0adf-4fcf-b4a5-1c8401957993-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:56:31.621167 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.621102 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a6c761-0adf-4fcf-b4a5-1c8401957993-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:56:31.621167 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.621119 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57a6c761-0adf-4fcf-b4a5-1c8401957993-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:56:31.621167 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.621134 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57a6c761-0adf-4fcf-b4a5-1c8401957993-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:56:31.621167 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.621149 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a6c761-0adf-4fcf-b4a5-1c8401957993-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:56:31.622807 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.622779 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57a6c761-0adf-4fcf-b4a5-1c8401957993-istio-envoy" pod "57a6c761-0adf-4fcf-b4a5-1c8401957993" (UID: "57a6c761-0adf-4fcf-b4a5-1c8401957993"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:56:31.622807 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.622795 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57a6c761-0adf-4fcf-b4a5-1c8401957993-kube-api-access-zss8r" pod "57a6c761-0adf-4fcf-b4a5-1c8401957993" (UID: "57a6c761-0adf-4fcf-b4a5-1c8401957993"). InnerVolumeSpecName "kube-api-access-zss8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:56:31.623444 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.623414 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57a6c761-0adf-4fcf-b4a5-1c8401957993-istio-token" pod "57a6c761-0adf-4fcf-b4a5-1c8401957993" (UID: "57a6c761-0adf-4fcf-b4a5-1c8401957993"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:56:31.629079 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.629053 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57a6c761-0adf-4fcf-b4a5-1c8401957993-istio-podinfo" pod "57a6c761-0adf-4fcf-b4a5-1c8401957993" (UID: "57a6c761-0adf-4fcf-b4a5-1c8401957993"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:56:31.652276 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.652247 6572 generic.go:356] "Generic (PLEG): container finished" podID="57a6c761-0adf-4fcf-b4a5-1c8401957993" containerID="7df91a2698213ba57e7ac907e2d2f52db95052cbbe079e62732ecf33e09b57be" exitCode=0 Oct 30 01:56:31.652370 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.652295 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" Oct 30 01:56:31.652370 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.652326 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" event={"ID":"57a6c761-0adf-4fcf-b4a5-1c8401957993","Type":"ContainerDied","Data":"7df91a2698213ba57e7ac907e2d2f52db95052cbbe079e62732ecf33e09b57be"} Oct 30 01:56:31.652459 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.652384 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps" event={"ID":"57a6c761-0adf-4fcf-b4a5-1c8401957993","Type":"ContainerDied","Data":"5a21cc359980029aa617bc742bddc6a6b2d24af9d938fd585ceda04e6626148e"} Oct 30 01:56:31.652459 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.652415 6572 scope.go:122] "RemoveContainer" containerID="7df91a2698213ba57e7ac907e2d2f52db95052cbbe079e62732ecf33e09b57be" Oct 30 01:56:31.663715 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.663698 6572 scope.go:122] "RemoveContainer" containerID="78a007e88a5b645bb18d9960a40bf7aef53d9b5ea63dd8ce0f599e527d84860d" Oct 30 01:56:31.674216 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.674195 6572 scope.go:122] "RemoveContainer" containerID="7df91a2698213ba57e7ac907e2d2f52db95052cbbe079e62732ecf33e09b57be" Oct 30 01:56:31.674480 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:56:31.674461 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7df91a2698213ba57e7ac907e2d2f52db95052cbbe079e62732ecf33e09b57be\": container with ID starting with 7df91a2698213ba57e7ac907e2d2f52db95052cbbe079e62732ecf33e09b57be not found: ID does not exist" containerID="7df91a2698213ba57e7ac907e2d2f52db95052cbbe079e62732ecf33e09b57be" Oct 30 01:56:31.674556 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.674490 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7df91a2698213ba57e7ac907e2d2f52db95052cbbe079e62732ecf33e09b57be"} err="failed to get container status \"7df91a2698213ba57e7ac907e2d2f52db95052cbbe079e62732ecf33e09b57be\": rpc error: code = NotFound desc = could not find container \"7df91a2698213ba57e7ac907e2d2f52db95052cbbe079e62732ecf33e09b57be\": container with ID starting with 7df91a2698213ba57e7ac907e2d2f52db95052cbbe079e62732ecf33e09b57be not found: ID does not exist" Oct 30 01:56:31.674556 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.674514 6572 scope.go:122] "RemoveContainer" containerID="78a007e88a5b645bb18d9960a40bf7aef53d9b5ea63dd8ce0f599e527d84860d" Oct 30 01:56:31.674781 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:56:31.674760 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78a007e88a5b645bb18d9960a40bf7aef53d9b5ea63dd8ce0f599e527d84860d\": container with ID starting with 78a007e88a5b645bb18d9960a40bf7aef53d9b5ea63dd8ce0f599e527d84860d not found: ID does not exist" containerID="78a007e88a5b645bb18d9960a40bf7aef53d9b5ea63dd8ce0f599e527d84860d" Oct 30 01:56:31.674831 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.674787 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78a007e88a5b645bb18d9960a40bf7aef53d9b5ea63dd8ce0f599e527d84860d"} err="failed to get container status \"78a007e88a5b645bb18d9960a40bf7aef53d9b5ea63dd8ce0f599e527d84860d\": rpc error: code = NotFound desc = could not find container \"78a007e88a5b645bb18d9960a40bf7aef53d9b5ea63dd8ce0f599e527d84860d\": container with ID starting with 78a007e88a5b645bb18d9960a40bf7aef53d9b5ea63dd8ce0f599e527d84860d not found: ID does not exist" Oct 30 01:56:31.705091 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.705065 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps"] Oct 30 01:56:31.714509 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.714484 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-0835dd-openshift-default-6fdf9cbb7-x4cps"] Oct 30 01:56:31.722780 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.722731 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zss8r\" (UniqueName: \"kubernetes.io/projected/57a6c761-0adf-4fcf-b4a5-1c8401957993-kube-api-access-zss8r\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:56:31.722780 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.722778 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57a6c761-0adf-4fcf-b4a5-1c8401957993-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:56:31.722953 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.722795 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57a6c761-0adf-4fcf-b4a5-1c8401957993-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:56:31.722953 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:31.722808 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57a6c761-0adf-4fcf-b4a5-1c8401957993-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:56:32.243530 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:32.243495 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:32.437817 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:32.437786 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:32.545980 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:32.545902 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57a6c761-0adf-4fcf-b4a5-1c8401957993" path="/var/lib/kubelet/pods/57a6c761-0adf-4fcf-b4a5-1c8401957993/volumes" Oct 30 01:56:33.384260 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:33.384228 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:33.591260 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:33.591226 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:33.795966 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:33.795932 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:34.142645 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:34.142561 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:34.379626 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:34.379589 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:34.854971 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:34.854933 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:35.064838 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:35.064801 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:35.121311 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:35.121233 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-817fd7-6f9f8d5c7f-x9gmq"] Oct 30 01:56:35.122017 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:35.121997 6572 cpu_manager.go:414] "containerMap: removing container" podUID="57a6c761-0adf-4fcf-b4a5-1c8401957993" containerName="istio-proxy" Oct 30 01:56:35.122017 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:35.122017 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57a6c761-0adf-4fcf-b4a5-1c8401957993" containerName="istio-proxy" Oct 30 01:56:35.122185 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:35.122049 6572 cpu_manager.go:414] "containerMap: removing container" podUID="57a6c761-0adf-4fcf-b4a5-1c8401957993" containerName="istio-proxy" Oct 30 01:56:35.122185 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:35.122054 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57a6c761-0adf-4fcf-b4a5-1c8401957993" containerName="istio-proxy" Oct 30 01:56:35.122263 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:35.122202 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="57a6c761-0adf-4fcf-b4a5-1c8401957993" containerName="istio-proxy" Oct 30 01:56:35.127542 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:35.127522 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-817fd7-6f9f8d5c7f-x9gmq" Oct 30 01:56:35.145560 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:35.145536 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-817fd7-6f9f8d5c7f-x9gmq"] Oct 30 01:56:35.148844 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:35.148823 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-817fd7-dockercfg-5cg7s\"" Oct 30 01:56:35.148981 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:35.148881 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-817fd7\"" Oct 30 01:56:35.209811 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:35.209774 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-817fd7-7dfc855d4b-4b72d"] Oct 30 01:56:35.211061 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:35.211025 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="57a6c761-0adf-4fcf-b4a5-1c8401957993" containerName="istio-proxy" Oct 30 01:56:35.218431 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:35.218391 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fbdq\" (UniqueName: \"kubernetes.io/projected/2db1738c-aff2-4494-a0a7-adb1d8a25ee2-kube-api-access-6fbdq\") pod \"payload-pre-processing-e2e-extmod-a-817fd7-6f9f8d5c7f-x9gmq\" (UID: \"2db1738c-aff2-4494-a0a7-adb1d8a25ee2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-817fd7-6f9f8d5c7f-x9gmq" Oct 30 01:56:35.219163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:35.219138 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2db1738c-aff2-4494-a0a7-adb1d8a25ee2-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-817fd7-6f9f8d5c7f-x9gmq\" (UID: \"2db1738c-aff2-4494-a0a7-adb1d8a25ee2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-817fd7-6f9f8d5c7f-x9gmq" Oct 30 01:56:35.254076 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:35.254047 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-817fd7-7dfc855d4b-4b72d" Oct 30 01:56:35.282633 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:35.282602 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:35.321335 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:35.321303 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jplk2\" (UniqueName: \"kubernetes.io/projected/874b4b98-1b9b-4c67-8204-fa621edf5b55-kube-api-access-jplk2\") pod \"payload-processing-e2e-extmod-a-817fd7-7dfc855d4b-4b72d\" (UID: \"874b4b98-1b9b-4c67-8204-fa621edf5b55\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-817fd7-7dfc855d4b-4b72d" Oct 30 01:56:35.321567 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:35.321546 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fbdq\" (UniqueName: \"kubernetes.io/projected/2db1738c-aff2-4494-a0a7-adb1d8a25ee2-kube-api-access-6fbdq\") pod \"payload-pre-processing-e2e-extmod-a-817fd7-6f9f8d5c7f-x9gmq\" (UID: \"2db1738c-aff2-4494-a0a7-adb1d8a25ee2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-817fd7-6f9f8d5c7f-x9gmq" Oct 30 01:56:35.321795 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:35.321777 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2db1738c-aff2-4494-a0a7-adb1d8a25ee2-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-817fd7-6f9f8d5c7f-x9gmq\" (UID: \"2db1738c-aff2-4494-a0a7-adb1d8a25ee2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-817fd7-6f9f8d5c7f-x9gmq" Oct 30 01:56:35.321979 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:35.321963 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/874b4b98-1b9b-4c67-8204-fa621edf5b55-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-817fd7-7dfc855d4b-4b72d\" (UID: \"874b4b98-1b9b-4c67-8204-fa621edf5b55\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-817fd7-7dfc855d4b-4b72d" Oct 30 01:56:35.346350 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:35.322487 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-817fd7-7dfc855d4b-4b72d"] Oct 30 01:56:35.347684 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:35.323344 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2db1738c-aff2-4494-a0a7-adb1d8a25ee2-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-817fd7-6f9f8d5c7f-x9gmq\" (UID: \"2db1738c-aff2-4494-a0a7-adb1d8a25ee2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-817fd7-6f9f8d5c7f-x9gmq" Oct 30 01:56:35.347919 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:35.347548 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fbdq\" (UniqueName: \"kubernetes.io/projected/2db1738c-aff2-4494-a0a7-adb1d8a25ee2-kube-api-access-6fbdq\") pod \"payload-pre-processing-e2e-extmod-a-817fd7-6f9f8d5c7f-x9gmq\" (UID: \"2db1738c-aff2-4494-a0a7-adb1d8a25ee2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-817fd7-6f9f8d5c7f-x9gmq" Oct 30 01:56:35.423389 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:35.423354 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/874b4b98-1b9b-4c67-8204-fa621edf5b55-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-817fd7-7dfc855d4b-4b72d\" (UID: \"874b4b98-1b9b-4c67-8204-fa621edf5b55\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-817fd7-7dfc855d4b-4b72d" Oct 30 01:56:35.423572 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:35.423487 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jplk2\" (UniqueName: \"kubernetes.io/projected/874b4b98-1b9b-4c67-8204-fa621edf5b55-kube-api-access-jplk2\") pod \"payload-processing-e2e-extmod-a-817fd7-7dfc855d4b-4b72d\" (UID: \"874b4b98-1b9b-4c67-8204-fa621edf5b55\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-817fd7-7dfc855d4b-4b72d" Oct 30 01:56:35.424117 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:35.424089 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/874b4b98-1b9b-4c67-8204-fa621edf5b55-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-817fd7-7dfc855d4b-4b72d\" (UID: \"874b4b98-1b9b-4c67-8204-fa621edf5b55\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-817fd7-7dfc855d4b-4b72d" Oct 30 01:56:35.444380 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:35.444357 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-817fd7-6f9f8d5c7f-x9gmq" Oct 30 01:56:35.445301 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:35.445272 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jplk2\" (UniqueName: \"kubernetes.io/projected/874b4b98-1b9b-4c67-8204-fa621edf5b55-kube-api-access-jplk2\") pod \"payload-processing-e2e-extmod-a-817fd7-7dfc855d4b-4b72d\" (UID: \"874b4b98-1b9b-4c67-8204-fa621edf5b55\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-817fd7-7dfc855d4b-4b72d" Oct 30 01:56:35.642133 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:35.642103 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-817fd7-7dfc855d4b-4b72d" Oct 30 01:56:35.774776 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:35.774712 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-817fd7-6f9f8d5c7f-x9gmq"] Oct 30 01:56:35.775012 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:56:35.774988 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2db1738c_aff2_4494_a0a7_adb1d8a25ee2.slice/crio-e5f6e2e08a0adfc836ee08d1e87c3398348042777a2e376fc9d089726b44428d WatchSource:0}: Error finding container e5f6e2e08a0adfc836ee08d1e87c3398348042777a2e376fc9d089726b44428d: Status 404 returned error can't find the container with id e5f6e2e08a0adfc836ee08d1e87c3398348042777a2e376fc9d089726b44428d Oct 30 01:56:35.943698 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:35.943668 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-817fd7-7dfc855d4b-4b72d"] Oct 30 01:56:35.944356 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:56:35.944333 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod874b4b98_1b9b_4c67_8204_fa621edf5b55.slice/crio-814e8bd98ee1006d90ce75c5fa106fddeb1807278168d43d7662ecf9e4170070 WatchSource:0}: Error finding container 814e8bd98ee1006d90ce75c5fa106fddeb1807278168d43d7662ecf9e4170070: Status 404 returned error can't find the container with id 814e8bd98ee1006d90ce75c5fa106fddeb1807278168d43d7662ecf9e4170070 Oct 30 01:56:35.997772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:35.996638 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:36.064826 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:36.064087 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ee021ae-l7zzf"] Oct 30 01:56:36.073720 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:36.073521 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ee021ae-l7zzf"] Oct 30 01:56:36.162288 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:36.162259 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w"] Oct 30 01:56:36.166724 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:36.166706 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w" Oct 30 01:56:36.181450 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:36.181384 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-cd8173\"/\"openshift-service-ca.crt\"" Oct 30 01:56:36.181450 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:36.181366 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-cd8173\"/\"kube-root-ca.crt\"" Oct 30 01:56:36.181450 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:36.181407 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-cd8173\"/\"payload-processing-external-model-plugins-e2e-extmod-b-cd8173\"" Oct 30 01:56:36.181450 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:36.181425 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-cd8173\"/\"routing-overlay\"" Oct 30 01:56:36.183710 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:36.183691 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w"] Oct 30 01:56:36.202954 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:36.202897 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-cd8173\"/\"agc-cred-f70ae1064daa\"" Oct 30 01:56:36.203053 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:36.203002 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-cd8173\"/\"payload-processing-external-model-e2e--1a03a560-dockercfg-dwfpl\"" Oct 30 01:56:36.245251 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:36.245219 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cf1131b-1eb3-4a4c-a612-abdc03e7f418-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w\" (UID: \"1cf1131b-1eb3-4a4c-a612-abdc03e7f418\") " pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w" Oct 30 01:56:36.245406 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:36.245375 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1cf1131b-1eb3-4a4c-a612-abdc03e7f418-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w\" (UID: \"1cf1131b-1eb3-4a4c-a612-abdc03e7f418\") " pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w" Oct 30 01:56:36.245524 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:36.245505 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgfjm\" (UniqueName: \"kubernetes.io/projected/1cf1131b-1eb3-4a4c-a612-abdc03e7f418-kube-api-access-kgfjm\") pod \"payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w\" (UID: \"1cf1131b-1eb3-4a4c-a612-abdc03e7f418\") " pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w" Oct 30 01:56:36.245598 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:36.245572 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1cf1131b-1eb3-4a4c-a612-abdc03e7f418-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w\" (UID: \"1cf1131b-1eb3-4a4c-a612-abdc03e7f418\") " pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w" Oct 30 01:56:36.350610 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:36.350568 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cf1131b-1eb3-4a4c-a612-abdc03e7f418-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w\" (UID: \"1cf1131b-1eb3-4a4c-a612-abdc03e7f418\") " pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w" Oct 30 01:56:36.350861 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:36.350685 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1cf1131b-1eb3-4a4c-a612-abdc03e7f418-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w\" (UID: \"1cf1131b-1eb3-4a4c-a612-abdc03e7f418\") " pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w" Oct 30 01:56:36.350946 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:36.350862 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgfjm\" (UniqueName: \"kubernetes.io/projected/1cf1131b-1eb3-4a4c-a612-abdc03e7f418-kube-api-access-kgfjm\") pod \"payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w\" (UID: \"1cf1131b-1eb3-4a4c-a612-abdc03e7f418\") " pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w" Oct 30 01:56:36.350946 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:36.350928 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1cf1131b-1eb3-4a4c-a612-abdc03e7f418-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w\" (UID: \"1cf1131b-1eb3-4a4c-a612-abdc03e7f418\") " pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w" Oct 30 01:56:36.352431 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:36.352400 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cf1131b-1eb3-4a4c-a612-abdc03e7f418-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w\" (UID: \"1cf1131b-1eb3-4a4c-a612-abdc03e7f418\") " pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w" Oct 30 01:56:36.357792 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:36.355464 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1cf1131b-1eb3-4a4c-a612-abdc03e7f418-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w\" (UID: \"1cf1131b-1eb3-4a4c-a612-abdc03e7f418\") " pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w" Oct 30 01:56:36.365792 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:36.358733 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1cf1131b-1eb3-4a4c-a612-abdc03e7f418-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w\" (UID: \"1cf1131b-1eb3-4a4c-a612-abdc03e7f418\") " pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w" Oct 30 01:56:36.384208 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:36.384147 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgfjm\" (UniqueName: \"kubernetes.io/projected/1cf1131b-1eb3-4a4c-a612-abdc03e7f418-kube-api-access-kgfjm\") pod \"payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w\" (UID: \"1cf1131b-1eb3-4a4c-a612-abdc03e7f418\") " pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w" Oct 30 01:56:36.483293 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:36.483213 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w" Oct 30 01:56:36.545013 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:36.544472 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68078396-b940-4d83-ac04-39f7ca2c535c" path="/var/lib/kubelet/pods/68078396-b940-4d83-ac04-39f7ca2c535c/volumes" Oct 30 01:56:36.712703 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:36.712667 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-817fd7-7dfc855d4b-4b72d" event={"ID":"874b4b98-1b9b-4c67-8204-fa621edf5b55","Type":"ContainerStarted","Data":"b6795ed4b258e4f130b254cc4e28dc22e957daa8aa367e8177ddad4bfc4a739c"} Oct 30 01:56:36.712901 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:36.712734 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-817fd7-7dfc855d4b-4b72d" Oct 30 01:56:36.712901 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:36.712803 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-817fd7-7dfc855d4b-4b72d" event={"ID":"874b4b98-1b9b-4c67-8204-fa621edf5b55","Type":"ContainerStarted","Data":"814e8bd98ee1006d90ce75c5fa106fddeb1807278168d43d7662ecf9e4170070"} Oct 30 01:56:36.715546 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:36.715516 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-817fd7-6f9f8d5c7f-x9gmq" event={"ID":"2db1738c-aff2-4494-a0a7-adb1d8a25ee2","Type":"ContainerStarted","Data":"f45d2efe2579a18fe031ce4c82b36b76e39a3c08092f0bd92a6ec3da8fcf6ac7"} Oct 30 01:56:36.715684 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:36.715578 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-817fd7-6f9f8d5c7f-x9gmq" event={"ID":"2db1738c-aff2-4494-a0a7-adb1d8a25ee2","Type":"ContainerStarted","Data":"e5f6e2e08a0adfc836ee08d1e87c3398348042777a2e376fc9d089726b44428d"} Oct 30 01:56:36.715684 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:36.715606 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-817fd7-6f9f8d5c7f-x9gmq" Oct 30 01:56:36.719948 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:36.719902 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:36.752319 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:36.752084 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-817fd7-7dfc855d4b-4b72d" podStartSLOduration=1.752070755 podStartE2EDuration="1.752070755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:56:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:56:36.749357769 +0000 UTC m=+2322.325783171" watchObservedRunningTime="2026-10-30 01:56:36.752070755 +0000 UTC m=+2322.328496155" Oct 30 01:56:36.798719 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:36.798654 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-817fd7-6f9f8d5c7f-x9gmq" podStartSLOduration=1.798635971 podStartE2EDuration="1.798635971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:56:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:56:36.786847927 +0000 UTC m=+2322.363273340" watchObservedRunningTime="2026-10-30 01:56:36.798635971 +0000 UTC m=+2322.375061374" Oct 30 01:56:36.806693 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:36.806663 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w"] Oct 30 01:56:36.808263 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:56:36.808238 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cf1131b_1eb3_4a4c_a612_abdc03e7f418.slice/crio-eb7d3a1855c984677e24582b5f5d7528bf2ba3d721d2fd25de2470948a15e21f WatchSource:0}: Error finding container eb7d3a1855c984677e24582b5f5d7528bf2ba3d721d2fd25de2470948a15e21f: Status 404 returned error can't find the container with id eb7d3a1855c984677e24582b5f5d7528bf2ba3d721d2fd25de2470948a15e21f Oct 30 01:56:37.724928 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:37.724883 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w" event={"ID":"1cf1131b-1eb3-4a4c-a612-abdc03e7f418","Type":"ContainerStarted","Data":"7b68aefafed2d08dcddf53d4c12d7920f3bfd9114b6637c1713f97a7022fa184"} Oct 30 01:56:37.725418 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:37.724948 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w" event={"ID":"1cf1131b-1eb3-4a4c-a612-abdc03e7f418","Type":"ContainerStarted","Data":"eb7d3a1855c984677e24582b5f5d7528bf2ba3d721d2fd25de2470948a15e21f"} Oct 30 01:56:37.725418 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:37.725093 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w" Oct 30 01:56:37.767521 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:37.767471 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w" podStartSLOduration=1.767456796 podStartE2EDuration="1.767456796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:56:37.763001051 +0000 UTC m=+2323.339426454" watchObservedRunningTime="2026-10-30 01:56:37.767456796 +0000 UTC m=+2323.343882197" Oct 30 01:56:38.357706 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:38.357671 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:39.960618 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:39.960579 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:40.144176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:40.144141 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:41.575487 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:41.575021 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:43.065458 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:43.065424 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:44.542595 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:44.542557 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:44.691519 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:56:44.691475 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:56:46.196159 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:46.195122 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:47.728491 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:47.728460 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-817fd7-7dfc855d4b-4b72d" Oct 30 01:56:47.729028 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:47.728530 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-817fd7-6f9f8d5c7f-x9gmq" Oct 30 01:56:47.889733 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:47.889687 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:48.737668 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:48.737642 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w" Oct 30 01:56:49.490113 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:49.490069 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:51.059528 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:51.058725 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:52.694189 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:52.694156 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:54.194135 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:54.194086 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:55.684089 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:55.684047 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:57.216461 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:57.216427 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:58.770760 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:56:58.770709 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:00.374249 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:00.374209 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:01.840380 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:01.840342 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:03.491527 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:03.491431 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:05.019612 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:05.019571 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:06.607205 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:06.607166 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:06.796293 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:06.796257 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:07.636591 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:07.636551 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f81dd7e1-4d5g6"] Oct 30 01:57:07.643499 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:07.643474 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81dd7e1-4d5g6" Oct 30 01:57:07.656478 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:07.656438 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f81dd7e1-4d5g6"] Oct 30 01:57:07.661388 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:07.661352 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:57:07.661574 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:07.661535 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:57:07.661648 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:07.661586 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:57:07.661714 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:07.661662 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:57:07.718963 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:07.718920 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4397e72b-fc07-423d-9682-7cb11feb5d5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f81dd7e1-4d5g6\" (UID: \"4397e72b-fc07-423d-9682-7cb11feb5d5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81dd7e1-4d5g6" Oct 30 01:57:07.718963 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:07.718971 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4397e72b-fc07-423d-9682-7cb11feb5d5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f81dd7e1-4d5g6\" (UID: \"4397e72b-fc07-423d-9682-7cb11feb5d5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81dd7e1-4d5g6" Oct 30 01:57:07.719235 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:07.719223 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndwsb\" (UniqueName: \"kubernetes.io/projected/4397e72b-fc07-423d-9682-7cb11feb5d5d-kube-api-access-ndwsb\") pod \"maas-api-revoke-sub-f81dd7e1-4d5g6\" (UID: \"4397e72b-fc07-423d-9682-7cb11feb5d5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81dd7e1-4d5g6" Oct 30 01:57:07.820730 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:07.820688 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndwsb\" (UniqueName: \"kubernetes.io/projected/4397e72b-fc07-423d-9682-7cb11feb5d5d-kube-api-access-ndwsb\") pod \"maas-api-revoke-sub-f81dd7e1-4d5g6\" (UID: \"4397e72b-fc07-423d-9682-7cb11feb5d5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81dd7e1-4d5g6" Oct 30 01:57:07.820949 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:07.820853 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4397e72b-fc07-423d-9682-7cb11feb5d5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f81dd7e1-4d5g6\" (UID: \"4397e72b-fc07-423d-9682-7cb11feb5d5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81dd7e1-4d5g6" Oct 30 01:57:07.820949 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:07.820877 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4397e72b-fc07-423d-9682-7cb11feb5d5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f81dd7e1-4d5g6\" (UID: \"4397e72b-fc07-423d-9682-7cb11feb5d5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81dd7e1-4d5g6" Oct 30 01:57:07.821758 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:07.821724 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4397e72b-fc07-423d-9682-7cb11feb5d5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f81dd7e1-4d5g6\" (UID: \"4397e72b-fc07-423d-9682-7cb11feb5d5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81dd7e1-4d5g6" Oct 30 01:57:07.823460 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:07.823433 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4397e72b-fc07-423d-9682-7cb11feb5d5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f81dd7e1-4d5g6\" (UID: \"4397e72b-fc07-423d-9682-7cb11feb5d5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81dd7e1-4d5g6" Oct 30 01:57:07.840245 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:07.840211 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndwsb\" (UniqueName: \"kubernetes.io/projected/4397e72b-fc07-423d-9682-7cb11feb5d5d-kube-api-access-ndwsb\") pod \"maas-api-revoke-sub-f81dd7e1-4d5g6\" (UID: \"4397e72b-fc07-423d-9682-7cb11feb5d5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81dd7e1-4d5g6" Oct 30 01:57:07.967556 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:07.967508 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81dd7e1-4d5g6" Oct 30 01:57:08.202482 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:08.202428 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:08.273809 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:08.273772 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f81dd7e1-4d5g6"] Oct 30 01:57:08.274137 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:57:08.274112 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4397e72b_fc07_423d_9682_7cb11feb5d5d.slice/crio-a634cd05c83e71461f84a8bb4fe9779cda18d1f256e0345c5e54f96bdf01c469 WatchSource:0}: Error finding container a634cd05c83e71461f84a8bb4fe9779cda18d1f256e0345c5e54f96bdf01c469: Status 404 returned error can't find the container with id a634cd05c83e71461f84a8bb4fe9779cda18d1f256e0345c5e54f96bdf01c469 Oct 30 01:57:09.035801 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:09.035758 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81dd7e1-4d5g6" event={"ID":"4397e72b-fc07-423d-9682-7cb11feb5d5d","Type":"ContainerStarted","Data":"7a22d530689c97a3c544f1ad8ea6e9000d5396fb57832a6b74528ade2b7d839e"} Oct 30 01:57:09.036242 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:09.035832 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81dd7e1-4d5g6" event={"ID":"4397e72b-fc07-423d-9682-7cb11feb5d5d","Type":"ContainerStarted","Data":"a634cd05c83e71461f84a8bb4fe9779cda18d1f256e0345c5e54f96bdf01c469"} Oct 30 01:57:09.074381 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:09.074321 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81dd7e1-4d5g6" podStartSLOduration=2.074305281 podStartE2EDuration="2.074305281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:57:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:57:09.072447679 +0000 UTC m=+2354.648873084" watchObservedRunningTime="2026-10-30 01:57:09.074305281 +0000 UTC m=+2354.650730683" Oct 30 01:57:09.800647 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:09.800605 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:10.046194 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:10.046159 6572 generic.go:356] "Generic (PLEG): container finished" podID="4397e72b-fc07-423d-9682-7cb11feb5d5d" containerID="7a22d530689c97a3c544f1ad8ea6e9000d5396fb57832a6b74528ade2b7d839e" exitCode=0 Oct 30 01:57:10.046609 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:10.046246 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81dd7e1-4d5g6" event={"ID":"4397e72b-fc07-423d-9682-7cb11feb5d5d","Type":"ContainerDied","Data":"7a22d530689c97a3c544f1ad8ea6e9000d5396fb57832a6b74528ade2b7d839e"} Oct 30 01:57:11.246678 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:11.246642 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81dd7e1-4d5g6" Oct 30 01:57:11.329558 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:11.329381 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:11.375804 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:11.375721 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4397e72b-fc07-423d-9682-7cb11feb5d5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4397e72b-fc07-423d-9682-7cb11feb5d5d-maas-api-xks-ca\") pod \"4397e72b-fc07-423d-9682-7cb11feb5d5d\" (UID: \"4397e72b-fc07-423d-9682-7cb11feb5d5d\") " Oct 30 01:57:11.376356 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:11.376326 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4397e72b-fc07-423d-9682-7cb11feb5d5d-kube-api-access-ndwsb\" (UniqueName: \"kubernetes.io/projected/4397e72b-fc07-423d-9682-7cb11feb5d5d-kube-api-access-ndwsb\") pod \"4397e72b-fc07-423d-9682-7cb11feb5d5d\" (UID: \"4397e72b-fc07-423d-9682-7cb11feb5d5d\") " Oct 30 01:57:11.376825 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:11.376801 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4397e72b-fc07-423d-9682-7cb11feb5d5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4397e72b-fc07-423d-9682-7cb11feb5d5d-maas-api-openshift-ca\") pod \"4397e72b-fc07-423d-9682-7cb11feb5d5d\" (UID: \"4397e72b-fc07-423d-9682-7cb11feb5d5d\") " Oct 30 01:57:11.378076 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:11.378052 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4397e72b-fc07-423d-9682-7cb11feb5d5d-maas-api-openshift-ca" pod "4397e72b-fc07-423d-9682-7cb11feb5d5d" (UID: "4397e72b-fc07-423d-9682-7cb11feb5d5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:57:11.389521 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:11.389468 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4397e72b-fc07-423d-9682-7cb11feb5d5d-maas-api-xks-ca" pod "4397e72b-fc07-423d-9682-7cb11feb5d5d" (UID: "4397e72b-fc07-423d-9682-7cb11feb5d5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:57:11.442016 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:11.441971 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4397e72b-fc07-423d-9682-7cb11feb5d5d-kube-api-access-ndwsb" pod "4397e72b-fc07-423d-9682-7cb11feb5d5d" (UID: "4397e72b-fc07-423d-9682-7cb11feb5d5d"). InnerVolumeSpecName "kube-api-access-ndwsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:57:11.479502 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:11.479464 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4397e72b-fc07-423d-9682-7cb11feb5d5d-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:57:11.479502 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:11.479499 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4397e72b-fc07-423d-9682-7cb11feb5d5d-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:57:11.479502 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:11.479508 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndwsb\" (UniqueName: \"kubernetes.io/projected/4397e72b-fc07-423d-9682-7cb11feb5d5d-kube-api-access-ndwsb\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:57:11.507362 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:11.507312 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:12.064670 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:12.064642 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81dd7e1-4d5g6" Oct 30 01:57:12.064995 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:12.064639 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81dd7e1-4d5g6" event={"ID":"4397e72b-fc07-423d-9682-7cb11feb5d5d","Type":"ContainerDied","Data":"a634cd05c83e71461f84a8bb4fe9779cda18d1f256e0345c5e54f96bdf01c469"} Oct 30 01:57:12.064995 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:12.064808 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a634cd05c83e71461f84a8bb4fe9779cda18d1f256e0345c5e54f96bdf01c469" Oct 30 01:57:12.877636 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:12.877599 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:14.028513 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:14.028475 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2561142-xl9mk"] Oct 30 01:57:14.040248 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:14.040210 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2561142-xl9mk"] Oct 30 01:57:14.491254 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:14.491223 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:14.546218 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:14.546186 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11" path="/var/lib/kubelet/pods/ad76cbaf-e21f-4ab4-bbcf-c04996fa8a11/volumes" Oct 30 01:57:14.689256 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:57:14.689212 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:57:16.023571 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:16.023530 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:17.615309 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:17.615273 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:18.914569 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:18.914504 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:18.947017 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:18.946984 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9"] Oct 30 01:57:19.101325 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:19.101289 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:19.298879 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:19.298539 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:19.692634 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:19.692603 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:20.770938 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:20.770878 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:22.308985 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:22.308941 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:23.793661 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:23.793628 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:23.947582 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:23.947539 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9" podUID="18bbd0d9-bc71-4f59-bc75-7a94156bca1a" containerName="payload-processing" containerID="cri-o://5f53e04575bbddb672b1eae4b8b9abe7818166bbf2d979cd52e81f7a54f109e1" gracePeriod=25 Oct 30 01:57:24.029456 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:24.029380 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe25dd41-ftbx9"] Oct 30 01:57:24.040602 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:24.040571 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe25dd41-ftbx9"] Oct 30 01:57:24.175343 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:24.175303 6572 generic.go:356] "Generic (PLEG): container finished" podID="18bbd0d9-bc71-4f59-bc75-7a94156bca1a" containerID="5f53e04575bbddb672b1eae4b8b9abe7818166bbf2d979cd52e81f7a54f109e1" exitCode=0 Oct 30 01:57:24.175517 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:24.175375 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9" event={"ID":"18bbd0d9-bc71-4f59-bc75-7a94156bca1a","Type":"ContainerDied","Data":"5f53e04575bbddb672b1eae4b8b9abe7818166bbf2d979cd52e81f7a54f109e1"} Oct 30 01:57:24.228024 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:24.227993 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9" Oct 30 01:57:24.283339 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:24.283302 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18bbd0d9-bc71-4f59-bc75-7a94156bca1a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/18bbd0d9-bc71-4f59-bc75-7a94156bca1a-routing-overlay-volume\") pod \"18bbd0d9-bc71-4f59-bc75-7a94156bca1a\" (UID: \"18bbd0d9-bc71-4f59-bc75-7a94156bca1a\") " Oct 30 01:57:24.283549 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:24.283416 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18bbd0d9-bc71-4f59-bc75-7a94156bca1a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18bbd0d9-bc71-4f59-bc75-7a94156bca1a-plugins-config-volume\") pod \"18bbd0d9-bc71-4f59-bc75-7a94156bca1a\" (UID: \"18bbd0d9-bc71-4f59-bc75-7a94156bca1a\") " Oct 30 01:57:24.283549 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:24.283488 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18bbd0d9-bc71-4f59-bc75-7a94156bca1a-kube-api-access-s4r99\" (UniqueName: \"kubernetes.io/projected/18bbd0d9-bc71-4f59-bc75-7a94156bca1a-kube-api-access-s4r99\") pod \"18bbd0d9-bc71-4f59-bc75-7a94156bca1a\" (UID: \"18bbd0d9-bc71-4f59-bc75-7a94156bca1a\") " Oct 30 01:57:24.283667 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:24.283572 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18bbd0d9-bc71-4f59-bc75-7a94156bca1a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/18bbd0d9-bc71-4f59-bc75-7a94156bca1a-provider-credentials\") pod \"18bbd0d9-bc71-4f59-bc75-7a94156bca1a\" (UID: \"18bbd0d9-bc71-4f59-bc75-7a94156bca1a\") " Oct 30 01:57:24.283965 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:24.283716 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18bbd0d9-bc71-4f59-bc75-7a94156bca1a-routing-overlay-volume" pod "18bbd0d9-bc71-4f59-bc75-7a94156bca1a" (UID: "18bbd0d9-bc71-4f59-bc75-7a94156bca1a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:57:24.284380 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:24.284354 6572 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/18bbd0d9-bc71-4f59-bc75-7a94156bca1a-routing-overlay-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:57:24.284525 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:24.284493 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18bbd0d9-bc71-4f59-bc75-7a94156bca1a-plugins-config-volume" pod "18bbd0d9-bc71-4f59-bc75-7a94156bca1a" (UID: "18bbd0d9-bc71-4f59-bc75-7a94156bca1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:57:24.286386 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:24.286349 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18bbd0d9-bc71-4f59-bc75-7a94156bca1a-provider-credentials" pod "18bbd0d9-bc71-4f59-bc75-7a94156bca1a" (UID: "18bbd0d9-bc71-4f59-bc75-7a94156bca1a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:57:24.286508 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:24.286483 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18bbd0d9-bc71-4f59-bc75-7a94156bca1a-kube-api-access-s4r99" pod "18bbd0d9-bc71-4f59-bc75-7a94156bca1a" (UID: "18bbd0d9-bc71-4f59-bc75-7a94156bca1a"). InnerVolumeSpecName "kube-api-access-s4r99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:57:24.385777 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:24.385663 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18bbd0d9-bc71-4f59-bc75-7a94156bca1a-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:57:24.385777 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:24.385690 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4r99\" (UniqueName: \"kubernetes.io/projected/18bbd0d9-bc71-4f59-bc75-7a94156bca1a-kube-api-access-s4r99\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:57:24.385777 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:24.385701 6572 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/18bbd0d9-bc71-4f59-bc75-7a94156bca1a-provider-credentials\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:57:24.544783 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:24.544713 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bb94b2d-4e93-47f8-a33c-f51f56b32222" path="/var/lib/kubelet/pods/9bb94b2d-4e93-47f8-a33c-f51f56b32222/volumes" Oct 30 01:57:25.185065 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:25.185036 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9" Oct 30 01:57:25.185065 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:25.185050 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9" event={"ID":"18bbd0d9-bc71-4f59-bc75-7a94156bca1a","Type":"ContainerDied","Data":"4aa8ddd1652cd8d05496a87501c4e1c93c693277ea83369b680c683a76e59c75"} Oct 30 01:57:25.185584 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:25.185115 6572 scope.go:122] "RemoveContainer" containerID="5f53e04575bbddb672b1eae4b8b9abe7818166bbf2d979cd52e81f7a54f109e1" Oct 30 01:57:25.249740 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:25.249702 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9"] Oct 30 01:57:25.265622 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:25.265575 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-7858wgvp9"] Oct 30 01:57:25.429983 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:25.429941 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:25.876487 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:25.876453 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:26.064588 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:26.064552 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:26.544546 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:26.544504 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18bbd0d9-bc71-4f59-bc75-7a94156bca1a" path="/var/lib/kubelet/pods/18bbd0d9-bc71-4f59-bc75-7a94156bca1a/volumes" Oct 30 01:57:27.149256 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:27.149216 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:28.890108 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:28.890060 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:30.611114 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:30.611075 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:31.827691 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:31.827651 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:32.056836 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:32.056796 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:32.279967 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:32.279921 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:32.492077 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:32.492028 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:32.774281 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:32.774247 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:33.422201 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:33.422161 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:33.617019 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:33.616969 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:34.195433 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:34.195391 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:34.693248 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:34.693208 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:34.853179 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:34.853149 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr"] Oct 30 01:57:34.853910 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:34.853893 6572 cpu_manager.go:414] "containerMap: removing container" podUID="18bbd0d9-bc71-4f59-bc75-7a94156bca1a" containerName="payload-processing" Oct 30 01:57:34.853971 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:34.853911 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18bbd0d9-bc71-4f59-bc75-7a94156bca1a" containerName="payload-processing" Oct 30 01:57:34.853971 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:34.853928 6572 cpu_manager.go:414] "containerMap: removing container" podUID="4397e72b-fc07-423d-9682-7cb11feb5d5d" containerName="revoke-keys" Oct 30 01:57:34.853971 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:34.853933 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4397e72b-fc07-423d-9682-7cb11feb5d5d" containerName="revoke-keys" Oct 30 01:57:34.854074 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:34.854052 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="18bbd0d9-bc71-4f59-bc75-7a94156bca1a" containerName="payload-processing" Oct 30 01:57:34.854074 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:34.854063 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="4397e72b-fc07-423d-9682-7cb11feb5d5d" containerName="revoke-keys" Oct 30 01:57:34.858063 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:34.858034 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr" Oct 30 01:57:34.879729 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:34.879697 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-92f0cc\"/\"agc-cred-2bf58c377925\"" Oct 30 01:57:34.891723 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:34.891694 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-92f0cc\"/\"payload-processing-external-model-e2e--c7e40432-dockercfg-fffn4\"" Oct 30 01:57:34.892448 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:34.892413 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-92f0cc\"/\"payload-processing-external-model-plugins-e2e-extmod-a-92f0cc\"" Oct 30 01:57:34.892612 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:34.892553 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-92f0cc\"/\"openshift-service-ca.crt\"" Oct 30 01:57:34.892673 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:34.892655 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-92f0cc\"/\"kube-root-ca.crt\"" Oct 30 01:57:34.892804 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:34.892761 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-92f0cc\"/\"routing-overlay\"" Oct 30 01:57:34.897870 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:34.897837 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr"] Oct 30 01:57:34.984899 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:34.982844 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44681477-4758-4150-88c9-a9fe7e70d6ee-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr\" (UID: \"44681477-4758-4150-88c9-a9fe7e70d6ee\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr" Oct 30 01:57:34.984899 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:34.982948 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/44681477-4758-4150-88c9-a9fe7e70d6ee-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr\" (UID: \"44681477-4758-4150-88c9-a9fe7e70d6ee\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr" Oct 30 01:57:34.984899 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:34.983045 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zvht\" (UniqueName: \"kubernetes.io/projected/44681477-4758-4150-88c9-a9fe7e70d6ee-kube-api-access-8zvht\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr\" (UID: \"44681477-4758-4150-88c9-a9fe7e70d6ee\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr" Oct 30 01:57:34.984899 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:34.983116 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44681477-4758-4150-88c9-a9fe7e70d6ee-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr\" (UID: \"44681477-4758-4150-88c9-a9fe7e70d6ee\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr" Oct 30 01:57:35.086782 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:35.085252 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44681477-4758-4150-88c9-a9fe7e70d6ee-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr\" (UID: \"44681477-4758-4150-88c9-a9fe7e70d6ee\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr" Oct 30 01:57:35.086782 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:35.085331 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/44681477-4758-4150-88c9-a9fe7e70d6ee-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr\" (UID: \"44681477-4758-4150-88c9-a9fe7e70d6ee\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr" Oct 30 01:57:35.086782 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:35.085395 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zvht\" (UniqueName: \"kubernetes.io/projected/44681477-4758-4150-88c9-a9fe7e70d6ee-kube-api-access-8zvht\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr\" (UID: \"44681477-4758-4150-88c9-a9fe7e70d6ee\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr" Oct 30 01:57:35.086782 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:35.085443 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44681477-4758-4150-88c9-a9fe7e70d6ee-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr\" (UID: \"44681477-4758-4150-88c9-a9fe7e70d6ee\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr" Oct 30 01:57:35.086782 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:35.086337 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44681477-4758-4150-88c9-a9fe7e70d6ee-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr\" (UID: \"44681477-4758-4150-88c9-a9fe7e70d6ee\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr" Oct 30 01:57:35.087181 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:35.086991 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44681477-4758-4150-88c9-a9fe7e70d6ee-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr\" (UID: \"44681477-4758-4150-88c9-a9fe7e70d6ee\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr" Oct 30 01:57:35.092417 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:35.092379 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/44681477-4758-4150-88c9-a9fe7e70d6ee-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr\" (UID: \"44681477-4758-4150-88c9-a9fe7e70d6ee\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr" Oct 30 01:57:35.110760 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:35.110612 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zvht\" (UniqueName: \"kubernetes.io/projected/44681477-4758-4150-88c9-a9fe7e70d6ee-kube-api-access-8zvht\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr\" (UID: \"44681477-4758-4150-88c9-a9fe7e70d6ee\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr" Oct 30 01:57:35.111789 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:35.111760 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:35.173374 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:35.173333 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr" Oct 30 01:57:35.483128 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:35.483093 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr"] Oct 30 01:57:35.483667 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:57:35.483642 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44681477_4758_4150_88c9_a9fe7e70d6ee.slice/crio-c522db075065e7dc1e4e11a96ab60e73bac426d8511590c984baf3b73736cc37 WatchSource:0}: Error finding container c522db075065e7dc1e4e11a96ab60e73bac426d8511590c984baf3b73736cc37: Status 404 returned error can't find the container with id c522db075065e7dc1e4e11a96ab60e73bac426d8511590c984baf3b73736cc37 Oct 30 01:57:35.982169 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:35.982133 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:36.292633 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:36.292524 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr" event={"ID":"44681477-4758-4150-88c9-a9fe7e70d6ee","Type":"ContainerStarted","Data":"70a7bf883a706846e42c4058e496d2b63ba0715ddba243eac820a40a3d8620db"} Oct 30 01:57:36.292633 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:36.292598 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr" event={"ID":"44681477-4758-4150-88c9-a9fe7e70d6ee","Type":"ContainerStarted","Data":"c522db075065e7dc1e4e11a96ab60e73bac426d8511590c984baf3b73736cc37"} Oct 30 01:57:36.292633 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:36.292632 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr" Oct 30 01:57:36.335435 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:36.335372 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr" podStartSLOduration=2.335354595 podStartE2EDuration="2.335354595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:57:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:57:36.331035903 +0000 UTC m=+2381.907461306" watchObservedRunningTime="2026-10-30 01:57:36.335354595 +0000 UTC m=+2381.911780012" Oct 30 01:57:37.501116 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:37.501080 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:39.217298 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:39.217253 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:40.790512 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:40.790476 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:42.527888 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:42.527838 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:44.186742 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:44.186696 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:44.374494 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:44.374452 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:44.690662 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:57:44.690621 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:57:45.813096 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:45.813052 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:46.007696 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:46.007646 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:47.314323 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:47.314291 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr" Oct 30 01:57:47.335408 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:47.335374 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:48.874658 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:48.874621 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:50.341904 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:50.341807 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:51.371870 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:51.371833 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p"] Oct 30 01:57:51.372290 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:51.372050 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" podUID="fdadf1cc-610e-4e7b-83cf-5a173f5779eb" containerName="istio-proxy" containerID="cri-o://96b0395c0ebede04db1c43677d9faa01e3353c3d8a987b373b2ceeb928c533b5" gracePeriod=30 Oct 30 01:57:51.888642 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:51.888608 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:53.368476 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:53.368432 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:54.916278 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:54.916240 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:55.102890 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:55.102854 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:56.275039 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.275007 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:56.497094 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.497065 6572 generic.go:356] "Generic (PLEG): container finished" podID="fdadf1cc-610e-4e7b-83cf-5a173f5779eb" containerID="96b0395c0ebede04db1c43677d9faa01e3353c3d8a987b373b2ceeb928c533b5" exitCode=0 Oct 30 01:57:56.497229 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.497135 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" event={"ID":"fdadf1cc-610e-4e7b-83cf-5a173f5779eb","Type":"ContainerDied","Data":"96b0395c0ebede04db1c43677d9faa01e3353c3d8a987b373b2ceeb928c533b5"} Oct 30 01:57:56.565367 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.565334 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:56.587755 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.587710 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn"] Oct 30 01:57:56.598090 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.598064 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 01:57:56.618421 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.618397 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn"] Oct 30 01:57:56.622622 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.622595 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-f6e11e-openshift-default-dockercfg-gd55x\"" Oct 30 01:57:56.669714 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.669692 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" Oct 30 01:57:56.741771 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.721061 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-istiod-ca-cert\") pod \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\" (UID: \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\") " Oct 30 01:57:56.742138 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.742109 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-istio-podinfo\") pod \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\" (UID: \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\") " Oct 30 01:57:56.743187 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.743160 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-workload-certs\") pod \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\" (UID: \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\") " Oct 30 01:57:56.743444 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.743424 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-istio-token\" (UniqueName: \"kubernetes.io/projected/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-istio-token\") pod \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\" (UID: \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\") " Oct 30 01:57:56.743585 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.743570 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-kube-api-access-wmrqr\" (UniqueName: \"kubernetes.io/projected/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-kube-api-access-wmrqr\") pod \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\" (UID: \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\") " Oct 30 01:57:56.743843 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.743821 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-credential-socket\") pod \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\" (UID: \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\") " Oct 30 01:57:56.744039 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.744022 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-istio-envoy\") pod \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\" (UID: \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\") " Oct 30 01:57:56.744192 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.744176 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-workload-socket\") pod \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\" (UID: \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\") " Oct 30 01:57:56.744352 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.744336 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-istio-data\") pod \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\" (UID: \"fdadf1cc-610e-4e7b-83cf-5a173f5779eb\") " Oct 30 01:57:56.777847 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.744702 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a341c867-33db-494c-b3e4-873415967cf0-istiod-ca-cert\") pod \"e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn\" (UID: \"a341c867-33db-494c-b3e4-873415967cf0\") " pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 01:57:56.777847 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.764930 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a341c867-33db-494c-b3e4-873415967cf0-credential-socket\") pod \"e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn\" (UID: \"a341c867-33db-494c-b3e4-873415967cf0\") " pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 01:57:56.777847 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.765057 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a341c867-33db-494c-b3e4-873415967cf0-workload-certs\") pod \"e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn\" (UID: \"a341c867-33db-494c-b3e4-873415967cf0\") " pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 01:57:56.777847 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.765100 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a341c867-33db-494c-b3e4-873415967cf0-istio-podinfo\") pod \"e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn\" (UID: \"a341c867-33db-494c-b3e4-873415967cf0\") " pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 01:57:56.777847 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.765134 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g6tf\" (UniqueName: \"kubernetes.io/projected/a341c867-33db-494c-b3e4-873415967cf0-kube-api-access-5g6tf\") pod \"e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn\" (UID: \"a341c867-33db-494c-b3e4-873415967cf0\") " pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 01:57:56.777847 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.765179 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a341c867-33db-494c-b3e4-873415967cf0-workload-socket\") pod \"e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn\" (UID: \"a341c867-33db-494c-b3e4-873415967cf0\") " pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 01:57:56.777847 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.765256 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a341c867-33db-494c-b3e4-873415967cf0-istio-envoy\") pod \"e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn\" (UID: \"a341c867-33db-494c-b3e4-873415967cf0\") " pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 01:57:56.777847 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.765318 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a341c867-33db-494c-b3e4-873415967cf0-istio-data\") pod \"e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn\" (UID: \"a341c867-33db-494c-b3e4-873415967cf0\") " pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 01:57:56.777847 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.765474 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a341c867-33db-494c-b3e4-873415967cf0-istio-token\") pod \"e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn\" (UID: \"a341c867-33db-494c-b3e4-873415967cf0\") " pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 01:57:56.777847 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.756011 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-credential-socket" pod "fdadf1cc-610e-4e7b-83cf-5a173f5779eb" (UID: "fdadf1cc-610e-4e7b-83cf-5a173f5779eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:57:56.777847 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.756337 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-workload-socket" pod "fdadf1cc-610e-4e7b-83cf-5a173f5779eb" (UID: "fdadf1cc-610e-4e7b-83cf-5a173f5779eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:57:56.777847 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.756602 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-workload-certs" pod "fdadf1cc-610e-4e7b-83cf-5a173f5779eb" (UID: "fdadf1cc-610e-4e7b-83cf-5a173f5779eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:57:56.777847 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.756894 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-istio-data" pod "fdadf1cc-610e-4e7b-83cf-5a173f5779eb" (UID: "fdadf1cc-610e-4e7b-83cf-5a173f5779eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:57:56.781779 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.781600 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:56.782120 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.782097 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-istiod-ca-cert" pod "fdadf1cc-610e-4e7b-83cf-5a173f5779eb" (UID: "fdadf1cc-610e-4e7b-83cf-5a173f5779eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:57:56.824007 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.822914 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-istio-token" pod "fdadf1cc-610e-4e7b-83cf-5a173f5779eb" (UID: "fdadf1cc-610e-4e7b-83cf-5a173f5779eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:57:56.868910 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.866761 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a341c867-33db-494c-b3e4-873415967cf0-istio-token\") pod \"e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn\" (UID: \"a341c867-33db-494c-b3e4-873415967cf0\") " pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 01:57:56.868910 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.866964 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a341c867-33db-494c-b3e4-873415967cf0-istiod-ca-cert\") pod \"e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn\" (UID: \"a341c867-33db-494c-b3e4-873415967cf0\") " pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 01:57:56.868910 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.867002 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a341c867-33db-494c-b3e4-873415967cf0-credential-socket\") pod \"e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn\" (UID: \"a341c867-33db-494c-b3e4-873415967cf0\") " pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 01:57:56.868910 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.867129 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a341c867-33db-494c-b3e4-873415967cf0-workload-certs\") pod \"e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn\" (UID: \"a341c867-33db-494c-b3e4-873415967cf0\") " pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 01:57:56.868910 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.867167 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a341c867-33db-494c-b3e4-873415967cf0-istio-podinfo\") pod \"e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn\" (UID: \"a341c867-33db-494c-b3e4-873415967cf0\") " pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 01:57:56.868910 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.867199 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g6tf\" (UniqueName: \"kubernetes.io/projected/a341c867-33db-494c-b3e4-873415967cf0-kube-api-access-5g6tf\") pod \"e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn\" (UID: \"a341c867-33db-494c-b3e4-873415967cf0\") " pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 01:57:56.868910 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.867242 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a341c867-33db-494c-b3e4-873415967cf0-workload-socket\") pod \"e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn\" (UID: \"a341c867-33db-494c-b3e4-873415967cf0\") " pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 01:57:56.868910 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.867308 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a341c867-33db-494c-b3e4-873415967cf0-istio-envoy\") pod \"e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn\" (UID: \"a341c867-33db-494c-b3e4-873415967cf0\") " pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 01:57:56.868910 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.867359 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a341c867-33db-494c-b3e4-873415967cf0-istio-data\") pod \"e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn\" (UID: \"a341c867-33db-494c-b3e4-873415967cf0\") " pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 01:57:56.868910 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.867478 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:57:56.868910 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.867496 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:57:56.868910 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.867512 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:57:56.868910 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.867526 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:57:56.868910 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.867538 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:57:56.868910 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.867551 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:57:56.868910 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.868026 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a341c867-33db-494c-b3e4-873415967cf0-istio-data\") pod \"e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn\" (UID: \"a341c867-33db-494c-b3e4-873415967cf0\") " pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 01:57:56.871163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.870329 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a341c867-33db-494c-b3e4-873415967cf0-workload-socket\") pod \"e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn\" (UID: \"a341c867-33db-494c-b3e4-873415967cf0\") " pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 01:57:56.871163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.870804 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a341c867-33db-494c-b3e4-873415967cf0-istiod-ca-cert\") pod \"e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn\" (UID: \"a341c867-33db-494c-b3e4-873415967cf0\") " pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 01:57:56.871163 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.871155 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-istio-podinfo" pod "fdadf1cc-610e-4e7b-83cf-5a173f5779eb" (UID: "fdadf1cc-610e-4e7b-83cf-5a173f5779eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:57:56.871650 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.871620 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a341c867-33db-494c-b3e4-873415967cf0-workload-certs\") pod \"e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn\" (UID: \"a341c867-33db-494c-b3e4-873415967cf0\") " pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 01:57:56.871886 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.871859 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a341c867-33db-494c-b3e4-873415967cf0-credential-socket\") pod \"e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn\" (UID: \"a341c867-33db-494c-b3e4-873415967cf0\") " pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 01:57:56.872558 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.872533 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a341c867-33db-494c-b3e4-873415967cf0-istio-podinfo\") pod \"e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn\" (UID: \"a341c867-33db-494c-b3e4-873415967cf0\") " pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 01:57:56.877248 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.877196 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a341c867-33db-494c-b3e4-873415967cf0-istio-envoy\") pod \"e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn\" (UID: \"a341c867-33db-494c-b3e4-873415967cf0\") " pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 01:57:56.877871 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.877826 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-kube-api-access-wmrqr" pod "fdadf1cc-610e-4e7b-83cf-5a173f5779eb" (UID: "fdadf1cc-610e-4e7b-83cf-5a173f5779eb"). InnerVolumeSpecName "kube-api-access-wmrqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:57:56.879445 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.879409 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-istio-envoy" pod "fdadf1cc-610e-4e7b-83cf-5a173f5779eb" (UID: "fdadf1cc-610e-4e7b-83cf-5a173f5779eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:57:56.925109 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.924572 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g6tf\" (UniqueName: \"kubernetes.io/projected/a341c867-33db-494c-b3e4-873415967cf0-kube-api-access-5g6tf\") pod \"e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn\" (UID: \"a341c867-33db-494c-b3e4-873415967cf0\") " pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 01:57:56.925109 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.925053 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a341c867-33db-494c-b3e4-873415967cf0-istio-token\") pod \"e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn\" (UID: \"a341c867-33db-494c-b3e4-873415967cf0\") " pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 01:57:56.969142 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.969105 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:57:56.969142 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.969144 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:57:56.969537 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.969161 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmrqr\" (UniqueName: \"kubernetes.io/projected/fdadf1cc-610e-4e7b-83cf-5a173f5779eb-kube-api-access-wmrqr\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:57:56.971439 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:56.971404 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:57.219029 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:57.218995 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 01:57:57.520699 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:57.520617 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" Oct 30 01:57:57.521144 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:57.520621 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p" event={"ID":"fdadf1cc-610e-4e7b-83cf-5a173f5779eb","Type":"ContainerDied","Data":"6961b05897ecb4a9256f116d2e5f7a4a37a2bd7bf24ac05c3648ffef1a09f893"} Oct 30 01:57:57.521144 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:57.520823 6572 scope.go:122] "RemoveContainer" containerID="96b0395c0ebede04db1c43677d9faa01e3353c3d8a987b373b2ceeb928c533b5" Oct 30 01:57:57.540417 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:57:57.540391 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda341c867_33db_494c_b3e4_873415967cf0.slice/crio-c17d7848b23d31220c9671a5319059ae031dc544e838eacf82cac5a9be2c165a WatchSource:0}: Error finding container c17d7848b23d31220c9671a5319059ae031dc544e838eacf82cac5a9be2c165a: Status 404 returned error can't find the container with id c17d7848b23d31220c9671a5319059ae031dc544e838eacf82cac5a9be2c165a Oct 30 01:57:57.541420 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:57.541399 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn"] Oct 30 01:57:57.543428 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:57.543390 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:57:57.543529 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:57.543470 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:57:57.543529 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:57.543512 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:57:57.575953 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:57.575930 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p"] Oct 30 01:57:57.581532 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:57.581510 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-9227p"] Oct 30 01:57:57.924227 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:57.924195 6572 scope.go:122] "RemoveContainer" containerID="c6574c63fe35ea428f30eb8b46912a7652f072b6fee155ca2f5f80e5850e9077" Oct 30 01:57:57.936666 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:57.936643 6572 scope.go:122] "RemoveContainer" containerID="545269faf33fc7f6b600e8611fa80a2e7bd8882493c52418dda3537fa9381c10" Oct 30 01:57:58.076890 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:58.076854 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:58.530464 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:58.530426 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" event={"ID":"a341c867-33db-494c-b3e4-873415967cf0","Type":"ContainerStarted","Data":"ad35146aaa1dd21e8cd4038b8e3ad2d4a22358aa3ac01d1cc3f633cb654cf713"} Oct 30 01:57:58.530901 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:58.530490 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" event={"ID":"a341c867-33db-494c-b3e4-873415967cf0","Type":"ContainerStarted","Data":"c17d7848b23d31220c9671a5319059ae031dc544e838eacf82cac5a9be2c165a"} Oct 30 01:57:58.544677 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:58.544647 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdadf1cc-610e-4e7b-83cf-5a173f5779eb" path="/var/lib/kubelet/pods/fdadf1cc-610e-4e7b-83cf-5a173f5779eb/volumes" Oct 30 01:57:58.577015 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:58.576964 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" podStartSLOduration=2.576949373 podStartE2EDuration="2.576949373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:57:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:57:58.573266619 +0000 UTC m=+2404.149692020" watchObservedRunningTime="2026-10-30 01:57:58.576949373 +0000 UTC m=+2404.153374845" Oct 30 01:57:59.219582 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:59.219546 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 01:57:59.225735 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:59.225711 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 01:57:59.539362 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:59.539269 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 01:57:59.540622 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:59.540601 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 01:57:59.805552 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:57:59.805479 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:01.131878 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:01.131829 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:01.338617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:01.338584 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:01.873720 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:01.873680 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:02.765174 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:02.765139 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:02.782295 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:02.782254 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f6e11e-7d74b77d6-w25pd"] Oct 30 01:58:02.783252 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:02.783226 6572 cpu_manager.go:414] "containerMap: removing container" podUID="fdadf1cc-610e-4e7b-83cf-5a173f5779eb" containerName="istio-proxy" Oct 30 01:58:02.783252 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:02.783249 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdadf1cc-610e-4e7b-83cf-5a173f5779eb" containerName="istio-proxy" Oct 30 01:58:02.783418 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:02.783405 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdadf1cc-610e-4e7b-83cf-5a173f5779eb" containerName="istio-proxy" Oct 30 01:58:02.788722 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:02.788705 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-f6e11e-7d74b77d6-w25pd" Oct 30 01:58:02.805734 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:02.805710 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f6e11e-7d74b77d6-w25pd"] Oct 30 01:58:02.811935 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:02.811900 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-f6e11e\"" Oct 30 01:58:02.812118 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:02.812029 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-f6e11e-dockercfg-rwfm5\"" Oct 30 01:58:02.871799 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:02.871756 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f6e11e-794f9f4fc5-z7q9b"] Oct 30 01:58:02.876431 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:02.876410 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f6e11e-794f9f4fc5-z7q9b" Oct 30 01:58:02.893642 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:02.893496 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddf6fa1c-dc69-45e9-b865-46d8b020731a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-f6e11e-7d74b77d6-w25pd\" (UID: \"ddf6fa1c-dc69-45e9-b865-46d8b020731a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f6e11e-7d74b77d6-w25pd" Oct 30 01:58:02.894176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:02.894139 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87xhh\" (UniqueName: \"kubernetes.io/projected/ddf6fa1c-dc69-45e9-b865-46d8b020731a-kube-api-access-87xhh\") pod \"payload-processing-e2e-extmod-b-f6e11e-7d74b77d6-w25pd\" (UID: \"ddf6fa1c-dc69-45e9-b865-46d8b020731a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f6e11e-7d74b77d6-w25pd" Oct 30 01:58:03.035823 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:03.031619 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f6e11e-794f9f4fc5-z7q9b"] Oct 30 01:58:03.060541 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:03.045722 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c2e3d3a-c68b-4dd6-94ea-e17f286b5738-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-f6e11e-794f9f4fc5-z7q9b\" (UID: \"9c2e3d3a-c68b-4dd6-94ea-e17f286b5738\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f6e11e-794f9f4fc5-z7q9b" Oct 30 01:58:03.061104 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:03.061065 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddf6fa1c-dc69-45e9-b865-46d8b020731a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-f6e11e-7d74b77d6-w25pd\" (UID: \"ddf6fa1c-dc69-45e9-b865-46d8b020731a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f6e11e-7d74b77d6-w25pd" Oct 30 01:58:03.061384 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:03.061366 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd46n\" (UniqueName: \"kubernetes.io/projected/9c2e3d3a-c68b-4dd6-94ea-e17f286b5738-kube-api-access-gd46n\") pod \"payload-pre-processing-e2e-extmod-b-f6e11e-794f9f4fc5-z7q9b\" (UID: \"9c2e3d3a-c68b-4dd6-94ea-e17f286b5738\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f6e11e-794f9f4fc5-z7q9b" Oct 30 01:58:03.061556 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:03.061541 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87xhh\" (UniqueName: \"kubernetes.io/projected/ddf6fa1c-dc69-45e9-b865-46d8b020731a-kube-api-access-87xhh\") pod \"payload-processing-e2e-extmod-b-f6e11e-7d74b77d6-w25pd\" (UID: \"ddf6fa1c-dc69-45e9-b865-46d8b020731a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f6e11e-7d74b77d6-w25pd" Oct 30 01:58:03.061728 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:03.061689 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:03.063120 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:03.063092 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddf6fa1c-dc69-45e9-b865-46d8b020731a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-f6e11e-7d74b77d6-w25pd\" (UID: \"ddf6fa1c-dc69-45e9-b865-46d8b020731a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f6e11e-7d74b77d6-w25pd" Oct 30 01:58:03.114690 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:03.114659 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87xhh\" (UniqueName: \"kubernetes.io/projected/ddf6fa1c-dc69-45e9-b865-46d8b020731a-kube-api-access-87xhh\") pod \"payload-processing-e2e-extmod-b-f6e11e-7d74b77d6-w25pd\" (UID: \"ddf6fa1c-dc69-45e9-b865-46d8b020731a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f6e11e-7d74b77d6-w25pd" Oct 30 01:58:03.162717 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:03.162675 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd46n\" (UniqueName: \"kubernetes.io/projected/9c2e3d3a-c68b-4dd6-94ea-e17f286b5738-kube-api-access-gd46n\") pod \"payload-pre-processing-e2e-extmod-b-f6e11e-794f9f4fc5-z7q9b\" (UID: \"9c2e3d3a-c68b-4dd6-94ea-e17f286b5738\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f6e11e-794f9f4fc5-z7q9b" Oct 30 01:58:03.162932 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:03.162812 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c2e3d3a-c68b-4dd6-94ea-e17f286b5738-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-f6e11e-794f9f4fc5-z7q9b\" (UID: \"9c2e3d3a-c68b-4dd6-94ea-e17f286b5738\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f6e11e-794f9f4fc5-z7q9b" Oct 30 01:58:03.163540 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:03.163520 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c2e3d3a-c68b-4dd6-94ea-e17f286b5738-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-f6e11e-794f9f4fc5-z7q9b\" (UID: \"9c2e3d3a-c68b-4dd6-94ea-e17f286b5738\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f6e11e-794f9f4fc5-z7q9b" Oct 30 01:58:03.182275 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:03.182242 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd46n\" (UniqueName: \"kubernetes.io/projected/9c2e3d3a-c68b-4dd6-94ea-e17f286b5738-kube-api-access-gd46n\") pod \"payload-pre-processing-e2e-extmod-b-f6e11e-794f9f4fc5-z7q9b\" (UID: \"9c2e3d3a-c68b-4dd6-94ea-e17f286b5738\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f6e11e-794f9f4fc5-z7q9b" Oct 30 01:58:03.194115 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:03.194083 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f6e11e-794f9f4fc5-z7q9b" Oct 30 01:58:03.404036 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:03.403941 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-f6e11e-7d74b77d6-w25pd" Oct 30 01:58:03.506277 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:03.506096 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f6e11e-794f9f4fc5-z7q9b"] Oct 30 01:58:03.510426 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:58:03.510392 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c2e3d3a_c68b_4dd6_94ea_e17f286b5738.slice/crio-64a01825610acb87e1ec7e0668c9dcc407991f907fdfe7e4f48dad8f9cf79150 WatchSource:0}: Error finding container 64a01825610acb87e1ec7e0668c9dcc407991f907fdfe7e4f48dad8f9cf79150: Status 404 returned error can't find the container with id 64a01825610acb87e1ec7e0668c9dcc407991f907fdfe7e4f48dad8f9cf79150 Oct 30 01:58:03.587642 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:03.587602 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f6e11e-794f9f4fc5-z7q9b" event={"ID":"9c2e3d3a-c68b-4dd6-94ea-e17f286b5738","Type":"ContainerStarted","Data":"64a01825610acb87e1ec7e0668c9dcc407991f907fdfe7e4f48dad8f9cf79150"} Oct 30 01:58:03.732981 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:03.732951 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f6e11e-7d74b77d6-w25pd"] Oct 30 01:58:03.733373 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:58:03.733351 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddf6fa1c_dc69_45e9_b865_46d8b020731a.slice/crio-59b3e8f8e8f14f4907b80646898f94afe7230be6583018d54df80bd908dd7592 WatchSource:0}: Error finding container 59b3e8f8e8f14f4907b80646898f94afe7230be6583018d54df80bd908dd7592: Status 404 returned error can't find the container with id 59b3e8f8e8f14f4907b80646898f94afe7230be6583018d54df80bd908dd7592 Oct 30 01:58:03.740161 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:03.740137 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:04.489879 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:04.489843 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:04.597877 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:04.597840 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f6e11e-794f9f4fc5-z7q9b" event={"ID":"9c2e3d3a-c68b-4dd6-94ea-e17f286b5738","Type":"ContainerStarted","Data":"d15215779b4c61392092e632c398d4317d118a00e16d2944c3dde15f9dac98be"} Oct 30 01:58:04.598088 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:04.597914 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f6e11e-794f9f4fc5-z7q9b" Oct 30 01:58:04.599764 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:04.599724 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-f6e11e-7d74b77d6-w25pd" event={"ID":"ddf6fa1c-dc69-45e9-b865-46d8b020731a","Type":"ContainerStarted","Data":"dcbb5ce83e85e3917446d92e2c7d0a21f216d489c345b08deb3b91e33a4684c8"} Oct 30 01:58:04.599940 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:04.599793 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-f6e11e-7d74b77d6-w25pd" event={"ID":"ddf6fa1c-dc69-45e9-b865-46d8b020731a","Type":"ContainerStarted","Data":"59b3e8f8e8f14f4907b80646898f94afe7230be6583018d54df80bd908dd7592"} Oct 30 01:58:04.599940 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:04.599825 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-f6e11e-7d74b77d6-w25pd" Oct 30 01:58:04.636362 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:04.636306 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f6e11e-794f9f4fc5-z7q9b" podStartSLOduration=2.636279068 podStartE2EDuration="2.636279068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:58:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:58:04.63481731 +0000 UTC m=+2410.211242711" watchObservedRunningTime="2026-10-30 01:58:04.636279068 +0000 UTC m=+2410.212704471" Oct 30 01:58:04.670291 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:04.670258 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:04.673311 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:04.673138 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-f6e11e-7d74b77d6-w25pd" podStartSLOduration=2.673120762 podStartE2EDuration="2.673120762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:58:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:58:04.672391694 +0000 UTC m=+2410.248817097" watchObservedRunningTime="2026-10-30 01:58:04.673120762 +0000 UTC m=+2410.249546169" Oct 30 01:58:05.184082 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:05.177973 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:05.309564 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:05.309531 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2eae39/payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz"] Oct 30 01:58:05.314682 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:05.314653 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2eae39/payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz" Oct 30 01:58:05.335797 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:05.330628 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2eae39\"/\"agc-cred-87ddc0e1a748\"" Oct 30 01:58:05.335797 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:05.334569 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2eae39/payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz"] Oct 30 01:58:05.338409 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:05.338378 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2eae39\"/\"payload-processing-external-model-e2e--297d8bbe-dockercfg-dvckm\"" Oct 30 01:58:05.338564 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:05.338445 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2eae39\"/\"kube-root-ca.crt\"" Oct 30 01:58:05.338564 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:05.338542 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2eae39\"/\"routing-overlay\"" Oct 30 01:58:05.338699 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:05.338577 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2eae39\"/\"openshift-service-ca.crt\"" Oct 30 01:58:05.338699 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:05.338649 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2eae39\"/\"payload-processing-external-model-plugins-e2e-extmod-a-2eae39\"" Oct 30 01:58:05.445994 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:05.440336 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/421ef56b-ad77-4985-ab23-792d7163901a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz\" (UID: \"421ef56b-ad77-4985-ab23-792d7163901a\") " pod="ai-tenant-e2e-extmod-a-2eae39/payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz" Oct 30 01:58:05.445994 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:05.440502 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25mj4\" (UniqueName: \"kubernetes.io/projected/421ef56b-ad77-4985-ab23-792d7163901a-kube-api-access-25mj4\") pod \"payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz\" (UID: \"421ef56b-ad77-4985-ab23-792d7163901a\") " pod="ai-tenant-e2e-extmod-a-2eae39/payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz" Oct 30 01:58:05.445994 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:05.440550 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/421ef56b-ad77-4985-ab23-792d7163901a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz\" (UID: \"421ef56b-ad77-4985-ab23-792d7163901a\") " pod="ai-tenant-e2e-extmod-a-2eae39/payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz" Oct 30 01:58:05.445994 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:05.440581 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/421ef56b-ad77-4985-ab23-792d7163901a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz\" (UID: \"421ef56b-ad77-4985-ab23-792d7163901a\") " pod="ai-tenant-e2e-extmod-a-2eae39/payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz" Oct 30 01:58:05.601773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:05.597871 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/421ef56b-ad77-4985-ab23-792d7163901a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz\" (UID: \"421ef56b-ad77-4985-ab23-792d7163901a\") " pod="ai-tenant-e2e-extmod-a-2eae39/payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz" Oct 30 01:58:05.601773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:05.597924 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/421ef56b-ad77-4985-ab23-792d7163901a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz\" (UID: \"421ef56b-ad77-4985-ab23-792d7163901a\") " pod="ai-tenant-e2e-extmod-a-2eae39/payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz" Oct 30 01:58:05.601773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:05.598115 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/421ef56b-ad77-4985-ab23-792d7163901a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz\" (UID: \"421ef56b-ad77-4985-ab23-792d7163901a\") " pod="ai-tenant-e2e-extmod-a-2eae39/payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz" Oct 30 01:58:05.601773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:05.598254 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25mj4\" (UniqueName: \"kubernetes.io/projected/421ef56b-ad77-4985-ab23-792d7163901a-kube-api-access-25mj4\") pod \"payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz\" (UID: \"421ef56b-ad77-4985-ab23-792d7163901a\") " pod="ai-tenant-e2e-extmod-a-2eae39/payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz" Oct 30 01:58:05.601773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:05.599445 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/421ef56b-ad77-4985-ab23-792d7163901a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz\" (UID: \"421ef56b-ad77-4985-ab23-792d7163901a\") " pod="ai-tenant-e2e-extmod-a-2eae39/payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz" Oct 30 01:58:05.625793 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:05.619704 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/421ef56b-ad77-4985-ab23-792d7163901a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz\" (UID: \"421ef56b-ad77-4985-ab23-792d7163901a\") " pod="ai-tenant-e2e-extmod-a-2eae39/payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz" Oct 30 01:58:05.654017 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:05.653949 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/421ef56b-ad77-4985-ab23-792d7163901a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz\" (UID: \"421ef56b-ad77-4985-ab23-792d7163901a\") " pod="ai-tenant-e2e-extmod-a-2eae39/payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz" Oct 30 01:58:05.684807 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:05.682617 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25mj4\" (UniqueName: \"kubernetes.io/projected/421ef56b-ad77-4985-ab23-792d7163901a-kube-api-access-25mj4\") pod \"payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz\" (UID: \"421ef56b-ad77-4985-ab23-792d7163901a\") " pod="ai-tenant-e2e-extmod-a-2eae39/payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz" Oct 30 01:58:05.770836 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:05.765186 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:05.934266 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:05.934228 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2eae39/payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz" Oct 30 01:58:06.238599 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:06.238567 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2eae39/payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz"] Oct 30 01:58:06.239469 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:58:06.239440 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod421ef56b_ad77_4985_ab23_792d7163901a.slice/crio-eb23c5e9e3499ccf83e2b9fb35fc65ea6e7ed84512843b1ce6ee328ebe03cc8b WatchSource:0}: Error finding container eb23c5e9e3499ccf83e2b9fb35fc65ea6e7ed84512843b1ce6ee328ebe03cc8b: Status 404 returned error can't find the container with id eb23c5e9e3499ccf83e2b9fb35fc65ea6e7ed84512843b1ce6ee328ebe03cc8b Oct 30 01:58:06.254133 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:06.254101 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:06.445417 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:06.445378 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:06.694574 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:06.694527 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2eae39/payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz" event={"ID":"421ef56b-ad77-4985-ab23-792d7163901a","Type":"ContainerStarted","Data":"b486bb8933390cc9b3cf7130e49d24b3ba6505d2113adf83f685a097c5376a27"} Oct 30 01:58:06.695100 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:06.694584 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2eae39/payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz" event={"ID":"421ef56b-ad77-4985-ab23-792d7163901a","Type":"ContainerStarted","Data":"eb23c5e9e3499ccf83e2b9fb35fc65ea6e7ed84512843b1ce6ee328ebe03cc8b"} Oct 30 01:58:06.695100 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:06.694617 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2eae39/payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz" Oct 30 01:58:06.732470 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:06.732410 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2eae39/payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz" podStartSLOduration=1.732394662 podStartE2EDuration="1.732394662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:58:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:58:06.731644632 +0000 UTC m=+2412.308070031" watchObservedRunningTime="2026-10-30 01:58:06.732394662 +0000 UTC m=+2412.308820066" Oct 30 01:58:06.812187 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:06.812140 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:07.028060 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:07.027976 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38e55eea-42q97"] Oct 30 01:58:07.031928 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:07.031905 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38e55eea-42q97"] Oct 30 01:58:07.591116 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:07.591047 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:08.552841 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:08.552806 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="496115bd-ea68-46f9-811d-40204f4ee403" path="/var/lib/kubelet/pods/496115bd-ea68-46f9-811d-40204f4ee403/volumes" Oct 30 01:58:09.123108 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:09.123070 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:10.028762 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:10.028727 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-523e05e3-h5j7x"] Oct 30 01:58:10.039028 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:10.038997 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-523e05e3-h5j7x"] Oct 30 01:58:10.544413 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:10.544382 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cba074e7-47f7-440b-bf10-db550a616970" path="/var/lib/kubelet/pods/cba074e7-47f7-440b-bf10-db550a616970/volumes" Oct 30 01:58:10.619338 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:10.619304 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:11.681848 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:11.681813 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2eae39/payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz"] Oct 30 01:58:11.685382 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:11.685361 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2eae39/payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz" Oct 30 01:58:11.850092 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:11.850059 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:12.165791 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:12.164763 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:12.464727 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:12.464497 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:14.042706 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:14.042653 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:14.245794 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:14.245743 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:14.689896 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:58:14.689861 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:58:14.789692 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:14.789603 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b765065e-524x2"] Oct 30 01:58:14.795957 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:14.795933 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b765065e-524x2" Oct 30 01:58:14.811855 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:14.811813 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b765065e-524x2"] Oct 30 01:58:14.818965 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:14.818855 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:58:14.818965 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:14.818906 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:58:14.818965 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:14.818940 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:58:14.821418 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:14.821384 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:58:14.883860 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:14.883795 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5175c99-df96-4d90-90ea-c2274424a8a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b765065e-524x2\" (UID: \"e5175c99-df96-4d90-90ea-c2274424a8a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b765065e-524x2" Oct 30 01:58:14.884069 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:14.883917 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5175c99-df96-4d90-90ea-c2274424a8a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b765065e-524x2\" (UID: \"e5175c99-df96-4d90-90ea-c2274424a8a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b765065e-524x2" Oct 30 01:58:14.884116 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:14.884072 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n96pn\" (UniqueName: \"kubernetes.io/projected/e5175c99-df96-4d90-90ea-c2274424a8a5-kube-api-access-n96pn\") pod \"maas-api-revoke-sub-b765065e-524x2\" (UID: \"e5175c99-df96-4d90-90ea-c2274424a8a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b765065e-524x2" Oct 30 01:58:14.985906 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:14.985870 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n96pn\" (UniqueName: \"kubernetes.io/projected/e5175c99-df96-4d90-90ea-c2274424a8a5-kube-api-access-n96pn\") pod \"maas-api-revoke-sub-b765065e-524x2\" (UID: \"e5175c99-df96-4d90-90ea-c2274424a8a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b765065e-524x2" Oct 30 01:58:14.986076 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:14.986036 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5175c99-df96-4d90-90ea-c2274424a8a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b765065e-524x2\" (UID: \"e5175c99-df96-4d90-90ea-c2274424a8a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b765065e-524x2" Oct 30 01:58:14.986076 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:14.986072 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5175c99-df96-4d90-90ea-c2274424a8a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b765065e-524x2\" (UID: \"e5175c99-df96-4d90-90ea-c2274424a8a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b765065e-524x2" Oct 30 01:58:14.986841 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:14.986817 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5175c99-df96-4d90-90ea-c2274424a8a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b765065e-524x2\" (UID: \"e5175c99-df96-4d90-90ea-c2274424a8a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b765065e-524x2" Oct 30 01:58:14.988656 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:14.988631 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5175c99-df96-4d90-90ea-c2274424a8a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b765065e-524x2\" (UID: \"e5175c99-df96-4d90-90ea-c2274424a8a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b765065e-524x2" Oct 30 01:58:15.004896 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:15.004869 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n96pn\" (UniqueName: \"kubernetes.io/projected/e5175c99-df96-4d90-90ea-c2274424a8a5-kube-api-access-n96pn\") pod \"maas-api-revoke-sub-b765065e-524x2\" (UID: \"e5175c99-df96-4d90-90ea-c2274424a8a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b765065e-524x2" Oct 30 01:58:15.113684 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:15.113597 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b765065e-524x2" Oct 30 01:58:15.467859 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:15.467826 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b765065e-524x2"] Oct 30 01:58:15.468026 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:58:15.468003 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5175c99_df96_4d90_90ea_c2274424a8a5.slice/crio-b38f57817fbc923c218a5947ea98a6ae8dc894c89017c7cf26ee63b0832a44dd WatchSource:0}: Error finding container b38f57817fbc923c218a5947ea98a6ae8dc894c89017c7cf26ee63b0832a44dd: Status 404 returned error can't find the container with id b38f57817fbc923c218a5947ea98a6ae8dc894c89017c7cf26ee63b0832a44dd Oct 30 01:58:15.688245 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:15.688211 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-f6e11e-7d74b77d6-w25pd" Oct 30 01:58:15.688466 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:15.688293 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f6e11e-794f9f4fc5-z7q9b" Oct 30 01:58:15.693246 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:15.693211 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:15.774162 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:15.774083 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b765065e-524x2" event={"ID":"e5175c99-df96-4d90-90ea-c2274424a8a5","Type":"ContainerStarted","Data":"b38f57817fbc923c218a5947ea98a6ae8dc894c89017c7cf26ee63b0832a44dd"} Oct 30 01:58:16.682511 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:16.682468 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2eae39/payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz" podUID="421ef56b-ad77-4985-ab23-792d7163901a" containerName="payload-processing" containerID="cri-o://b486bb8933390cc9b3cf7130e49d24b3ba6505d2113adf83f685a097c5376a27" gracePeriod=25 Oct 30 01:58:16.784563 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:16.784458 6572 generic.go:356] "Generic (PLEG): container finished" podID="e5175c99-df96-4d90-90ea-c2274424a8a5" containerID="a10dbcef00425120267d35a4ea67600d2ba257465768385c55f8c3bdbacaa3bd" exitCode=0 Oct 30 01:58:16.784563 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:16.784539 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b765065e-524x2" event={"ID":"e5175c99-df96-4d90-90ea-c2274424a8a5","Type":"ContainerDied","Data":"a10dbcef00425120267d35a4ea67600d2ba257465768385c55f8c3bdbacaa3bd"} Oct 30 01:58:17.010176 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:17.010143 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2eae39/payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz" Oct 30 01:58:17.032637 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:17.032597 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb95e69a-rdmmv"] Oct 30 01:58:17.036575 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:17.036514 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb95e69a-rdmmv"] Oct 30 01:58:17.120547 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:17.120500 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/421ef56b-ad77-4985-ab23-792d7163901a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/421ef56b-ad77-4985-ab23-792d7163901a-plugins-config-volume\") pod \"421ef56b-ad77-4985-ab23-792d7163901a\" (UID: \"421ef56b-ad77-4985-ab23-792d7163901a\") " Oct 30 01:58:17.120731 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:17.120665 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/421ef56b-ad77-4985-ab23-792d7163901a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/421ef56b-ad77-4985-ab23-792d7163901a-provider-credentials\") pod \"421ef56b-ad77-4985-ab23-792d7163901a\" (UID: \"421ef56b-ad77-4985-ab23-792d7163901a\") " Oct 30 01:58:17.120907 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:17.120727 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/421ef56b-ad77-4985-ab23-792d7163901a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/421ef56b-ad77-4985-ab23-792d7163901a-routing-overlay-volume\") pod \"421ef56b-ad77-4985-ab23-792d7163901a\" (UID: \"421ef56b-ad77-4985-ab23-792d7163901a\") " Oct 30 01:58:17.120907 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:17.120882 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/421ef56b-ad77-4985-ab23-792d7163901a-kube-api-access-25mj4\" (UniqueName: \"kubernetes.io/projected/421ef56b-ad77-4985-ab23-792d7163901a-kube-api-access-25mj4\") pod \"421ef56b-ad77-4985-ab23-792d7163901a\" (UID: \"421ef56b-ad77-4985-ab23-792d7163901a\") " Oct 30 01:58:17.121228 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:17.121199 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/421ef56b-ad77-4985-ab23-792d7163901a-plugins-config-volume" pod "421ef56b-ad77-4985-ab23-792d7163901a" (UID: "421ef56b-ad77-4985-ab23-792d7163901a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:58:17.121357 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:17.121244 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/421ef56b-ad77-4985-ab23-792d7163901a-routing-overlay-volume" pod "421ef56b-ad77-4985-ab23-792d7163901a" (UID: "421ef56b-ad77-4985-ab23-792d7163901a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:58:17.121574 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:17.121546 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/421ef56b-ad77-4985-ab23-792d7163901a-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:58:17.121649 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:17.121574 6572 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/421ef56b-ad77-4985-ab23-792d7163901a-routing-overlay-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:58:17.123212 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:17.123187 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/421ef56b-ad77-4985-ab23-792d7163901a-provider-credentials" pod "421ef56b-ad77-4985-ab23-792d7163901a" (UID: "421ef56b-ad77-4985-ab23-792d7163901a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:58:17.123490 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:17.123468 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/421ef56b-ad77-4985-ab23-792d7163901a-kube-api-access-25mj4" pod "421ef56b-ad77-4985-ab23-792d7163901a" (UID: "421ef56b-ad77-4985-ab23-792d7163901a"). InnerVolumeSpecName "kube-api-access-25mj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:58:17.223550 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:17.223511 6572 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/421ef56b-ad77-4985-ab23-792d7163901a-provider-credentials\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:58:17.223550 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:17.223551 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25mj4\" (UniqueName: \"kubernetes.io/projected/421ef56b-ad77-4985-ab23-792d7163901a-kube-api-access-25mj4\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:58:17.283333 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:17.283296 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:17.794197 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:17.794156 6572 generic.go:356] "Generic (PLEG): container finished" podID="421ef56b-ad77-4985-ab23-792d7163901a" containerID="b486bb8933390cc9b3cf7130e49d24b3ba6505d2113adf83f685a097c5376a27" exitCode=0 Oct 30 01:58:17.794673 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:17.794209 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2eae39/payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz" Oct 30 01:58:17.794673 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:17.794240 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2eae39/payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz" event={"ID":"421ef56b-ad77-4985-ab23-792d7163901a","Type":"ContainerDied","Data":"b486bb8933390cc9b3cf7130e49d24b3ba6505d2113adf83f685a097c5376a27"} Oct 30 01:58:17.794673 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:17.794316 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2eae39/payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz" event={"ID":"421ef56b-ad77-4985-ab23-792d7163901a","Type":"ContainerDied","Data":"eb23c5e9e3499ccf83e2b9fb35fc65ea6e7ed84512843b1ce6ee328ebe03cc8b"} Oct 30 01:58:17.794673 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:17.794346 6572 scope.go:122] "RemoveContainer" containerID="b486bb8933390cc9b3cf7130e49d24b3ba6505d2113adf83f685a097c5376a27" Oct 30 01:58:17.807448 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:17.807407 6572 scope.go:122] "RemoveContainer" containerID="b486bb8933390cc9b3cf7130e49d24b3ba6505d2113adf83f685a097c5376a27" Oct 30 01:58:17.807866 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:58:17.807835 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b486bb8933390cc9b3cf7130e49d24b3ba6505d2113adf83f685a097c5376a27\": container with ID starting with b486bb8933390cc9b3cf7130e49d24b3ba6505d2113adf83f685a097c5376a27 not found: ID does not exist" containerID="b486bb8933390cc9b3cf7130e49d24b3ba6505d2113adf83f685a097c5376a27" Oct 30 01:58:17.807974 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:17.807876 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b486bb8933390cc9b3cf7130e49d24b3ba6505d2113adf83f685a097c5376a27"} err="failed to get container status \"b486bb8933390cc9b3cf7130e49d24b3ba6505d2113adf83f685a097c5376a27\": rpc error: code = NotFound desc = could not find container \"b486bb8933390cc9b3cf7130e49d24b3ba6505d2113adf83f685a097c5376a27\": container with ID starting with b486bb8933390cc9b3cf7130e49d24b3ba6505d2113adf83f685a097c5376a27 not found: ID does not exist" Oct 30 01:58:17.849040 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:17.849004 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2eae39/payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz"] Oct 30 01:58:17.864140 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:17.864106 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2eae39/payload-processing-external-model-e2e-extmod-a-2eae39-64dfxkhlz"] Oct 30 01:58:17.980215 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:17.980182 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b765065e-524x2" Oct 30 01:58:18.036202 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:18.036165 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5175c99-df96-4d90-90ea-c2274424a8a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5175c99-df96-4d90-90ea-c2274424a8a5-maas-api-openshift-ca\") pod \"e5175c99-df96-4d90-90ea-c2274424a8a5\" (UID: \"e5175c99-df96-4d90-90ea-c2274424a8a5\") " Oct 30 01:58:18.036431 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:18.036278 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5175c99-df96-4d90-90ea-c2274424a8a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5175c99-df96-4d90-90ea-c2274424a8a5-maas-api-xks-ca\") pod \"e5175c99-df96-4d90-90ea-c2274424a8a5\" (UID: \"e5175c99-df96-4d90-90ea-c2274424a8a5\") " Oct 30 01:58:18.036431 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:18.036307 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5175c99-df96-4d90-90ea-c2274424a8a5-kube-api-access-n96pn\" (UniqueName: \"kubernetes.io/projected/e5175c99-df96-4d90-90ea-c2274424a8a5-kube-api-access-n96pn\") pod \"e5175c99-df96-4d90-90ea-c2274424a8a5\" (UID: \"e5175c99-df96-4d90-90ea-c2274424a8a5\") " Oct 30 01:58:18.036626 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:18.036596 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5175c99-df96-4d90-90ea-c2274424a8a5-maas-api-openshift-ca" pod "e5175c99-df96-4d90-90ea-c2274424a8a5" (UID: "e5175c99-df96-4d90-90ea-c2274424a8a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:58:18.037111 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:18.037087 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5175c99-df96-4d90-90ea-c2274424a8a5-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:58:18.038493 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:18.038468 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5175c99-df96-4d90-90ea-c2274424a8a5-maas-api-xks-ca" pod "e5175c99-df96-4d90-90ea-c2274424a8a5" (UID: "e5175c99-df96-4d90-90ea-c2274424a8a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:58:18.038652 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:18.038637 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5175c99-df96-4d90-90ea-c2274424a8a5-kube-api-access-n96pn" pod "e5175c99-df96-4d90-90ea-c2274424a8a5" (UID: "e5175c99-df96-4d90-90ea-c2274424a8a5"). InnerVolumeSpecName "kube-api-access-n96pn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:58:18.139077 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:18.139033 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5175c99-df96-4d90-90ea-c2274424a8a5-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:58:18.139077 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:18.139073 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n96pn\" (UniqueName: \"kubernetes.io/projected/e5175c99-df96-4d90-90ea-c2274424a8a5-kube-api-access-n96pn\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:58:18.546858 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:18.546824 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cde2113-52f8-40d6-a1ee-fb94a2d497a2" path="/var/lib/kubelet/pods/3cde2113-52f8-40d6-a1ee-fb94a2d497a2/volumes" Oct 30 01:58:18.547283 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:18.547268 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="421ef56b-ad77-4985-ab23-792d7163901a" path="/var/lib/kubelet/pods/421ef56b-ad77-4985-ab23-792d7163901a/volumes" Oct 30 01:58:18.833990 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:18.833887 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:18.835132 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:18.835109 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b765065e-524x2" Oct 30 01:58:18.835255 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:18.835125 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b765065e-524x2" event={"ID":"e5175c99-df96-4d90-90ea-c2274424a8a5","Type":"ContainerDied","Data":"b38f57817fbc923c218a5947ea98a6ae8dc894c89017c7cf26ee63b0832a44dd"} Oct 30 01:58:18.835255 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:18.835191 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b38f57817fbc923c218a5947ea98a6ae8dc894c89017c7cf26ee63b0832a44dd" Oct 30 01:58:20.074460 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:20.074417 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f579307-5p8ng"] Oct 30 01:58:20.085858 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:20.085822 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f579307-5p8ng"] Oct 30 01:58:20.491843 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:20.491815 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:20.544415 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:20.544380 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="928c4da4-c1a1-4f68-9852-d1e80408a421" path="/var/lib/kubelet/pods/928c4da4-c1a1-4f68-9852-d1e80408a421/volumes" Oct 30 01:58:20.896832 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:20.892843 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:21.076740 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:21.076707 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:22.088275 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:22.088233 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:22.304229 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:22.304192 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-171c395f-9mw5n"] Oct 30 01:58:22.305036 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:22.305019 6572 cpu_manager.go:414] "containerMap: removing container" podUID="421ef56b-ad77-4985-ab23-792d7163901a" containerName="payload-processing" Oct 30 01:58:22.305113 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:22.305037 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="421ef56b-ad77-4985-ab23-792d7163901a" containerName="payload-processing" Oct 30 01:58:22.305113 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:22.305061 6572 cpu_manager.go:414] "containerMap: removing container" podUID="e5175c99-df96-4d90-90ea-c2274424a8a5" containerName="revoke-keys" Oct 30 01:58:22.305113 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:22.305067 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5175c99-df96-4d90-90ea-c2274424a8a5" containerName="revoke-keys" Oct 30 01:58:22.305211 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:22.305179 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="421ef56b-ad77-4985-ab23-792d7163901a" containerName="payload-processing" Oct 30 01:58:22.305211 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:22.305193 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5175c99-df96-4d90-90ea-c2274424a8a5" containerName="revoke-keys" Oct 30 01:58:22.308973 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:22.308956 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-171c395f-9mw5n" Oct 30 01:58:22.327506 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:22.327478 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-171c395f-9mw5n"] Oct 30 01:58:22.333805 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:22.333778 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:58:22.333805 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:22.333800 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:58:22.333993 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:22.333776 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:58:22.335249 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:22.335232 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:58:22.409419 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:22.409327 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11f367bb-5a71-4dad-8f72-88ef98f109f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-171c395f-9mw5n\" (UID: \"11f367bb-5a71-4dad-8f72-88ef98f109f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-171c395f-9mw5n" Oct 30 01:58:22.409419 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:22.409374 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwzp8\" (UniqueName: \"kubernetes.io/projected/11f367bb-5a71-4dad-8f72-88ef98f109f9-kube-api-access-jwzp8\") pod \"maas-api-revoke-sub-171c395f-9mw5n\" (UID: \"11f367bb-5a71-4dad-8f72-88ef98f109f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-171c395f-9mw5n" Oct 30 01:58:22.409664 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:22.409516 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11f367bb-5a71-4dad-8f72-88ef98f109f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-171c395f-9mw5n\" (UID: \"11f367bb-5a71-4dad-8f72-88ef98f109f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-171c395f-9mw5n" Oct 30 01:58:22.511699 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:22.511630 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11f367bb-5a71-4dad-8f72-88ef98f109f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-171c395f-9mw5n\" (UID: \"11f367bb-5a71-4dad-8f72-88ef98f109f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-171c395f-9mw5n" Oct 30 01:58:22.512151 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:22.512127 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11f367bb-5a71-4dad-8f72-88ef98f109f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-171c395f-9mw5n\" (UID: \"11f367bb-5a71-4dad-8f72-88ef98f109f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-171c395f-9mw5n" Oct 30 01:58:22.513769 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:22.513727 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwzp8\" (UniqueName: \"kubernetes.io/projected/11f367bb-5a71-4dad-8f72-88ef98f109f9-kube-api-access-jwzp8\") pod \"maas-api-revoke-sub-171c395f-9mw5n\" (UID: \"11f367bb-5a71-4dad-8f72-88ef98f109f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-171c395f-9mw5n" Oct 30 01:58:22.514335 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:22.514314 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11f367bb-5a71-4dad-8f72-88ef98f109f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-171c395f-9mw5n\" (UID: \"11f367bb-5a71-4dad-8f72-88ef98f109f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-171c395f-9mw5n" Oct 30 01:58:22.575505 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:22.568522 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwzp8\" (UniqueName: \"kubernetes.io/projected/11f367bb-5a71-4dad-8f72-88ef98f109f9-kube-api-access-jwzp8\") pod \"maas-api-revoke-sub-171c395f-9mw5n\" (UID: \"11f367bb-5a71-4dad-8f72-88ef98f109f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-171c395f-9mw5n" Oct 30 01:58:22.585302 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:22.582648 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11f367bb-5a71-4dad-8f72-88ef98f109f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-171c395f-9mw5n\" (UID: \"11f367bb-5a71-4dad-8f72-88ef98f109f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-171c395f-9mw5n" Oct 30 01:58:22.629126 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:22.629082 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-171c395f-9mw5n" Oct 30 01:58:22.948759 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:22.948718 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-171c395f-9mw5n"] Oct 30 01:58:22.950101 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:58:22.950069 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11f367bb_5a71_4dad_8f72_88ef98f109f9.slice/crio-5418988084f52f04290c9d78aa55e5c9d127dad8e4e7836e67e8fb870aceaf1d WatchSource:0}: Error finding container 5418988084f52f04290c9d78aa55e5c9d127dad8e4e7836e67e8fb870aceaf1d: Status 404 returned error can't find the container with id 5418988084f52f04290c9d78aa55e5c9d127dad8e4e7836e67e8fb870aceaf1d Oct 30 01:58:23.683985 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:23.683946 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:23.885284 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:23.885250 6572 generic.go:356] "Generic (PLEG): container finished" podID="11f367bb-5a71-4dad-8f72-88ef98f109f9" containerID="11137f13516f3739c874a1f44790c96e36e8174840bf1b7821c262e6e7955d17" exitCode=0 Oct 30 01:58:23.885447 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:23.885339 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-171c395f-9mw5n" event={"ID":"11f367bb-5a71-4dad-8f72-88ef98f109f9","Type":"ContainerDied","Data":"11137f13516f3739c874a1f44790c96e36e8174840bf1b7821c262e6e7955d17"} Oct 30 01:58:23.885447 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:23.885399 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-171c395f-9mw5n" event={"ID":"11f367bb-5a71-4dad-8f72-88ef98f109f9","Type":"ContainerStarted","Data":"5418988084f52f04290c9d78aa55e5c9d127dad8e4e7836e67e8fb870aceaf1d"} Oct 30 01:58:25.071635 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:25.071613 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-171c395f-9mw5n" Oct 30 01:58:25.171399 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:25.171371 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11f367bb-5a71-4dad-8f72-88ef98f109f9-kube-api-access-jwzp8\" (UniqueName: \"kubernetes.io/projected/11f367bb-5a71-4dad-8f72-88ef98f109f9-kube-api-access-jwzp8\") pod \"11f367bb-5a71-4dad-8f72-88ef98f109f9\" (UID: \"11f367bb-5a71-4dad-8f72-88ef98f109f9\") " Oct 30 01:58:25.171573 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:25.171561 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11f367bb-5a71-4dad-8f72-88ef98f109f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11f367bb-5a71-4dad-8f72-88ef98f109f9-maas-api-openshift-ca\") pod \"11f367bb-5a71-4dad-8f72-88ef98f109f9\" (UID: \"11f367bb-5a71-4dad-8f72-88ef98f109f9\") " Oct 30 01:58:25.171644 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:25.171631 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11f367bb-5a71-4dad-8f72-88ef98f109f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11f367bb-5a71-4dad-8f72-88ef98f109f9-maas-api-xks-ca\") pod \"11f367bb-5a71-4dad-8f72-88ef98f109f9\" (UID: \"11f367bb-5a71-4dad-8f72-88ef98f109f9\") " Oct 30 01:58:25.171960 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:25.171935 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11f367bb-5a71-4dad-8f72-88ef98f109f9-maas-api-openshift-ca" pod "11f367bb-5a71-4dad-8f72-88ef98f109f9" (UID: "11f367bb-5a71-4dad-8f72-88ef98f109f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:58:25.172234 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:25.172216 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11f367bb-5a71-4dad-8f72-88ef98f109f9-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:58:25.173492 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:25.173469 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f367bb-5a71-4dad-8f72-88ef98f109f9-kube-api-access-jwzp8" pod "11f367bb-5a71-4dad-8f72-88ef98f109f9" (UID: "11f367bb-5a71-4dad-8f72-88ef98f109f9"). InnerVolumeSpecName "kube-api-access-jwzp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:58:25.173684 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:25.173664 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11f367bb-5a71-4dad-8f72-88ef98f109f9-maas-api-xks-ca" pod "11f367bb-5a71-4dad-8f72-88ef98f109f9" (UID: "11f367bb-5a71-4dad-8f72-88ef98f109f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:58:25.273484 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:25.273456 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11f367bb-5a71-4dad-8f72-88ef98f109f9-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:58:25.273484 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:25.273484 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwzp8\" (UniqueName: \"kubernetes.io/projected/11f367bb-5a71-4dad-8f72-88ef98f109f9-kube-api-access-jwzp8\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:58:25.356633 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:25.356600 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:25.904466 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:25.904432 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-171c395f-9mw5n" event={"ID":"11f367bb-5a71-4dad-8f72-88ef98f109f9","Type":"ContainerDied","Data":"5418988084f52f04290c9d78aa55e5c9d127dad8e4e7836e67e8fb870aceaf1d"} Oct 30 01:58:25.904466 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:25.904460 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-171c395f-9mw5n" Oct 30 01:58:25.904689 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:25.904486 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5418988084f52f04290c9d78aa55e5c9d127dad8e4e7836e67e8fb870aceaf1d" Oct 30 01:58:26.055476 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:26.055437 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst"] Oct 30 01:58:26.222174 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:26.222144 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:26.422766 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:26.422528 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:26.648362 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:26.645319 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:26.936473 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:26.936435 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:27.130411 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:27.130379 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:27.319650 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:27.319562 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:27.617383 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:27.617304 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:27.815596 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:27.815556 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:28.345890 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:28.345855 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:28.572405 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:28.572360 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:29.207117 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:29.207074 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:29.399624 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:29.399575 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:29.590393 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:29.590309 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:29.784555 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:29.784510 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:30.427691 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:30.427646 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx"] Oct 30 01:58:30.429462 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:30.429429 6572 cpu_manager.go:414] "containerMap: removing container" podUID="11f367bb-5a71-4dad-8f72-88ef98f109f9" containerName="revoke-keys" Oct 30 01:58:30.429462 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:30.429461 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11f367bb-5a71-4dad-8f72-88ef98f109f9" containerName="revoke-keys" Oct 30 01:58:30.429884 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:30.429864 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="11f367bb-5a71-4dad-8f72-88ef98f109f9" containerName="revoke-keys" Oct 30 01:58:30.435269 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:30.435246 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx" Oct 30 01:58:30.450630 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:30.450604 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx"] Oct 30 01:58:30.456904 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:30.456877 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-76a19e\"/\"agc-cred-147ea441a8c3\"" Oct 30 01:58:30.457966 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:30.457940 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-76a19e\"/\"agc-cred-1786453aff48\"" Oct 30 01:58:30.458088 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:30.458002 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-76a19e\"/\"payload-processing-external-model-e2e--70e20018-dockercfg-mldgh\"" Oct 30 01:58:30.502535 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:30.502493 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx\" (UID: \"e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx" Oct 30 01:58:30.502727 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:30.502606 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx\" (UID: \"e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx" Oct 30 01:58:30.502727 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:30.502664 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx\" (UID: \"e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx" Oct 30 01:58:30.502727 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:30.502686 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btvll\" (UniqueName: \"kubernetes.io/projected/e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a-kube-api-access-btvll\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx\" (UID: \"e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx" Oct 30 01:58:30.604672 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:30.604632 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx\" (UID: \"e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx" Oct 30 01:58:30.604952 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:30.604703 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx\" (UID: \"e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx" Oct 30 01:58:30.604952 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:30.604803 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx\" (UID: \"e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx" Oct 30 01:58:30.604952 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:30.604839 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btvll\" (UniqueName: \"kubernetes.io/projected/e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a-kube-api-access-btvll\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx\" (UID: \"e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx" Oct 30 01:58:30.605424 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:30.605400 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx\" (UID: \"e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx" Oct 30 01:58:30.605621 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:30.605601 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx\" (UID: \"e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx" Oct 30 01:58:30.607549 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:30.607524 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx\" (UID: \"e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx" Oct 30 01:58:30.628099 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:30.628071 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btvll\" (UniqueName: \"kubernetes.io/projected/e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a-kube-api-access-btvll\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx\" (UID: \"e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx" Oct 30 01:58:30.752563 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:30.752478 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx" Oct 30 01:58:30.987587 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:30.987531 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:31.056645 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:31.056595 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst" podUID="cc1f69a6-3a72-4be6-b33e-c3b87abffcde" containerName="payload-processing" containerID="cri-o://e0e51a8b3113f9dbe9c8be6dc0302cffaa00d2a21f18f821c1d422693c4350b0" gracePeriod=25 Oct 30 01:58:31.057594 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:31.057570 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx"] Oct 30 01:58:31.058097 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:58:31.058074 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode73dcd7f_a5c3_4468_a15e_d98a9bf94c5a.slice/crio-b58fd9e11f1fe2d6fb3237095678cbf2102028d33c2b7f554a73ea4dd70bfc9b WatchSource:0}: Error finding container b58fd9e11f1fe2d6fb3237095678cbf2102028d33c2b7f554a73ea4dd70bfc9b: Status 404 returned error can't find the container with id b58fd9e11f1fe2d6fb3237095678cbf2102028d33c2b7f554a73ea4dd70bfc9b Oct 30 01:58:31.323637 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:31.323611 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst" Oct 30 01:58:31.416062 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:31.416031 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc1f69a6-3a72-4be6-b33e-c3b87abffcde-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc1f69a6-3a72-4be6-b33e-c3b87abffcde-routing-overlay-volume\") pod \"cc1f69a6-3a72-4be6-b33e-c3b87abffcde\" (UID: \"cc1f69a6-3a72-4be6-b33e-c3b87abffcde\") " Oct 30 01:58:31.416236 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:31.416107 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc1f69a6-3a72-4be6-b33e-c3b87abffcde-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc1f69a6-3a72-4be6-b33e-c3b87abffcde-plugins-config-volume\") pod \"cc1f69a6-3a72-4be6-b33e-c3b87abffcde\" (UID: \"cc1f69a6-3a72-4be6-b33e-c3b87abffcde\") " Oct 30 01:58:31.416236 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:31.416196 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc1f69a6-3a72-4be6-b33e-c3b87abffcde-kube-api-access-dfl7n\" (UniqueName: \"kubernetes.io/projected/cc1f69a6-3a72-4be6-b33e-c3b87abffcde-kube-api-access-dfl7n\") pod \"cc1f69a6-3a72-4be6-b33e-c3b87abffcde\" (UID: \"cc1f69a6-3a72-4be6-b33e-c3b87abffcde\") " Oct 30 01:58:31.416416 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:31.416387 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc1f69a6-3a72-4be6-b33e-c3b87abffcde-routing-overlay-volume" pod "cc1f69a6-3a72-4be6-b33e-c3b87abffcde" (UID: "cc1f69a6-3a72-4be6-b33e-c3b87abffcde"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:58:31.416489 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:31.416399 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc1f69a6-3a72-4be6-b33e-c3b87abffcde-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc1f69a6-3a72-4be6-b33e-c3b87abffcde-provider-credentials\") pod \"cc1f69a6-3a72-4be6-b33e-c3b87abffcde\" (UID: \"cc1f69a6-3a72-4be6-b33e-c3b87abffcde\") " Oct 30 01:58:31.416582 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:31.416553 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc1f69a6-3a72-4be6-b33e-c3b87abffcde-plugins-config-volume" pod "cc1f69a6-3a72-4be6-b33e-c3b87abffcde" (UID: "cc1f69a6-3a72-4be6-b33e-c3b87abffcde"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:58:31.416993 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:31.416974 6572 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc1f69a6-3a72-4be6-b33e-c3b87abffcde-routing-overlay-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:58:31.416993 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:31.416992 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc1f69a6-3a72-4be6-b33e-c3b87abffcde-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:58:31.418538 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:31.418510 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc1f69a6-3a72-4be6-b33e-c3b87abffcde-kube-api-access-dfl7n" pod "cc1f69a6-3a72-4be6-b33e-c3b87abffcde" (UID: "cc1f69a6-3a72-4be6-b33e-c3b87abffcde"). InnerVolumeSpecName "kube-api-access-dfl7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:58:31.418648 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:31.418571 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc1f69a6-3a72-4be6-b33e-c3b87abffcde-provider-credentials" pod "cc1f69a6-3a72-4be6-b33e-c3b87abffcde" (UID: "cc1f69a6-3a72-4be6-b33e-c3b87abffcde"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:58:31.518884 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:31.518839 6572 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc1f69a6-3a72-4be6-b33e-c3b87abffcde-provider-credentials\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:58:31.518884 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:31.518875 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfl7n\" (UniqueName: \"kubernetes.io/projected/cc1f69a6-3a72-4be6-b33e-c3b87abffcde-kube-api-access-dfl7n\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:58:32.004382 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:32.004343 6572 generic.go:356] "Generic (PLEG): container finished" podID="cc1f69a6-3a72-4be6-b33e-c3b87abffcde" containerID="e0e51a8b3113f9dbe9c8be6dc0302cffaa00d2a21f18f821c1d422693c4350b0" exitCode=0 Oct 30 01:58:32.004573 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:32.004392 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst" Oct 30 01:58:32.004573 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:32.004425 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst" event={"ID":"cc1f69a6-3a72-4be6-b33e-c3b87abffcde","Type":"ContainerDied","Data":"e0e51a8b3113f9dbe9c8be6dc0302cffaa00d2a21f18f821c1d422693c4350b0"} Oct 30 01:58:32.004573 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:32.004483 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst" event={"ID":"cc1f69a6-3a72-4be6-b33e-c3b87abffcde","Type":"ContainerDied","Data":"3ac505a8d9fbf67b6398015ebc16a85cc657246b002afe90cd12d3b2652ca1b4"} Oct 30 01:58:32.004573 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:32.004507 6572 scope.go:122] "RemoveContainer" containerID="e0e51a8b3113f9dbe9c8be6dc0302cffaa00d2a21f18f821c1d422693c4350b0" Oct 30 01:58:32.006643 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:32.006616 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx" event={"ID":"e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a","Type":"ContainerStarted","Data":"644163eb4e418e8be8f8f796e5f6ac1f00562d2a4eca65ca3998b867dbf2e16d"} Oct 30 01:58:32.006798 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:32.006705 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx" Oct 30 01:58:32.006798 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:32.006735 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx" event={"ID":"e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a","Type":"ContainerStarted","Data":"b58fd9e11f1fe2d6fb3237095678cbf2102028d33c2b7f554a73ea4dd70bfc9b"} Oct 30 01:58:32.016251 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:32.016228 6572 scope.go:122] "RemoveContainer" containerID="e0e51a8b3113f9dbe9c8be6dc0302cffaa00d2a21f18f821c1d422693c4350b0" Oct 30 01:58:32.016543 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:58:32.016524 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0e51a8b3113f9dbe9c8be6dc0302cffaa00d2a21f18f821c1d422693c4350b0\": container with ID starting with e0e51a8b3113f9dbe9c8be6dc0302cffaa00d2a21f18f821c1d422693c4350b0 not found: ID does not exist" containerID="e0e51a8b3113f9dbe9c8be6dc0302cffaa00d2a21f18f821c1d422693c4350b0" Oct 30 01:58:32.016597 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:32.016553 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0e51a8b3113f9dbe9c8be6dc0302cffaa00d2a21f18f821c1d422693c4350b0"} err="failed to get container status \"e0e51a8b3113f9dbe9c8be6dc0302cffaa00d2a21f18f821c1d422693c4350b0\": rpc error: code = NotFound desc = could not find container \"e0e51a8b3113f9dbe9c8be6dc0302cffaa00d2a21f18f821c1d422693c4350b0\": container with ID starting with e0e51a8b3113f9dbe9c8be6dc0302cffaa00d2a21f18f821c1d422693c4350b0 not found: ID does not exist" Oct 30 01:58:32.063896 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:32.063838 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx" podStartSLOduration=2.063823783 podStartE2EDuration="2.063823783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:58:32.059970184 +0000 UTC m=+2437.636395586" watchObservedRunningTime="2026-10-30 01:58:32.063823783 +0000 UTC m=+2437.640249226" Oct 30 01:58:32.097765 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:32.097719 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst"] Oct 30 01:58:32.101914 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:32.101887 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-58f89kcst"] Oct 30 01:58:32.538415 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:32.538382 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:32.544019 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:32.543993 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc1f69a6-3a72-4be6-b33e-c3b87abffcde" path="/var/lib/kubelet/pods/cc1f69a6-3a72-4be6-b33e-c3b87abffcde/volumes" Oct 30 01:58:33.570901 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:33.570825 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w"] Oct 30 01:58:33.671126 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:33.671096 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:33.866280 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:33.866199 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:34.297092 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:34.297053 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:34.486849 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:34.486815 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:35.908619 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:35.908571 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:37.370936 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:37.370906 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:38.571427 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:38.571359 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w" podUID="1cf1131b-1eb3-4a4c-a612-abdc03e7f418" containerName="payload-processing" containerID="cri-o://7b68aefafed2d08dcddf53d4c12d7920f3bfd9114b6637c1713f97a7022fa184" gracePeriod=25 Oct 30 01:58:38.734450 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:58:38.734422 6572 logging.go:55] [core] [Channel #6701 SubChannel #6702]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.17:50052", ServerName: "10.133.1.17:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.17:50052: connect: connection refused" Oct 30 01:58:38.886660 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:38.886631 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w" Oct 30 01:58:38.890480 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:38.890453 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:39.043518 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:39.043478 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cf1131b-1eb3-4a4c-a612-abdc03e7f418-kube-api-access-kgfjm\" (UniqueName: \"kubernetes.io/projected/1cf1131b-1eb3-4a4c-a612-abdc03e7f418-kube-api-access-kgfjm\") pod \"1cf1131b-1eb3-4a4c-a612-abdc03e7f418\" (UID: \"1cf1131b-1eb3-4a4c-a612-abdc03e7f418\") " Oct 30 01:58:39.043702 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:39.043643 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cf1131b-1eb3-4a4c-a612-abdc03e7f418-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1cf1131b-1eb3-4a4c-a612-abdc03e7f418-routing-overlay-volume\") pod \"1cf1131b-1eb3-4a4c-a612-abdc03e7f418\" (UID: \"1cf1131b-1eb3-4a4c-a612-abdc03e7f418\") " Oct 30 01:58:39.043702 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:39.043689 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cf1131b-1eb3-4a4c-a612-abdc03e7f418-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1cf1131b-1eb3-4a4c-a612-abdc03e7f418-provider-credentials\") pod \"1cf1131b-1eb3-4a4c-a612-abdc03e7f418\" (UID: \"1cf1131b-1eb3-4a4c-a612-abdc03e7f418\") " Oct 30 01:58:39.043862 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:39.043832 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cf1131b-1eb3-4a4c-a612-abdc03e7f418-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cf1131b-1eb3-4a4c-a612-abdc03e7f418-plugins-config-volume\") pod \"1cf1131b-1eb3-4a4c-a612-abdc03e7f418\" (UID: \"1cf1131b-1eb3-4a4c-a612-abdc03e7f418\") " Oct 30 01:58:39.044118 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:39.044078 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cf1131b-1eb3-4a4c-a612-abdc03e7f418-routing-overlay-volume" pod "1cf1131b-1eb3-4a4c-a612-abdc03e7f418" (UID: "1cf1131b-1eb3-4a4c-a612-abdc03e7f418"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:58:39.044268 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:39.044246 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cf1131b-1eb3-4a4c-a612-abdc03e7f418-plugins-config-volume" pod "1cf1131b-1eb3-4a4c-a612-abdc03e7f418" (UID: "1cf1131b-1eb3-4a4c-a612-abdc03e7f418"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:58:39.044659 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:39.044637 6572 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1cf1131b-1eb3-4a4c-a612-abdc03e7f418-routing-overlay-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:58:39.044709 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:39.044670 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cf1131b-1eb3-4a4c-a612-abdc03e7f418-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:58:39.045948 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:39.045927 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cf1131b-1eb3-4a4c-a612-abdc03e7f418-kube-api-access-kgfjm" pod "1cf1131b-1eb3-4a4c-a612-abdc03e7f418" (UID: "1cf1131b-1eb3-4a4c-a612-abdc03e7f418"). InnerVolumeSpecName "kube-api-access-kgfjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:58:39.046125 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:39.046101 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cf1131b-1eb3-4a4c-a612-abdc03e7f418-provider-credentials" pod "1cf1131b-1eb3-4a4c-a612-abdc03e7f418" (UID: "1cf1131b-1eb3-4a4c-a612-abdc03e7f418"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:58:39.082665 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:39.082635 6572 generic.go:356] "Generic (PLEG): container finished" podID="1cf1131b-1eb3-4a4c-a612-abdc03e7f418" containerID="7b68aefafed2d08dcddf53d4c12d7920f3bfd9114b6637c1713f97a7022fa184" exitCode=0 Oct 30 01:58:39.082818 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:39.082680 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w" Oct 30 01:58:39.082818 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:39.082716 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w" event={"ID":"1cf1131b-1eb3-4a4c-a612-abdc03e7f418","Type":"ContainerDied","Data":"7b68aefafed2d08dcddf53d4c12d7920f3bfd9114b6637c1713f97a7022fa184"} Oct 30 01:58:39.082818 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:39.082789 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w" event={"ID":"1cf1131b-1eb3-4a4c-a612-abdc03e7f418","Type":"ContainerDied","Data":"eb7d3a1855c984677e24582b5f5d7528bf2ba3d721d2fd25de2470948a15e21f"} Oct 30 01:58:39.082818 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:39.082814 6572 scope.go:122] "RemoveContainer" containerID="7b68aefafed2d08dcddf53d4c12d7920f3bfd9114b6637c1713f97a7022fa184" Oct 30 01:58:39.094924 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:39.094879 6572 scope.go:122] "RemoveContainer" containerID="7b68aefafed2d08dcddf53d4c12d7920f3bfd9114b6637c1713f97a7022fa184" Oct 30 01:58:39.095196 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:58:39.095173 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b68aefafed2d08dcddf53d4c12d7920f3bfd9114b6637c1713f97a7022fa184\": container with ID starting with 7b68aefafed2d08dcddf53d4c12d7920f3bfd9114b6637c1713f97a7022fa184 not found: ID does not exist" containerID="7b68aefafed2d08dcddf53d4c12d7920f3bfd9114b6637c1713f97a7022fa184" Oct 30 01:58:39.095241 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:39.095206 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b68aefafed2d08dcddf53d4c12d7920f3bfd9114b6637c1713f97a7022fa184"} err="failed to get container status \"7b68aefafed2d08dcddf53d4c12d7920f3bfd9114b6637c1713f97a7022fa184\": rpc error: code = NotFound desc = could not find container \"7b68aefafed2d08dcddf53d4c12d7920f3bfd9114b6637c1713f97a7022fa184\": container with ID starting with 7b68aefafed2d08dcddf53d4c12d7920f3bfd9114b6637c1713f97a7022fa184 not found: ID does not exist" Oct 30 01:58:39.135175 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:39.135145 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w"] Oct 30 01:58:39.141646 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:39.141620 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w"] Oct 30 01:58:39.145643 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:39.145618 6572 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1cf1131b-1eb3-4a4c-a612-abdc03e7f418-provider-credentials\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:58:39.145643 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:39.145643 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgfjm\" (UniqueName: \"kubernetes.io/projected/1cf1131b-1eb3-4a4c-a612-abdc03e7f418-kube-api-access-kgfjm\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:58:39.734870 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:39.734826 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-57c94kn5w" podUID="1cf1131b-1eb3-4a4c-a612-abdc03e7f418" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.1.17:50052\" within 1s: context deadline exceeded" Oct 30 01:58:39.735329 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:58:39.735017 6572 logging.go:55] [core] [Channel #6701 SubChannel #6702]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.17:50052", ServerName: "10.133.1.17:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.17:50052: operation was canceled" Oct 30 01:58:40.331204 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:40.331164 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:40.525549 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:40.525511 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:40.545376 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:40.545343 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cf1131b-1eb3-4a4c-a612-abdc03e7f418" path="/var/lib/kubelet/pods/1cf1131b-1eb3-4a4c-a612-abdc03e7f418/volumes" Oct 30 01:58:42.130033 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:42.130001 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:42.329026 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:42.328990 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:43.020246 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:43.020207 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx" Oct 30 01:58:43.756156 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:43.756119 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:44.687409 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:58:44.687369 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:58:45.310559 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:45.310527 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:46.783193 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:46.783156 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:46.973963 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:46.973928 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:47.732879 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:47.732845 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:47.922799 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:47.922767 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:48.772222 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:48.772185 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:49.066825 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:49.065397 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:49.218391 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:49.218356 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-546cdc55c7-d64gv"] Oct 30 01:58:49.219103 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:49.219087 6572 cpu_manager.go:414] "containerMap: removing container" podUID="cc1f69a6-3a72-4be6-b33e-c3b87abffcde" containerName="payload-processing" Oct 30 01:58:49.219152 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:49.219104 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc1f69a6-3a72-4be6-b33e-c3b87abffcde" containerName="payload-processing" Oct 30 01:58:49.219152 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:49.219127 6572 cpu_manager.go:414] "containerMap: removing container" podUID="1cf1131b-1eb3-4a4c-a612-abdc03e7f418" containerName="payload-processing" Oct 30 01:58:49.219152 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:49.219132 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cf1131b-1eb3-4a4c-a612-abdc03e7f418" containerName="payload-processing" Oct 30 01:58:49.219270 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:49.219260 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc1f69a6-3a72-4be6-b33e-c3b87abffcde" containerName="payload-processing" Oct 30 01:58:49.219309 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:49.219276 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cf1131b-1eb3-4a4c-a612-abdc03e7f418" containerName="payload-processing" Oct 30 01:58:49.223036 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:49.223016 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-546cdc55c7-d64gv" Oct 30 01:58:49.237898 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:49.237862 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-284e3d9c18c5\"" Oct 30 01:58:49.249585 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:49.249398 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 01:58:49.249585 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:49.249454 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 01:58:49.249585 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:49.249472 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 01:58:49.249585 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:49.249559 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-nfgtq\"" Oct 30 01:58:49.250008 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:49.249979 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 01:58:49.254629 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:49.254609 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-546cdc55c7-d64gv"] Oct 30 01:58:49.447740 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:49.445580 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r286c\" (UniqueName: \"kubernetes.io/projected/6130dbc6-b1f1-4672-90cc-7e3be07be0e1-kube-api-access-r286c\") pod \"payload-processing-external-model-546cdc55c7-d64gv\" (UID: \"6130dbc6-b1f1-4672-90cc-7e3be07be0e1\") " pod="models-as-a-service/payload-processing-external-model-546cdc55c7-d64gv" Oct 30 01:58:49.447740 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:49.445666 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6130dbc6-b1f1-4672-90cc-7e3be07be0e1-routing-overlay-volume\") pod \"payload-processing-external-model-546cdc55c7-d64gv\" (UID: \"6130dbc6-b1f1-4672-90cc-7e3be07be0e1\") " pod="models-as-a-service/payload-processing-external-model-546cdc55c7-d64gv" Oct 30 01:58:49.447740 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:49.445825 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6130dbc6-b1f1-4672-90cc-7e3be07be0e1-plugins-config-volume\") pod \"payload-processing-external-model-546cdc55c7-d64gv\" (UID: \"6130dbc6-b1f1-4672-90cc-7e3be07be0e1\") " pod="models-as-a-service/payload-processing-external-model-546cdc55c7-d64gv" Oct 30 01:58:49.447740 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:49.445890 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6130dbc6-b1f1-4672-90cc-7e3be07be0e1-provider-credentials\") pod \"payload-processing-external-model-546cdc55c7-d64gv\" (UID: \"6130dbc6-b1f1-4672-90cc-7e3be07be0e1\") " pod="models-as-a-service/payload-processing-external-model-546cdc55c7-d64gv" Oct 30 01:58:49.546799 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:49.546767 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r286c\" (UniqueName: \"kubernetes.io/projected/6130dbc6-b1f1-4672-90cc-7e3be07be0e1-kube-api-access-r286c\") pod \"payload-processing-external-model-546cdc55c7-d64gv\" (UID: \"6130dbc6-b1f1-4672-90cc-7e3be07be0e1\") " pod="models-as-a-service/payload-processing-external-model-546cdc55c7-d64gv" Oct 30 01:58:49.546981 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:49.546813 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6130dbc6-b1f1-4672-90cc-7e3be07be0e1-routing-overlay-volume\") pod \"payload-processing-external-model-546cdc55c7-d64gv\" (UID: \"6130dbc6-b1f1-4672-90cc-7e3be07be0e1\") " pod="models-as-a-service/payload-processing-external-model-546cdc55c7-d64gv" Oct 30 01:58:49.546981 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:49.546911 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6130dbc6-b1f1-4672-90cc-7e3be07be0e1-plugins-config-volume\") pod \"payload-processing-external-model-546cdc55c7-d64gv\" (UID: \"6130dbc6-b1f1-4672-90cc-7e3be07be0e1\") " pod="models-as-a-service/payload-processing-external-model-546cdc55c7-d64gv" Oct 30 01:58:49.546981 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:49.546960 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6130dbc6-b1f1-4672-90cc-7e3be07be0e1-provider-credentials\") pod \"payload-processing-external-model-546cdc55c7-d64gv\" (UID: \"6130dbc6-b1f1-4672-90cc-7e3be07be0e1\") " pod="models-as-a-service/payload-processing-external-model-546cdc55c7-d64gv" Oct 30 01:58:49.547525 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:49.547498 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6130dbc6-b1f1-4672-90cc-7e3be07be0e1-routing-overlay-volume\") pod \"payload-processing-external-model-546cdc55c7-d64gv\" (UID: \"6130dbc6-b1f1-4672-90cc-7e3be07be0e1\") " pod="models-as-a-service/payload-processing-external-model-546cdc55c7-d64gv" Oct 30 01:58:49.547735 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:49.547715 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6130dbc6-b1f1-4672-90cc-7e3be07be0e1-plugins-config-volume\") pod \"payload-processing-external-model-546cdc55c7-d64gv\" (UID: \"6130dbc6-b1f1-4672-90cc-7e3be07be0e1\") " pod="models-as-a-service/payload-processing-external-model-546cdc55c7-d64gv" Oct 30 01:58:49.549368 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:49.549342 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6130dbc6-b1f1-4672-90cc-7e3be07be0e1-provider-credentials\") pod \"payload-processing-external-model-546cdc55c7-d64gv\" (UID: \"6130dbc6-b1f1-4672-90cc-7e3be07be0e1\") " pod="models-as-a-service/payload-processing-external-model-546cdc55c7-d64gv" Oct 30 01:58:49.565143 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:49.565124 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r286c\" (UniqueName: \"kubernetes.io/projected/6130dbc6-b1f1-4672-90cc-7e3be07be0e1-kube-api-access-r286c\") pod \"payload-processing-external-model-546cdc55c7-d64gv\" (UID: \"6130dbc6-b1f1-4672-90cc-7e3be07be0e1\") " pod="models-as-a-service/payload-processing-external-model-546cdc55c7-d64gv" Oct 30 01:58:49.839148 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:49.839075 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-546cdc55c7-d64gv" Oct 30 01:58:50.142365 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:50.142334 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-546cdc55c7-d64gv"] Oct 30 01:58:50.142731 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:58:50.142634 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6130dbc6_b1f1_4672_90cc_7e3be07be0e1.slice/crio-faae4b324bb3e6e3f74d27da8fcc7668444c763c68be477da68f320bd242839c WatchSource:0}: Error finding container faae4b324bb3e6e3f74d27da8fcc7668444c763c68be477da68f320bd242839c: Status 404 returned error can't find the container with id faae4b324bb3e6e3f74d27da8fcc7668444c763c68be477da68f320bd242839c Oct 30 01:58:50.186289 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:50.186261 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-546cdc55c7-d64gv" event={"ID":"6130dbc6-b1f1-4672-90cc-7e3be07be0e1","Type":"ContainerStarted","Data":"faae4b324bb3e6e3f74d27da8fcc7668444c763c68be477da68f320bd242839c"} Oct 30 01:58:50.458133 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:50.458097 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:51.200001 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:51.199967 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-546cdc55c7-d64gv" event={"ID":"6130dbc6-b1f1-4672-90cc-7e3be07be0e1","Type":"ContainerStarted","Data":"3cbd133dbe2bc8a965078eaac06c8f939bff5fa44198ff2b90fd47c1e3b9c518"} Oct 30 01:58:51.200405 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:51.200098 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-546cdc55c7-d64gv" Oct 30 01:58:51.238100 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:51.238046 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-546cdc55c7-d64gv" podStartSLOduration=2.238031187 podStartE2EDuration="2.238031187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:58:51.237781929 +0000 UTC m=+2456.814207333" watchObservedRunningTime="2026-10-30 01:58:51.238031187 +0000 UTC m=+2456.814456590" Oct 30 01:58:52.079610 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:52.079574 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:53.694302 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:53.694267 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:55.395219 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:55.395181 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:55.779294 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:55.779250 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:56.011785 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:56.006399 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:57.664684 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:57.664652 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:57.901318 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:57.901193 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:57.993631 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:57.993543 6572 scope.go:122] "RemoveContainer" containerID="9c69b978275ac4ffbe40cf489ca075e8855b01ef7417af6f68bb8ab4923dcf96" Oct 30 01:58:58.005588 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:58.005562 6572 scope.go:122] "RemoveContainer" containerID="b09de6cb4d593027d42fdcc87a63ac592a74b3fbd52c30fd3f20ce49cbeaf2ed" Oct 30 01:58:58.071089 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:58.071064 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d44413/payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4"] Oct 30 01:58:58.077652 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:58.077627 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d44413/payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4" Oct 30 01:58:58.091663 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:58.091640 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d44413\"/\"agc-cred-7552499b51f4\"" Oct 30 01:58:58.091803 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:58.091729 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d44413\"/\"payload-processing-external-model-e2e--20faa50d-dockercfg-t9xmd\"" Oct 30 01:58:58.093022 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:58.093004 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d44413/payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4"] Oct 30 01:58:58.141742 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:58.141707 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d44413\"/\"kube-root-ca.crt\"" Oct 30 01:58:58.141742 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:58.141730 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d44413\"/\"routing-overlay\"" Oct 30 01:58:58.141985 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:58.141739 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d44413\"/\"openshift-service-ca.crt\"" Oct 30 01:58:58.141985 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:58.141739 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d44413\"/\"payload-processing-external-model-plugins-e2e-extmod-a-d44413\"" Oct 30 01:58:58.167321 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:58.167277 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34415c1c-7ed9-40f5-b644-4b798018d895-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4\" (UID: \"34415c1c-7ed9-40f5-b644-4b798018d895\") " pod="ai-tenant-e2e-extmod-a-d44413/payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4" Oct 30 01:58:58.167542 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:58.167427 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34415c1c-7ed9-40f5-b644-4b798018d895-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4\" (UID: \"34415c1c-7ed9-40f5-b644-4b798018d895\") " pod="ai-tenant-e2e-extmod-a-d44413/payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4" Oct 30 01:58:58.167613 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:58.167540 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd5pc\" (UniqueName: \"kubernetes.io/projected/34415c1c-7ed9-40f5-b644-4b798018d895-kube-api-access-qd5pc\") pod \"payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4\" (UID: \"34415c1c-7ed9-40f5-b644-4b798018d895\") " pod="ai-tenant-e2e-extmod-a-d44413/payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4" Oct 30 01:58:58.167613 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:58.167597 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34415c1c-7ed9-40f5-b644-4b798018d895-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4\" (UID: \"34415c1c-7ed9-40f5-b644-4b798018d895\") " pod="ai-tenant-e2e-extmod-a-d44413/payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4" Oct 30 01:58:58.270831 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:58.269101 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34415c1c-7ed9-40f5-b644-4b798018d895-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4\" (UID: \"34415c1c-7ed9-40f5-b644-4b798018d895\") " pod="ai-tenant-e2e-extmod-a-d44413/payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4" Oct 30 01:58:58.270831 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:58.269221 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34415c1c-7ed9-40f5-b644-4b798018d895-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4\" (UID: \"34415c1c-7ed9-40f5-b644-4b798018d895\") " pod="ai-tenant-e2e-extmod-a-d44413/payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4" Oct 30 01:58:58.270831 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:58.269305 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34415c1c-7ed9-40f5-b644-4b798018d895-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4\" (UID: \"34415c1c-7ed9-40f5-b644-4b798018d895\") " pod="ai-tenant-e2e-extmod-a-d44413/payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4" Oct 30 01:58:58.270831 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:58.269410 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd5pc\" (UniqueName: \"kubernetes.io/projected/34415c1c-7ed9-40f5-b644-4b798018d895-kube-api-access-qd5pc\") pod \"payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4\" (UID: \"34415c1c-7ed9-40f5-b644-4b798018d895\") " pod="ai-tenant-e2e-extmod-a-d44413/payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4" Oct 30 01:58:58.270831 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:58.270510 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34415c1c-7ed9-40f5-b644-4b798018d895-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4\" (UID: \"34415c1c-7ed9-40f5-b644-4b798018d895\") " pod="ai-tenant-e2e-extmod-a-d44413/payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4" Oct 30 01:58:58.271251 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:58.271041 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34415c1c-7ed9-40f5-b644-4b798018d895-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4\" (UID: \"34415c1c-7ed9-40f5-b644-4b798018d895\") " pod="ai-tenant-e2e-extmod-a-d44413/payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4" Oct 30 01:58:58.318148 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:58.318107 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34415c1c-7ed9-40f5-b644-4b798018d895-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4\" (UID: \"34415c1c-7ed9-40f5-b644-4b798018d895\") " pod="ai-tenant-e2e-extmod-a-d44413/payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4" Oct 30 01:58:58.318869 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:58.318825 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd5pc\" (UniqueName: \"kubernetes.io/projected/34415c1c-7ed9-40f5-b644-4b798018d895-kube-api-access-qd5pc\") pod \"payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4\" (UID: \"34415c1c-7ed9-40f5-b644-4b798018d895\") " pod="ai-tenant-e2e-extmod-a-d44413/payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4" Oct 30 01:58:58.392649 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:58.392616 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d44413/payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4" Oct 30 01:58:58.704169 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:58.704140 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d44413/payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4"] Oct 30 01:58:58.704486 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:58:58.704397 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34415c1c_7ed9_40f5_b644_4b798018d895.slice/crio-f288048a1158f31147619f6ff5ce9110fab75b9c818ca715c3c6adf294f5b5e2 WatchSource:0}: Error finding container f288048a1158f31147619f6ff5ce9110fab75b9c818ca715c3c6adf294f5b5e2: Status 404 returned error can't find the container with id f288048a1158f31147619f6ff5ce9110fab75b9c818ca715c3c6adf294f5b5e2 Oct 30 01:58:58.905000 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:58.904953 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:59.092835 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:59.090694 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:59.365716 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:59.365619 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d44413/payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4" event={"ID":"34415c1c-7ed9-40f5-b644-4b798018d895","Type":"ContainerStarted","Data":"bdf5cd104e2e519339764d20add06ce678d69dc40c9a134ffaf11a6322815e79"} Oct 30 01:58:59.365716 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:59.365678 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d44413/payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4" event={"ID":"34415c1c-7ed9-40f5-b644-4b798018d895","Type":"ContainerStarted","Data":"f288048a1158f31147619f6ff5ce9110fab75b9c818ca715c3c6adf294f5b5e2"} Oct 30 01:58:59.366001 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:59.365723 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d44413/payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4" Oct 30 01:58:59.408065 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:59.408003 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d44413/payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4" podStartSLOduration=1.407984208 podStartE2EDuration="1.407984208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:58:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:58:59.405492311 +0000 UTC m=+2464.981917713" watchObservedRunningTime="2026-10-30 01:58:59.407984208 +0000 UTC m=+2464.984409614" Oct 30 01:58:59.576944 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:58:59.576900 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:01.086151 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:01.086109 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:02.212106 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:02.212072 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-546cdc55c7-d64gv" Oct 30 01:59:02.673716 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:02.673685 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:04.284645 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:04.284608 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:04.332393 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:04.332360 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d44413/payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4"] Oct 30 01:59:04.335234 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:04.335209 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-d44413/payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4" Oct 30 01:59:04.487487 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:04.487444 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:04.680252 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:04.680210 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:05.476330 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:05.476296 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7"] Oct 30 01:59:05.482475 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:05.482450 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7" Oct 30 01:59:05.501239 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:05.501208 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-92f0cc\"/\"agc-cred-ce62c5464e20\"" Oct 30 01:59:05.501524 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:05.501508 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-92f0cc\"/\"agc-cred-edfce4f81270\"" Oct 30 01:59:05.508548 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:05.508516 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7"] Oct 30 01:59:05.596649 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:05.596609 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0c61263-a7c1-4792-8970-6fe6388f11bf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7\" (UID: \"b0c61263-a7c1-4792-8970-6fe6388f11bf\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7" Oct 30 01:59:05.596895 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:05.596671 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b0c61263-a7c1-4792-8970-6fe6388f11bf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7\" (UID: \"b0c61263-a7c1-4792-8970-6fe6388f11bf\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7" Oct 30 01:59:05.596895 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:05.596886 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b0c61263-a7c1-4792-8970-6fe6388f11bf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7\" (UID: \"b0c61263-a7c1-4792-8970-6fe6388f11bf\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7" Oct 30 01:59:05.597040 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:05.596972 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5dzj\" (UniqueName: \"kubernetes.io/projected/b0c61263-a7c1-4792-8970-6fe6388f11bf-kube-api-access-m5dzj\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7\" (UID: \"b0c61263-a7c1-4792-8970-6fe6388f11bf\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7" Oct 30 01:59:05.698579 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:05.698536 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b0c61263-a7c1-4792-8970-6fe6388f11bf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7\" (UID: \"b0c61263-a7c1-4792-8970-6fe6388f11bf\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7" Oct 30 01:59:05.698819 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:05.698589 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5dzj\" (UniqueName: \"kubernetes.io/projected/b0c61263-a7c1-4792-8970-6fe6388f11bf-kube-api-access-m5dzj\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7\" (UID: \"b0c61263-a7c1-4792-8970-6fe6388f11bf\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7" Oct 30 01:59:05.698819 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:05.698716 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0c61263-a7c1-4792-8970-6fe6388f11bf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7\" (UID: \"b0c61263-a7c1-4792-8970-6fe6388f11bf\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7" Oct 30 01:59:05.698819 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:05.698799 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b0c61263-a7c1-4792-8970-6fe6388f11bf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7\" (UID: \"b0c61263-a7c1-4792-8970-6fe6388f11bf\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7" Oct 30 01:59:05.699378 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:05.699353 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b0c61263-a7c1-4792-8970-6fe6388f11bf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7\" (UID: \"b0c61263-a7c1-4792-8970-6fe6388f11bf\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7" Oct 30 01:59:05.699514 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:05.699492 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0c61263-a7c1-4792-8970-6fe6388f11bf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7\" (UID: \"b0c61263-a7c1-4792-8970-6fe6388f11bf\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7" Oct 30 01:59:05.701423 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:05.701398 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b0c61263-a7c1-4792-8970-6fe6388f11bf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7\" (UID: \"b0c61263-a7c1-4792-8970-6fe6388f11bf\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7" Oct 30 01:59:05.717963 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:05.717933 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5dzj\" (UniqueName: \"kubernetes.io/projected/b0c61263-a7c1-4792-8970-6fe6388f11bf-kube-api-access-m5dzj\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7\" (UID: \"b0c61263-a7c1-4792-8970-6fe6388f11bf\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7" Oct 30 01:59:05.807478 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:05.807378 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7" Oct 30 01:59:06.088992 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:06.088896 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:06.117448 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:06.117415 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7"] Oct 30 01:59:06.117584 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:59:06.117509 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c61263_a7c1_4792_8970_6fe6388f11bf.slice/crio-39ad8178957e7880c0562d51ec4d998387c592aa56d8945bc7992399998d075e WatchSource:0}: Error finding container 39ad8178957e7880c0562d51ec4d998387c592aa56d8945bc7992399998d075e: Status 404 returned error can't find the container with id 39ad8178957e7880c0562d51ec4d998387c592aa56d8945bc7992399998d075e Oct 30 01:59:06.430302 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:06.430258 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7" event={"ID":"b0c61263-a7c1-4792-8970-6fe6388f11bf","Type":"ContainerStarted","Data":"c879fb15956d07e3ba835063190599f74656dea2ee33241c2854154d20dd3cb5"} Oct 30 01:59:06.430484 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:06.430332 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7" Oct 30 01:59:06.430484 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:06.430354 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7" event={"ID":"b0c61263-a7c1-4792-8970-6fe6388f11bf","Type":"ContainerStarted","Data":"39ad8178957e7880c0562d51ec4d998387c592aa56d8945bc7992399998d075e"} Oct 30 01:59:06.470887 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:06.470833 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7" podStartSLOduration=1.470817107 podStartE2EDuration="1.470817107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:59:06.469161534 +0000 UTC m=+2472.045586948" watchObservedRunningTime="2026-10-30 01:59:06.470817107 +0000 UTC m=+2472.047242554" Oct 30 01:59:06.916527 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:06.916491 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:07.693005 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:07.692969 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:07.798392 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:07.798344 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9"] Oct 30 01:59:07.798598 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:07.798571 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" podUID="780b98b5-6658-4a59-83cd-ae57c033bde9" containerName="istio-proxy" containerID="cri-o://a4e5d9d81321b10b45cc71be49e95a704a8a75e97577e3462a1166693faaad86" gracePeriod=30 Oct 30 01:59:09.172882 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:09.172848 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:09.332947 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:09.332895 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d44413/payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4" podUID="34415c1c-7ed9-40f5-b644-4b798018d895" containerName="payload-processing" containerID="cri-o://bdf5cd104e2e519339764d20add06ce678d69dc40c9a134ffaf11a6322815e79" gracePeriod=25 Oct 30 01:59:09.461172 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:09.461136 6572 generic.go:356] "Generic (PLEG): container finished" podID="34415c1c-7ed9-40f5-b644-4b798018d895" containerID="bdf5cd104e2e519339764d20add06ce678d69dc40c9a134ffaf11a6322815e79" exitCode=0 Oct 30 01:59:09.461375 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:09.461217 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d44413/payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4" event={"ID":"34415c1c-7ed9-40f5-b644-4b798018d895","Type":"ContainerDied","Data":"bdf5cd104e2e519339764d20add06ce678d69dc40c9a134ffaf11a6322815e79"} Oct 30 01:59:09.621875 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:09.621841 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d44413/payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4" Oct 30 01:59:09.773552 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:09.773453 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34415c1c-7ed9-40f5-b644-4b798018d895-kube-api-access-qd5pc\" (UniqueName: \"kubernetes.io/projected/34415c1c-7ed9-40f5-b644-4b798018d895-kube-api-access-qd5pc\") pod \"34415c1c-7ed9-40f5-b644-4b798018d895\" (UID: \"34415c1c-7ed9-40f5-b644-4b798018d895\") " Oct 30 01:59:09.773712 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:09.773641 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34415c1c-7ed9-40f5-b644-4b798018d895-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34415c1c-7ed9-40f5-b644-4b798018d895-routing-overlay-volume\") pod \"34415c1c-7ed9-40f5-b644-4b798018d895\" (UID: \"34415c1c-7ed9-40f5-b644-4b798018d895\") " Oct 30 01:59:09.773826 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:09.773739 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34415c1c-7ed9-40f5-b644-4b798018d895-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34415c1c-7ed9-40f5-b644-4b798018d895-plugins-config-volume\") pod \"34415c1c-7ed9-40f5-b644-4b798018d895\" (UID: \"34415c1c-7ed9-40f5-b644-4b798018d895\") " Oct 30 01:59:09.773910 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:09.773843 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34415c1c-7ed9-40f5-b644-4b798018d895-provider-credentials\" (UniqueName: \"kubernetes.io/projected/34415c1c-7ed9-40f5-b644-4b798018d895-provider-credentials\") pod \"34415c1c-7ed9-40f5-b644-4b798018d895\" (UID: \"34415c1c-7ed9-40f5-b644-4b798018d895\") " Oct 30 01:59:09.774109 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:09.774079 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34415c1c-7ed9-40f5-b644-4b798018d895-routing-overlay-volume" pod "34415c1c-7ed9-40f5-b644-4b798018d895" (UID: "34415c1c-7ed9-40f5-b644-4b798018d895"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:59:09.774243 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:09.774185 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34415c1c-7ed9-40f5-b644-4b798018d895-plugins-config-volume" pod "34415c1c-7ed9-40f5-b644-4b798018d895" (UID: "34415c1c-7ed9-40f5-b644-4b798018d895"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:59:09.774512 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:09.774493 6572 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34415c1c-7ed9-40f5-b644-4b798018d895-routing-overlay-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:59:09.774570 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:09.774515 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34415c1c-7ed9-40f5-b644-4b798018d895-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:59:09.776059 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:09.776030 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34415c1c-7ed9-40f5-b644-4b798018d895-kube-api-access-qd5pc" pod "34415c1c-7ed9-40f5-b644-4b798018d895" (UID: "34415c1c-7ed9-40f5-b644-4b798018d895"). InnerVolumeSpecName "kube-api-access-qd5pc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:09.776170 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:09.776066 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34415c1c-7ed9-40f5-b644-4b798018d895-provider-credentials" pod "34415c1c-7ed9-40f5-b644-4b798018d895" (UID: "34415c1c-7ed9-40f5-b644-4b798018d895"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:09.875505 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:09.875460 6572 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34415c1c-7ed9-40f5-b644-4b798018d895-provider-credentials\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:59:09.875505 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:09.875498 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd5pc\" (UniqueName: \"kubernetes.io/projected/34415c1c-7ed9-40f5-b644-4b798018d895-kube-api-access-qd5pc\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:59:10.471424 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:10.471389 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d44413/payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4" Oct 30 01:59:10.471424 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:10.471413 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d44413/payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4" event={"ID":"34415c1c-7ed9-40f5-b644-4b798018d895","Type":"ContainerDied","Data":"f288048a1158f31147619f6ff5ce9110fab75b9c818ca715c3c6adf294f5b5e2"} Oct 30 01:59:10.472032 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:10.471478 6572 scope.go:122] "RemoveContainer" containerID="bdf5cd104e2e519339764d20add06ce678d69dc40c9a134ffaf11a6322815e79" Oct 30 01:59:10.524488 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:10.524447 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d44413/payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4"] Oct 30 01:59:10.544269 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:10.544240 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d44413/payload-processing-external-model-e2e-extmod-a-d44413-7575ckrn4"] Oct 30 01:59:10.763376 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:10.763289 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:12.464903 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:12.464858 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:12.544521 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:12.544485 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34415c1c-7ed9-40f5-b644-4b798018d895" path="/var/lib/kubelet/pods/34415c1c-7ed9-40f5-b644-4b798018d895/volumes" Oct 30 01:59:13.097251 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.097173 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" Oct 30 01:59:13.181190 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.181145 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:13.221268 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.221236 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/780b98b5-6658-4a59-83cd-ae57c033bde9-kube-api-access-tjsjt\" (UniqueName: \"kubernetes.io/projected/780b98b5-6658-4a59-83cd-ae57c033bde9-kube-api-access-tjsjt\") pod \"780b98b5-6658-4a59-83cd-ae57c033bde9\" (UID: \"780b98b5-6658-4a59-83cd-ae57c033bde9\") " Oct 30 01:59:13.221457 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.221373 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780b98b5-6658-4a59-83cd-ae57c033bde9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/780b98b5-6658-4a59-83cd-ae57c033bde9-istio-envoy\") pod \"780b98b5-6658-4a59-83cd-ae57c033bde9\" (UID: \"780b98b5-6658-4a59-83cd-ae57c033bde9\") " Oct 30 01:59:13.221457 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.221436 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780b98b5-6658-4a59-83cd-ae57c033bde9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/780b98b5-6658-4a59-83cd-ae57c033bde9-workload-certs\") pod \"780b98b5-6658-4a59-83cd-ae57c033bde9\" (UID: \"780b98b5-6658-4a59-83cd-ae57c033bde9\") " Oct 30 01:59:13.221571 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.221491 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780b98b5-6658-4a59-83cd-ae57c033bde9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/780b98b5-6658-4a59-83cd-ae57c033bde9-workload-socket\") pod \"780b98b5-6658-4a59-83cd-ae57c033bde9\" (UID: \"780b98b5-6658-4a59-83cd-ae57c033bde9\") " Oct 30 01:59:13.221571 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.221551 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/780b98b5-6658-4a59-83cd-ae57c033bde9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/780b98b5-6658-4a59-83cd-ae57c033bde9-istio-podinfo\") pod \"780b98b5-6658-4a59-83cd-ae57c033bde9\" (UID: \"780b98b5-6658-4a59-83cd-ae57c033bde9\") " Oct 30 01:59:13.221695 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.221573 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/780b98b5-6658-4a59-83cd-ae57c033bde9-istio-token\" (UniqueName: \"kubernetes.io/projected/780b98b5-6658-4a59-83cd-ae57c033bde9-istio-token\") pod \"780b98b5-6658-4a59-83cd-ae57c033bde9\" (UID: \"780b98b5-6658-4a59-83cd-ae57c033bde9\") " Oct 30 01:59:13.221695 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.221615 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780b98b5-6658-4a59-83cd-ae57c033bde9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/780b98b5-6658-4a59-83cd-ae57c033bde9-credential-socket\") pod \"780b98b5-6658-4a59-83cd-ae57c033bde9\" (UID: \"780b98b5-6658-4a59-83cd-ae57c033bde9\") " Oct 30 01:59:13.221695 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.221687 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/780b98b5-6658-4a59-83cd-ae57c033bde9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/780b98b5-6658-4a59-83cd-ae57c033bde9-istiod-ca-cert\") pod \"780b98b5-6658-4a59-83cd-ae57c033bde9\" (UID: \"780b98b5-6658-4a59-83cd-ae57c033bde9\") " Oct 30 01:59:13.222070 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.221771 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/780b98b5-6658-4a59-83cd-ae57c033bde9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/780b98b5-6658-4a59-83cd-ae57c033bde9-istio-data\") pod \"780b98b5-6658-4a59-83cd-ae57c033bde9\" (UID: \"780b98b5-6658-4a59-83cd-ae57c033bde9\") " Oct 30 01:59:13.222070 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.221931 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780b98b5-6658-4a59-83cd-ae57c033bde9-workload-certs" pod "780b98b5-6658-4a59-83cd-ae57c033bde9" (UID: "780b98b5-6658-4a59-83cd-ae57c033bde9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:13.222070 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.222045 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780b98b5-6658-4a59-83cd-ae57c033bde9-credential-socket" pod "780b98b5-6658-4a59-83cd-ae57c033bde9" (UID: "780b98b5-6658-4a59-83cd-ae57c033bde9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:13.222290 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.222267 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780b98b5-6658-4a59-83cd-ae57c033bde9-istio-data" pod "780b98b5-6658-4a59-83cd-ae57c033bde9" (UID: "780b98b5-6658-4a59-83cd-ae57c033bde9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:13.222361 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.222326 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/780b98b5-6658-4a59-83cd-ae57c033bde9-istiod-ca-cert" pod "780b98b5-6658-4a59-83cd-ae57c033bde9" (UID: "780b98b5-6658-4a59-83cd-ae57c033bde9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:59:13.222522 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.222487 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780b98b5-6658-4a59-83cd-ae57c033bde9-workload-socket" pod "780b98b5-6658-4a59-83cd-ae57c033bde9" (UID: "780b98b5-6658-4a59-83cd-ae57c033bde9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:13.222656 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.222555 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/780b98b5-6658-4a59-83cd-ae57c033bde9-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:59:13.222656 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.222577 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/780b98b5-6658-4a59-83cd-ae57c033bde9-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:59:13.222656 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.222594 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/780b98b5-6658-4a59-83cd-ae57c033bde9-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:59:13.222656 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.222607 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/780b98b5-6658-4a59-83cd-ae57c033bde9-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:59:13.223690 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.223669 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/780b98b5-6658-4a59-83cd-ae57c033bde9-kube-api-access-tjsjt" pod "780b98b5-6658-4a59-83cd-ae57c033bde9" (UID: "780b98b5-6658-4a59-83cd-ae57c033bde9"). InnerVolumeSpecName "kube-api-access-tjsjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:13.223869 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.223852 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780b98b5-6658-4a59-83cd-ae57c033bde9-istio-envoy" pod "780b98b5-6658-4a59-83cd-ae57c033bde9" (UID: "780b98b5-6658-4a59-83cd-ae57c033bde9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:13.223972 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.223953 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/780b98b5-6658-4a59-83cd-ae57c033bde9-istio-podinfo" pod "780b98b5-6658-4a59-83cd-ae57c033bde9" (UID: "780b98b5-6658-4a59-83cd-ae57c033bde9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:59:13.224013 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.223987 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/780b98b5-6658-4a59-83cd-ae57c033bde9-istio-token" pod "780b98b5-6658-4a59-83cd-ae57c033bde9" (UID: "780b98b5-6658-4a59-83cd-ae57c033bde9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:13.323613 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.323576 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjsjt\" (UniqueName: \"kubernetes.io/projected/780b98b5-6658-4a59-83cd-ae57c033bde9-kube-api-access-tjsjt\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:59:13.323613 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.323607 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/780b98b5-6658-4a59-83cd-ae57c033bde9-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:59:13.323613 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.323616 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/780b98b5-6658-4a59-83cd-ae57c033bde9-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:59:13.323909 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.323624 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/780b98b5-6658-4a59-83cd-ae57c033bde9-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:59:13.323909 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.323635 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/780b98b5-6658-4a59-83cd-ae57c033bde9-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:59:13.499705 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.499673 6572 generic.go:356] "Generic (PLEG): container finished" podID="780b98b5-6658-4a59-83cd-ae57c033bde9" containerID="a4e5d9d81321b10b45cc71be49e95a704a8a75e97577e3462a1166693faaad86" exitCode=0 Oct 30 01:59:13.500183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.499725 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" Oct 30 01:59:13.500183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.499781 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" event={"ID":"780b98b5-6658-4a59-83cd-ae57c033bde9","Type":"ContainerDied","Data":"a4e5d9d81321b10b45cc71be49e95a704a8a75e97577e3462a1166693faaad86"} Oct 30 01:59:13.500183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.499836 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9" event={"ID":"780b98b5-6658-4a59-83cd-ae57c033bde9","Type":"ContainerDied","Data":"3ba3ed720d85298493bb9df76f308f038cab9de909ad15e17a33569686ef433e"} Oct 30 01:59:13.500183 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.499858 6572 scope.go:122] "RemoveContainer" containerID="a4e5d9d81321b10b45cc71be49e95a704a8a75e97577e3462a1166693faaad86" Oct 30 01:59:13.512975 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.512956 6572 scope.go:122] "RemoveContainer" containerID="a4e5d9d81321b10b45cc71be49e95a704a8a75e97577e3462a1166693faaad86" Oct 30 01:59:13.513275 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:59:13.513253 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4e5d9d81321b10b45cc71be49e95a704a8a75e97577e3462a1166693faaad86\": container with ID starting with a4e5d9d81321b10b45cc71be49e95a704a8a75e97577e3462a1166693faaad86 not found: ID does not exist" containerID="a4e5d9d81321b10b45cc71be49e95a704a8a75e97577e3462a1166693faaad86" Oct 30 01:59:13.513324 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.513284 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4e5d9d81321b10b45cc71be49e95a704a8a75e97577e3462a1166693faaad86"} err="failed to get container status \"a4e5d9d81321b10b45cc71be49e95a704a8a75e97577e3462a1166693faaad86\": rpc error: code = NotFound desc = could not find container \"a4e5d9d81321b10b45cc71be49e95a704a8a75e97577e3462a1166693faaad86\": container with ID starting with a4e5d9d81321b10b45cc71be49e95a704a8a75e97577e3462a1166693faaad86 not found: ID does not exist" Oct 30 01:59:13.553252 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.553219 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9"] Oct 30 01:59:13.559286 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:13.559253 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-rdzs9"] Oct 30 01:59:14.191974 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:14.191936 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:14.571259 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:14.571225 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="780b98b5-6658-4a59-83cd-ae57c033bde9" path="/var/lib/kubelet/pods/780b98b5-6658-4a59-83cd-ae57c033bde9/volumes" Oct 30 01:59:14.692944 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:59:14.692901 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:59:15.901138 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:15.892406 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:17.491222 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:17.489958 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7" Oct 30 01:59:17.577366 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:17.577327 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr"] Oct 30 01:59:17.885133 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:17.884800 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:19.278585 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:19.278550 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:20.885848 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:20.885731 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:22.439001 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:22.438915 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:22.579617 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:22.579559 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr" podUID="44681477-4758-4150-88c9-a9fe7e70d6ee" containerName="payload-processing" containerID="cri-o://70a7bf883a706846e42c4058e496d2b63ba0715ddba243eac820a40a3d8620db" gracePeriod=25 Oct 30 01:59:22.915165 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:22.915141 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr" Oct 30 01:59:22.950507 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:22.950473 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44681477-4758-4150-88c9-a9fe7e70d6ee-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44681477-4758-4150-88c9-a9fe7e70d6ee-routing-overlay-volume\") pod \"44681477-4758-4150-88c9-a9fe7e70d6ee\" (UID: \"44681477-4758-4150-88c9-a9fe7e70d6ee\") " Oct 30 01:59:22.950665 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:22.950541 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44681477-4758-4150-88c9-a9fe7e70d6ee-kube-api-access-8zvht\" (UniqueName: \"kubernetes.io/projected/44681477-4758-4150-88c9-a9fe7e70d6ee-kube-api-access-8zvht\") pod \"44681477-4758-4150-88c9-a9fe7e70d6ee\" (UID: \"44681477-4758-4150-88c9-a9fe7e70d6ee\") " Oct 30 01:59:22.950665 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:22.950579 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44681477-4758-4150-88c9-a9fe7e70d6ee-provider-credentials\" (UniqueName: \"kubernetes.io/projected/44681477-4758-4150-88c9-a9fe7e70d6ee-provider-credentials\") pod \"44681477-4758-4150-88c9-a9fe7e70d6ee\" (UID: \"44681477-4758-4150-88c9-a9fe7e70d6ee\") " Oct 30 01:59:22.950811 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:22.950795 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44681477-4758-4150-88c9-a9fe7e70d6ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44681477-4758-4150-88c9-a9fe7e70d6ee-plugins-config-volume\") pod \"44681477-4758-4150-88c9-a9fe7e70d6ee\" (UID: \"44681477-4758-4150-88c9-a9fe7e70d6ee\") " Oct 30 01:59:22.950919 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:22.950890 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44681477-4758-4150-88c9-a9fe7e70d6ee-routing-overlay-volume" pod "44681477-4758-4150-88c9-a9fe7e70d6ee" (UID: "44681477-4758-4150-88c9-a9fe7e70d6ee"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:59:22.951219 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:22.951193 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44681477-4758-4150-88c9-a9fe7e70d6ee-plugins-config-volume" pod "44681477-4758-4150-88c9-a9fe7e70d6ee" (UID: "44681477-4758-4150-88c9-a9fe7e70d6ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:59:22.951465 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:22.951449 6572 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44681477-4758-4150-88c9-a9fe7e70d6ee-routing-overlay-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:59:22.951510 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:22.951467 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44681477-4758-4150-88c9-a9fe7e70d6ee-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:59:22.953603 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:22.953563 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44681477-4758-4150-88c9-a9fe7e70d6ee-kube-api-access-8zvht" pod "44681477-4758-4150-88c9-a9fe7e70d6ee" (UID: "44681477-4758-4150-88c9-a9fe7e70d6ee"). InnerVolumeSpecName "kube-api-access-8zvht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:22.953952 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:22.953911 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44681477-4758-4150-88c9-a9fe7e70d6ee-provider-credentials" pod "44681477-4758-4150-88c9-a9fe7e70d6ee" (UID: "44681477-4758-4150-88c9-a9fe7e70d6ee"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:22.980820 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:22.980259 6572 generic.go:356] "Generic (PLEG): container finished" podID="44681477-4758-4150-88c9-a9fe7e70d6ee" containerID="70a7bf883a706846e42c4058e496d2b63ba0715ddba243eac820a40a3d8620db" exitCode=0 Oct 30 01:59:22.980820 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:22.980384 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr" event={"ID":"44681477-4758-4150-88c9-a9fe7e70d6ee","Type":"ContainerDied","Data":"70a7bf883a706846e42c4058e496d2b63ba0715ddba243eac820a40a3d8620db"} Oct 30 01:59:22.980820 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:22.980444 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr" event={"ID":"44681477-4758-4150-88c9-a9fe7e70d6ee","Type":"ContainerDied","Data":"c522db075065e7dc1e4e11a96ab60e73bac426d8511590c984baf3b73736cc37"} Oct 30 01:59:22.980820 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:22.980475 6572 scope.go:122] "RemoveContainer" containerID="70a7bf883a706846e42c4058e496d2b63ba0715ddba243eac820a40a3d8620db" Oct 30 01:59:22.983770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:22.981337 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr" Oct 30 01:59:23.014894 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:23.014865 6572 scope.go:122] "RemoveContainer" containerID="70a7bf883a706846e42c4058e496d2b63ba0715ddba243eac820a40a3d8620db" Oct 30 01:59:23.015286 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:59:23.015256 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70a7bf883a706846e42c4058e496d2b63ba0715ddba243eac820a40a3d8620db\": container with ID starting with 70a7bf883a706846e42c4058e496d2b63ba0715ddba243eac820a40a3d8620db not found: ID does not exist" containerID="70a7bf883a706846e42c4058e496d2b63ba0715ddba243eac820a40a3d8620db" Oct 30 01:59:23.015373 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:23.015300 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70a7bf883a706846e42c4058e496d2b63ba0715ddba243eac820a40a3d8620db"} err="failed to get container status \"70a7bf883a706846e42c4058e496d2b63ba0715ddba243eac820a40a3d8620db\": rpc error: code = NotFound desc = could not find container \"70a7bf883a706846e42c4058e496d2b63ba0715ddba243eac820a40a3d8620db\": container with ID starting with 70a7bf883a706846e42c4058e496d2b63ba0715ddba243eac820a40a3d8620db not found: ID does not exist" Oct 30 01:59:23.043273 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:23.043242 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr"] Oct 30 01:59:23.049640 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:23.049615 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbg6clr"] Oct 30 01:59:23.052948 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:23.052922 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zvht\" (UniqueName: \"kubernetes.io/projected/44681477-4758-4150-88c9-a9fe7e70d6ee-kube-api-access-8zvht\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:59:23.052948 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:23.052947 6572 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/44681477-4758-4150-88c9-a9fe7e70d6ee-provider-credentials\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:59:23.988738 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:23.988264 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:24.547356 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:24.547322 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44681477-4758-4150-88c9-a9fe7e70d6ee" path="/var/lib/kubelet/pods/44681477-4758-4150-88c9-a9fe7e70d6ee/volumes" Oct 30 01:59:25.009230 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:25.009198 6572 generic.go:356] "Generic (PLEG): container finished" podID="a341c867-33db-494c-b3e4-873415967cf0" containerID="ad35146aaa1dd21e8cd4038b8e3ad2d4a22358aa3ac01d1cc3f633cb654cf713" exitCode=0 Oct 30 01:59:25.009622 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:25.009271 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" event={"ID":"a341c867-33db-494c-b3e4-873415967cf0","Type":"ContainerDied","Data":"ad35146aaa1dd21e8cd4038b8e3ad2d4a22358aa3ac01d1cc3f633cb654cf713"} Oct 30 01:59:25.009734 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:25.009711 6572 scope.go:122] "RemoveContainer" containerID="ad35146aaa1dd21e8cd4038b8e3ad2d4a22358aa3ac01d1cc3f633cb654cf713" Oct 30 01:59:25.011459 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:25.011428 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:59:25.011528 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:25.011501 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:59:25.011571 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:25.011534 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:59:25.575065 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:25.575034 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:26.039943 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:26.039905 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" event={"ID":"a341c867-33db-494c-b3e4-873415967cf0","Type":"ContainerStarted","Data":"d9440f31cdf646bc27031e114d9b9f724b896f8a03b03b44d33b71beff6a6797"} Oct 30 01:59:26.227773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:26.225515 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 01:59:26.231181 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:26.230872 6572 patch_prober.go:29] interesting pod/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body= Oct 30 01:59:26.231181 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:26.230932 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" podUID="a341c867-33db-494c-b3e4-873415967cf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" Oct 30 01:59:27.175494 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:27.175459 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:27.228780 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:27.224162 6572 patch_prober.go:29] interesting pod/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body= Oct 30 01:59:27.228780 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:27.224227 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" podUID="a341c867-33db-494c-b3e4-873415967cf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" Oct 30 01:59:27.972989 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:27.972955 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:28.223342 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:28.223232 6572 patch_prober.go:29] interesting pod/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body= Oct 30 01:59:28.224047 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:28.224010 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" podUID="a341c867-33db-494c-b3e4-873415967cf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" Oct 30 01:59:28.784882 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:28.784846 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:29.164680 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:29.164651 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:29.229797 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:29.229344 6572 patch_prober.go:29] interesting pod/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body= Oct 30 01:59:29.229797 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:29.229409 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" podUID="a341c867-33db-494c-b3e4-873415967cf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" Oct 30 01:59:29.356604 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:29.356560 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:29.539813 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:29.539722 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 01:59:29.627772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:29.627282 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:29.856789 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:29.856673 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:30.067084 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:30.067048 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:30.222773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:30.222217 6572 patch_prober.go:29] interesting pod/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body= Oct 30 01:59:30.222773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:30.222299 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" podUID="a341c867-33db-494c-b3e4-873415967cf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" Oct 30 01:59:30.531929 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:30.531832 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:30.732199 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:30.732157 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:31.230773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:31.225175 6572 patch_prober.go:29] interesting pod/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body= Oct 30 01:59:31.230773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:31.225255 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" podUID="a341c867-33db-494c-b3e4-873415967cf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" Oct 30 01:59:31.410294 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:31.410255 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:31.605555 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:31.605460 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:32.009195 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:32.009158 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:32.225140 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:32.225110 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 01:59:32.226246 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:32.226227 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 01:59:32.250065 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:32.250032 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:33.808719 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:33.808674 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:35.398795 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:35.398727 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:37.094034 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:37.093986 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:38.435724 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:38.435689 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:38.618401 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:38.618370 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:38.803865 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:38.803741 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:40.376524 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:40.376446 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:41.900737 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:41.900695 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:43.456951 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:43.456914 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:44.688550 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:59:44.688511 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 01:59:44.784965 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:44.784930 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:44.993433 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:44.993345 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:45.192477 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:45.192433 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:45.402214 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:45.401959 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:45.610933 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:45.610895 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:45.804440 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:45.804406 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:46.300129 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:46.300074 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:46.492848 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:46.492818 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:47.171219 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:47.171190 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:47.543331 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:47.543202 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:47.702082 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:47.702049 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr"] Oct 30 01:59:47.702836 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:47.702819 6572 cpu_manager.go:414] "containerMap: removing container" podUID="34415c1c-7ed9-40f5-b644-4b798018d895" containerName="payload-processing" Oct 30 01:59:47.702913 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:47.702837 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34415c1c-7ed9-40f5-b644-4b798018d895" containerName="payload-processing" Oct 30 01:59:47.702913 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:47.702880 6572 cpu_manager.go:414] "containerMap: removing container" podUID="44681477-4758-4150-88c9-a9fe7e70d6ee" containerName="payload-processing" Oct 30 01:59:47.702913 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:47.702885 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44681477-4758-4150-88c9-a9fe7e70d6ee" containerName="payload-processing" Oct 30 01:59:47.703022 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:47.702916 6572 cpu_manager.go:414] "containerMap: removing container" podUID="780b98b5-6658-4a59-83cd-ae57c033bde9" containerName="istio-proxy" Oct 30 01:59:47.703022 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:47.702922 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="780b98b5-6658-4a59-83cd-ae57c033bde9" containerName="istio-proxy" Oct 30 01:59:47.703088 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:47.703036 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="44681477-4758-4150-88c9-a9fe7e70d6ee" containerName="payload-processing" Oct 30 01:59:47.703088 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:47.703051 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="780b98b5-6658-4a59-83cd-ae57c033bde9" containerName="istio-proxy" Oct 30 01:59:47.703088 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:47.703059 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="34415c1c-7ed9-40f5-b644-4b798018d895" containerName="payload-processing" Oct 30 01:59:47.705946 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:47.705920 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr" Oct 30 01:59:47.721848 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:47.721804 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-817fd7\"/\"agc-cred-08a8e8dfe09b\"" Oct 30 01:59:47.722047 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:47.721837 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-817fd7\"/\"routing-overlay\"" Oct 30 01:59:47.733881 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:47.733851 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-817fd7\"/\"kube-root-ca.crt\"" Oct 30 01:59:47.734124 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:47.733908 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-817fd7\"/\"payload-processing-external-model-plugins-e2e-extmod-a-817fd7\"" Oct 30 01:59:47.734124 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:47.733909 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-817fd7\"/\"openshift-service-ca.crt\"" Oct 30 01:59:47.734319 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:47.733961 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-817fd7\"/\"payload-processing-external-model-e2e--6dbff1fe-dockercfg-5jctl\"" Oct 30 01:59:47.740862 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:47.740831 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr"] Oct 30 01:59:47.819830 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:47.815388 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg694\" (UniqueName: \"kubernetes.io/projected/18a3aece-618b-4a9a-8fc9-28606bddc3ce-kube-api-access-bg694\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr\" (UID: \"18a3aece-618b-4a9a-8fc9-28606bddc3ce\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr" Oct 30 01:59:47.819830 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:47.815509 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/18a3aece-618b-4a9a-8fc9-28606bddc3ce-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr\" (UID: \"18a3aece-618b-4a9a-8fc9-28606bddc3ce\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr" Oct 30 01:59:47.819830 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:47.815560 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/18a3aece-618b-4a9a-8fc9-28606bddc3ce-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr\" (UID: \"18a3aece-618b-4a9a-8fc9-28606bddc3ce\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr" Oct 30 01:59:47.819830 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:47.815630 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18a3aece-618b-4a9a-8fc9-28606bddc3ce-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr\" (UID: \"18a3aece-618b-4a9a-8fc9-28606bddc3ce\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr" Oct 30 01:59:47.917774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:47.916884 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/18a3aece-618b-4a9a-8fc9-28606bddc3ce-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr\" (UID: \"18a3aece-618b-4a9a-8fc9-28606bddc3ce\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr" Oct 30 01:59:47.917774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:47.916967 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/18a3aece-618b-4a9a-8fc9-28606bddc3ce-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr\" (UID: \"18a3aece-618b-4a9a-8fc9-28606bddc3ce\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr" Oct 30 01:59:47.917774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:47.917048 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18a3aece-618b-4a9a-8fc9-28606bddc3ce-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr\" (UID: \"18a3aece-618b-4a9a-8fc9-28606bddc3ce\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr" Oct 30 01:59:47.917774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:47.917245 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg694\" (UniqueName: \"kubernetes.io/projected/18a3aece-618b-4a9a-8fc9-28606bddc3ce-kube-api-access-bg694\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr\" (UID: \"18a3aece-618b-4a9a-8fc9-28606bddc3ce\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr" Oct 30 01:59:47.919126 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:47.919098 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/18a3aece-618b-4a9a-8fc9-28606bddc3ce-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr\" (UID: \"18a3aece-618b-4a9a-8fc9-28606bddc3ce\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr" Oct 30 01:59:47.922038 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:47.921997 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18a3aece-618b-4a9a-8fc9-28606bddc3ce-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr\" (UID: \"18a3aece-618b-4a9a-8fc9-28606bddc3ce\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr" Oct 30 01:59:47.931564 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:47.931541 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/18a3aece-618b-4a9a-8fc9-28606bddc3ce-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr\" (UID: \"18a3aece-618b-4a9a-8fc9-28606bddc3ce\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr" Oct 30 01:59:47.936816 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:47.936789 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg694\" (UniqueName: \"kubernetes.io/projected/18a3aece-618b-4a9a-8fc9-28606bddc3ce-kube-api-access-bg694\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr\" (UID: \"18a3aece-618b-4a9a-8fc9-28606bddc3ce\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr" Oct 30 01:59:48.020050 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:48.020019 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr" Oct 30 01:59:48.323058 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:48.323031 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr"] Oct 30 01:59:48.324416 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:59:48.324388 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a3aece_618b_4a9a_8fc9_28606bddc3ce.slice/crio-7dad09b3c2c17b2c564516b23cd8be070bd82553d2961f855e7ab8c5eb5719d5 WatchSource:0}: Error finding container 7dad09b3c2c17b2c564516b23cd8be070bd82553d2961f855e7ab8c5eb5719d5: Status 404 returned error can't find the container with id 7dad09b3c2c17b2c564516b23cd8be070bd82553d2961f855e7ab8c5eb5719d5 Oct 30 01:59:48.740672 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:48.740633 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:48.923666 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:48.923627 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:49.116349 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.116241 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:49.317494 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.315044 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:49.337840 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.337802 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr" event={"ID":"18a3aece-618b-4a9a-8fc9-28606bddc3ce","Type":"ContainerStarted","Data":"f4f77eb8c90187c02cfa5691c32d49290342add6e4117bbb57ab0454d0748510"} Oct 30 01:59:49.348847 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.348815 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr" Oct 30 01:59:49.349193 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.349159 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr" event={"ID":"18a3aece-618b-4a9a-8fc9-28606bddc3ce","Type":"ContainerStarted","Data":"7dad09b3c2c17b2c564516b23cd8be070bd82553d2961f855e7ab8c5eb5719d5"} Oct 30 01:59:49.402484 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.391584 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr" podStartSLOduration=2.391564398 podStartE2EDuration="2.391564398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:59:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:59:49.387425586 +0000 UTC m=+2514.963850990" watchObservedRunningTime="2026-10-30 01:59:49.391564398 +0000 UTC m=+2514.967989802" Oct 30 01:59:49.496872 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:59:49.484893 6572 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-817fd7/agc-cred-08a8e8dfe09b: secret "agc-cred-08a8e8dfe09b" not found Oct 30 01:59:49.496872 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:59:49.484923 6572 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr: secret "agc-cred-08a8e8dfe09b" not found Oct 30 01:59:49.496872 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:59:49.485004 6572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18a3aece-618b-4a9a-8fc9-28606bddc3ce-provider-credentials podName:18a3aece-618b-4a9a-8fc9-28606bddc3ce nodeName:}" failed. No retries permitted until 2026-10-30 01:59:49.98498258 +0000 UTC m=+2515.561407973 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/18a3aece-618b-4a9a-8fc9-28606bddc3ce-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr" (UID: "18a3aece-618b-4a9a-8fc9-28606bddc3ce") : secret "agc-cred-08a8e8dfe09b" not found Oct 30 01:59:49.539789 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.518936 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:49.715489 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.715454 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2"] Oct 30 01:59:49.720588 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.720563 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" Oct 30 01:59:49.745080 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.745042 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2"] Oct 30 01:59:49.750338 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.750298 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw"] Oct 30 01:59:49.757682 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.757634 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" Oct 30 01:59:49.795297 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.795245 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw"] Oct 30 01:59:49.796033 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.796002 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f3448e6-c22d-466c-a485-fbe2e676c250-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rl4q2\" (UID: \"5f3448e6-c22d-466c-a485-fbe2e676c250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" Oct 30 01:59:49.796165 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.796057 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f3448e6-c22d-466c-a485-fbe2e676c250-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rl4q2\" (UID: \"5f3448e6-c22d-466c-a485-fbe2e676c250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" Oct 30 01:59:49.796270 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.796246 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f3448e6-c22d-466c-a485-fbe2e676c250-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rl4q2\" (UID: \"5f3448e6-c22d-466c-a485-fbe2e676c250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" Oct 30 01:59:49.796334 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.796306 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f3448e6-c22d-466c-a485-fbe2e676c250-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rl4q2\" (UID: \"5f3448e6-c22d-466c-a485-fbe2e676c250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" Oct 30 01:59:49.796431 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.796406 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f3448e6-c22d-466c-a485-fbe2e676c250-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rl4q2\" (UID: \"5f3448e6-c22d-466c-a485-fbe2e676c250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" Oct 30 01:59:49.796507 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.796488 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvbbv\" (UniqueName: \"kubernetes.io/projected/5f3448e6-c22d-466c-a485-fbe2e676c250-kube-api-access-mvbbv\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rl4q2\" (UID: \"5f3448e6-c22d-466c-a485-fbe2e676c250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" Oct 30 01:59:49.796624 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.796607 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f3448e6-c22d-466c-a485-fbe2e676c250-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rl4q2\" (UID: \"5f3448e6-c22d-466c-a485-fbe2e676c250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" Oct 30 01:59:49.796684 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.796655 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f3448e6-c22d-466c-a485-fbe2e676c250-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rl4q2\" (UID: \"5f3448e6-c22d-466c-a485-fbe2e676c250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" Oct 30 01:59:49.796739 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.796707 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f3448e6-c22d-466c-a485-fbe2e676c250-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rl4q2\" (UID: \"5f3448e6-c22d-466c-a485-fbe2e676c250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" Oct 30 01:59:49.898164 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.898118 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/230f0407-5c25-44aa-9a97-1de91305e4f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tzxqw\" (UID: \"230f0407-5c25-44aa-9a97-1de91305e4f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" Oct 30 01:59:49.898164 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.898166 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f3448e6-c22d-466c-a485-fbe2e676c250-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rl4q2\" (UID: \"5f3448e6-c22d-466c-a485-fbe2e676c250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" Oct 30 01:59:49.898436 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.898242 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f3448e6-c22d-466c-a485-fbe2e676c250-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rl4q2\" (UID: \"5f3448e6-c22d-466c-a485-fbe2e676c250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" Oct 30 01:59:49.898436 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.898292 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f3448e6-c22d-466c-a485-fbe2e676c250-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rl4q2\" (UID: \"5f3448e6-c22d-466c-a485-fbe2e676c250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" Oct 30 01:59:49.898436 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.898337 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/230f0407-5c25-44aa-9a97-1de91305e4f3-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tzxqw\" (UID: \"230f0407-5c25-44aa-9a97-1de91305e4f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" Oct 30 01:59:49.898436 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.898392 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvbbv\" (UniqueName: \"kubernetes.io/projected/5f3448e6-c22d-466c-a485-fbe2e676c250-kube-api-access-mvbbv\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rl4q2\" (UID: \"5f3448e6-c22d-466c-a485-fbe2e676c250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" Oct 30 01:59:49.898661 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.898526 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/230f0407-5c25-44aa-9a97-1de91305e4f3-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tzxqw\" (UID: \"230f0407-5c25-44aa-9a97-1de91305e4f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" Oct 30 01:59:49.898661 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.898575 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f3448e6-c22d-466c-a485-fbe2e676c250-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rl4q2\" (UID: \"5f3448e6-c22d-466c-a485-fbe2e676c250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" Oct 30 01:59:49.898661 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.898642 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f3448e6-c22d-466c-a485-fbe2e676c250-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rl4q2\" (UID: \"5f3448e6-c22d-466c-a485-fbe2e676c250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" Oct 30 01:59:49.898847 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.898655 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f3448e6-c22d-466c-a485-fbe2e676c250-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rl4q2\" (UID: \"5f3448e6-c22d-466c-a485-fbe2e676c250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" Oct 30 01:59:49.898847 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.898687 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f3448e6-c22d-466c-a485-fbe2e676c250-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rl4q2\" (UID: \"5f3448e6-c22d-466c-a485-fbe2e676c250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" Oct 30 01:59:49.898847 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.898782 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f3448e6-c22d-466c-a485-fbe2e676c250-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rl4q2\" (UID: \"5f3448e6-c22d-466c-a485-fbe2e676c250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" Oct 30 01:59:49.898991 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.898866 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/230f0407-5c25-44aa-9a97-1de91305e4f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tzxqw\" (UID: \"230f0407-5c25-44aa-9a97-1de91305e4f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" Oct 30 01:59:49.899028 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.898977 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/230f0407-5c25-44aa-9a97-1de91305e4f3-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tzxqw\" (UID: \"230f0407-5c25-44aa-9a97-1de91305e4f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" Oct 30 01:59:49.899028 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.898986 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f3448e6-c22d-466c-a485-fbe2e676c250-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rl4q2\" (UID: \"5f3448e6-c22d-466c-a485-fbe2e676c250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" Oct 30 01:59:49.899099 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.899061 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f3448e6-c22d-466c-a485-fbe2e676c250-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rl4q2\" (UID: \"5f3448e6-c22d-466c-a485-fbe2e676c250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" Oct 30 01:59:49.899135 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.899092 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/230f0407-5c25-44aa-9a97-1de91305e4f3-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tzxqw\" (UID: \"230f0407-5c25-44aa-9a97-1de91305e4f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" Oct 30 01:59:49.899170 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.899152 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f3448e6-c22d-466c-a485-fbe2e676c250-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rl4q2\" (UID: \"5f3448e6-c22d-466c-a485-fbe2e676c250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" Oct 30 01:59:49.899208 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.899193 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f3448e6-c22d-466c-a485-fbe2e676c250-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rl4q2\" (UID: \"5f3448e6-c22d-466c-a485-fbe2e676c250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" Oct 30 01:59:49.899260 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.899230 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/230f0407-5c25-44aa-9a97-1de91305e4f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tzxqw\" (UID: \"230f0407-5c25-44aa-9a97-1de91305e4f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" Oct 30 01:59:49.899313 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.899299 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/230f0407-5c25-44aa-9a97-1de91305e4f3-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tzxqw\" (UID: \"230f0407-5c25-44aa-9a97-1de91305e4f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" Oct 30 01:59:49.899355 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.899329 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w49f8\" (UniqueName: \"kubernetes.io/projected/230f0407-5c25-44aa-9a97-1de91305e4f3-kube-api-access-w49f8\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tzxqw\" (UID: \"230f0407-5c25-44aa-9a97-1de91305e4f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" Oct 30 01:59:49.899496 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.899480 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f3448e6-c22d-466c-a485-fbe2e676c250-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rl4q2\" (UID: \"5f3448e6-c22d-466c-a485-fbe2e676c250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" Oct 30 01:59:49.901095 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.901071 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f3448e6-c22d-466c-a485-fbe2e676c250-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rl4q2\" (UID: \"5f3448e6-c22d-466c-a485-fbe2e676c250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" Oct 30 01:59:49.901552 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.901534 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f3448e6-c22d-466c-a485-fbe2e676c250-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rl4q2\" (UID: \"5f3448e6-c22d-466c-a485-fbe2e676c250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" Oct 30 01:59:49.925654 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.925620 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f3448e6-c22d-466c-a485-fbe2e676c250-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rl4q2\" (UID: \"5f3448e6-c22d-466c-a485-fbe2e676c250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" Oct 30 01:59:49.930827 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:49.930796 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvbbv\" (UniqueName: \"kubernetes.io/projected/5f3448e6-c22d-466c-a485-fbe2e676c250-kube-api-access-mvbbv\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rl4q2\" (UID: \"5f3448e6-c22d-466c-a485-fbe2e676c250\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" Oct 30 01:59:50.000551 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.000442 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/230f0407-5c25-44aa-9a97-1de91305e4f3-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tzxqw\" (UID: \"230f0407-5c25-44aa-9a97-1de91305e4f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" Oct 30 01:59:50.000551 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.000545 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/230f0407-5c25-44aa-9a97-1de91305e4f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tzxqw\" (UID: \"230f0407-5c25-44aa-9a97-1de91305e4f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" Oct 30 01:59:50.000865 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:59:50.000565 6572 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-817fd7/agc-cred-08a8e8dfe09b: secret "agc-cred-08a8e8dfe09b" not found Oct 30 01:59:50.000865 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:59:50.000587 6572 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr: secret "agc-cred-08a8e8dfe09b" not found Oct 30 01:59:50.000865 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.000595 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/230f0407-5c25-44aa-9a97-1de91305e4f3-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tzxqw\" (UID: \"230f0407-5c25-44aa-9a97-1de91305e4f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" Oct 30 01:59:50.000865 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:59:50.000638 6572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18a3aece-618b-4a9a-8fc9-28606bddc3ce-provider-credentials podName:18a3aece-618b-4a9a-8fc9-28606bddc3ce nodeName:}" failed. No retries permitted until 2026-10-30 01:59:51.000620939 +0000 UTC m=+2516.577046331 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/18a3aece-618b-4a9a-8fc9-28606bddc3ce-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr" (UID: "18a3aece-618b-4a9a-8fc9-28606bddc3ce") : secret "agc-cred-08a8e8dfe09b" not found Oct 30 01:59:50.000865 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.000686 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/230f0407-5c25-44aa-9a97-1de91305e4f3-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tzxqw\" (UID: \"230f0407-5c25-44aa-9a97-1de91305e4f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" Oct 30 01:59:50.000865 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.000742 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/230f0407-5c25-44aa-9a97-1de91305e4f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tzxqw\" (UID: \"230f0407-5c25-44aa-9a97-1de91305e4f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" Oct 30 01:59:50.000865 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.000832 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/230f0407-5c25-44aa-9a97-1de91305e4f3-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tzxqw\" (UID: \"230f0407-5c25-44aa-9a97-1de91305e4f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" Oct 30 01:59:50.000865 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.000856 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w49f8\" (UniqueName: \"kubernetes.io/projected/230f0407-5c25-44aa-9a97-1de91305e4f3-kube-api-access-w49f8\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tzxqw\" (UID: \"230f0407-5c25-44aa-9a97-1de91305e4f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" Oct 30 01:59:50.001334 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.000907 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/230f0407-5c25-44aa-9a97-1de91305e4f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tzxqw\" (UID: \"230f0407-5c25-44aa-9a97-1de91305e4f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" Oct 30 01:59:50.001334 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.001006 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/230f0407-5c25-44aa-9a97-1de91305e4f3-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tzxqw\" (UID: \"230f0407-5c25-44aa-9a97-1de91305e4f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" Oct 30 01:59:50.001334 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.001005 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/230f0407-5c25-44aa-9a97-1de91305e4f3-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tzxqw\" (UID: \"230f0407-5c25-44aa-9a97-1de91305e4f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" Oct 30 01:59:50.001334 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.001073 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/230f0407-5c25-44aa-9a97-1de91305e4f3-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tzxqw\" (UID: \"230f0407-5c25-44aa-9a97-1de91305e4f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" Oct 30 01:59:50.001334 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.001276 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/230f0407-5c25-44aa-9a97-1de91305e4f3-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tzxqw\" (UID: \"230f0407-5c25-44aa-9a97-1de91305e4f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" Oct 30 01:59:50.001608 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.001339 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/230f0407-5c25-44aa-9a97-1de91305e4f3-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tzxqw\" (UID: \"230f0407-5c25-44aa-9a97-1de91305e4f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" Oct 30 01:59:50.001608 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.001394 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/230f0407-5c25-44aa-9a97-1de91305e4f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tzxqw\" (UID: \"230f0407-5c25-44aa-9a97-1de91305e4f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" Oct 30 01:59:50.003379 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.003355 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/230f0407-5c25-44aa-9a97-1de91305e4f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tzxqw\" (UID: \"230f0407-5c25-44aa-9a97-1de91305e4f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" Oct 30 01:59:50.003626 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.003606 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/230f0407-5c25-44aa-9a97-1de91305e4f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tzxqw\" (UID: \"230f0407-5c25-44aa-9a97-1de91305e4f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" Oct 30 01:59:50.021640 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.021608 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/230f0407-5c25-44aa-9a97-1de91305e4f3-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tzxqw\" (UID: \"230f0407-5c25-44aa-9a97-1de91305e4f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" Oct 30 01:59:50.021846 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.021794 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w49f8\" (UniqueName: \"kubernetes.io/projected/230f0407-5c25-44aa-9a97-1de91305e4f3-kube-api-access-w49f8\") pod \"maas-default-gateway-openshift-default-5bc884dd87-tzxqw\" (UID: \"230f0407-5c25-44aa-9a97-1de91305e4f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" Oct 30 01:59:50.038492 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.038457 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" Oct 30 01:59:50.084575 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.084526 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m"] Oct 30 01:59:50.090604 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.090575 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m" Oct 30 01:59:50.097282 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.097216 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" Oct 30 01:59:50.114863 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.104988 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-817fd7\"/\"e2e-cfl-bdaa0d-provider-cred\"" Oct 30 01:59:50.124250 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.115999 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m"] Oct 30 01:59:50.207791 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.205568 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e90574-f0e0-40dd-8ada-bf19c3839a6d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m\" (UID: \"17e90574-f0e0-40dd-8ada-bf19c3839a6d\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m" Oct 30 01:59:50.207791 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.205821 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvcmv\" (UniqueName: \"kubernetes.io/projected/17e90574-f0e0-40dd-8ada-bf19c3839a6d-kube-api-access-nvcmv\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m\" (UID: \"17e90574-f0e0-40dd-8ada-bf19c3839a6d\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m" Oct 30 01:59:50.207791 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.205899 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/17e90574-f0e0-40dd-8ada-bf19c3839a6d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m\" (UID: \"17e90574-f0e0-40dd-8ada-bf19c3839a6d\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m" Oct 30 01:59:50.207791 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.205933 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/17e90574-f0e0-40dd-8ada-bf19c3839a6d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m\" (UID: \"17e90574-f0e0-40dd-8ada-bf19c3839a6d\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m" Oct 30 01:59:50.307942 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.307847 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e90574-f0e0-40dd-8ada-bf19c3839a6d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m\" (UID: \"17e90574-f0e0-40dd-8ada-bf19c3839a6d\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m" Oct 30 01:59:50.308184 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.308017 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvcmv\" (UniqueName: \"kubernetes.io/projected/17e90574-f0e0-40dd-8ada-bf19c3839a6d-kube-api-access-nvcmv\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m\" (UID: \"17e90574-f0e0-40dd-8ada-bf19c3839a6d\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m" Oct 30 01:59:50.308184 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.308075 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/17e90574-f0e0-40dd-8ada-bf19c3839a6d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m\" (UID: \"17e90574-f0e0-40dd-8ada-bf19c3839a6d\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m" Oct 30 01:59:50.308184 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.308109 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/17e90574-f0e0-40dd-8ada-bf19c3839a6d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m\" (UID: \"17e90574-f0e0-40dd-8ada-bf19c3839a6d\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m" Oct 30 01:59:50.308894 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.308868 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e90574-f0e0-40dd-8ada-bf19c3839a6d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m\" (UID: \"17e90574-f0e0-40dd-8ada-bf19c3839a6d\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m" Oct 30 01:59:50.309000 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.308919 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/17e90574-f0e0-40dd-8ada-bf19c3839a6d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m\" (UID: \"17e90574-f0e0-40dd-8ada-bf19c3839a6d\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m" Oct 30 01:59:50.310783 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.310743 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/17e90574-f0e0-40dd-8ada-bf19c3839a6d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m\" (UID: \"17e90574-f0e0-40dd-8ada-bf19c3839a6d\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m" Oct 30 01:59:50.327916 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.327892 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvcmv\" (UniqueName: \"kubernetes.io/projected/17e90574-f0e0-40dd-8ada-bf19c3839a6d-kube-api-access-nvcmv\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m\" (UID: \"17e90574-f0e0-40dd-8ada-bf19c3839a6d\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m" Oct 30 01:59:50.362433 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.362404 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2"] Oct 30 01:59:50.363818 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:59:50.363790 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f3448e6_c22d_466c_a485_fbe2e676c250.slice/crio-b5fa13bdadc6ab30b79f4001275e85ee681fcb877ab3ff13a8ef837b28cf47ca WatchSource:0}: Error finding container b5fa13bdadc6ab30b79f4001275e85ee681fcb877ab3ff13a8ef837b28cf47ca: Status 404 returned error can't find the container with id b5fa13bdadc6ab30b79f4001275e85ee681fcb877ab3ff13a8ef837b28cf47ca Oct 30 01:59:50.366865 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.366833 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:59:50.366953 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.366905 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:59:50.366953 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.366938 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:59:50.426499 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:59:50.426473 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod230f0407_5c25_44aa_9a97_1de91305e4f3.slice/crio-8876f28a99416f33b6f854397e1d5bf50f4567f3d6f82f3dd077d40ea8c96910 WatchSource:0}: Error finding container 8876f28a99416f33b6f854397e1d5bf50f4567f3d6f82f3dd077d40ea8c96910: Status 404 returned error can't find the container with id 8876f28a99416f33b6f854397e1d5bf50f4567f3d6f82f3dd077d40ea8c96910 Oct 30 01:59:50.426782 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.426742 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw"] Oct 30 01:59:50.429701 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.429664 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:59:50.429829 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.429764 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:59:50.429829 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.429811 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:59:50.432252 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.432218 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m" Oct 30 01:59:50.760403 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.760373 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m"] Oct 30 01:59:50.762243 ip-10-0-138-223 kubenswrapper[6572]: W1030 01:59:50.762216 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e90574_f0e0_40dd_8ada_bf19c3839a6d.slice/crio-ebf40f31e661ad6feb76b38a42bf895d800ef5c70d6692c50d6b09cb2efce9c8 WatchSource:0}: Error finding container ebf40f31e661ad6feb76b38a42bf895d800ef5c70d6692c50d6b09cb2efce9c8: Status 404 returned error can't find the container with id ebf40f31e661ad6feb76b38a42bf895d800ef5c70d6692c50d6b09cb2efce9c8 Oct 30 01:59:50.970815 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:50.968384 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:51.053171 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:59:51.053081 6572 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-817fd7/agc-cred-08a8e8dfe09b: secret "agc-cred-08a8e8dfe09b" not found Oct 30 01:59:51.053171 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:59:51.053111 6572 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr: secret "agc-cred-08a8e8dfe09b" not found Oct 30 01:59:51.053171 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:59:51.053166 6572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18a3aece-618b-4a9a-8fc9-28606bddc3ce-provider-credentials podName:18a3aece-618b-4a9a-8fc9-28606bddc3ce nodeName:}" failed. No retries permitted until 2026-10-30 01:59:53.05315061 +0000 UTC m=+2518.629576004 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/18a3aece-618b-4a9a-8fc9-28606bddc3ce-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr" (UID: "18a3aece-618b-4a9a-8fc9-28606bddc3ce") : secret "agc-cred-08a8e8dfe09b" not found Oct 30 01:59:51.358779 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:51.358641 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" event={"ID":"230f0407-5c25-44aa-9a97-1de91305e4f3","Type":"ContainerStarted","Data":"c2d23000006f5624e239b5fb1e018bd43cb1905d4614a9bd0f09a4ba5f9da268"} Oct 30 01:59:51.358779 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:51.358741 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" event={"ID":"230f0407-5c25-44aa-9a97-1de91305e4f3","Type":"ContainerStarted","Data":"8876f28a99416f33b6f854397e1d5bf50f4567f3d6f82f3dd077d40ea8c96910"} Oct 30 01:59:51.360588 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:51.360559 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" event={"ID":"5f3448e6-c22d-466c-a485-fbe2e676c250","Type":"ContainerStarted","Data":"0635c8e2fdad88fff1b57a81a5208e77d93cfa9be156bb2af249114f0913534e"} Oct 30 01:59:51.360737 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:51.360607 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" event={"ID":"5f3448e6-c22d-466c-a485-fbe2e676c250","Type":"ContainerStarted","Data":"b5fa13bdadc6ab30b79f4001275e85ee681fcb877ab3ff13a8ef837b28cf47ca"} Oct 30 01:59:51.363105 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:51.363078 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m" event={"ID":"17e90574-f0e0-40dd-8ada-bf19c3839a6d","Type":"ContainerStarted","Data":"c65589b423f010aa4404a0bdc3d09d4c8a12c3619ee1a988dbcc49a638d68916"} Oct 30 01:59:51.363537 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:51.363151 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m" event={"ID":"17e90574-f0e0-40dd-8ada-bf19c3839a6d","Type":"ContainerStarted","Data":"ebf40f31e661ad6feb76b38a42bf895d800ef5c70d6692c50d6b09cb2efce9c8"} Oct 30 01:59:51.363537 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:51.363348 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m" Oct 30 01:59:51.405740 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:51.405674 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" podStartSLOduration=2.405652803 podStartE2EDuration="2.405652803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:59:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:59:51.400346521 +0000 UTC m=+2516.976771923" watchObservedRunningTime="2026-10-30 01:59:51.405652803 +0000 UTC m=+2516.982078206" Oct 30 01:59:51.441344 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:51.441287 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m" podStartSLOduration=1.441274097 podStartE2EDuration="1.441274097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:59:51.439423796 +0000 UTC m=+2517.015849197" watchObservedRunningTime="2026-10-30 01:59:51.441274097 +0000 UTC m=+2517.017699499" Oct 30 01:59:51.479953 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:51.479888 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" podStartSLOduration=2.479873539 podStartE2EDuration="2.479873539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:59:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:59:51.47881228 +0000 UTC m=+2517.055237685" watchObservedRunningTime="2026-10-30 01:59:51.479873539 +0000 UTC m=+2517.056298941" Oct 30 01:59:52.049841 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:52.049804 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" Oct 30 01:59:52.050199 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:52.050117 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-rl4q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body= Oct 30 01:59:52.050391 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:52.050366 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" podUID="5f3448e6-c22d-466c-a485-fbe2e676c250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" Oct 30 01:59:52.098004 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:52.097976 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" Oct 30 01:59:52.099077 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:52.099054 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-tzxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body= Oct 30 01:59:52.099205 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:52.099107 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" podUID="230f0407-5c25-44aa-9a97-1de91305e4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" Oct 30 01:59:52.513770 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:52.497570 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:53.046219 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:53.046177 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-rl4q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body= Oct 30 01:59:53.046510 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:53.046479 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" podUID="5f3448e6-c22d-466c-a485-fbe2e676c250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" Oct 30 01:59:53.090271 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:59:53.090244 6572 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-817fd7/agc-cred-08a8e8dfe09b: secret "agc-cred-08a8e8dfe09b" not found Oct 30 01:59:53.090460 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:59:53.090450 6572 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr: secret "agc-cred-08a8e8dfe09b" not found Oct 30 01:59:53.090582 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:59:53.090574 6572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18a3aece-618b-4a9a-8fc9-28606bddc3ce-provider-credentials podName:18a3aece-618b-4a9a-8fc9-28606bddc3ce nodeName:}" failed. No retries permitted until 2026-10-30 01:59:57.090557677 +0000 UTC m=+2522.666983070 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/18a3aece-618b-4a9a-8fc9-28606bddc3ce-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr" (UID: "18a3aece-618b-4a9a-8fc9-28606bddc3ce") : secret "agc-cred-08a8e8dfe09b" not found Oct 30 01:59:53.098244 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:53.098181 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-tzxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body= Oct 30 01:59:53.098456 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:53.098438 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" podUID="230f0407-5c25-44aa-9a97-1de91305e4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" Oct 30 01:59:54.041772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:54.040195 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-rl4q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body= Oct 30 01:59:54.041772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:54.040258 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" podUID="5f3448e6-c22d-466c-a485-fbe2e676c250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" Oct 30 01:59:54.098295 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:54.098252 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-tzxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body= Oct 30 01:59:54.098493 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:54.098315 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" podUID="230f0407-5c25-44aa-9a97-1de91305e4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" Oct 30 01:59:54.301831 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:54.297379 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:55.042773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:55.039259 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-rl4q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body= Oct 30 01:59:55.042773 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:55.039345 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" podUID="5f3448e6-c22d-466c-a485-fbe2e676c250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" Oct 30 01:59:55.098513 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:55.098476 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-tzxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body= Oct 30 01:59:55.098698 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:55.098534 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" podUID="230f0407-5c25-44aa-9a97-1de91305e4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" Oct 30 01:59:55.958774 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:55.958079 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:56.040778 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:56.040021 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-rl4q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body= Oct 30 01:59:56.040778 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:56.040075 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" podUID="5f3448e6-c22d-466c-a485-fbe2e676c250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" Oct 30 01:59:56.098269 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:56.098226 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-tzxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body= Oct 30 01:59:56.098817 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:56.098279 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" podUID="230f0407-5c25-44aa-9a97-1de91305e4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" Oct 30 01:59:56.659353 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:56.659299 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38bc62b8-zrngw"] Oct 30 01:59:56.666203 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:56.666174 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bc62b8-zrngw" Oct 30 01:59:56.681451 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:56.681421 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38bc62b8-zrngw"] Oct 30 01:59:56.685071 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:56.685023 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:59:56.685071 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:56.685038 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:59:56.685277 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:56.685168 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:59:56.692252 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:56.692227 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:59:56.757940 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:56.757893 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31878162-1468-4639-826b-a725564a1d41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38bc62b8-zrngw\" (UID: \"31878162-1468-4639-826b-a725564a1d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bc62b8-zrngw" Oct 30 01:59:56.758313 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:56.758288 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31878162-1468-4639-826b-a725564a1d41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38bc62b8-zrngw\" (UID: \"31878162-1468-4639-826b-a725564a1d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bc62b8-zrngw" Oct 30 01:59:56.758779 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:56.758731 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5474\" (UniqueName: \"kubernetes.io/projected/31878162-1468-4639-826b-a725564a1d41-kube-api-access-d5474\") pod \"maas-api-revoke-sub-38bc62b8-zrngw\" (UID: \"31878162-1468-4639-826b-a725564a1d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bc62b8-zrngw" Oct 30 01:59:56.862174 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:56.860153 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31878162-1468-4639-826b-a725564a1d41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38bc62b8-zrngw\" (UID: \"31878162-1468-4639-826b-a725564a1d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bc62b8-zrngw" Oct 30 01:59:56.862174 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:56.860210 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31878162-1468-4639-826b-a725564a1d41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38bc62b8-zrngw\" (UID: \"31878162-1468-4639-826b-a725564a1d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bc62b8-zrngw" Oct 30 01:59:56.862174 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:56.860375 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5474\" (UniqueName: \"kubernetes.io/projected/31878162-1468-4639-826b-a725564a1d41-kube-api-access-d5474\") pod \"maas-api-revoke-sub-38bc62b8-zrngw\" (UID: \"31878162-1468-4639-826b-a725564a1d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bc62b8-zrngw" Oct 30 01:59:56.862174 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:56.861880 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31878162-1468-4639-826b-a725564a1d41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38bc62b8-zrngw\" (UID: \"31878162-1468-4639-826b-a725564a1d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bc62b8-zrngw" Oct 30 01:59:56.863989 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:56.863965 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31878162-1468-4639-826b-a725564a1d41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38bc62b8-zrngw\" (UID: \"31878162-1468-4639-826b-a725564a1d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bc62b8-zrngw" Oct 30 01:59:56.883030 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:56.883002 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5474\" (UniqueName: \"kubernetes.io/projected/31878162-1468-4639-826b-a725564a1d41-kube-api-access-d5474\") pod \"maas-api-revoke-sub-38bc62b8-zrngw\" (UID: \"31878162-1468-4639-826b-a725564a1d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bc62b8-zrngw" Oct 30 01:59:56.983889 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:56.983857 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bc62b8-zrngw" Oct 30 01:59:57.042772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:57.042173 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-rl4q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body= Oct 30 01:59:57.042772 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:57.042234 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" podUID="5f3448e6-c22d-466c-a485-fbe2e676c250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" Oct 30 01:59:57.097922 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:57.097888 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-tzxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body= Oct 30 01:59:57.098038 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:57.097962 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" podUID="230f0407-5c25-44aa-9a97-1de91305e4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" Oct 30 01:59:57.168790 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:59:57.167967 6572 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-817fd7/agc-cred-08a8e8dfe09b: secret "agc-cred-08a8e8dfe09b" not found Oct 30 01:59:57.168790 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:59:57.167997 6572 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr: secret "agc-cred-08a8e8dfe09b" not found Oct 30 01:59:57.168790 ip-10-0-138-223 kubenswrapper[6572]: E1030 01:59:57.168120 6572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18a3aece-618b-4a9a-8fc9-28606bddc3ce-provider-credentials podName:18a3aece-618b-4a9a-8fc9-28606bddc3ce nodeName:}" failed. No retries permitted until 2026-10-30 02:00:05.168100529 +0000 UTC m=+2530.744525913 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/18a3aece-618b-4a9a-8fc9-28606bddc3ce-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr" (UID: "18a3aece-618b-4a9a-8fc9-28606bddc3ce") : secret "agc-cred-08a8e8dfe09b" not found Oct 30 01:59:57.349790 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:57.349720 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38bc62b8-zrngw"] Oct 30 01:59:57.526829 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:57.524091 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:57.535290 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:57.535252 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bc62b8-zrngw" event={"ID":"31878162-1468-4639-826b-a725564a1d41","Type":"ContainerStarted","Data":"ba4b43927d294ecab9eb4cd2a2ea9cd74fcd6875bb160502060d7ea9215eed28"} Oct 30 01:59:58.039087 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:58.039042 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-rl4q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body= Oct 30 01:59:58.039312 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:58.039128 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" podUID="5f3448e6-c22d-466c-a485-fbe2e676c250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" Oct 30 01:59:58.062626 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:58.062593 6572 scope.go:122] "RemoveContainer" containerID="e3533a68fa2b05877137216ac0c005ca03fc36112581c1213c9633090a3f45d8" Oct 30 01:59:58.078618 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:58.078589 6572 scope.go:122] "RemoveContainer" containerID="4d458581ea8ab8b57265dd4da2742c6b22e53ed55d18537df49cce0130c296d1" Oct 30 01:59:58.091318 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:58.091293 6572 scope.go:122] "RemoveContainer" containerID="4ebf6f2755c92bf9bff4017d44017db4c4d0eff92259a295a3bc09e8ddc6c5db" Oct 30 01:59:58.097814 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:58.097780 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-tzxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body= Oct 30 01:59:58.097936 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:58.097838 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" podUID="230f0407-5c25-44aa-9a97-1de91305e4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" Oct 30 01:59:58.104698 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:58.104678 6572 scope.go:122] "RemoveContainer" containerID="f1578462e840b980d2d3160338344aeab13c6381881ba13b47dfe1e6d8c3581b" Oct 30 01:59:58.552078 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:58.552037 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bc62b8-zrngw" event={"ID":"31878162-1468-4639-826b-a725564a1d41","Type":"ContainerStarted","Data":"cb423b65b755a1494b02905b7372fab803ac8e37df949e6b7ee7df95ccff9655"} Oct 30 01:59:59.101822 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:59.098874 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-tzxqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body= Oct 30 01:59:59.101822 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:59.098942 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" podUID="230f0407-5c25-44aa-9a97-1de91305e4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" Oct 30 01:59:59.119291 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:59.116740 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:59.573510 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:59.573465 6572 generic.go:356] "Generic (PLEG): container finished" podID="31878162-1468-4639-826b-a725564a1d41" containerID="cb423b65b755a1494b02905b7372fab803ac8e37df949e6b7ee7df95ccff9655" exitCode=0 Oct 30 01:59:59.574564 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:59.573845 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bc62b8-zrngw" event={"ID":"31878162-1468-4639-826b-a725564a1d41","Type":"ContainerDied","Data":"cb423b65b755a1494b02905b7372fab803ac8e37df949e6b7ee7df95ccff9655"} Oct 30 01:59:59.883985 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:59.883942 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bc62b8-zrngw" Oct 30 01:59:59.916704 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:59.916655 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31878162-1468-4639-826b-a725564a1d41-kube-api-access-d5474\" (UniqueName: \"kubernetes.io/projected/31878162-1468-4639-826b-a725564a1d41-kube-api-access-d5474\") pod \"31878162-1468-4639-826b-a725564a1d41\" (UID: \"31878162-1468-4639-826b-a725564a1d41\") " Oct 30 01:59:59.916931 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:59.916789 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31878162-1468-4639-826b-a725564a1d41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31878162-1468-4639-826b-a725564a1d41-maas-api-openshift-ca\") pod \"31878162-1468-4639-826b-a725564a1d41\" (UID: \"31878162-1468-4639-826b-a725564a1d41\") " Oct 30 01:59:59.916931 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:59.916906 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31878162-1468-4639-826b-a725564a1d41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31878162-1468-4639-826b-a725564a1d41-maas-api-xks-ca\") pod \"31878162-1468-4639-826b-a725564a1d41\" (UID: \"31878162-1468-4639-826b-a725564a1d41\") " Oct 30 01:59:59.917502 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:59.917463 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31878162-1468-4639-826b-a725564a1d41-maas-api-openshift-ca" pod "31878162-1468-4639-826b-a725564a1d41" (UID: "31878162-1468-4639-826b-a725564a1d41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:59:59.917801 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:59.917778 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31878162-1468-4639-826b-a725564a1d41-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 01:59:59.920064 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:59.920021 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31878162-1468-4639-826b-a725564a1d41-kube-api-access-d5474" pod "31878162-1468-4639-826b-a725564a1d41" (UID: "31878162-1468-4639-826b-a725564a1d41"). InnerVolumeSpecName "kube-api-access-d5474". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:59.921326 ip-10-0-138-223 kubenswrapper[6572]: I1030 01:59:59.920939 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31878162-1468-4639-826b-a725564a1d41-maas-api-xks-ca" pod "31878162-1468-4639-826b-a725564a1d41" (UID: "31878162-1468-4639-826b-a725564a1d41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:00:00.019282 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.019213 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5474\" (UniqueName: \"kubernetes.io/projected/31878162-1468-4639-826b-a725564a1d41-kube-api-access-d5474\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:00.019282 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.019259 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31878162-1468-4639-826b-a725564a1d41-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:00.039095 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.039061 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-rl4q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:00:00.039275 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.039114 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" podUID="5f3448e6-c22d-466c-a485-fbe2e676c250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:00:00.115431 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.115326 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" Oct 30 02:00:00.115942 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.115915 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" Oct 30 02:00:00.117902 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.117856 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" Oct 30 02:00:00.150775 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.150545 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888760-wkrkz"] Oct 30 02:00:00.151797 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.151758 6572 cpu_manager.go:414] "containerMap: removing container" podUID="31878162-1468-4639-826b-a725564a1d41" containerName="revoke-keys" Oct 30 02:00:00.151797 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.151784 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31878162-1468-4639-826b-a725564a1d41" containerName="revoke-keys" Oct 30 02:00:00.152010 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.151990 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="31878162-1468-4639-826b-a725564a1d41" containerName="revoke-keys" Oct 30 02:00:00.156303 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.156280 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888760-wkrkz" Oct 30 02:00:00.157595 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.157396 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298"] Oct 30 02:00:00.174788 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.172189 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" Oct 30 02:00:00.174788 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.172233 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2eae39-29888760-t6tz5"] Oct 30 02:00:00.174788 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.172377 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298" Oct 30 02:00:00.189058 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.188921 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888760-6h8kw"] Oct 30 02:00:00.189193 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.189026 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2eae39-29888760-t6tz5" Oct 30 02:00:00.196010 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.195981 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888760-j979j"] Oct 30 02:00:00.196508 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.196487 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888760-6h8kw" Oct 30 02:00:00.202635 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.202216 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888760-wkrkz"] Oct 30 02:00:00.202635 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.202269 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" Oct 30 02:00:00.202635 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.202325 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" Oct 30 02:00:00.202635 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.202417 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888760-j979j" Oct 30 02:00:00.214327 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.214296 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888760-hvkkt"] Oct 30 02:00:00.224153 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.224119 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888760-hvkkt" Oct 30 02:00:00.231402 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.227277 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgxxb\" (UniqueName: \"kubernetes.io/projected/32354403-56a7-4f01-a228-122805b44ba0-kube-api-access-pgxxb\") pod \"maas-api-key-cleanup-e2e-extmod-b-093d23-29888760-wkrkz\" (UID: \"32354403-56a7-4f01-a228-122805b44ba0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888760-wkrkz" Oct 30 02:00:00.231402 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.227893 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jft2l\" (UniqueName: \"kubernetes.io/projected/ea84dc40-9569-4a87-a537-a8e0c0fb119d-kube-api-access-jft2l\") pod \"maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888760-hvkkt\" (UID: \"ea84dc40-9569-4a87-a537-a8e0c0fb119d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888760-hvkkt" Oct 30 02:00:00.231402 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.227919 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xmxt\" (UniqueName: \"kubernetes.io/projected/ad3ca000-d9f7-4d97-9a05-b39362550ba0-kube-api-access-6xmxt\") pod \"maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298\" (UID: \"ad3ca000-d9f7-4d97-9a05-b39362550ba0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298" Oct 30 02:00:00.231402 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.228370 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmvqr\" (UniqueName: \"kubernetes.io/projected/bc3321aa-698f-4c74-94ae-95e8729d9f27-kube-api-access-kmvqr\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-292018-29888760-j979j\" (UID: \"bc3321aa-698f-4c74-94ae-95e8729d9f27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888760-j979j" Oct 30 02:00:00.231402 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.228400 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32354403-56a7-4f01-a228-122805b44ba0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-093d23-29888760-wkrkz\" (UID: \"32354403-56a7-4f01-a228-122805b44ba0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888760-wkrkz" Oct 30 02:00:00.231402 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.228539 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dc37d0b-7989-4437-9902-df0a34eb682a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2eae39-29888760-t6tz5\" (UID: \"9dc37d0b-7989-4437-9902-df0a34eb682a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2eae39-29888760-t6tz5" Oct 30 02:00:00.231402 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.228634 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfrph\" (UniqueName: \"kubernetes.io/projected/9dc37d0b-7989-4437-9902-df0a34eb682a-kube-api-access-zfrph\") pod \"maas-api-key-cleanup-e2e-extmod-a-2eae39-29888760-t6tz5\" (UID: \"9dc37d0b-7989-4437-9902-df0a34eb682a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2eae39-29888760-t6tz5" Oct 30 02:00:00.231402 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.228693 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888760-6h8kw\" (UID: \"d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888760-6h8kw" Oct 30 02:00:00.231402 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.228725 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc3321aa-698f-4c74-94ae-95e8729d9f27-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-292018-29888760-j979j\" (UID: \"bc3321aa-698f-4c74-94ae-95e8729d9f27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888760-j979j" Oct 30 02:00:00.231402 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.229271 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn6rk\" (UniqueName: \"kubernetes.io/projected/d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b-kube-api-access-cn6rk\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888760-6h8kw\" (UID: \"d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888760-6h8kw" Oct 30 02:00:00.231402 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.229401 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea84dc40-9569-4a87-a537-a8e0c0fb119d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888760-hvkkt\" (UID: \"ea84dc40-9569-4a87-a537-a8e0c0fb119d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888760-hvkkt" Oct 30 02:00:00.245018 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.244971 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad3ca000-d9f7-4d97-9a05-b39362550ba0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298\" (UID: \"ad3ca000-d9f7-4d97-9a05-b39362550ba0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298" Oct 30 02:00:00.250132 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.250103 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888760-xhtnj"] Oct 30 02:00:00.285686 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.283722 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888760-m68hl"] Oct 30 02:00:00.293030 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.287428 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888760-xhtnj" Oct 30 02:00:00.299490 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.297217 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888760-5xgtv"] Oct 30 02:00:00.299490 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.297290 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888760-m68hl" Oct 30 02:00:00.312131 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.312094 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888760-kwp4k"] Oct 30 02:00:00.312317 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.312279 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-5xgtv" Oct 30 02:00:00.321992 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.321961 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298"] Oct 30 02:00:00.324255 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.322180 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888760-kwp4k" Oct 30 02:00:00.332940 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.330817 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2eae39-29888760-t6tz5"] Oct 30 02:00:00.342624 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.342599 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888760-j979j"] Oct 30 02:00:00.346427 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.346395 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb5qv\" (UniqueName: \"kubernetes.io/projected/45e5935e-478f-4fec-beae-0cae2ea14c1b-kube-api-access-xb5qv\") pod \"maas-api-key-cleanup-e2e-extmod-a-76a19e-29888760-xhtnj\" (UID: \"45e5935e-478f-4fec-beae-0cae2ea14c1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888760-xhtnj" Oct 30 02:00:00.346953 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.346926 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad3ca000-d9f7-4d97-9a05-b39362550ba0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298\" (UID: \"ad3ca000-d9f7-4d97-9a05-b39362550ba0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298" Oct 30 02:00:00.347147 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.347127 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgxxb\" (UniqueName: \"kubernetes.io/projected/32354403-56a7-4f01-a228-122805b44ba0-kube-api-access-pgxxb\") pod \"maas-api-key-cleanup-e2e-extmod-b-093d23-29888760-wkrkz\" (UID: \"32354403-56a7-4f01-a228-122805b44ba0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888760-wkrkz" Oct 30 02:00:00.348494 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.348228 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad3ca000-d9f7-4d97-9a05-b39362550ba0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298\" (UID: \"ad3ca000-d9f7-4d97-9a05-b39362550ba0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298" Oct 30 02:00:00.348630 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.348485 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jft2l\" (UniqueName: \"kubernetes.io/projected/ea84dc40-9569-4a87-a537-a8e0c0fb119d-kube-api-access-jft2l\") pod \"maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888760-hvkkt\" (UID: \"ea84dc40-9569-4a87-a537-a8e0c0fb119d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888760-hvkkt" Oct 30 02:00:00.348810 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.348734 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xmxt\" (UniqueName: \"kubernetes.io/projected/ad3ca000-d9f7-4d97-9a05-b39362550ba0-kube-api-access-6xmxt\") pod \"maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298\" (UID: \"ad3ca000-d9f7-4d97-9a05-b39362550ba0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298" Oct 30 02:00:00.348953 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.348937 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8e5ee17-3f55-4488-aba3-2a81eceb15a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888760-5xgtv\" (UID: \"b8e5ee17-3f55-4488-aba3-2a81eceb15a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-5xgtv" Oct 30 02:00:00.349139 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.349118 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmvqr\" (UniqueName: \"kubernetes.io/projected/bc3321aa-698f-4c74-94ae-95e8729d9f27-kube-api-access-kmvqr\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-292018-29888760-j979j\" (UID: \"bc3321aa-698f-4c74-94ae-95e8729d9f27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888760-j979j" Oct 30 02:00:00.349283 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.349266 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32354403-56a7-4f01-a228-122805b44ba0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-093d23-29888760-wkrkz\" (UID: \"32354403-56a7-4f01-a228-122805b44ba0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888760-wkrkz" Oct 30 02:00:00.349425 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.349407 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mj5k\" (UniqueName: \"kubernetes.io/projected/228bfbbd-21b3-4f89-b247-962811a659e6-kube-api-access-7mj5k\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888760-m68hl\" (UID: \"228bfbbd-21b3-4f89-b247-962811a659e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888760-m68hl" Oct 30 02:00:00.349585 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.349568 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6txhh\" (UniqueName: \"kubernetes.io/projected/b8e5ee17-3f55-4488-aba3-2a81eceb15a4-kube-api-access-6txhh\") pod \"maas-api-key-cleanup-29888760-5xgtv\" (UID: \"b8e5ee17-3f55-4488-aba3-2a81eceb15a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-5xgtv" Oct 30 02:00:00.349819 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.349732 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dc37d0b-7989-4437-9902-df0a34eb682a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2eae39-29888760-t6tz5\" (UID: \"9dc37d0b-7989-4437-9902-df0a34eb682a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2eae39-29888760-t6tz5" Oct 30 02:00:00.349819 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.349817 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfrph\" (UniqueName: \"kubernetes.io/projected/9dc37d0b-7989-4437-9902-df0a34eb682a-kube-api-access-zfrph\") pod \"maas-api-key-cleanup-e2e-extmod-a-2eae39-29888760-t6tz5\" (UID: \"9dc37d0b-7989-4437-9902-df0a34eb682a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2eae39-29888760-t6tz5" Oct 30 02:00:00.350034 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.349850 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqkbf\" (UniqueName: \"kubernetes.io/projected/05cb266a-e1fe-4b51-a18f-96d85a78ab16-kube-api-access-jqkbf\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888760-kwp4k\" (UID: \"05cb266a-e1fe-4b51-a18f-96d85a78ab16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888760-kwp4k" Oct 30 02:00:00.350034 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.349884 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888760-6h8kw\" (UID: \"d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888760-6h8kw" Oct 30 02:00:00.350034 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.349912 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc3321aa-698f-4c74-94ae-95e8729d9f27-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-292018-29888760-j979j\" (UID: \"bc3321aa-698f-4c74-94ae-95e8729d9f27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888760-j979j" Oct 30 02:00:00.350034 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.349958 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn6rk\" (UniqueName: \"kubernetes.io/projected/d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b-kube-api-access-cn6rk\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888760-6h8kw\" (UID: \"d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888760-6h8kw" Oct 30 02:00:00.354274 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.350045 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea84dc40-9569-4a87-a537-a8e0c0fb119d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888760-hvkkt\" (UID: \"ea84dc40-9569-4a87-a537-a8e0c0fb119d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888760-hvkkt" Oct 30 02:00:00.354274 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.350123 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/228bfbbd-21b3-4f89-b247-962811a659e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888760-m68hl\" (UID: \"228bfbbd-21b3-4f89-b247-962811a659e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888760-m68hl" Oct 30 02:00:00.354274 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.350169 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45e5935e-478f-4fec-beae-0cae2ea14c1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-76a19e-29888760-xhtnj\" (UID: \"45e5935e-478f-4fec-beae-0cae2ea14c1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888760-xhtnj" Oct 30 02:00:00.354274 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.350282 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05cb266a-e1fe-4b51-a18f-96d85a78ab16-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888760-kwp4k\" (UID: \"05cb266a-e1fe-4b51-a18f-96d85a78ab16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888760-kwp4k" Oct 30 02:00:00.354274 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.351294 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32354403-56a7-4f01-a228-122805b44ba0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-093d23-29888760-wkrkz\" (UID: \"32354403-56a7-4f01-a228-122805b44ba0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888760-wkrkz" Oct 30 02:00:00.354274 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.351294 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888760-6h8kw\" (UID: \"d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888760-6h8kw" Oct 30 02:00:00.354274 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.351860 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc3321aa-698f-4c74-94ae-95e8729d9f27-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-292018-29888760-j979j\" (UID: \"bc3321aa-698f-4c74-94ae-95e8729d9f27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888760-j979j" Oct 30 02:00:00.354274 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.351992 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dc37d0b-7989-4437-9902-df0a34eb682a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2eae39-29888760-t6tz5\" (UID: \"9dc37d0b-7989-4437-9902-df0a34eb682a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2eae39-29888760-t6tz5" Oct 30 02:00:00.354274 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.352293 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea84dc40-9569-4a87-a537-a8e0c0fb119d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888760-hvkkt\" (UID: \"ea84dc40-9569-4a87-a537-a8e0c0fb119d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888760-hvkkt" Oct 30 02:00:00.354952 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.354401 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888760-6h8kw"] Oct 30 02:00:00.371848 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.369166 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888760-hvkkt"] Oct 30 02:00:00.375330 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.375284 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888760-z8976"] Oct 30 02:00:00.389549 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.389511 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn6rk\" (UniqueName: \"kubernetes.io/projected/d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b-kube-api-access-cn6rk\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888760-6h8kw\" (UID: \"d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888760-6h8kw" Oct 30 02:00:00.389957 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.389932 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888760-lnmg5"] Oct 30 02:00:00.391790 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.391215 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888760-z8976" Oct 30 02:00:00.391790 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.391308 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgxxb\" (UniqueName: \"kubernetes.io/projected/32354403-56a7-4f01-a228-122805b44ba0-kube-api-access-pgxxb\") pod \"maas-api-key-cleanup-e2e-extmod-b-093d23-29888760-wkrkz\" (UID: \"32354403-56a7-4f01-a228-122805b44ba0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888760-wkrkz" Oct 30 02:00:00.393149 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.393040 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xmxt\" (UniqueName: \"kubernetes.io/projected/ad3ca000-d9f7-4d97-9a05-b39362550ba0-kube-api-access-6xmxt\") pod \"maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298\" (UID: \"ad3ca000-d9f7-4d97-9a05-b39362550ba0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298" Oct 30 02:00:00.394366 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.394339 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmvqr\" (UniqueName: \"kubernetes.io/projected/bc3321aa-698f-4c74-94ae-95e8729d9f27-kube-api-access-kmvqr\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-292018-29888760-j979j\" (UID: \"bc3321aa-698f-4c74-94ae-95e8729d9f27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888760-j979j" Oct 30 02:00:00.396655 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.396630 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfrph\" (UniqueName: \"kubernetes.io/projected/9dc37d0b-7989-4437-9902-df0a34eb682a-kube-api-access-zfrph\") pod \"maas-api-key-cleanup-e2e-extmod-a-2eae39-29888760-t6tz5\" (UID: \"9dc37d0b-7989-4437-9902-df0a34eb682a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2eae39-29888760-t6tz5" Oct 30 02:00:00.397273 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.397253 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888760-lnmg5" Oct 30 02:00:00.398005 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.397979 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888760-qczr8"] Oct 30 02:00:00.404472 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.404317 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888760-qczr8" Oct 30 02:00:00.404657 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.404224 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888760-5xgtv"] Oct 30 02:00:00.408503 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.408476 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jft2l\" (UniqueName: \"kubernetes.io/projected/ea84dc40-9569-4a87-a537-a8e0c0fb119d-kube-api-access-jft2l\") pod \"maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888760-hvkkt\" (UID: \"ea84dc40-9569-4a87-a537-a8e0c0fb119d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888760-hvkkt" Oct 30 02:00:00.414028 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.413922 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888760-m68hl"] Oct 30 02:00:00.421260 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.421177 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888760-xhtnj"] Oct 30 02:00:00.429275 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.429246 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888760-kwp4k"] Oct 30 02:00:00.440220 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.440190 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888760-5h4tv"] Oct 30 02:00:00.452176 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.452102 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/228bfbbd-21b3-4f89-b247-962811a659e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888760-m68hl\" (UID: \"228bfbbd-21b3-4f89-b247-962811a659e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888760-m68hl" Oct 30 02:00:00.452373 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.452217 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkxx9\" (UniqueName: \"kubernetes.io/projected/dff6267c-a0a8-4d35-badf-3032bfb3e979-kube-api-access-tkxx9\") pod \"maas-api-key-cleanup-e2e-extmod-a-d44413-29888760-z8976\" (UID: \"dff6267c-a0a8-4d35-badf-3032bfb3e979\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888760-z8976" Oct 30 02:00:00.452373 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.452255 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45e5935e-478f-4fec-beae-0cae2ea14c1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-76a19e-29888760-xhtnj\" (UID: \"45e5935e-478f-4fec-beae-0cae2ea14c1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888760-xhtnj" Oct 30 02:00:00.452502 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.452394 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2956c338-3b66-48aa-b3cc-521dbdef04ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888760-lnmg5\" (UID: \"2956c338-3b66-48aa-b3cc-521dbdef04ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888760-lnmg5" Oct 30 02:00:00.452502 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.452449 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05cb266a-e1fe-4b51-a18f-96d85a78ab16-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888760-kwp4k\" (UID: \"05cb266a-e1fe-4b51-a18f-96d85a78ab16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888760-kwp4k" Oct 30 02:00:00.452620 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.452501 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dff6267c-a0a8-4d35-badf-3032bfb3e979-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d44413-29888760-z8976\" (UID: \"dff6267c-a0a8-4d35-badf-3032bfb3e979\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888760-z8976" Oct 30 02:00:00.452620 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.452559 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb5qv\" (UniqueName: \"kubernetes.io/projected/45e5935e-478f-4fec-beae-0cae2ea14c1b-kube-api-access-xb5qv\") pod \"maas-api-key-cleanup-e2e-extmod-a-76a19e-29888760-xhtnj\" (UID: \"45e5935e-478f-4fec-beae-0cae2ea14c1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888760-xhtnj" Oct 30 02:00:00.452620 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.452596 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72n44\" (UniqueName: \"kubernetes.io/projected/2956c338-3b66-48aa-b3cc-521dbdef04ce-kube-api-access-72n44\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888760-lnmg5\" (UID: \"2956c338-3b66-48aa-b3cc-521dbdef04ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888760-lnmg5" Oct 30 02:00:00.452834 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.452639 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f818d22e-27ff-4caa-a48d-352af6583de4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888760-qczr8\" (UID: \"f818d22e-27ff-4caa-a48d-352af6583de4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888760-qczr8" Oct 30 02:00:00.452834 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.452674 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gxdh\" (UniqueName: \"kubernetes.io/projected/f818d22e-27ff-4caa-a48d-352af6583de4-kube-api-access-7gxdh\") pod \"maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888760-qczr8\" (UID: \"f818d22e-27ff-4caa-a48d-352af6583de4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888760-qczr8" Oct 30 02:00:00.452834 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.452810 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8e5ee17-3f55-4488-aba3-2a81eceb15a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888760-5xgtv\" (UID: \"b8e5ee17-3f55-4488-aba3-2a81eceb15a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-5xgtv" Oct 30 02:00:00.453011 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.452905 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mj5k\" (UniqueName: \"kubernetes.io/projected/228bfbbd-21b3-4f89-b247-962811a659e6-kube-api-access-7mj5k\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888760-m68hl\" (UID: \"228bfbbd-21b3-4f89-b247-962811a659e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888760-m68hl" Oct 30 02:00:00.453011 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.452943 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/228bfbbd-21b3-4f89-b247-962811a659e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888760-m68hl\" (UID: \"228bfbbd-21b3-4f89-b247-962811a659e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888760-m68hl" Oct 30 02:00:00.453011 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.452966 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6txhh\" (UniqueName: \"kubernetes.io/projected/b8e5ee17-3f55-4488-aba3-2a81eceb15a4-kube-api-access-6txhh\") pod \"maas-api-key-cleanup-29888760-5xgtv\" (UID: \"b8e5ee17-3f55-4488-aba3-2a81eceb15a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-5xgtv" Oct 30 02:00:00.453778 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.453712 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45e5935e-478f-4fec-beae-0cae2ea14c1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-76a19e-29888760-xhtnj\" (UID: \"45e5935e-478f-4fec-beae-0cae2ea14c1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888760-xhtnj" Oct 30 02:00:00.453778 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.453712 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8e5ee17-3f55-4488-aba3-2a81eceb15a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888760-5xgtv\" (UID: \"b8e5ee17-3f55-4488-aba3-2a81eceb15a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-5xgtv" Oct 30 02:00:00.453984 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.453897 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05cb266a-e1fe-4b51-a18f-96d85a78ab16-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888760-kwp4k\" (UID: \"05cb266a-e1fe-4b51-a18f-96d85a78ab16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888760-kwp4k" Oct 30 02:00:00.453984 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.453901 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqkbf\" (UniqueName: \"kubernetes.io/projected/05cb266a-e1fe-4b51-a18f-96d85a78ab16-kube-api-access-jqkbf\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888760-kwp4k\" (UID: \"05cb266a-e1fe-4b51-a18f-96d85a78ab16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888760-kwp4k" Oct 30 02:00:00.455198 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.455180 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888760-z8976"] Oct 30 02:00:00.455326 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.455305 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888760-5h4tv" Oct 30 02:00:00.459336 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.459313 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888760-545xr"] Oct 30 02:00:00.464923 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.464896 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888760-lnmg5"] Oct 30 02:00:00.465071 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.465016 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888760-545xr" Oct 30 02:00:00.469726 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.469621 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888760-dfspx"] Oct 30 02:00:00.489848 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.488841 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888760-wkrkz" Oct 30 02:00:00.490300 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.490034 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb5qv\" (UniqueName: \"kubernetes.io/projected/45e5935e-478f-4fec-beae-0cae2ea14c1b-kube-api-access-xb5qv\") pod \"maas-api-key-cleanup-e2e-extmod-a-76a19e-29888760-xhtnj\" (UID: \"45e5935e-478f-4fec-beae-0cae2ea14c1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888760-xhtnj" Oct 30 02:00:00.491541 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.491509 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888760-k4ntx"] Oct 30 02:00:00.492804 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.492170 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888760-dfspx" Oct 30 02:00:00.493788 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.493723 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mj5k\" (UniqueName: \"kubernetes.io/projected/228bfbbd-21b3-4f89-b247-962811a659e6-kube-api-access-7mj5k\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888760-m68hl\" (UID: \"228bfbbd-21b3-4f89-b247-962811a659e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888760-m68hl" Oct 30 02:00:00.495055 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.495033 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6txhh\" (UniqueName: \"kubernetes.io/projected/b8e5ee17-3f55-4488-aba3-2a81eceb15a4-kube-api-access-6txhh\") pod \"maas-api-key-cleanup-29888760-5xgtv\" (UID: \"b8e5ee17-3f55-4488-aba3-2a81eceb15a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-5xgtv" Oct 30 02:00:00.496988 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.496964 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqkbf\" (UniqueName: \"kubernetes.io/projected/05cb266a-e1fe-4b51-a18f-96d85a78ab16-kube-api-access-jqkbf\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888760-kwp4k\" (UID: \"05cb266a-e1fe-4b51-a18f-96d85a78ab16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888760-kwp4k" Oct 30 02:00:00.498268 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.498242 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888760-5h4tv"] Oct 30 02:00:00.498360 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.498292 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888760-dfspx"] Oct 30 02:00:00.498360 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.498341 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888760-k4ntx" Oct 30 02:00:00.500456 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.500437 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888760-btnnt"] Oct 30 02:00:00.505733 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.505710 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888760-qczr8"] Oct 30 02:00:00.505864 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.505828 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888760-btnnt" Oct 30 02:00:00.510350 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.510322 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888760-k4ntx"] Oct 30 02:00:00.511391 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.511354 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298" Oct 30 02:00:00.515570 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.515546 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888760-545xr"] Oct 30 02:00:00.521168 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.521124 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888760-btnnt"] Oct 30 02:00:00.530517 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.530306 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2eae39-29888760-t6tz5" Oct 30 02:00:00.561270 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.558767 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc58a71f-7f5f-4f68-aa97-a22aa15b854e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888760-btnnt\" (UID: \"dc58a71f-7f5f-4f68-aa97-a22aa15b854e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888760-btnnt" Oct 30 02:00:00.561270 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.558847 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-558327-29888760-k4ntx\" (UID: \"ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888760-k4ntx" Oct 30 02:00:00.561270 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.558894 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkxx9\" (UniqueName: \"kubernetes.io/projected/dff6267c-a0a8-4d35-badf-3032bfb3e979-kube-api-access-tkxx9\") pod \"maas-api-key-cleanup-e2e-extmod-a-d44413-29888760-z8976\" (UID: \"dff6267c-a0a8-4d35-badf-3032bfb3e979\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888760-z8976" Oct 30 02:00:00.561270 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.558927 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16ce2d50-2777-4827-80ec-025ff18072ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888760-545xr\" (UID: \"16ce2d50-2777-4827-80ec-025ff18072ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888760-545xr" Oct 30 02:00:00.561270 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.558962 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m8sn\" (UniqueName: \"kubernetes.io/projected/419770d0-51af-4843-9ab2-8cbbe8215357-kube-api-access-6m8sn\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888760-5h4tv\" (UID: \"419770d0-51af-4843-9ab2-8cbbe8215357\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888760-5h4tv" Oct 30 02:00:00.561270 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.559040 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2956c338-3b66-48aa-b3cc-521dbdef04ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888760-lnmg5\" (UID: \"2956c338-3b66-48aa-b3cc-521dbdef04ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888760-lnmg5" Oct 30 02:00:00.561270 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.559071 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/419770d0-51af-4843-9ab2-8cbbe8215357-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888760-5h4tv\" (UID: \"419770d0-51af-4843-9ab2-8cbbe8215357\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888760-5h4tv" Oct 30 02:00:00.561270 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.559099 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tftl5\" (UniqueName: \"kubernetes.io/projected/dc58a71f-7f5f-4f68-aa97-a22aa15b854e-kube-api-access-tftl5\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888760-btnnt\" (UID: \"dc58a71f-7f5f-4f68-aa97-a22aa15b854e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888760-btnnt" Oct 30 02:00:00.561270 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.559155 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw5bp\" (UniqueName: \"kubernetes.io/projected/ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c-kube-api-access-zw5bp\") pod \"maas-api-key-cleanup-e2e-worker-w3-558327-29888760-k4ntx\" (UID: \"ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888760-k4ntx" Oct 30 02:00:00.561270 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.559194 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbd7b9d4-eb19-4605-bdc9-21940dad8c92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888760-dfspx\" (UID: \"dbd7b9d4-eb19-4605-bdc9-21940dad8c92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888760-dfspx" Oct 30 02:00:00.561270 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.559227 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dff6267c-a0a8-4d35-badf-3032bfb3e979-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d44413-29888760-z8976\" (UID: \"dff6267c-a0a8-4d35-badf-3032bfb3e979\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888760-z8976" Oct 30 02:00:00.561270 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.559286 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd9rh\" (UniqueName: \"kubernetes.io/projected/dbd7b9d4-eb19-4605-bdc9-21940dad8c92-kube-api-access-qd9rh\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888760-dfspx\" (UID: \"dbd7b9d4-eb19-4605-bdc9-21940dad8c92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888760-dfspx" Oct 30 02:00:00.561270 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.559326 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72n44\" (UniqueName: \"kubernetes.io/projected/2956c338-3b66-48aa-b3cc-521dbdef04ce-kube-api-access-72n44\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888760-lnmg5\" (UID: \"2956c338-3b66-48aa-b3cc-521dbdef04ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888760-lnmg5" Oct 30 02:00:00.561270 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.559394 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f818d22e-27ff-4caa-a48d-352af6583de4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888760-qczr8\" (UID: \"f818d22e-27ff-4caa-a48d-352af6583de4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888760-qczr8" Oct 30 02:00:00.561270 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.559435 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gxdh\" (UniqueName: \"kubernetes.io/projected/f818d22e-27ff-4caa-a48d-352af6583de4-kube-api-access-7gxdh\") pod \"maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888760-qczr8\" (UID: \"f818d22e-27ff-4caa-a48d-352af6583de4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888760-qczr8" Oct 30 02:00:00.561270 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.559533 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbkvl\" (UniqueName: \"kubernetes.io/projected/16ce2d50-2777-4827-80ec-025ff18072ed-kube-api-access-cbkvl\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888760-545xr\" (UID: \"16ce2d50-2777-4827-80ec-025ff18072ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888760-545xr" Oct 30 02:00:00.561270 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.560735 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dff6267c-a0a8-4d35-badf-3032bfb3e979-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d44413-29888760-z8976\" (UID: \"dff6267c-a0a8-4d35-badf-3032bfb3e979\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888760-z8976" Oct 30 02:00:00.561270 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.561035 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2956c338-3b66-48aa-b3cc-521dbdef04ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888760-lnmg5\" (UID: \"2956c338-3b66-48aa-b3cc-521dbdef04ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888760-lnmg5" Oct 30 02:00:00.562459 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.561663 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f818d22e-27ff-4caa-a48d-352af6583de4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888760-qczr8\" (UID: \"f818d22e-27ff-4caa-a48d-352af6583de4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888760-qczr8" Oct 30 02:00:00.581363 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.580815 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72n44\" (UniqueName: \"kubernetes.io/projected/2956c338-3b66-48aa-b3cc-521dbdef04ce-kube-api-access-72n44\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888760-lnmg5\" (UID: \"2956c338-3b66-48aa-b3cc-521dbdef04ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888760-lnmg5" Oct 30 02:00:00.583995 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.582474 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gxdh\" (UniqueName: \"kubernetes.io/projected/f818d22e-27ff-4caa-a48d-352af6583de4-kube-api-access-7gxdh\") pod \"maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888760-qczr8\" (UID: \"f818d22e-27ff-4caa-a48d-352af6583de4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888760-qczr8" Oct 30 02:00:00.589504 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.589476 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888760-6h8kw" Oct 30 02:00:00.604576 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.604328 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888760-j979j" Oct 30 02:00:00.607308 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.606908 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bc62b8-zrngw" Oct 30 02:00:00.607831 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.607519 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38bc62b8-zrngw" event={"ID":"31878162-1468-4639-826b-a725564a1d41","Type":"ContainerDied","Data":"ba4b43927d294ecab9eb4cd2a2ea9cd74fcd6875bb160502060d7ea9215eed28"} Oct 30 02:00:00.607831 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.607582 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba4b43927d294ecab9eb4cd2a2ea9cd74fcd6875bb160502060d7ea9215eed28" Oct 30 02:00:00.634006 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.631412 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888760-hvkkt" Oct 30 02:00:00.643285 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.643212 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkxx9\" (UniqueName: \"kubernetes.io/projected/dff6267c-a0a8-4d35-badf-3032bfb3e979-kube-api-access-tkxx9\") pod \"maas-api-key-cleanup-e2e-extmod-a-d44413-29888760-z8976\" (UID: \"dff6267c-a0a8-4d35-badf-3032bfb3e979\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888760-z8976" Oct 30 02:00:00.645873 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.645603 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888760-xhtnj" Oct 30 02:00:00.662866 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.661783 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbkvl\" (UniqueName: \"kubernetes.io/projected/16ce2d50-2777-4827-80ec-025ff18072ed-kube-api-access-cbkvl\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888760-545xr\" (UID: \"16ce2d50-2777-4827-80ec-025ff18072ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888760-545xr" Oct 30 02:00:00.662866 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.662333 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc58a71f-7f5f-4f68-aa97-a22aa15b854e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888760-btnnt\" (UID: \"dc58a71f-7f5f-4f68-aa97-a22aa15b854e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888760-btnnt" Oct 30 02:00:00.662866 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.662406 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-558327-29888760-k4ntx\" (UID: \"ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888760-k4ntx" Oct 30 02:00:00.662866 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.662455 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16ce2d50-2777-4827-80ec-025ff18072ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888760-545xr\" (UID: \"16ce2d50-2777-4827-80ec-025ff18072ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888760-545xr" Oct 30 02:00:00.662866 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.662487 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m8sn\" (UniqueName: \"kubernetes.io/projected/419770d0-51af-4843-9ab2-8cbbe8215357-kube-api-access-6m8sn\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888760-5h4tv\" (UID: \"419770d0-51af-4843-9ab2-8cbbe8215357\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888760-5h4tv" Oct 30 02:00:00.662866 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.662585 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/419770d0-51af-4843-9ab2-8cbbe8215357-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888760-5h4tv\" (UID: \"419770d0-51af-4843-9ab2-8cbbe8215357\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888760-5h4tv" Oct 30 02:00:00.662866 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.662616 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tftl5\" (UniqueName: \"kubernetes.io/projected/dc58a71f-7f5f-4f68-aa97-a22aa15b854e-kube-api-access-tftl5\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888760-btnnt\" (UID: \"dc58a71f-7f5f-4f68-aa97-a22aa15b854e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888760-btnnt" Oct 30 02:00:00.662866 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.662716 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw5bp\" (UniqueName: \"kubernetes.io/projected/ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c-kube-api-access-zw5bp\") pod \"maas-api-key-cleanup-e2e-worker-w3-558327-29888760-k4ntx\" (UID: \"ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888760-k4ntx" Oct 30 02:00:00.662866 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.662788 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbd7b9d4-eb19-4605-bdc9-21940dad8c92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888760-dfspx\" (UID: \"dbd7b9d4-eb19-4605-bdc9-21940dad8c92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888760-dfspx" Oct 30 02:00:00.664535 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.663557 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd9rh\" (UniqueName: \"kubernetes.io/projected/dbd7b9d4-eb19-4605-bdc9-21940dad8c92-kube-api-access-qd9rh\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888760-dfspx\" (UID: \"dbd7b9d4-eb19-4605-bdc9-21940dad8c92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888760-dfspx" Oct 30 02:00:00.665356 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.665327 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc58a71f-7f5f-4f68-aa97-a22aa15b854e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888760-btnnt\" (UID: \"dc58a71f-7f5f-4f68-aa97-a22aa15b854e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888760-btnnt" Oct 30 02:00:00.665599 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.665575 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/419770d0-51af-4843-9ab2-8cbbe8215357-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888760-5h4tv\" (UID: \"419770d0-51af-4843-9ab2-8cbbe8215357\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888760-5h4tv" Oct 30 02:00:00.666423 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.666399 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-558327-29888760-k4ntx\" (UID: \"ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888760-k4ntx" Oct 30 02:00:00.666849 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.666820 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbd7b9d4-eb19-4605-bdc9-21940dad8c92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888760-dfspx\" (UID: \"dbd7b9d4-eb19-4605-bdc9-21940dad8c92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888760-dfspx" Oct 30 02:00:00.666849 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.666845 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16ce2d50-2777-4827-80ec-025ff18072ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888760-545xr\" (UID: \"16ce2d50-2777-4827-80ec-025ff18072ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888760-545xr" Oct 30 02:00:00.672336 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.671250 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888760-m68hl" Oct 30 02:00:00.677988 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.677784 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-5xgtv" Oct 30 02:00:00.685766 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.685694 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd9rh\" (UniqueName: \"kubernetes.io/projected/dbd7b9d4-eb19-4605-bdc9-21940dad8c92-kube-api-access-qd9rh\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888760-dfspx\" (UID: \"dbd7b9d4-eb19-4605-bdc9-21940dad8c92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888760-dfspx" Oct 30 02:00:00.694512 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.694339 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888760-kwp4k" Oct 30 02:00:00.695473 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.695420 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m8sn\" (UniqueName: \"kubernetes.io/projected/419770d0-51af-4843-9ab2-8cbbe8215357-kube-api-access-6m8sn\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888760-5h4tv\" (UID: \"419770d0-51af-4843-9ab2-8cbbe8215357\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888760-5h4tv" Oct 30 02:00:00.695473 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.695451 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbkvl\" (UniqueName: \"kubernetes.io/projected/16ce2d50-2777-4827-80ec-025ff18072ed-kube-api-access-cbkvl\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888760-545xr\" (UID: \"16ce2d50-2777-4827-80ec-025ff18072ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888760-545xr" Oct 30 02:00:00.697303 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.696503 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tftl5\" (UniqueName: \"kubernetes.io/projected/dc58a71f-7f5f-4f68-aa97-a22aa15b854e-kube-api-access-tftl5\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888760-btnnt\" (UID: \"dc58a71f-7f5f-4f68-aa97-a22aa15b854e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888760-btnnt" Oct 30 02:00:00.698431 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.698153 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw5bp\" (UniqueName: \"kubernetes.io/projected/ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c-kube-api-access-zw5bp\") pod \"maas-api-key-cleanup-e2e-worker-w3-558327-29888760-k4ntx\" (UID: \"ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888760-k4ntx" Oct 30 02:00:00.775641 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.775598 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:00.783605 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.783213 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888760-5h4tv" Oct 30 02:00:00.790397 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.786125 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888760-z8976" Oct 30 02:00:00.790397 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.786489 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888760-qczr8" Oct 30 02:00:00.790397 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.786615 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888760-lnmg5" Oct 30 02:00:00.797118 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.797051 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888760-545xr" Oct 30 02:00:00.838051 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.834602 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888760-dfspx" Oct 30 02:00:00.838051 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.836909 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888760-btnnt" Oct 30 02:00:00.838051 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:00.837002 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888760-k4ntx" Oct 30 02:00:01.027547 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:01.027512 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888760-wkrkz"] Oct 30 02:00:01.113347 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:00:01.113294 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32354403_56a7_4f01_a228_122805b44ba0.slice/crio-224f90d1755c20fbf5f1580d3975caffe0bba7adc0f4518dc85c0f3526033d4b WatchSource:0}: Error finding container 224f90d1755c20fbf5f1580d3975caffe0bba7adc0f4518dc85c0f3526033d4b: Status 404 returned error can't find the container with id 224f90d1755c20fbf5f1580d3975caffe0bba7adc0f4518dc85c0f3526033d4b Oct 30 02:00:01.367388 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:01.367360 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr" Oct 30 02:00:01.501237 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:01.501203 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298"] Oct 30 02:00:01.510873 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:00:01.510846 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad3ca000_d9f7_4d97_9a05_b39362550ba0.slice/crio-8b716c75d607574fef235f1a2604d0bc12726058674dc8dc26dc4930cb60c831 WatchSource:0}: Error finding container 8b716c75d607574fef235f1a2604d0bc12726058674dc8dc26dc4930cb60c831: Status 404 returned error can't find the container with id 8b716c75d607574fef235f1a2604d0bc12726058674dc8dc26dc4930cb60c831 Oct 30 02:00:01.553415 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:01.553275 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888760-hvkkt"] Oct 30 02:00:01.558071 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:01.558047 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2eae39-29888760-t6tz5"] Oct 30 02:00:01.574680 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:01.574656 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888760-6h8kw"] Oct 30 02:00:01.579539 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:01.579515 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888760-xhtnj"] Oct 30 02:00:01.599234 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:01.599210 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888760-m68hl"] Oct 30 02:00:01.617349 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:00:01.617323 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea84dc40_9569_4a87_a537_a8e0c0fb119d.slice/crio-ec5c831b4661f2c60e6ce44377ac1de0e1ac8666f09d4a48daa332cc5a2529ac WatchSource:0}: Error finding container ec5c831b4661f2c60e6ce44377ac1de0e1ac8666f09d4a48daa332cc5a2529ac: Status 404 returned error can't find the container with id ec5c831b4661f2c60e6ce44377ac1de0e1ac8666f09d4a48daa332cc5a2529ac Oct 30 02:00:01.618239 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:01.618154 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888760-wkrkz" event={"ID":"32354403-56a7-4f01-a228-122805b44ba0","Type":"ContainerStarted","Data":"224f90d1755c20fbf5f1580d3975caffe0bba7adc0f4518dc85c0f3526033d4b"} Oct 30 02:00:01.622337 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:01.622248 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298" event={"ID":"ad3ca000-d9f7-4d97-9a05-b39362550ba0","Type":"ContainerStarted","Data":"8b716c75d607574fef235f1a2604d0bc12726058674dc8dc26dc4930cb60c831"} Oct 30 02:00:01.649763 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:01.649723 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888760-j979j"] Oct 30 02:00:01.749259 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:01.748594 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888760-qczr8"] Oct 30 02:00:01.751887 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:00:01.751814 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf818d22e_27ff_4caa_a48d_352af6583de4.slice/crio-6b42e2e0100c1f49070c6b68b80c12949aa33533281852116058370f33e490d5 WatchSource:0}: Error finding container 6b42e2e0100c1f49070c6b68b80c12949aa33533281852116058370f33e490d5: Status 404 returned error can't find the container with id 6b42e2e0100c1f49070c6b68b80c12949aa33533281852116058370f33e490d5 Oct 30 02:00:01.754781 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:00:01.754735 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded66d3d7_b7a2_4dbf_8ea6_5144e1cc879c.slice/crio-a2665ee7958ead93588c8fb2e74904526df5b41ee7806de5fe0f9d4f7bf67750 WatchSource:0}: Error finding container a2665ee7958ead93588c8fb2e74904526df5b41ee7806de5fe0f9d4f7bf67750: Status 404 returned error can't find the container with id a2665ee7958ead93588c8fb2e74904526df5b41ee7806de5fe0f9d4f7bf67750 Oct 30 02:00:01.755696 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:01.755672 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888760-lnmg5"] Oct 30 02:00:01.761945 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:01.761920 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888760-k4ntx"] Oct 30 02:00:01.766318 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:00:01.766278 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2956c338_3b66_48aa_b3cc_521dbdef04ce.slice/crio-969b4bfaf990edc19893d6ece9ab06ab08d440c270f2b34d2bf2a1502bc9842b WatchSource:0}: Error finding container 969b4bfaf990edc19893d6ece9ab06ab08d440c270f2b34d2bf2a1502bc9842b: Status 404 returned error can't find the container with id 969b4bfaf990edc19893d6ece9ab06ab08d440c270f2b34d2bf2a1502bc9842b Oct 30 02:00:01.771821 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:01.771791 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888760-5h4tv"] Oct 30 02:00:01.779773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:01.779452 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888760-545xr"] Oct 30 02:00:01.785471 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:01.785448 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888760-dfspx"] Oct 30 02:00:01.791869 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:00:01.791781 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbd7b9d4_eb19_4605_bdc9_21940dad8c92.slice/crio-babc29cf7fa79f9d2afe4f2c40bde16f4cef974cbcee257b78f7b35857c2942c WatchSource:0}: Error finding container babc29cf7fa79f9d2afe4f2c40bde16f4cef974cbcee257b78f7b35857c2942c: Status 404 returned error can't find the container with id babc29cf7fa79f9d2afe4f2c40bde16f4cef974cbcee257b78f7b35857c2942c Oct 30 02:00:01.876306 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:01.876279 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888760-z8976"] Oct 30 02:00:01.898618 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:01.898595 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888760-5xgtv"] Oct 30 02:00:01.903047 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:01.903019 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888760-btnnt"] Oct 30 02:00:01.914264 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:01.914224 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888760-kwp4k"] Oct 30 02:00:01.916395 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:00:01.916354 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddff6267c_a0a8_4d35_badf_3032bfb3e979.slice/crio-fc4b9409cbac951e81f66e021b0ea63eb985800948b48a82f5e8feb5b6f70413 WatchSource:0}: Error finding container fc4b9409cbac951e81f66e021b0ea63eb985800948b48a82f5e8feb5b6f70413: Status 404 returned error can't find the container with id fc4b9409cbac951e81f66e021b0ea63eb985800948b48a82f5e8feb5b6f70413 Oct 30 02:00:02.010212 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:00:02.010128 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc58a71f_7f5f_4f68_aa97_a22aa15b854e.slice/crio-1e120c094ddf77870464de59f7029d72e1e7626451526bc1fa3ed904459eb6ca WatchSource:0}: Error finding container 1e120c094ddf77870464de59f7029d72e1e7626451526bc1fa3ed904459eb6ca: Status 404 returned error can't find the container with id 1e120c094ddf77870464de59f7029d72e1e7626451526bc1fa3ed904459eb6ca Oct 30 02:00:02.012540 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:00:02.012315 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8e5ee17_3f55_4488_aba3_2a81eceb15a4.slice/crio-e63efb694621c6844de4606ed90a3e9723514ba6df6f1763af7707f8af5d6c6b WatchSource:0}: Error finding container e63efb694621c6844de4606ed90a3e9723514ba6df6f1763af7707f8af5d6c6b: Status 404 returned error can't find the container with id e63efb694621c6844de4606ed90a3e9723514ba6df6f1763af7707f8af5d6c6b Oct 30 02:00:02.014909 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:00:02.014864 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05cb266a_e1fe_4b51_a18f_96d85a78ab16.slice/crio-4e6467d19876d2700ab0b46f04bf4e83599bb6537b25613d3514bc72c9ad4be5 WatchSource:0}: Error finding container 4e6467d19876d2700ab0b46f04bf4e83599bb6537b25613d3514bc72c9ad4be5: Status 404 returned error can't find the container with id 4e6467d19876d2700ab0b46f04bf4e83599bb6537b25613d3514bc72c9ad4be5 Oct 30 02:00:02.376037 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:02.376002 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m" Oct 30 02:00:02.446003 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:02.445971 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr"] Oct 30 02:00:02.494680 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:02.494629 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:02.641540 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:02.641512 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888760-k4ntx" event={"ID":"ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c","Type":"ContainerStarted","Data":"a2665ee7958ead93588c8fb2e74904526df5b41ee7806de5fe0f9d4f7bf67750"} Oct 30 02:00:02.645126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:02.645091 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888760-5h4tv" event={"ID":"419770d0-51af-4843-9ab2-8cbbe8215357","Type":"ContainerStarted","Data":"8fec76af3acc2551971fd9ca7e144653748ff1ed9bdb9ab3f330d2917ab6566a"} Oct 30 02:00:02.647999 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:02.647971 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888760-m68hl" event={"ID":"228bfbbd-21b3-4f89-b247-962811a659e6","Type":"ContainerStarted","Data":"fabaa8c1ca3dcb84ed16689e9a9e5fa1451e7fb1fdb08f5f46595ae9e7ec6061"} Oct 30 02:00:02.648161 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:02.648024 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888760-m68hl" event={"ID":"228bfbbd-21b3-4f89-b247-962811a659e6","Type":"ContainerStarted","Data":"59a9b25a5a6067af42f28c3aaa6014335eec3037051baf002afbd08134b5b19d"} Oct 30 02:00:02.650017 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:02.649651 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-5xgtv" event={"ID":"b8e5ee17-3f55-4488-aba3-2a81eceb15a4","Type":"ContainerStarted","Data":"e63efb694621c6844de4606ed90a3e9723514ba6df6f1763af7707f8af5d6c6b"} Oct 30 02:00:02.651019 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:02.650997 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888760-z8976" event={"ID":"dff6267c-a0a8-4d35-badf-3032bfb3e979","Type":"ContainerStarted","Data":"fc4b9409cbac951e81f66e021b0ea63eb985800948b48a82f5e8feb5b6f70413"} Oct 30 02:00:02.652260 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:02.652234 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888760-btnnt" event={"ID":"dc58a71f-7f5f-4f68-aa97-a22aa15b854e","Type":"ContainerStarted","Data":"1e120c094ddf77870464de59f7029d72e1e7626451526bc1fa3ed904459eb6ca"} Oct 30 02:00:02.653861 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:02.653838 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888760-6h8kw" event={"ID":"d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b","Type":"ContainerStarted","Data":"0628039ef6f3839f190386fe610deea13cd7c7d0786c6f982286a464699fd13d"} Oct 30 02:00:02.655255 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:02.655236 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2eae39-29888760-t6tz5" event={"ID":"9dc37d0b-7989-4437-9902-df0a34eb682a","Type":"ContainerStarted","Data":"5a2df92cf71ae8c940ae45055c823287af02c86acf1d9ffed22a65a37573e6c2"} Oct 30 02:00:02.656492 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:02.656472 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888760-lnmg5" event={"ID":"2956c338-3b66-48aa-b3cc-521dbdef04ce","Type":"ContainerStarted","Data":"969b4bfaf990edc19893d6ece9ab06ab08d440c270f2b34d2bf2a1502bc9842b"} Oct 30 02:00:02.657647 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:02.657627 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888760-545xr" event={"ID":"16ce2d50-2777-4827-80ec-025ff18072ed","Type":"ContainerStarted","Data":"f4da7ab21469b510e241e70cdc19c7b5f0c31f5fdd405937ba72a9f3a77bd561"} Oct 30 02:00:02.658787 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:02.658767 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888760-hvkkt" event={"ID":"ea84dc40-9569-4a87-a537-a8e0c0fb119d","Type":"ContainerStarted","Data":"ec5c831b4661f2c60e6ce44377ac1de0e1ac8666f09d4a48daa332cc5a2529ac"} Oct 30 02:00:02.660266 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:02.660242 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298" event={"ID":"ad3ca000-d9f7-4d97-9a05-b39362550ba0","Type":"ContainerStarted","Data":"29063cda7e8d3f7adc7d286e10e5bb7154d4d1cdda72c515d40a66007e0cf460"} Oct 30 02:00:02.661601 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:02.661578 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888760-xhtnj" event={"ID":"45e5935e-478f-4fec-beae-0cae2ea14c1b","Type":"ContainerStarted","Data":"fee24ed70ae7528e112201ccb54a72293fa1ba817cfe0acc6a5c9ceca52b4e11"} Oct 30 02:00:02.666798 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:02.663881 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888760-dfspx" event={"ID":"dbd7b9d4-eb19-4605-bdc9-21940dad8c92","Type":"ContainerStarted","Data":"babc29cf7fa79f9d2afe4f2c40bde16f4cef974cbcee257b78f7b35857c2942c"} Oct 30 02:00:02.666798 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:02.666697 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888760-wkrkz" event={"ID":"32354403-56a7-4f01-a228-122805b44ba0","Type":"ContainerStarted","Data":"a1623006f06a65d603906d4b5f6776ec33775daabe65d570515ff008e4c91b3d"} Oct 30 02:00:02.677165 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:02.677134 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888760-qczr8" event={"ID":"f818d22e-27ff-4caa-a48d-352af6583de4","Type":"ContainerStarted","Data":"6b42e2e0100c1f49070c6b68b80c12949aa33533281852116058370f33e490d5"} Oct 30 02:00:02.678367 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:02.678341 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888760-j979j" event={"ID":"bc3321aa-698f-4c74-94ae-95e8729d9f27","Type":"ContainerStarted","Data":"32823aabb63ee1ba008169b6f2af6dc2ce2f61d6c9c07fe75d719a2115befd89"} Oct 30 02:00:02.679531 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:02.679511 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888760-kwp4k" event={"ID":"05cb266a-e1fe-4b51-a18f-96d85a78ab16","Type":"ContainerStarted","Data":"4e6467d19876d2700ab0b46f04bf4e83599bb6537b25613d3514bc72c9ad4be5"} Oct 30 02:00:02.689976 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:02.689924 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888760-m68hl" podStartSLOduration=2.689893591 podStartE2EDuration="2.689893591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:02.686011851 +0000 UTC m=+2528.262437252" watchObservedRunningTime="2026-10-30 02:00:02.689893591 +0000 UTC m=+2528.266318985" Oct 30 02:00:02.728539 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:02.728488 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888760-wkrkz" podStartSLOduration=2.728473751 podStartE2EDuration="2.728473751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:02.72500851 +0000 UTC m=+2528.301433910" watchObservedRunningTime="2026-10-30 02:00:02.728473751 +0000 UTC m=+2528.304899153" Oct 30 02:00:02.762437 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:02.762386 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298" podStartSLOduration=2.762370815 podStartE2EDuration="2.762370815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:02.760813533 +0000 UTC m=+2528.337238933" watchObservedRunningTime="2026-10-30 02:00:02.762370815 +0000 UTC m=+2528.338796236" Oct 30 02:00:02.803876 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:02.803707 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888760-6h8kw" podStartSLOduration=2.803692388 podStartE2EDuration="2.803692388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:02.798278143 +0000 UTC m=+2528.374703582" watchObservedRunningTime="2026-10-30 02:00:02.803692388 +0000 UTC m=+2528.380117789" Oct 30 02:00:03.691803 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.691774 6572 generic.go:356] "Generic (PLEG): container finished" podID="32354403-56a7-4f01-a228-122805b44ba0" containerID="a1623006f06a65d603906d4b5f6776ec33775daabe65d570515ff008e4c91b3d" exitCode=60 Oct 30 02:00:03.692204 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.691862 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888760-wkrkz" event={"ID":"32354403-56a7-4f01-a228-122805b44ba0","Type":"ContainerDied","Data":"a1623006f06a65d603906d4b5f6776ec33775daabe65d570515ff008e4c91b3d"} Oct 30 02:00:03.692204 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.692152 6572 scope.go:122] "RemoveContainer" containerID="a1623006f06a65d603906d4b5f6776ec33775daabe65d570515ff008e4c91b3d" Oct 30 02:00:03.693786 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.693731 6572 generic.go:356] "Generic (PLEG): container finished" podID="f818d22e-27ff-4caa-a48d-352af6583de4" containerID="d5d8e837e66047efee29435b6ace98017ab04806517c8bdddfe85be52706e9b2" exitCode=60 Oct 30 02:00:03.693786 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.693772 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888760-qczr8" event={"ID":"f818d22e-27ff-4caa-a48d-352af6583de4","Type":"ContainerDied","Data":"d5d8e837e66047efee29435b6ace98017ab04806517c8bdddfe85be52706e9b2"} Oct 30 02:00:03.694166 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.694076 6572 scope.go:122] "RemoveContainer" containerID="d5d8e837e66047efee29435b6ace98017ab04806517c8bdddfe85be52706e9b2" Oct 30 02:00:03.696109 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.696083 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888760-j979j" event={"ID":"bc3321aa-698f-4c74-94ae-95e8729d9f27","Type":"ContainerStarted","Data":"ff2c3c18b1280eb0ff670b77fcc11a3fab7224df7a215ba385acb070cfa55e58"} Oct 30 02:00:03.697994 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.697974 6572 generic.go:356] "Generic (PLEG): container finished" podID="05cb266a-e1fe-4b51-a18f-96d85a78ab16" containerID="7fd65067a43596d9b5f8c44a53b7dbabbd0e27b790ba511d997b50f6d9ef983b" exitCode=60 Oct 30 02:00:03.698074 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.698031 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888760-kwp4k" event={"ID":"05cb266a-e1fe-4b51-a18f-96d85a78ab16","Type":"ContainerDied","Data":"7fd65067a43596d9b5f8c44a53b7dbabbd0e27b790ba511d997b50f6d9ef983b"} Oct 30 02:00:03.698268 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.698253 6572 scope.go:122] "RemoveContainer" containerID="7fd65067a43596d9b5f8c44a53b7dbabbd0e27b790ba511d997b50f6d9ef983b" Oct 30 02:00:03.699856 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.699836 6572 generic.go:356] "Generic (PLEG): container finished" podID="ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c" containerID="93e2719df595b90689297fa962bfd0f0474dbd911c7dba7b287f2b29e46741a7" exitCode=60 Oct 30 02:00:03.699950 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.699859 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888760-k4ntx" event={"ID":"ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c","Type":"ContainerDied","Data":"93e2719df595b90689297fa962bfd0f0474dbd911c7dba7b287f2b29e46741a7"} Oct 30 02:00:03.700124 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.700110 6572 scope.go:122] "RemoveContainer" containerID="93e2719df595b90689297fa962bfd0f0474dbd911c7dba7b287f2b29e46741a7" Oct 30 02:00:03.701647 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.701608 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888760-5h4tv" event={"ID":"419770d0-51af-4843-9ab2-8cbbe8215357","Type":"ContainerStarted","Data":"e2d82f5b85e7d6cdd8c3ce587fa51ee41a893324f829acbf991fa9ff1a7e0a2d"} Oct 30 02:00:03.703235 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.703207 6572 generic.go:356] "Generic (PLEG): container finished" podID="228bfbbd-21b3-4f89-b247-962811a659e6" containerID="fabaa8c1ca3dcb84ed16689e9a9e5fa1451e7fb1fdb08f5f46595ae9e7ec6061" exitCode=60 Oct 30 02:00:03.703315 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.703278 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888760-m68hl" event={"ID":"228bfbbd-21b3-4f89-b247-962811a659e6","Type":"ContainerDied","Data":"fabaa8c1ca3dcb84ed16689e9a9e5fa1451e7fb1fdb08f5f46595ae9e7ec6061"} Oct 30 02:00:03.703453 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.703440 6572 scope.go:122] "RemoveContainer" containerID="fabaa8c1ca3dcb84ed16689e9a9e5fa1451e7fb1fdb08f5f46595ae9e7ec6061" Oct 30 02:00:03.705015 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.704994 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-5xgtv" event={"ID":"b8e5ee17-3f55-4488-aba3-2a81eceb15a4","Type":"ContainerStarted","Data":"df3376e7086da732df49ad98df67c5ad6b17e92ed1820443b22f24b017bad29c"} Oct 30 02:00:03.709050 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.709024 6572 generic.go:356] "Generic (PLEG): container finished" podID="dff6267c-a0a8-4d35-badf-3032bfb3e979" containerID="ca586e9c26d2d05c0c37d6d0eec03aaf62c3bcf49e43679307f3d6051bfbd76a" exitCode=60 Oct 30 02:00:03.709139 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.709129 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888760-z8976" event={"ID":"dff6267c-a0a8-4d35-badf-3032bfb3e979","Type":"ContainerDied","Data":"ca586e9c26d2d05c0c37d6d0eec03aaf62c3bcf49e43679307f3d6051bfbd76a"} Oct 30 02:00:03.709304 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.709285 6572 scope.go:122] "RemoveContainer" containerID="ca586e9c26d2d05c0c37d6d0eec03aaf62c3bcf49e43679307f3d6051bfbd76a" Oct 30 02:00:03.711600 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.711565 6572 generic.go:356] "Generic (PLEG): container finished" podID="dc58a71f-7f5f-4f68-aa97-a22aa15b854e" containerID="3b95e491fd59d13a45036afeb0e73ce3648df7a3468a8059e5adf91a95ee451d" exitCode=60 Oct 30 02:00:03.711795 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.711696 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888760-btnnt" event={"ID":"dc58a71f-7f5f-4f68-aa97-a22aa15b854e","Type":"ContainerDied","Data":"3b95e491fd59d13a45036afeb0e73ce3648df7a3468a8059e5adf91a95ee451d"} Oct 30 02:00:03.711902 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.711885 6572 scope.go:122] "RemoveContainer" containerID="3b95e491fd59d13a45036afeb0e73ce3648df7a3468a8059e5adf91a95ee451d" Oct 30 02:00:03.714384 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.714364 6572 generic.go:356] "Generic (PLEG): container finished" podID="d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b" containerID="3120b3018e87291a9a7327c1e66b6cc28b1745f4072878e46a516b0228fd2353" exitCode=60 Oct 30 02:00:03.714487 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.714443 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888760-6h8kw" event={"ID":"d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b","Type":"ContainerDied","Data":"3120b3018e87291a9a7327c1e66b6cc28b1745f4072878e46a516b0228fd2353"} Oct 30 02:00:03.714624 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.714610 6572 scope.go:122] "RemoveContainer" containerID="3120b3018e87291a9a7327c1e66b6cc28b1745f4072878e46a516b0228fd2353" Oct 30 02:00:03.716203 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.716179 6572 generic.go:356] "Generic (PLEG): container finished" podID="9dc37d0b-7989-4437-9902-df0a34eb682a" containerID="93464f0634469ebdd2a1b0cd017375b582aafdc3ffc28489db742673faf317a0" exitCode=60 Oct 30 02:00:03.716311 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.716252 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2eae39-29888760-t6tz5" event={"ID":"9dc37d0b-7989-4437-9902-df0a34eb682a","Type":"ContainerDied","Data":"93464f0634469ebdd2a1b0cd017375b582aafdc3ffc28489db742673faf317a0"} Oct 30 02:00:03.716431 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.716416 6572 scope.go:122] "RemoveContainer" containerID="93464f0634469ebdd2a1b0cd017375b582aafdc3ffc28489db742673faf317a0" Oct 30 02:00:03.718210 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.718189 6572 generic.go:356] "Generic (PLEG): container finished" podID="2956c338-3b66-48aa-b3cc-521dbdef04ce" containerID="2519b21a590340a8f185e88578269974470efdfb1e219f3920e74f93f5b2e4c2" exitCode=60 Oct 30 02:00:03.718309 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.718266 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888760-lnmg5" event={"ID":"2956c338-3b66-48aa-b3cc-521dbdef04ce","Type":"ContainerDied","Data":"2519b21a590340a8f185e88578269974470efdfb1e219f3920e74f93f5b2e4c2"} Oct 30 02:00:03.718470 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.718450 6572 scope.go:122] "RemoveContainer" containerID="2519b21a590340a8f185e88578269974470efdfb1e219f3920e74f93f5b2e4c2" Oct 30 02:00:03.720206 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.720181 6572 generic.go:356] "Generic (PLEG): container finished" podID="16ce2d50-2777-4827-80ec-025ff18072ed" containerID="f6752c8dbc3dc85684aac0056a98fe91586b075d215a958edbed03268d1ffe2b" exitCode=60 Oct 30 02:00:03.720294 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.720271 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888760-545xr" event={"ID":"16ce2d50-2777-4827-80ec-025ff18072ed","Type":"ContainerDied","Data":"f6752c8dbc3dc85684aac0056a98fe91586b075d215a958edbed03268d1ffe2b"} Oct 30 02:00:03.720517 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.720502 6572 scope.go:122] "RemoveContainer" containerID="f6752c8dbc3dc85684aac0056a98fe91586b075d215a958edbed03268d1ffe2b" Oct 30 02:00:03.729717 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.729696 6572 generic.go:356] "Generic (PLEG): container finished" podID="ea84dc40-9569-4a87-a537-a8e0c0fb119d" containerID="5eecc7665ef4149a2dfd70fbbc172f181fe8e5684d06aee9f04af51727ac216a" exitCode=60 Oct 30 02:00:03.729860 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.729813 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888760-hvkkt" event={"ID":"ea84dc40-9569-4a87-a537-a8e0c0fb119d","Type":"ContainerDied","Data":"5eecc7665ef4149a2dfd70fbbc172f181fe8e5684d06aee9f04af51727ac216a"} Oct 30 02:00:03.730626 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.730140 6572 scope.go:122] "RemoveContainer" containerID="5eecc7665ef4149a2dfd70fbbc172f181fe8e5684d06aee9f04af51727ac216a" Oct 30 02:00:03.734120 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.732661 6572 generic.go:356] "Generic (PLEG): container finished" podID="ad3ca000-d9f7-4d97-9a05-b39362550ba0" containerID="29063cda7e8d3f7adc7d286e10e5bb7154d4d1cdda72c515d40a66007e0cf460" exitCode=60 Oct 30 02:00:03.734120 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.732800 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298" event={"ID":"ad3ca000-d9f7-4d97-9a05-b39362550ba0","Type":"ContainerDied","Data":"29063cda7e8d3f7adc7d286e10e5bb7154d4d1cdda72c515d40a66007e0cf460"} Oct 30 02:00:03.734120 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.733041 6572 scope.go:122] "RemoveContainer" containerID="29063cda7e8d3f7adc7d286e10e5bb7154d4d1cdda72c515d40a66007e0cf460" Oct 30 02:00:03.740156 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.740135 6572 generic.go:356] "Generic (PLEG): container finished" podID="45e5935e-478f-4fec-beae-0cae2ea14c1b" containerID="2ce5810ee15cfd14b7db2c46502132a164e6258a539222a78e4669ad54adec83" exitCode=60 Oct 30 02:00:03.740354 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.740336 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888760-xhtnj" event={"ID":"45e5935e-478f-4fec-beae-0cae2ea14c1b","Type":"ContainerDied","Data":"2ce5810ee15cfd14b7db2c46502132a164e6258a539222a78e4669ad54adec83"} Oct 30 02:00:03.744150 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.740523 6572 scope.go:122] "RemoveContainer" containerID="2ce5810ee15cfd14b7db2c46502132a164e6258a539222a78e4669ad54adec83" Oct 30 02:00:03.748582 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.747843 6572 generic.go:356] "Generic (PLEG): container finished" podID="dbd7b9d4-eb19-4605-bdc9-21940dad8c92" containerID="9479136954bb4962503e3db897ee0f67444fb7f887bc1b2b79ba13bf93c1bd15" exitCode=60 Oct 30 02:00:03.748582 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.747897 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888760-dfspx" event={"ID":"dbd7b9d4-eb19-4605-bdc9-21940dad8c92","Type":"ContainerDied","Data":"9479136954bb4962503e3db897ee0f67444fb7f887bc1b2b79ba13bf93c1bd15"} Oct 30 02:00:03.748582 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.748249 6572 scope.go:122] "RemoveContainer" containerID="9479136954bb4962503e3db897ee0f67444fb7f887bc1b2b79ba13bf93c1bd15" Oct 30 02:00:03.846184 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:03.845815 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-5xgtv" podStartSLOduration=3.845799206 podStartE2EDuration="3.845799206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:03.844404928 +0000 UTC m=+2529.420830330" watchObservedRunningTime="2026-10-30 02:00:03.845799206 +0000 UTC m=+2529.422224609" Oct 30 02:00:04.060815 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:04.060571 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:04.202718 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:04.202660 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888760-5h4tv" podStartSLOduration=4.202645959 podStartE2EDuration="4.202645959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:04.201350576 +0000 UTC m=+2529.777775977" watchObservedRunningTime="2026-10-30 02:00:04.202645959 +0000 UTC m=+2529.779071406" Oct 30 02:00:04.242105 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:04.242046 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888760-j979j" podStartSLOduration=4.242029015 podStartE2EDuration="4.242029015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:04.237870535 +0000 UTC m=+2529.814295951" watchObservedRunningTime="2026-10-30 02:00:04.242029015 +0000 UTC m=+2529.818454417" Oct 30 02:00:04.759464 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:04.759429 6572 generic.go:356] "Generic (PLEG): container finished" podID="bc3321aa-698f-4c74-94ae-95e8729d9f27" containerID="ff2c3c18b1280eb0ff670b77fcc11a3fab7224df7a215ba385acb070cfa55e58" exitCode=60 Oct 30 02:00:04.759965 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:04.759518 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888760-j979j" event={"ID":"bc3321aa-698f-4c74-94ae-95e8729d9f27","Type":"ContainerDied","Data":"ff2c3c18b1280eb0ff670b77fcc11a3fab7224df7a215ba385acb070cfa55e58"} Oct 30 02:00:04.759965 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:04.759893 6572 scope.go:122] "RemoveContainer" containerID="ff2c3c18b1280eb0ff670b77fcc11a3fab7224df7a215ba385acb070cfa55e58" Oct 30 02:00:04.761712 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:04.761687 6572 generic.go:356] "Generic (PLEG): container finished" podID="419770d0-51af-4843-9ab2-8cbbe8215357" containerID="e2d82f5b85e7d6cdd8c3ce587fa51ee41a893324f829acbf991fa9ff1a7e0a2d" exitCode=60 Oct 30 02:00:04.761860 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:04.761797 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888760-5h4tv" event={"ID":"419770d0-51af-4843-9ab2-8cbbe8215357","Type":"ContainerDied","Data":"e2d82f5b85e7d6cdd8c3ce587fa51ee41a893324f829acbf991fa9ff1a7e0a2d"} Oct 30 02:00:04.762070 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:04.762052 6572 scope.go:122] "RemoveContainer" containerID="e2d82f5b85e7d6cdd8c3ce587fa51ee41a893324f829acbf991fa9ff1a7e0a2d" Oct 30 02:00:04.763892 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:04.763808 6572 generic.go:356] "Generic (PLEG): container finished" podID="b8e5ee17-3f55-4488-aba3-2a81eceb15a4" containerID="df3376e7086da732df49ad98df67c5ad6b17e92ed1820443b22f24b017bad29c" exitCode=60 Oct 30 02:00:04.763892 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:04.763839 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-5xgtv" event={"ID":"b8e5ee17-3f55-4488-aba3-2a81eceb15a4","Type":"ContainerDied","Data":"df3376e7086da732df49ad98df67c5ad6b17e92ed1820443b22f24b017bad29c"} Oct 30 02:00:04.764155 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:04.764072 6572 scope.go:122] "RemoveContainer" containerID="df3376e7086da732df49ad98df67c5ad6b17e92ed1820443b22f24b017bad29c" Oct 30 02:00:05.177503 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:05.177458 6572 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-817fd7/agc-cred-08a8e8dfe09b: secret "agc-cred-08a8e8dfe09b" not found Oct 30 02:00:05.177503 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:05.177493 6572 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr: secret "agc-cred-08a8e8dfe09b" not found Oct 30 02:00:05.177714 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:05.177558 6572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18a3aece-618b-4a9a-8fc9-28606bddc3ce-provider-credentials podName:18a3aece-618b-4a9a-8fc9-28606bddc3ce nodeName:}" failed. No retries permitted until 2026-10-30 02:00:21.17754018 +0000 UTC m=+2546.753965578 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/18a3aece-618b-4a9a-8fc9-28606bddc3ce-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr" (UID: "18a3aece-618b-4a9a-8fc9-28606bddc3ce") : secret "agc-cred-08a8e8dfe09b" not found Oct 30 02:00:05.694131 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:05.693903 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:05.779173 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:05.779121 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888760-xhtnj" event={"ID":"45e5935e-478f-4fec-beae-0cae2ea14c1b","Type":"ContainerStarted","Data":"1e662ebe72d1fee4bc9b8ca9fa336dab1854a1d142a147cdf9ce430811d39534"} Oct 30 02:00:05.781495 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:05.781463 6572 generic.go:356] "Generic (PLEG): container finished" podID="dbd7b9d4-eb19-4605-bdc9-21940dad8c92" containerID="8939cae34cbc595709b337be49308d7a38faba8304ea3ed336874047e0a08770" exitCode=60 Oct 30 02:00:05.781667 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:05.781535 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888760-dfspx" event={"ID":"dbd7b9d4-eb19-4605-bdc9-21940dad8c92","Type":"ContainerDied","Data":"8939cae34cbc595709b337be49308d7a38faba8304ea3ed336874047e0a08770"} Oct 30 02:00:05.781667 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:05.781590 6572 scope.go:122] "RemoveContainer" containerID="9479136954bb4962503e3db897ee0f67444fb7f887bc1b2b79ba13bf93c1bd15" Oct 30 02:00:05.781955 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:05.781936 6572 scope.go:122] "RemoveContainer" containerID="8939cae34cbc595709b337be49308d7a38faba8304ea3ed336874047e0a08770" Oct 30 02:00:05.782359 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:05.782325 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888760-dfspx_odh-ai-gateway-infra(dbd7b9d4-eb19-4605-bdc9-21940dad8c92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888760-dfspx" podUID="dbd7b9d4-eb19-4605-bdc9-21940dad8c92" Oct 30 02:00:05.784150 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:05.784125 6572 generic.go:356] "Generic (PLEG): container finished" podID="32354403-56a7-4f01-a228-122805b44ba0" containerID="caf476fa4ec68421ea80a109161831a97e219c1373aba2b5b445ed7fef6ffa57" exitCode=60 Oct 30 02:00:05.784234 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:05.784203 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888760-wkrkz" event={"ID":"32354403-56a7-4f01-a228-122805b44ba0","Type":"ContainerDied","Data":"caf476fa4ec68421ea80a109161831a97e219c1373aba2b5b445ed7fef6ffa57"} Oct 30 02:00:05.784549 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:05.784532 6572 scope.go:122] "RemoveContainer" containerID="caf476fa4ec68421ea80a109161831a97e219c1373aba2b5b445ed7fef6ffa57" Oct 30 02:00:05.784951 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:05.784923 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-093d23-29888760-wkrkz_odh-ai-gateway-infra(32354403-56a7-4f01-a228-122805b44ba0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888760-wkrkz" podUID="32354403-56a7-4f01-a228-122805b44ba0" Oct 30 02:00:05.787279 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:05.786493 6572 generic.go:356] "Generic (PLEG): container finished" podID="f818d22e-27ff-4caa-a48d-352af6583de4" containerID="6a722909d05e844bddde7b8f6eb001aa8a0ba7d690948a707eeb9e600264a12e" exitCode=60 Oct 30 02:00:05.787279 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:05.786569 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888760-qczr8" event={"ID":"f818d22e-27ff-4caa-a48d-352af6583de4","Type":"ContainerDied","Data":"6a722909d05e844bddde7b8f6eb001aa8a0ba7d690948a707eeb9e600264a12e"} Oct 30 02:00:05.787279 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:05.786889 6572 scope.go:122] "RemoveContainer" containerID="6a722909d05e844bddde7b8f6eb001aa8a0ba7d690948a707eeb9e600264a12e" Oct 30 02:00:05.787279 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:05.787235 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888760-qczr8_odh-ai-gateway-infra(f818d22e-27ff-4caa-a48d-352af6583de4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888760-qczr8" podUID="f818d22e-27ff-4caa-a48d-352af6583de4" Oct 30 02:00:05.790040 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:05.789522 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888760-j979j" event={"ID":"bc3321aa-698f-4c74-94ae-95e8729d9f27","Type":"ContainerStarted","Data":"f84020310d4354eef143b703b9fbacb18c17129b9377cb3c0ddf0fbe582f04f1"} Oct 30 02:00:05.791798 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:05.791765 6572 generic.go:356] "Generic (PLEG): container finished" podID="05cb266a-e1fe-4b51-a18f-96d85a78ab16" containerID="47c1389175e0ef1883f749a545a7b9e93b925273a24a210d8bef2456a195b6d3" exitCode=60 Oct 30 02:00:05.791906 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:05.791869 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888760-kwp4k" event={"ID":"05cb266a-e1fe-4b51-a18f-96d85a78ab16","Type":"ContainerDied","Data":"47c1389175e0ef1883f749a545a7b9e93b925273a24a210d8bef2456a195b6d3"} Oct 30 02:00:05.792115 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:05.792097 6572 scope.go:122] "RemoveContainer" containerID="47c1389175e0ef1883f749a545a7b9e93b925273a24a210d8bef2456a195b6d3" Oct 30 02:00:05.792439 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:05.792413 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888760-kwp4k_odh-ai-gateway-infra(05cb266a-e1fe-4b51-a18f-96d85a78ab16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888760-kwp4k" podUID="05cb266a-e1fe-4b51-a18f-96d85a78ab16" Oct 30 02:00:05.794085 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:05.794064 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888760-k4ntx" event={"ID":"ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c","Type":"ContainerStarted","Data":"a0004f97efb7d21fd811411afb300589e3722b1cb0b6c31a03d8db0371854b1f"} Oct 30 02:00:05.799630 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:05.799606 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888760-m68hl" event={"ID":"228bfbbd-21b3-4f89-b247-962811a659e6","Type":"ContainerStarted","Data":"76f9cdf06582c5cf138fdb50682247b526d369e63f13905b9680773bde9f4ea7"} Oct 30 02:00:05.801885 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:05.801853 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-5xgtv" event={"ID":"b8e5ee17-3f55-4488-aba3-2a81eceb15a4","Type":"ContainerStarted","Data":"8801fe0095f947337dc65df5994bacac5f5e8762b8ad984070e0f7a19b3448bb"} Oct 30 02:00:05.804030 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:05.804004 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888760-z8976" event={"ID":"dff6267c-a0a8-4d35-badf-3032bfb3e979","Type":"ContainerStarted","Data":"1986b21d7033b3ad98fc116384ff6f8464734f43788d3768b8403bbb15e3a969"} Oct 30 02:00:05.807155 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:05.806613 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888760-btnnt" event={"ID":"dc58a71f-7f5f-4f68-aa97-a22aa15b854e","Type":"ContainerStarted","Data":"4f1d458927bf8584661488a67b20a1685b3f498956eb66d616fd60a897aa0e0c"} Oct 30 02:00:05.811465 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:05.811443 6572 generic.go:356] "Generic (PLEG): container finished" podID="d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b" containerID="e26bfa123910ccde51549e589499fadc6a958c5ceae45035e29fd83115da0060" exitCode=60 Oct 30 02:00:05.811644 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:05.811621 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888760-6h8kw" event={"ID":"d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b","Type":"ContainerDied","Data":"e26bfa123910ccde51549e589499fadc6a958c5ceae45035e29fd83115da0060"} Oct 30 02:00:05.812360 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:05.812339 6572 scope.go:122] "RemoveContainer" containerID="e26bfa123910ccde51549e589499fadc6a958c5ceae45035e29fd83115da0060" Oct 30 02:00:05.817830 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:05.817794 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888760-6h8kw_odh-ai-gateway-infra(d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888760-6h8kw" podUID="d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b" Oct 30 02:00:05.823807 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:05.823396 6572 generic.go:356] "Generic (PLEG): container finished" podID="9dc37d0b-7989-4437-9902-df0a34eb682a" containerID="f73df7b446098ac9a80890f69a0a6a6942226e9a5e1c714291020f300ee6dbf9" exitCode=60 Oct 30 02:00:05.823807 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:05.823520 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2eae39-29888760-t6tz5" event={"ID":"9dc37d0b-7989-4437-9902-df0a34eb682a","Type":"ContainerDied","Data":"f73df7b446098ac9a80890f69a0a6a6942226e9a5e1c714291020f300ee6dbf9"} Oct 30 02:00:05.823807 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:05.823554 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888760-xhtnj" podStartSLOduration=5.823539527 podStartE2EDuration="5.823539527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:05.820904628 +0000 UTC m=+2531.397330032" watchObservedRunningTime="2026-10-30 02:00:05.823539527 +0000 UTC m=+2531.399964920" Oct 30 02:00:05.824423 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:05.824399 6572 scope.go:122] "RemoveContainer" containerID="f73df7b446098ac9a80890f69a0a6a6942226e9a5e1c714291020f300ee6dbf9" Oct 30 02:00:05.824865 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:05.824838 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2eae39-29888760-t6tz5_odh-ai-gateway-infra(9dc37d0b-7989-4437-9902-df0a34eb682a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2eae39-29888760-t6tz5" podUID="9dc37d0b-7989-4437-9902-df0a34eb682a" Oct 30 02:00:05.836134 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:05.835167 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888760-lnmg5" event={"ID":"2956c338-3b66-48aa-b3cc-521dbdef04ce","Type":"ContainerStarted","Data":"a8cbbe235a83b98ca3292e27b04467422a611d467d913ca771d04cb3b7c65e56"} Oct 30 02:00:05.840177 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:05.840147 6572 generic.go:356] "Generic (PLEG): container finished" podID="16ce2d50-2777-4827-80ec-025ff18072ed" containerID="5a27bddcd7b87b779e0da47909c1b6db34fe0d3f1bd22389da15918a6b9ebade" exitCode=60 Oct 30 02:00:05.840404 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:05.840226 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888760-545xr" event={"ID":"16ce2d50-2777-4827-80ec-025ff18072ed","Type":"ContainerDied","Data":"5a27bddcd7b87b779e0da47909c1b6db34fe0d3f1bd22389da15918a6b9ebade"} Oct 30 02:00:05.840633 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:05.840602 6572 scope.go:122] "RemoveContainer" containerID="5a27bddcd7b87b779e0da47909c1b6db34fe0d3f1bd22389da15918a6b9ebade" Oct 30 02:00:05.841125 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:05.841094 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888760-545xr_odh-ai-gateway-infra(16ce2d50-2777-4827-80ec-025ff18072ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888760-545xr" podUID="16ce2d50-2777-4827-80ec-025ff18072ed" Oct 30 02:00:05.843966 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:05.843942 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888760-hvkkt" event={"ID":"ea84dc40-9569-4a87-a537-a8e0c0fb119d","Type":"ContainerStarted","Data":"7945297f774a9cac967f628c6177e8e6610feb1ec5114975ee2e000ee361ac9c"} Oct 30 02:00:05.847312 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:05.847287 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298" event={"ID":"ad3ca000-d9f7-4d97-9a05-b39362550ba0","Type":"ContainerStarted","Data":"4244a0c770ddc623a6d17cd44a402801aa19635ec8680525b5e6da6d65f927eb"} Oct 30 02:00:05.848248 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:05.848226 6572 scope.go:122] "RemoveContainer" containerID="4244a0c770ddc623a6d17cd44a402801aa19635ec8680525b5e6da6d65f927eb" Oct 30 02:00:05.848640 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:05.848616 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298_odh-ai-gateway-infra(ad3ca000-d9f7-4d97-9a05-b39362550ba0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298" podUID="ad3ca000-d9f7-4d97-9a05-b39362550ba0" Oct 30 02:00:05.860604 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:05.860546 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888760-btnnt" podStartSLOduration=5.860529071 podStartE2EDuration="5.860529071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:05.859095017 +0000 UTC m=+2531.435520423" watchObservedRunningTime="2026-10-30 02:00:05.860529071 +0000 UTC m=+2531.436954473" Oct 30 02:00:06.104992 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.104935 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888760-z8976" podStartSLOduration=6.10492059 podStartE2EDuration="6.10492059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:06.100909078 +0000 UTC m=+2531.677334470" watchObservedRunningTime="2026-10-30 02:00:06.10492059 +0000 UTC m=+2531.681345993" Oct 30 02:00:06.215885 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.215823 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888760-k4ntx" podStartSLOduration=6.21580772 podStartE2EDuration="6.21580772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:06.212048783 +0000 UTC m=+2531.788474185" watchObservedRunningTime="2026-10-30 02:00:06.21580772 +0000 UTC m=+2531.792233121" Oct 30 02:00:06.436847 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.436790 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888760-hvkkt" podStartSLOduration=6.436734208 podStartE2EDuration="6.436734208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:06.434132622 +0000 UTC m=+2532.010558024" watchObservedRunningTime="2026-10-30 02:00:06.436734208 +0000 UTC m=+2532.013159650" Oct 30 02:00:06.476056 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.475994 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888760-lnmg5" podStartSLOduration=6.475975628 podStartE2EDuration="6.475975628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:06.470625777 +0000 UTC m=+2532.047051179" watchObservedRunningTime="2026-10-30 02:00:06.475975628 +0000 UTC m=+2532.052401031" Oct 30 02:00:06.626100 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.626075 6572 scope.go:122] "RemoveContainer" containerID="a1623006f06a65d603906d4b5f6776ec33775daabe65d570515ff008e4c91b3d" Oct 30 02:00:06.641805 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.641783 6572 scope.go:122] "RemoveContainer" containerID="d5d8e837e66047efee29435b6ace98017ab04806517c8bdddfe85be52706e9b2" Oct 30 02:00:06.657891 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.657868 6572 scope.go:122] "RemoveContainer" containerID="7fd65067a43596d9b5f8c44a53b7dbabbd0e27b790ba511d997b50f6d9ef983b" Oct 30 02:00:06.674253 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.674229 6572 scope.go:122] "RemoveContainer" containerID="3120b3018e87291a9a7327c1e66b6cc28b1745f4072878e46a516b0228fd2353" Oct 30 02:00:06.690930 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.690906 6572 scope.go:122] "RemoveContainer" containerID="93464f0634469ebdd2a1b0cd017375b582aafdc3ffc28489db742673faf317a0" Oct 30 02:00:06.706245 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.706224 6572 scope.go:122] "RemoveContainer" containerID="f6752c8dbc3dc85684aac0056a98fe91586b075d215a958edbed03268d1ffe2b" Oct 30 02:00:06.862206 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.862093 6572 scope.go:122] "RemoveContainer" containerID="6a722909d05e844bddde7b8f6eb001aa8a0ba7d690948a707eeb9e600264a12e" Oct 30 02:00:06.862607 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:06.862406 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888760-qczr8_odh-ai-gateway-infra(f818d22e-27ff-4caa-a48d-352af6583de4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888760-qczr8" podUID="f818d22e-27ff-4caa-a48d-352af6583de4" Oct 30 02:00:06.864042 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.864014 6572 generic.go:356] "Generic (PLEG): container finished" podID="bc3321aa-698f-4c74-94ae-95e8729d9f27" containerID="f84020310d4354eef143b703b9fbacb18c17129b9377cb3c0ddf0fbe582f04f1" exitCode=60 Oct 30 02:00:06.864185 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.864097 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888760-j979j" event={"ID":"bc3321aa-698f-4c74-94ae-95e8729d9f27","Type":"ContainerDied","Data":"f84020310d4354eef143b703b9fbacb18c17129b9377cb3c0ddf0fbe582f04f1"} Oct 30 02:00:06.864185 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.864159 6572 scope.go:122] "RemoveContainer" containerID="ff2c3c18b1280eb0ff670b77fcc11a3fab7224df7a215ba385acb070cfa55e58" Oct 30 02:00:06.864366 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.864349 6572 scope.go:122] "RemoveContainer" containerID="f84020310d4354eef143b703b9fbacb18c17129b9377cb3c0ddf0fbe582f04f1" Oct 30 02:00:06.864704 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:06.864683 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-292018-29888760-j979j_odh-ai-gateway-infra(bc3321aa-698f-4c74-94ae-95e8729d9f27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888760-j979j" podUID="bc3321aa-698f-4c74-94ae-95e8729d9f27" Oct 30 02:00:06.866284 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.866267 6572 scope.go:122] "RemoveContainer" containerID="47c1389175e0ef1883f749a545a7b9e93b925273a24a210d8bef2456a195b6d3" Oct 30 02:00:06.866577 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:06.866556 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888760-kwp4k_odh-ai-gateway-infra(05cb266a-e1fe-4b51-a18f-96d85a78ab16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888760-kwp4k" podUID="05cb266a-e1fe-4b51-a18f-96d85a78ab16" Oct 30 02:00:06.868454 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.868432 6572 generic.go:356] "Generic (PLEG): container finished" podID="ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c" containerID="a0004f97efb7d21fd811411afb300589e3722b1cb0b6c31a03d8db0371854b1f" exitCode=60 Oct 30 02:00:06.868549 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.868502 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888760-k4ntx" event={"ID":"ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c","Type":"ContainerDied","Data":"a0004f97efb7d21fd811411afb300589e3722b1cb0b6c31a03d8db0371854b1f"} Oct 30 02:00:06.868734 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.868716 6572 scope.go:122] "RemoveContainer" containerID="a0004f97efb7d21fd811411afb300589e3722b1cb0b6c31a03d8db0371854b1f" Oct 30 02:00:06.869036 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:06.869002 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-558327-29888760-k4ntx_odh-ai-gateway-infra(ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888760-k4ntx" podUID="ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c" Oct 30 02:00:06.870645 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.870624 6572 generic.go:356] "Generic (PLEG): container finished" podID="419770d0-51af-4843-9ab2-8cbbe8215357" containerID="b533070cefc7fbeb5246f2601194c6317be5034aea4ae85f2935ba9de8f26c28" exitCode=60 Oct 30 02:00:06.870729 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.870699 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888760-5h4tv" event={"ID":"419770d0-51af-4843-9ab2-8cbbe8215357","Type":"ContainerDied","Data":"b533070cefc7fbeb5246f2601194c6317be5034aea4ae85f2935ba9de8f26c28"} Oct 30 02:00:06.871096 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.871078 6572 scope.go:122] "RemoveContainer" containerID="b533070cefc7fbeb5246f2601194c6317be5034aea4ae85f2935ba9de8f26c28" Oct 30 02:00:06.871386 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:06.871360 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888760-5h4tv_odh-ai-gateway-infra(419770d0-51af-4843-9ab2-8cbbe8215357)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888760-5h4tv" podUID="419770d0-51af-4843-9ab2-8cbbe8215357" Oct 30 02:00:06.872767 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.872729 6572 generic.go:356] "Generic (PLEG): container finished" podID="228bfbbd-21b3-4f89-b247-962811a659e6" containerID="76f9cdf06582c5cf138fdb50682247b526d369e63f13905b9680773bde9f4ea7" exitCode=60 Oct 30 02:00:06.872877 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.872857 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888760-m68hl" event={"ID":"228bfbbd-21b3-4f89-b247-962811a659e6","Type":"ContainerDied","Data":"76f9cdf06582c5cf138fdb50682247b526d369e63f13905b9680773bde9f4ea7"} Oct 30 02:00:06.873217 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.873196 6572 scope.go:122] "RemoveContainer" containerID="76f9cdf06582c5cf138fdb50682247b526d369e63f13905b9680773bde9f4ea7" Oct 30 02:00:06.873482 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:06.873461 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888760-m68hl_odh-ai-gateway-infra(228bfbbd-21b3-4f89-b247-962811a659e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888760-m68hl" podUID="228bfbbd-21b3-4f89-b247-962811a659e6" Oct 30 02:00:06.875088 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.875066 6572 generic.go:356] "Generic (PLEG): container finished" podID="b8e5ee17-3f55-4488-aba3-2a81eceb15a4" containerID="8801fe0095f947337dc65df5994bacac5f5e8762b8ad984070e0f7a19b3448bb" exitCode=60 Oct 30 02:00:06.875192 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.875137 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-5xgtv" event={"ID":"b8e5ee17-3f55-4488-aba3-2a81eceb15a4","Type":"ContainerDied","Data":"8801fe0095f947337dc65df5994bacac5f5e8762b8ad984070e0f7a19b3448bb"} Oct 30 02:00:06.875356 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.875337 6572 scope.go:122] "RemoveContainer" containerID="8801fe0095f947337dc65df5994bacac5f5e8762b8ad984070e0f7a19b3448bb" Oct 30 02:00:06.875672 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:06.875652 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888760-5xgtv_odh-ai-gateway-infra(b8e5ee17-3f55-4488-aba3-2a81eceb15a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-5xgtv" podUID="b8e5ee17-3f55-4488-aba3-2a81eceb15a4" Oct 30 02:00:06.877325 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.877302 6572 generic.go:356] "Generic (PLEG): container finished" podID="dff6267c-a0a8-4d35-badf-3032bfb3e979" containerID="1986b21d7033b3ad98fc116384ff6f8464734f43788d3768b8403bbb15e3a969" exitCode=60 Oct 30 02:00:06.877417 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.877369 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888760-z8976" event={"ID":"dff6267c-a0a8-4d35-badf-3032bfb3e979","Type":"ContainerDied","Data":"1986b21d7033b3ad98fc116384ff6f8464734f43788d3768b8403bbb15e3a969"} Oct 30 02:00:06.877563 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.877549 6572 scope.go:122] "RemoveContainer" containerID="1986b21d7033b3ad98fc116384ff6f8464734f43788d3768b8403bbb15e3a969" Oct 30 02:00:06.877807 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:06.877774 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d44413-29888760-z8976_odh-ai-gateway-infra(dff6267c-a0a8-4d35-badf-3032bfb3e979)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888760-z8976" podUID="dff6267c-a0a8-4d35-badf-3032bfb3e979" Oct 30 02:00:06.879217 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.879197 6572 generic.go:356] "Generic (PLEG): container finished" podID="dc58a71f-7f5f-4f68-aa97-a22aa15b854e" containerID="4f1d458927bf8584661488a67b20a1685b3f498956eb66d616fd60a897aa0e0c" exitCode=60 Oct 30 02:00:06.879335 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.879280 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888760-btnnt" event={"ID":"dc58a71f-7f5f-4f68-aa97-a22aa15b854e","Type":"ContainerDied","Data":"4f1d458927bf8584661488a67b20a1685b3f498956eb66d616fd60a897aa0e0c"} Oct 30 02:00:06.879465 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.879451 6572 scope.go:122] "RemoveContainer" containerID="4f1d458927bf8584661488a67b20a1685b3f498956eb66d616fd60a897aa0e0c" Oct 30 02:00:06.879685 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:06.879664 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888760-btnnt_odh-ai-gateway-infra(dc58a71f-7f5f-4f68-aa97-a22aa15b854e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888760-btnnt" podUID="dc58a71f-7f5f-4f68-aa97-a22aa15b854e" Oct 30 02:00:06.881202 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.881186 6572 scope.go:122] "RemoveContainer" containerID="e26bfa123910ccde51549e589499fadc6a958c5ceae45035e29fd83115da0060" Oct 30 02:00:06.881430 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:06.881412 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888760-6h8kw_odh-ai-gateway-infra(d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888760-6h8kw" podUID="d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b" Oct 30 02:00:06.882206 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.882188 6572 scope.go:122] "RemoveContainer" containerID="93e2719df595b90689297fa962bfd0f0474dbd911c7dba7b287f2b29e46741a7" Oct 30 02:00:06.883197 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.883095 6572 scope.go:122] "RemoveContainer" containerID="f73df7b446098ac9a80890f69a0a6a6942226e9a5e1c714291020f300ee6dbf9" Oct 30 02:00:06.883358 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:06.883336 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2eae39-29888760-t6tz5_odh-ai-gateway-infra(9dc37d0b-7989-4437-9902-df0a34eb682a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2eae39-29888760-t6tz5" podUID="9dc37d0b-7989-4437-9902-df0a34eb682a" Oct 30 02:00:06.884823 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.884800 6572 generic.go:356] "Generic (PLEG): container finished" podID="2956c338-3b66-48aa-b3cc-521dbdef04ce" containerID="a8cbbe235a83b98ca3292e27b04467422a611d467d913ca771d04cb3b7c65e56" exitCode=60 Oct 30 02:00:06.884923 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.884883 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888760-lnmg5" event={"ID":"2956c338-3b66-48aa-b3cc-521dbdef04ce","Type":"ContainerDied","Data":"a8cbbe235a83b98ca3292e27b04467422a611d467d913ca771d04cb3b7c65e56"} Oct 30 02:00:06.885057 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.885042 6572 scope.go:122] "RemoveContainer" containerID="a8cbbe235a83b98ca3292e27b04467422a611d467d913ca771d04cb3b7c65e56" Oct 30 02:00:06.885342 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:06.885321 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888760-lnmg5_odh-ai-gateway-infra(2956c338-3b66-48aa-b3cc-521dbdef04ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888760-lnmg5" podUID="2956c338-3b66-48aa-b3cc-521dbdef04ce" Oct 30 02:00:06.886648 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.886630 6572 scope.go:122] "RemoveContainer" containerID="5a27bddcd7b87b779e0da47909c1b6db34fe0d3f1bd22389da15918a6b9ebade" Oct 30 02:00:06.886916 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:06.886891 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888760-545xr_odh-ai-gateway-infra(16ce2d50-2777-4827-80ec-025ff18072ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888760-545xr" podUID="16ce2d50-2777-4827-80ec-025ff18072ed" Oct 30 02:00:06.888669 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.888649 6572 generic.go:356] "Generic (PLEG): container finished" podID="ea84dc40-9569-4a87-a537-a8e0c0fb119d" containerID="7945297f774a9cac967f628c6177e8e6610feb1ec5114975ee2e000ee361ac9c" exitCode=60 Oct 30 02:00:06.888829 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.888715 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888760-hvkkt" event={"ID":"ea84dc40-9569-4a87-a537-a8e0c0fb119d","Type":"ContainerDied","Data":"7945297f774a9cac967f628c6177e8e6610feb1ec5114975ee2e000ee361ac9c"} Oct 30 02:00:06.889071 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.889050 6572 scope.go:122] "RemoveContainer" containerID="7945297f774a9cac967f628c6177e8e6610feb1ec5114975ee2e000ee361ac9c" Oct 30 02:00:06.889342 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:06.889316 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888760-hvkkt_odh-ai-gateway-infra(ea84dc40-9569-4a87-a537-a8e0c0fb119d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888760-hvkkt" podUID="ea84dc40-9569-4a87-a537-a8e0c0fb119d" Oct 30 02:00:06.892024 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.892002 6572 generic.go:356] "Generic (PLEG): container finished" podID="ad3ca000-d9f7-4d97-9a05-b39362550ba0" containerID="4244a0c770ddc623a6d17cd44a402801aa19635ec8680525b5e6da6d65f927eb" exitCode=60 Oct 30 02:00:06.892156 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.892050 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298" event={"ID":"ad3ca000-d9f7-4d97-9a05-b39362550ba0","Type":"ContainerDied","Data":"4244a0c770ddc623a6d17cd44a402801aa19635ec8680525b5e6da6d65f927eb"} Oct 30 02:00:06.892339 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.892323 6572 scope.go:122] "RemoveContainer" containerID="4244a0c770ddc623a6d17cd44a402801aa19635ec8680525b5e6da6d65f927eb" Oct 30 02:00:06.892582 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:06.892561 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298_odh-ai-gateway-infra(ad3ca000-d9f7-4d97-9a05-b39362550ba0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298" podUID="ad3ca000-d9f7-4d97-9a05-b39362550ba0" Oct 30 02:00:06.894443 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.894417 6572 generic.go:356] "Generic (PLEG): container finished" podID="45e5935e-478f-4fec-beae-0cae2ea14c1b" containerID="1e662ebe72d1fee4bc9b8ca9fa336dab1854a1d142a147cdf9ce430811d39534" exitCode=60 Oct 30 02:00:06.894535 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.894496 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888760-xhtnj" event={"ID":"45e5935e-478f-4fec-beae-0cae2ea14c1b","Type":"ContainerDied","Data":"1e662ebe72d1fee4bc9b8ca9fa336dab1854a1d142a147cdf9ce430811d39534"} Oct 30 02:00:06.894721 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.894701 6572 scope.go:122] "RemoveContainer" containerID="1e662ebe72d1fee4bc9b8ca9fa336dab1854a1d142a147cdf9ce430811d39534" Oct 30 02:00:06.894995 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:06.894973 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-76a19e-29888760-xhtnj_odh-ai-gateway-infra(45e5935e-478f-4fec-beae-0cae2ea14c1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888760-xhtnj" podUID="45e5935e-478f-4fec-beae-0cae2ea14c1b" Oct 30 02:00:06.896491 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.896473 6572 scope.go:122] "RemoveContainer" containerID="8939cae34cbc595709b337be49308d7a38faba8304ea3ed336874047e0a08770" Oct 30 02:00:06.896706 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:06.896686 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888760-dfspx_odh-ai-gateway-infra(dbd7b9d4-eb19-4605-bdc9-21940dad8c92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888760-dfspx" podUID="dbd7b9d4-eb19-4605-bdc9-21940dad8c92" Oct 30 02:00:06.898326 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.898308 6572 scope.go:122] "RemoveContainer" containerID="caf476fa4ec68421ea80a109161831a97e219c1373aba2b5b445ed7fef6ffa57" Oct 30 02:00:06.898540 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:06.898521 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-093d23-29888760-wkrkz_odh-ai-gateway-infra(32354403-56a7-4f01-a228-122805b44ba0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888760-wkrkz" podUID="32354403-56a7-4f01-a228-122805b44ba0" Oct 30 02:00:06.901291 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.901236 6572 scope.go:122] "RemoveContainer" containerID="e2d82f5b85e7d6cdd8c3ce587fa51ee41a893324f829acbf991fa9ff1a7e0a2d" Oct 30 02:00:06.917126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.917107 6572 scope.go:122] "RemoveContainer" containerID="fabaa8c1ca3dcb84ed16689e9a9e5fa1451e7fb1fdb08f5f46595ae9e7ec6061" Oct 30 02:00:06.940149 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.940116 6572 scope.go:122] "RemoveContainer" containerID="df3376e7086da732df49ad98df67c5ad6b17e92ed1820443b22f24b017bad29c" Oct 30 02:00:06.958694 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.958662 6572 scope.go:122] "RemoveContainer" containerID="ca586e9c26d2d05c0c37d6d0eec03aaf62c3bcf49e43679307f3d6051bfbd76a" Oct 30 02:00:06.974968 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.974937 6572 scope.go:122] "RemoveContainer" containerID="3b95e491fd59d13a45036afeb0e73ce3648df7a3468a8059e5adf91a95ee451d" Oct 30 02:00:06.991495 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:06.991474 6572 scope.go:122] "RemoveContainer" containerID="2519b21a590340a8f185e88578269974470efdfb1e219f3920e74f93f5b2e4c2" Oct 30 02:00:07.007452 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:07.007430 6572 scope.go:122] "RemoveContainer" containerID="5eecc7665ef4149a2dfd70fbbc172f181fe8e5684d06aee9f04af51727ac216a" Oct 30 02:00:07.023505 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:07.023481 6572 scope.go:122] "RemoveContainer" containerID="29063cda7e8d3f7adc7d286e10e5bb7154d4d1cdda72c515d40a66007e0cf460" Oct 30 02:00:07.039684 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:07.039662 6572 scope.go:122] "RemoveContainer" containerID="2ce5810ee15cfd14b7db2c46502132a164e6258a539222a78e4669ad54adec83" Oct 30 02:00:07.052310 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:07.052264 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:07.432793 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:07.432713 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:07.447075 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:07.447035 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr" podUID="18a3aece-618b-4a9a-8fc9-28606bddc3ce" containerName="payload-processing" containerID="cri-o://f4f77eb8c90187c02cfa5691c32d49290342add6e4117bbb57ab0454d0748510" gracePeriod=25 Oct 30 02:00:07.771547 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:07.771511 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr" Oct 30 02:00:07.823228 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:07.823196 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18a3aece-618b-4a9a-8fc9-28606bddc3ce-provider-credentials\" (UniqueName: \"kubernetes.io/projected/18a3aece-618b-4a9a-8fc9-28606bddc3ce-provider-credentials\") pod \"18a3aece-618b-4a9a-8fc9-28606bddc3ce\" (UID: \"18a3aece-618b-4a9a-8fc9-28606bddc3ce\") " Oct 30 02:00:07.823414 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:07.823339 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18a3aece-618b-4a9a-8fc9-28606bddc3ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18a3aece-618b-4a9a-8fc9-28606bddc3ce-plugins-config-volume\") pod \"18a3aece-618b-4a9a-8fc9-28606bddc3ce\" (UID: \"18a3aece-618b-4a9a-8fc9-28606bddc3ce\") " Oct 30 02:00:07.823482 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:07.823426 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18a3aece-618b-4a9a-8fc9-28606bddc3ce-kube-api-access-bg694\" (UniqueName: \"kubernetes.io/projected/18a3aece-618b-4a9a-8fc9-28606bddc3ce-kube-api-access-bg694\") pod \"18a3aece-618b-4a9a-8fc9-28606bddc3ce\" (UID: \"18a3aece-618b-4a9a-8fc9-28606bddc3ce\") " Oct 30 02:00:07.823632 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:07.823610 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18a3aece-618b-4a9a-8fc9-28606bddc3ce-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/18a3aece-618b-4a9a-8fc9-28606bddc3ce-routing-overlay-volume\") pod \"18a3aece-618b-4a9a-8fc9-28606bddc3ce\" (UID: \"18a3aece-618b-4a9a-8fc9-28606bddc3ce\") " Oct 30 02:00:07.823983 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:07.823805 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18a3aece-618b-4a9a-8fc9-28606bddc3ce-plugins-config-volume" pod "18a3aece-618b-4a9a-8fc9-28606bddc3ce" (UID: "18a3aece-618b-4a9a-8fc9-28606bddc3ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:07.824262 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:07.824240 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18a3aece-618b-4a9a-8fc9-28606bddc3ce-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:07.824262 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:07.824249 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18a3aece-618b-4a9a-8fc9-28606bddc3ce-routing-overlay-volume" pod "18a3aece-618b-4a9a-8fc9-28606bddc3ce" (UID: "18a3aece-618b-4a9a-8fc9-28606bddc3ce"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:07.826970 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:07.826934 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a3aece-618b-4a9a-8fc9-28606bddc3ce-provider-credentials" pod "18a3aece-618b-4a9a-8fc9-28606bddc3ce" (UID: "18a3aece-618b-4a9a-8fc9-28606bddc3ce"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:07.827288 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:07.827259 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a3aece-618b-4a9a-8fc9-28606bddc3ce-kube-api-access-bg694" pod "18a3aece-618b-4a9a-8fc9-28606bddc3ce" (UID: "18a3aece-618b-4a9a-8fc9-28606bddc3ce"). InnerVolumeSpecName "kube-api-access-bg694". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:07.915996 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:07.915963 6572 scope.go:122] "RemoveContainer" containerID="f84020310d4354eef143b703b9fbacb18c17129b9377cb3c0ddf0fbe582f04f1" Oct 30 02:00:07.916397 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:07.916325 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-292018-29888760-j979j_odh-ai-gateway-infra(bc3321aa-698f-4c74-94ae-95e8729d9f27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888760-j979j" podUID="bc3321aa-698f-4c74-94ae-95e8729d9f27" Oct 30 02:00:07.918558 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:07.918530 6572 scope.go:122] "RemoveContainer" containerID="a0004f97efb7d21fd811411afb300589e3722b1cb0b6c31a03d8db0371854b1f" Oct 30 02:00:07.918972 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:07.918943 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-558327-29888760-k4ntx_odh-ai-gateway-infra(ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888760-k4ntx" podUID="ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c" Oct 30 02:00:07.923578 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:07.923552 6572 scope.go:122] "RemoveContainer" containerID="76f9cdf06582c5cf138fdb50682247b526d369e63f13905b9680773bde9f4ea7" Oct 30 02:00:07.924032 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:07.924004 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888760-m68hl_odh-ai-gateway-infra(228bfbbd-21b3-4f89-b247-962811a659e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888760-m68hl" podUID="228bfbbd-21b3-4f89-b247-962811a659e6" Oct 30 02:00:07.925590 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:07.925472 6572 scope.go:122] "RemoveContainer" containerID="8801fe0095f947337dc65df5994bacac5f5e8762b8ad984070e0f7a19b3448bb" Oct 30 02:00:07.925590 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:07.925572 6572 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/18a3aece-618b-4a9a-8fc9-28606bddc3ce-provider-credentials\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:07.925828 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:07.925783 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg694\" (UniqueName: \"kubernetes.io/projected/18a3aece-618b-4a9a-8fc9-28606bddc3ce-kube-api-access-bg694\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:07.925828 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:07.925804 6572 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/18a3aece-618b-4a9a-8fc9-28606bddc3ce-routing-overlay-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:07.925928 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:07.925841 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888760-5xgtv_odh-ai-gateway-infra(b8e5ee17-3f55-4488-aba3-2a81eceb15a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-5xgtv" podUID="b8e5ee17-3f55-4488-aba3-2a81eceb15a4" Oct 30 02:00:07.927427 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:07.927407 6572 scope.go:122] "RemoveContainer" containerID="1986b21d7033b3ad98fc116384ff6f8464734f43788d3768b8403bbb15e3a969" Oct 30 02:00:07.927734 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:07.927707 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d44413-29888760-z8976_odh-ai-gateway-infra(dff6267c-a0a8-4d35-badf-3032bfb3e979)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888760-z8976" podUID="dff6267c-a0a8-4d35-badf-3032bfb3e979" Oct 30 02:00:07.928517 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:07.928496 6572 generic.go:356] "Generic (PLEG): container finished" podID="18a3aece-618b-4a9a-8fc9-28606bddc3ce" containerID="f4f77eb8c90187c02cfa5691c32d49290342add6e4117bbb57ab0454d0748510" exitCode=0 Oct 30 02:00:07.928592 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:07.928539 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr" Oct 30 02:00:07.928592 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:07.928565 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr" event={"ID":"18a3aece-618b-4a9a-8fc9-28606bddc3ce","Type":"ContainerDied","Data":"f4f77eb8c90187c02cfa5691c32d49290342add6e4117bbb57ab0454d0748510"} Oct 30 02:00:07.928703 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:07.928648 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr" event={"ID":"18a3aece-618b-4a9a-8fc9-28606bddc3ce","Type":"ContainerDied","Data":"7dad09b3c2c17b2c564516b23cd8be070bd82553d2961f855e7ab8c5eb5719d5"} Oct 30 02:00:07.928703 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:07.928683 6572 scope.go:122] "RemoveContainer" containerID="f4f77eb8c90187c02cfa5691c32d49290342add6e4117bbb57ab0454d0748510" Oct 30 02:00:07.931341 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:07.931320 6572 scope.go:122] "RemoveContainer" containerID="4f1d458927bf8584661488a67b20a1685b3f498956eb66d616fd60a897aa0e0c" Oct 30 02:00:07.931619 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:07.931596 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888760-btnnt_odh-ai-gateway-infra(dc58a71f-7f5f-4f68-aa97-a22aa15b854e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888760-btnnt" podUID="dc58a71f-7f5f-4f68-aa97-a22aa15b854e" Oct 30 02:00:07.933823 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:07.933806 6572 scope.go:122] "RemoveContainer" containerID="a8cbbe235a83b98ca3292e27b04467422a611d467d913ca771d04cb3b7c65e56" Oct 30 02:00:07.934103 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:07.934081 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888760-lnmg5_odh-ai-gateway-infra(2956c338-3b66-48aa-b3cc-521dbdef04ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888760-lnmg5" podUID="2956c338-3b66-48aa-b3cc-521dbdef04ce" Oct 30 02:00:07.936906 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:07.936885 6572 scope.go:122] "RemoveContainer" containerID="7945297f774a9cac967f628c6177e8e6610feb1ec5114975ee2e000ee361ac9c" Oct 30 02:00:07.937244 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:07.937220 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888760-hvkkt_odh-ai-gateway-infra(ea84dc40-9569-4a87-a537-a8e0c0fb119d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888760-hvkkt" podUID="ea84dc40-9569-4a87-a537-a8e0c0fb119d" Oct 30 02:00:07.939333 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:07.939181 6572 scope.go:122] "RemoveContainer" containerID="4244a0c770ddc623a6d17cd44a402801aa19635ec8680525b5e6da6d65f927eb" Oct 30 02:00:07.939473 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:07.939449 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298_odh-ai-gateway-infra(ad3ca000-d9f7-4d97-9a05-b39362550ba0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298" podUID="ad3ca000-d9f7-4d97-9a05-b39362550ba0" Oct 30 02:00:07.943545 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:07.943523 6572 scope.go:122] "RemoveContainer" containerID="1e662ebe72d1fee4bc9b8ca9fa336dab1854a1d142a147cdf9ce430811d39534" Oct 30 02:00:07.944860 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:07.943966 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-76a19e-29888760-xhtnj_odh-ai-gateway-infra(45e5935e-478f-4fec-beae-0cae2ea14c1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888760-xhtnj" podUID="45e5935e-478f-4fec-beae-0cae2ea14c1b" Oct 30 02:00:07.945133 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:07.945110 6572 scope.go:122] "RemoveContainer" containerID="f4f77eb8c90187c02cfa5691c32d49290342add6e4117bbb57ab0454d0748510" Oct 30 02:00:07.945486 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:07.945463 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4f77eb8c90187c02cfa5691c32d49290342add6e4117bbb57ab0454d0748510\": container with ID starting with f4f77eb8c90187c02cfa5691c32d49290342add6e4117bbb57ab0454d0748510 not found: ID does not exist" containerID="f4f77eb8c90187c02cfa5691c32d49290342add6e4117bbb57ab0454d0748510" Oct 30 02:00:07.945558 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:07.945496 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4f77eb8c90187c02cfa5691c32d49290342add6e4117bbb57ab0454d0748510"} err="failed to get container status \"f4f77eb8c90187c02cfa5691c32d49290342add6e4117bbb57ab0454d0748510\": rpc error: code = NotFound desc = could not find container \"f4f77eb8c90187c02cfa5691c32d49290342add6e4117bbb57ab0454d0748510\": container with ID starting with f4f77eb8c90187c02cfa5691c32d49290342add6e4117bbb57ab0454d0748510 not found: ID does not exist" Oct 30 02:00:07.999383 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:07.999275 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr"] Oct 30 02:00:08.004839 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:08.004740 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f88s7qr"] Oct 30 02:00:08.185770 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:08.171170 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:08.351533 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:08.351439 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:08.426182 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:08.426147 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s"] Oct 30 02:00:08.427107 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:08.427090 6572 cpu_manager.go:414] "containerMap: removing container" podUID="18a3aece-618b-4a9a-8fc9-28606bddc3ce" containerName="payload-processing" Oct 30 02:00:08.427163 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:08.427108 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18a3aece-618b-4a9a-8fc9-28606bddc3ce" containerName="payload-processing" Oct 30 02:00:08.427277 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:08.427268 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="18a3aece-618b-4a9a-8fc9-28606bddc3ce" containerName="payload-processing" Oct 30 02:00:08.432698 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:08.432672 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s" Oct 30 02:00:08.448262 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:08.448220 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-093d23\"/\"kube-root-ca.crt\"" Oct 30 02:00:08.448433 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:08.448310 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-093d23\"/\"agc-cred-1eeb94896bc0\"" Oct 30 02:00:08.455918 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:08.449466 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-093d23\"/\"payload-processing-external-model-plugins-e2e-extmod-b-093d23\"" Oct 30 02:00:08.455918 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:08.449684 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-093d23\"/\"payload-processing-external-model-e2e--4282dfa5-dockercfg-q42pg\"" Oct 30 02:00:08.455918 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:08.450450 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-093d23\"/\"openshift-service-ca.crt\"" Oct 30 02:00:08.455918 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:08.450958 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-093d23\"/\"agc-cred-8b0342fa1e4d\"" Oct 30 02:00:08.462197 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:08.456384 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-093d23\"/\"routing-overlay\"" Oct 30 02:00:08.462197 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:08.461450 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s"] Oct 30 02:00:08.547569 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:08.547532 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d818fec7-418c-447e-bd2e-b64e07fdbb86-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s\" (UID: \"d818fec7-418c-447e-bd2e-b64e07fdbb86\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s" Oct 30 02:00:08.547831 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:08.547811 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcjv8\" (UniqueName: \"kubernetes.io/projected/d818fec7-418c-447e-bd2e-b64e07fdbb86-kube-api-access-dcjv8\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s\" (UID: \"d818fec7-418c-447e-bd2e-b64e07fdbb86\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s" Oct 30 02:00:08.547893 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:08.547874 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18a3aece-618b-4a9a-8fc9-28606bddc3ce" path="/var/lib/kubelet/pods/18a3aece-618b-4a9a-8fc9-28606bddc3ce/volumes" Oct 30 02:00:08.547974 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:08.547950 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d818fec7-418c-447e-bd2e-b64e07fdbb86-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s\" (UID: \"d818fec7-418c-447e-bd2e-b64e07fdbb86\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s" Oct 30 02:00:08.548083 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:08.548055 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d818fec7-418c-447e-bd2e-b64e07fdbb86-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s\" (UID: \"d818fec7-418c-447e-bd2e-b64e07fdbb86\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s" Oct 30 02:00:08.663828 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:08.662974 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcjv8\" (UniqueName: \"kubernetes.io/projected/d818fec7-418c-447e-bd2e-b64e07fdbb86-kube-api-access-dcjv8\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s\" (UID: \"d818fec7-418c-447e-bd2e-b64e07fdbb86\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s" Oct 30 02:00:08.663828 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:08.663128 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d818fec7-418c-447e-bd2e-b64e07fdbb86-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s\" (UID: \"d818fec7-418c-447e-bd2e-b64e07fdbb86\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s" Oct 30 02:00:08.663828 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:08.663182 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d818fec7-418c-447e-bd2e-b64e07fdbb86-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s\" (UID: \"d818fec7-418c-447e-bd2e-b64e07fdbb86\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s" Oct 30 02:00:08.663828 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:08.663363 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d818fec7-418c-447e-bd2e-b64e07fdbb86-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s\" (UID: \"d818fec7-418c-447e-bd2e-b64e07fdbb86\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s" Oct 30 02:00:08.684776 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:08.664474 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d818fec7-418c-447e-bd2e-b64e07fdbb86-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s\" (UID: \"d818fec7-418c-447e-bd2e-b64e07fdbb86\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s" Oct 30 02:00:08.684776 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:08.665737 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d818fec7-418c-447e-bd2e-b64e07fdbb86-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s\" (UID: \"d818fec7-418c-447e-bd2e-b64e07fdbb86\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s" Oct 30 02:00:08.700777 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:08.692526 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcjv8\" (UniqueName: \"kubernetes.io/projected/d818fec7-418c-447e-bd2e-b64e07fdbb86-kube-api-access-dcjv8\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s\" (UID: \"d818fec7-418c-447e-bd2e-b64e07fdbb86\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s" Oct 30 02:00:08.702842 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:08.702807 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d818fec7-418c-447e-bd2e-b64e07fdbb86-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s\" (UID: \"d818fec7-418c-447e-bd2e-b64e07fdbb86\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s" Oct 30 02:00:08.803342 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:08.789412 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s" Oct 30 02:00:09.162809 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:00:09.162776 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd818fec7_418c_447e_bd2e_b64e07fdbb86.slice/crio-75ba27ac60df1fd1f8806195bfbaa523d7c237d39563a812e55da185f3fdeae7 WatchSource:0}: Error finding container 75ba27ac60df1fd1f8806195bfbaa523d7c237d39563a812e55da185f3fdeae7: Status 404 returned error can't find the container with id 75ba27ac60df1fd1f8806195bfbaa523d7c237d39563a812e55da185f3fdeae7 Oct 30 02:00:09.166269 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:09.166124 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s"] Oct 30 02:00:09.454837 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:09.454694 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:09.972809 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:09.972743 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s" event={"ID":"d818fec7-418c-447e-bd2e-b64e07fdbb86","Type":"ContainerStarted","Data":"1fe3028850ee4cc59a0ed9533c4a01140b8ccf6577fec972b976e0588a015c15"} Oct 30 02:00:09.973019 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:09.972832 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s" event={"ID":"d818fec7-418c-447e-bd2e-b64e07fdbb86","Type":"ContainerStarted","Data":"75ba27ac60df1fd1f8806195bfbaa523d7c237d39563a812e55da185f3fdeae7"} Oct 30 02:00:09.973019 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:09.972856 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s" Oct 30 02:00:10.014226 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:10.014171 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s" podStartSLOduration=2.014152608 podStartE2EDuration="2.014152608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:10.01110462 +0000 UTC m=+2535.587530044" watchObservedRunningTime="2026-10-30 02:00:10.014152608 +0000 UTC m=+2535.590578011" Oct 30 02:00:11.084778 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:11.084225 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:12.677394 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:12.677354 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:14.161296 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:14.160586 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:14.692952 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:14.692918 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:00:15.805044 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:15.804870 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:16.535524 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:16.535487 6572 scope.go:122] "RemoveContainer" containerID="caf476fa4ec68421ea80a109161831a97e219c1373aba2b5b445ed7fef6ffa57" Oct 30 02:00:16.536910 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:16.536887 6572 scope.go:122] "RemoveContainer" containerID="a8cbbe235a83b98ca3292e27b04467422a611d467d913ca771d04cb3b7c65e56" Oct 30 02:00:16.537897 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:16.537879 6572 scope.go:122] "RemoveContainer" containerID="1986b21d7033b3ad98fc116384ff6f8464734f43788d3768b8403bbb15e3a969" Oct 30 02:00:17.483498 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:17.483385 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:17.535104 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:17.535063 6572 scope.go:122] "RemoveContainer" containerID="76f9cdf06582c5cf138fdb50682247b526d369e63f13905b9680773bde9f4ea7" Oct 30 02:00:17.535393 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:17.535375 6572 scope.go:122] "RemoveContainer" containerID="f84020310d4354eef143b703b9fbacb18c17129b9377cb3c0ddf0fbe582f04f1" Oct 30 02:00:17.535605 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:17.535567 6572 scope.go:122] "RemoveContainer" containerID="4f1d458927bf8584661488a67b20a1685b3f498956eb66d616fd60a897aa0e0c" Oct 30 02:00:17.535787 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:17.535618 6572 scope.go:122] "RemoveContainer" containerID="4244a0c770ddc623a6d17cd44a402801aa19635ec8680525b5e6da6d65f927eb" Oct 30 02:00:18.060550 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:18.060519 6572 generic.go:356] "Generic (PLEG): container finished" podID="dff6267c-a0a8-4d35-badf-3032bfb3e979" containerID="c99f99d79c6618d97b53b2f8f1f193e13f0f3ff5fae0dc5006b2e3a4f578dd31" exitCode=60 Oct 30 02:00:18.060712 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:18.060599 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888760-z8976" event={"ID":"dff6267c-a0a8-4d35-badf-3032bfb3e979","Type":"ContainerDied","Data":"c99f99d79c6618d97b53b2f8f1f193e13f0f3ff5fae0dc5006b2e3a4f578dd31"} Oct 30 02:00:18.060712 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:18.060679 6572 scope.go:122] "RemoveContainer" containerID="1986b21d7033b3ad98fc116384ff6f8464734f43788d3768b8403bbb15e3a969" Oct 30 02:00:18.061177 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:18.061160 6572 scope.go:122] "RemoveContainer" containerID="c99f99d79c6618d97b53b2f8f1f193e13f0f3ff5fae0dc5006b2e3a4f578dd31" Oct 30 02:00:18.061543 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:18.061526 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d44413-29888760-z8976_odh-ai-gateway-infra(dff6267c-a0a8-4d35-badf-3032bfb3e979)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888760-z8976" podUID="dff6267c-a0a8-4d35-badf-3032bfb3e979" Oct 30 02:00:18.065559 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:18.065537 6572 generic.go:356] "Generic (PLEG): container finished" podID="2956c338-3b66-48aa-b3cc-521dbdef04ce" containerID="d691202ed78cbfff7334feb918c2992df87ed55da968b6284c2ed7ac8e59d550" exitCode=60 Oct 30 02:00:18.065659 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:18.065607 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888760-lnmg5" event={"ID":"2956c338-3b66-48aa-b3cc-521dbdef04ce","Type":"ContainerDied","Data":"d691202ed78cbfff7334feb918c2992df87ed55da968b6284c2ed7ac8e59d550"} Oct 30 02:00:18.066118 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:18.066097 6572 scope.go:122] "RemoveContainer" containerID="d691202ed78cbfff7334feb918c2992df87ed55da968b6284c2ed7ac8e59d550" Oct 30 02:00:18.066552 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:18.066511 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888760-lnmg5_odh-ai-gateway-infra(2956c338-3b66-48aa-b3cc-521dbdef04ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888760-lnmg5" podUID="2956c338-3b66-48aa-b3cc-521dbdef04ce" Oct 30 02:00:18.068454 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:18.068428 6572 generic.go:356] "Generic (PLEG): container finished" podID="32354403-56a7-4f01-a228-122805b44ba0" containerID="6f15c3365220a63beb0b815cd73bd14ef816500d5817e83084ad271c15b5b215" exitCode=60 Oct 30 02:00:18.068534 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:18.068510 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888760-wkrkz" event={"ID":"32354403-56a7-4f01-a228-122805b44ba0","Type":"ContainerDied","Data":"6f15c3365220a63beb0b815cd73bd14ef816500d5817e83084ad271c15b5b215"} Oct 30 02:00:18.068985 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:18.068959 6572 scope.go:122] "RemoveContainer" containerID="6f15c3365220a63beb0b815cd73bd14ef816500d5817e83084ad271c15b5b215" Oct 30 02:00:18.069305 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:18.069281 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-093d23-29888760-wkrkz_odh-ai-gateway-infra(32354403-56a7-4f01-a228-122805b44ba0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888760-wkrkz" podUID="32354403-56a7-4f01-a228-122805b44ba0" Oct 30 02:00:18.192190 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:18.192161 6572 scope.go:122] "RemoveContainer" containerID="a8cbbe235a83b98ca3292e27b04467422a611d467d913ca771d04cb3b7c65e56" Oct 30 02:00:18.212226 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:18.212200 6572 scope.go:122] "RemoveContainer" containerID="caf476fa4ec68421ea80a109161831a97e219c1373aba2b5b445ed7fef6ffa57" Oct 30 02:00:18.535583 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:18.535550 6572 scope.go:122] "RemoveContainer" containerID="b533070cefc7fbeb5246f2601194c6317be5034aea4ae85f2935ba9de8f26c28" Oct 30 02:00:18.536099 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:18.535656 6572 scope.go:122] "RemoveContainer" containerID="6a722909d05e844bddde7b8f6eb001aa8a0ba7d690948a707eeb9e600264a12e" Oct 30 02:00:18.536099 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:18.535825 6572 scope.go:122] "RemoveContainer" containerID="a0004f97efb7d21fd811411afb300589e3722b1cb0b6c31a03d8db0371854b1f" Oct 30 02:00:18.536099 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:18.536027 6572 scope.go:122] "RemoveContainer" containerID="7945297f774a9cac967f628c6177e8e6610feb1ec5114975ee2e000ee361ac9c" Oct 30 02:00:18.985063 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:18.978934 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:19.084313 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.084287 6572 generic.go:356] "Generic (PLEG): container finished" podID="bc3321aa-698f-4c74-94ae-95e8729d9f27" containerID="60e68186facd694ebacbcb41fdeb90a593ef4ff08eefc030711ce407d2ed8f08" exitCode=60 Oct 30 02:00:19.084395 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.084357 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888760-j979j" event={"ID":"bc3321aa-698f-4c74-94ae-95e8729d9f27","Type":"ContainerDied","Data":"60e68186facd694ebacbcb41fdeb90a593ef4ff08eefc030711ce407d2ed8f08"} Oct 30 02:00:19.084434 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.084421 6572 scope.go:122] "RemoveContainer" containerID="f84020310d4354eef143b703b9fbacb18c17129b9377cb3c0ddf0fbe582f04f1" Oct 30 02:00:19.084994 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.084962 6572 scope.go:122] "RemoveContainer" containerID="60e68186facd694ebacbcb41fdeb90a593ef4ff08eefc030711ce407d2ed8f08" Oct 30 02:00:19.085414 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:19.085389 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-292018-29888760-j979j_odh-ai-gateway-infra(bc3321aa-698f-4c74-94ae-95e8729d9f27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888760-j979j" podUID="bc3321aa-698f-4c74-94ae-95e8729d9f27" Oct 30 02:00:19.087048 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.087026 6572 generic.go:356] "Generic (PLEG): container finished" podID="228bfbbd-21b3-4f89-b247-962811a659e6" containerID="6464156bb5fa6011f863d7851d003914b7a15e7f1341cfd85e0e2ca0ddfdba18" exitCode=60 Oct 30 02:00:19.087143 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.087096 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888760-m68hl" event={"ID":"228bfbbd-21b3-4f89-b247-962811a659e6","Type":"ContainerDied","Data":"6464156bb5fa6011f863d7851d003914b7a15e7f1341cfd85e0e2ca0ddfdba18"} Oct 30 02:00:19.087556 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.087521 6572 scope.go:122] "RemoveContainer" containerID="6464156bb5fa6011f863d7851d003914b7a15e7f1341cfd85e0e2ca0ddfdba18" Oct 30 02:00:19.088043 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:19.088019 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888760-m68hl_odh-ai-gateway-infra(228bfbbd-21b3-4f89-b247-962811a659e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888760-m68hl" podUID="228bfbbd-21b3-4f89-b247-962811a659e6" Oct 30 02:00:19.091387 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.091360 6572 generic.go:356] "Generic (PLEG): container finished" podID="dc58a71f-7f5f-4f68-aa97-a22aa15b854e" containerID="cd7a5a5205b70ca54435db09d40617d7b2b4c16ed1c7b52157aad55f5d137470" exitCode=60 Oct 30 02:00:19.091500 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.091431 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888760-btnnt" event={"ID":"dc58a71f-7f5f-4f68-aa97-a22aa15b854e","Type":"ContainerDied","Data":"cd7a5a5205b70ca54435db09d40617d7b2b4c16ed1c7b52157aad55f5d137470"} Oct 30 02:00:19.091870 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.091851 6572 scope.go:122] "RemoveContainer" containerID="cd7a5a5205b70ca54435db09d40617d7b2b4c16ed1c7b52157aad55f5d137470" Oct 30 02:00:19.092182 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:19.092138 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888760-btnnt_odh-ai-gateway-infra(dc58a71f-7f5f-4f68-aa97-a22aa15b854e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888760-btnnt" podUID="dc58a71f-7f5f-4f68-aa97-a22aa15b854e" Oct 30 02:00:19.095988 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.095956 6572 generic.go:356] "Generic (PLEG): container finished" podID="ad3ca000-d9f7-4d97-9a05-b39362550ba0" containerID="65f84e27d0953ceb2bcc066dcef193288d30e67322356ce32e7dc368ad911a97" exitCode=60 Oct 30 02:00:19.096159 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.096029 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298" event={"ID":"ad3ca000-d9f7-4d97-9a05-b39362550ba0","Type":"ContainerDied","Data":"65f84e27d0953ceb2bcc066dcef193288d30e67322356ce32e7dc368ad911a97"} Oct 30 02:00:19.096523 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.096503 6572 scope.go:122] "RemoveContainer" containerID="65f84e27d0953ceb2bcc066dcef193288d30e67322356ce32e7dc368ad911a97" Oct 30 02:00:19.096951 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:19.096916 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298_odh-ai-gateway-infra(ad3ca000-d9f7-4d97-9a05-b39362550ba0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298" podUID="ad3ca000-d9f7-4d97-9a05-b39362550ba0" Oct 30 02:00:19.112437 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.112401 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888760-z8976"] Oct 30 02:00:19.167840 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.167810 6572 scope.go:122] "RemoveContainer" containerID="76f9cdf06582c5cf138fdb50682247b526d369e63f13905b9680773bde9f4ea7" Oct 30 02:00:19.178181 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.178148 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888760-lnmg5"] Oct 30 02:00:19.203536 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.203490 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888760-wkrkz"] Oct 30 02:00:19.226152 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.217842 6572 scope.go:122] "RemoveContainer" containerID="4f1d458927bf8584661488a67b20a1685b3f498956eb66d616fd60a897aa0e0c" Oct 30 02:00:19.265553 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.265479 6572 scope.go:122] "RemoveContainer" containerID="4244a0c770ddc623a6d17cd44a402801aa19635ec8680525b5e6da6d65f927eb" Oct 30 02:00:19.382384 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.382353 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888760-z8976" Oct 30 02:00:19.443941 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.443869 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888760-lnmg5" Oct 30 02:00:19.499095 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.499052 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dff6267c-a0a8-4d35-badf-3032bfb3e979-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dff6267c-a0a8-4d35-badf-3032bfb3e979-maas-api-service-ca\") pod \"dff6267c-a0a8-4d35-badf-3032bfb3e979\" (UID: \"dff6267c-a0a8-4d35-badf-3032bfb3e979\") " Oct 30 02:00:19.499314 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.499146 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dff6267c-a0a8-4d35-badf-3032bfb3e979-kube-api-access-tkxx9\" (UniqueName: \"kubernetes.io/projected/dff6267c-a0a8-4d35-badf-3032bfb3e979-kube-api-access-tkxx9\") pod \"dff6267c-a0a8-4d35-badf-3032bfb3e979\" (UID: \"dff6267c-a0a8-4d35-badf-3032bfb3e979\") " Oct 30 02:00:19.499930 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.499905 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dff6267c-a0a8-4d35-badf-3032bfb3e979-maas-api-service-ca" pod "dff6267c-a0a8-4d35-badf-3032bfb3e979" (UID: "dff6267c-a0a8-4d35-badf-3032bfb3e979"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:19.501662 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.501629 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff6267c-a0a8-4d35-badf-3032bfb3e979-kube-api-access-tkxx9" pod "dff6267c-a0a8-4d35-badf-3032bfb3e979" (UID: "dff6267c-a0a8-4d35-badf-3032bfb3e979"). InnerVolumeSpecName "kube-api-access-tkxx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:19.533476 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.533442 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888760-wkrkz" Oct 30 02:00:19.535911 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.535880 6572 scope.go:122] "RemoveContainer" containerID="8939cae34cbc595709b337be49308d7a38faba8304ea3ed336874047e0a08770" Oct 30 02:00:19.536369 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.536059 6572 scope.go:122] "RemoveContainer" containerID="1e662ebe72d1fee4bc9b8ca9fa336dab1854a1d142a147cdf9ce430811d39534" Oct 30 02:00:19.536436 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.536373 6572 scope.go:122] "RemoveContainer" containerID="5a27bddcd7b87b779e0da47909c1b6db34fe0d3f1bd22389da15918a6b9ebade" Oct 30 02:00:19.537374 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.537094 6572 scope.go:122] "RemoveContainer" containerID="47c1389175e0ef1883f749a545a7b9e93b925273a24a210d8bef2456a195b6d3" Oct 30 02:00:19.537374 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.537244 6572 scope.go:122] "RemoveContainer" containerID="e26bfa123910ccde51549e589499fadc6a958c5ceae45035e29fd83115da0060" Oct 30 02:00:19.601207 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.601168 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32354403-56a7-4f01-a228-122805b44ba0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32354403-56a7-4f01-a228-122805b44ba0-maas-api-service-ca\") pod \"32354403-56a7-4f01-a228-122805b44ba0\" (UID: \"32354403-56a7-4f01-a228-122805b44ba0\") " Oct 30 02:00:19.601466 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.601445 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2956c338-3b66-48aa-b3cc-521dbdef04ce-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2956c338-3b66-48aa-b3cc-521dbdef04ce-maas-api-service-ca\") pod \"2956c338-3b66-48aa-b3cc-521dbdef04ce\" (UID: \"2956c338-3b66-48aa-b3cc-521dbdef04ce\") " Oct 30 02:00:19.601583 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.601567 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32354403-56a7-4f01-a228-122805b44ba0-kube-api-access-pgxxb\" (UniqueName: \"kubernetes.io/projected/32354403-56a7-4f01-a228-122805b44ba0-kube-api-access-pgxxb\") pod \"32354403-56a7-4f01-a228-122805b44ba0\" (UID: \"32354403-56a7-4f01-a228-122805b44ba0\") " Oct 30 02:00:19.602074 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.601711 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2956c338-3b66-48aa-b3cc-521dbdef04ce-kube-api-access-72n44\" (UniqueName: \"kubernetes.io/projected/2956c338-3b66-48aa-b3cc-521dbdef04ce-kube-api-access-72n44\") pod \"2956c338-3b66-48aa-b3cc-521dbdef04ce\" (UID: \"2956c338-3b66-48aa-b3cc-521dbdef04ce\") " Oct 30 02:00:19.602074 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.602067 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32354403-56a7-4f01-a228-122805b44ba0-maas-api-service-ca" pod "32354403-56a7-4f01-a228-122805b44ba0" (UID: "32354403-56a7-4f01-a228-122805b44ba0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:19.602820 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.602796 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2956c338-3b66-48aa-b3cc-521dbdef04ce-maas-api-service-ca" pod "2956c338-3b66-48aa-b3cc-521dbdef04ce" (UID: "2956c338-3b66-48aa-b3cc-521dbdef04ce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:19.603109 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.602913 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32354403-56a7-4f01-a228-122805b44ba0-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:19.603337 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.603316 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dff6267c-a0a8-4d35-badf-3032bfb3e979-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:19.603478 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.603460 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkxx9\" (UniqueName: \"kubernetes.io/projected/dff6267c-a0a8-4d35-badf-3032bfb3e979-kube-api-access-tkxx9\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:19.605118 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.605078 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32354403-56a7-4f01-a228-122805b44ba0-kube-api-access-pgxxb" pod "32354403-56a7-4f01-a228-122805b44ba0" (UID: "32354403-56a7-4f01-a228-122805b44ba0"). InnerVolumeSpecName "kube-api-access-pgxxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:19.610150 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.610108 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2956c338-3b66-48aa-b3cc-521dbdef04ce-kube-api-access-72n44" pod "2956c338-3b66-48aa-b3cc-521dbdef04ce" (UID: "2956c338-3b66-48aa-b3cc-521dbdef04ce"). InnerVolumeSpecName "kube-api-access-72n44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:19.705632 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.705533 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgxxb\" (UniqueName: \"kubernetes.io/projected/32354403-56a7-4f01-a228-122805b44ba0-kube-api-access-pgxxb\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:19.705632 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.705571 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72n44\" (UniqueName: \"kubernetes.io/projected/2956c338-3b66-48aa-b3cc-521dbdef04ce-kube-api-access-72n44\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:19.705632 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:19.705584 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2956c338-3b66-48aa-b3cc-521dbdef04ce-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:20.106886 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.106843 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888760-wkrkz" event={"ID":"32354403-56a7-4f01-a228-122805b44ba0","Type":"ContainerDied","Data":"224f90d1755c20fbf5f1580d3975caffe0bba7adc0f4518dc85c0f3526033d4b"} Oct 30 02:00:20.107063 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.106924 6572 scope.go:122] "RemoveContainer" containerID="6f15c3365220a63beb0b815cd73bd14ef816500d5817e83084ad271c15b5b215" Oct 30 02:00:20.107063 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.107032 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888760-wkrkz" Oct 30 02:00:20.113910 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.113875 6572 generic.go:356] "Generic (PLEG): container finished" podID="f818d22e-27ff-4caa-a48d-352af6583de4" containerID="08737f497b2d983f762cd7df00c78c07c2014adaa0abd5706b21e9ff768e4c4a" exitCode=60 Oct 30 02:00:20.114038 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.113974 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888760-qczr8" event={"ID":"f818d22e-27ff-4caa-a48d-352af6583de4","Type":"ContainerDied","Data":"08737f497b2d983f762cd7df00c78c07c2014adaa0abd5706b21e9ff768e4c4a"} Oct 30 02:00:20.114432 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.114408 6572 scope.go:122] "RemoveContainer" containerID="08737f497b2d983f762cd7df00c78c07c2014adaa0abd5706b21e9ff768e4c4a" Oct 30 02:00:20.114904 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:20.114826 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888760-qczr8_odh-ai-gateway-infra(f818d22e-27ff-4caa-a48d-352af6583de4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888760-qczr8" podUID="f818d22e-27ff-4caa-a48d-352af6583de4" Oct 30 02:00:20.118968 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.118884 6572 generic.go:356] "Generic (PLEG): container finished" podID="ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c" containerID="c75763091396844c9163dceb400dcf5b6d38521579f2d18b1da45e8954d18638" exitCode=60 Oct 30 02:00:20.118968 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.118955 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888760-k4ntx" event={"ID":"ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c","Type":"ContainerDied","Data":"c75763091396844c9163dceb400dcf5b6d38521579f2d18b1da45e8954d18638"} Oct 30 02:00:20.119458 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.119434 6572 scope.go:122] "RemoveContainer" containerID="c75763091396844c9163dceb400dcf5b6d38521579f2d18b1da45e8954d18638" Oct 30 02:00:20.120039 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:20.119799 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-558327-29888760-k4ntx_odh-ai-gateway-infra(ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888760-k4ntx" podUID="ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c" Oct 30 02:00:20.121826 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.121709 6572 generic.go:356] "Generic (PLEG): container finished" podID="419770d0-51af-4843-9ab2-8cbbe8215357" containerID="9a34470cbc34e4a2868d5cdeafe1052054730841d5309098780b32119f52b52b" exitCode=60 Oct 30 02:00:20.121895 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.121858 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888760-5h4tv" event={"ID":"419770d0-51af-4843-9ab2-8cbbe8215357","Type":"ContainerDied","Data":"9a34470cbc34e4a2868d5cdeafe1052054730841d5309098780b32119f52b52b"} Oct 30 02:00:20.122704 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.122287 6572 scope.go:122] "RemoveContainer" containerID="9a34470cbc34e4a2868d5cdeafe1052054730841d5309098780b32119f52b52b" Oct 30 02:00:20.122704 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:20.122634 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888760-5h4tv_odh-ai-gateway-infra(419770d0-51af-4843-9ab2-8cbbe8215357)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888760-5h4tv" podUID="419770d0-51af-4843-9ab2-8cbbe8215357" Oct 30 02:00:20.222274 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.222226 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888760-z8976" event={"ID":"dff6267c-a0a8-4d35-badf-3032bfb3e979","Type":"ContainerDied","Data":"fc4b9409cbac951e81f66e021b0ea63eb985800948b48a82f5e8feb5b6f70413"} Oct 30 02:00:20.223541 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.223515 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:20.223816 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.223566 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888760-z8976" Oct 30 02:00:20.249835 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.232593 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888760-j979j"] Oct 30 02:00:20.268166 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.261516 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888760-lnmg5" event={"ID":"2956c338-3b66-48aa-b3cc-521dbdef04ce","Type":"ContainerDied","Data":"969b4bfaf990edc19893d6ece9ab06ab08d440c270f2b34d2bf2a1502bc9842b"} Oct 30 02:00:20.268166 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.261650 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888760-lnmg5" Oct 30 02:00:20.306428 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.304595 6572 generic.go:356] "Generic (PLEG): container finished" podID="ea84dc40-9569-4a87-a537-a8e0c0fb119d" containerID="448b6cf2427529aecb68ad76e2cb583a19f749ea5475f0c012084ea3f0d6ee24" exitCode=60 Oct 30 02:00:20.306428 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.304740 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888760-hvkkt" event={"ID":"ea84dc40-9569-4a87-a537-a8e0c0fb119d","Type":"ContainerDied","Data":"448b6cf2427529aecb68ad76e2cb583a19f749ea5475f0c012084ea3f0d6ee24"} Oct 30 02:00:20.306428 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.305294 6572 scope.go:122] "RemoveContainer" containerID="448b6cf2427529aecb68ad76e2cb583a19f749ea5475f0c012084ea3f0d6ee24" Oct 30 02:00:20.306428 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:20.305722 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888760-hvkkt_odh-ai-gateway-infra(ea84dc40-9569-4a87-a537-a8e0c0fb119d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888760-hvkkt" podUID="ea84dc40-9569-4a87-a537-a8e0c0fb119d" Oct 30 02:00:20.363773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.358402 6572 scope.go:122] "RemoveContainer" containerID="6a722909d05e844bddde7b8f6eb001aa8a0ba7d690948a707eeb9e600264a12e" Oct 30 02:00:20.390015 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.389973 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888760-btnnt"] Oct 30 02:00:20.415846 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.414487 6572 scope.go:122] "RemoveContainer" containerID="a0004f97efb7d21fd811411afb300589e3722b1cb0b6c31a03d8db0371854b1f" Oct 30 02:00:20.421046 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.420872 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:20.421046 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.421021 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298"] Oct 30 02:00:20.445545 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.442036 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888760-m68hl"] Oct 30 02:00:20.455427 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.455402 6572 scope.go:122] "RemoveContainer" containerID="b533070cefc7fbeb5246f2601194c6317be5034aea4ae85f2935ba9de8f26c28" Oct 30 02:00:20.494441 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.494351 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888760-wkrkz"] Oct 30 02:00:20.546637 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.538221 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888760-wkrkz"] Oct 30 02:00:20.553320 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.553294 6572 scope.go:122] "RemoveContainer" containerID="c99f99d79c6618d97b53b2f8f1f193e13f0f3ff5fae0dc5006b2e3a4f578dd31" Oct 30 02:00:20.554243 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.554226 6572 scope.go:122] "RemoveContainer" containerID="8801fe0095f947337dc65df5994bacac5f5e8762b8ad984070e0f7a19b3448bb" Oct 30 02:00:20.555386 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.555370 6572 scope.go:122] "RemoveContainer" containerID="f73df7b446098ac9a80890f69a0a6a6942226e9a5e1c714291020f300ee6dbf9" Oct 30 02:00:20.612311 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.609540 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32354403-56a7-4f01-a228-122805b44ba0" path="/var/lib/kubelet/pods/32354403-56a7-4f01-a228-122805b44ba0/volumes" Oct 30 02:00:20.790765 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.761795 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888760-z8976"] Oct 30 02:00:20.886679 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.886623 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888760-z8976"] Oct 30 02:00:20.920490 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.920256 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888760-lnmg5"] Oct 30 02:00:20.937077 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.932555 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888760-lnmg5"] Oct 30 02:00:20.944112 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.942979 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c12916d7-2cprr"] Oct 30 02:00:20.944923 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.944440 6572 cpu_manager.go:414] "containerMap: removing container" podUID="dff6267c-a0a8-4d35-badf-3032bfb3e979" containerName="cleanup" Oct 30 02:00:20.944923 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.944465 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dff6267c-a0a8-4d35-badf-3032bfb3e979" containerName="cleanup" Oct 30 02:00:20.944923 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.944489 6572 cpu_manager.go:414] "containerMap: removing container" podUID="32354403-56a7-4f01-a228-122805b44ba0" containerName="cleanup" Oct 30 02:00:20.944923 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.944496 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32354403-56a7-4f01-a228-122805b44ba0" containerName="cleanup" Oct 30 02:00:20.944923 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.944541 6572 cpu_manager.go:414] "containerMap: removing container" podUID="32354403-56a7-4f01-a228-122805b44ba0" containerName="cleanup" Oct 30 02:00:20.944923 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.944549 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32354403-56a7-4f01-a228-122805b44ba0" containerName="cleanup" Oct 30 02:00:20.944923 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.944560 6572 cpu_manager.go:414] "containerMap: removing container" podUID="2956c338-3b66-48aa-b3cc-521dbdef04ce" containerName="cleanup" Oct 30 02:00:20.944923 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.944566 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2956c338-3b66-48aa-b3cc-521dbdef04ce" containerName="cleanup" Oct 30 02:00:20.944923 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.944591 6572 cpu_manager.go:414] "containerMap: removing container" podUID="32354403-56a7-4f01-a228-122805b44ba0" containerName="cleanup" Oct 30 02:00:20.944923 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.944600 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32354403-56a7-4f01-a228-122805b44ba0" containerName="cleanup" Oct 30 02:00:20.944923 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.944624 6572 cpu_manager.go:414] "containerMap: removing container" podUID="dff6267c-a0a8-4d35-badf-3032bfb3e979" containerName="cleanup" Oct 30 02:00:20.944923 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.944631 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dff6267c-a0a8-4d35-badf-3032bfb3e979" containerName="cleanup" Oct 30 02:00:20.944923 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.944643 6572 cpu_manager.go:414] "containerMap: removing container" podUID="2956c338-3b66-48aa-b3cc-521dbdef04ce" containerName="cleanup" Oct 30 02:00:20.944923 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.944650 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2956c338-3b66-48aa-b3cc-521dbdef04ce" containerName="cleanup" Oct 30 02:00:20.944923 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.944666 6572 cpu_manager.go:414] "containerMap: removing container" podUID="2956c338-3b66-48aa-b3cc-521dbdef04ce" containerName="cleanup" Oct 30 02:00:20.944923 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.944673 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2956c338-3b66-48aa-b3cc-521dbdef04ce" containerName="cleanup" Oct 30 02:00:20.944923 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.944894 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="32354403-56a7-4f01-a228-122805b44ba0" containerName="cleanup" Oct 30 02:00:20.944923 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.944911 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="dff6267c-a0a8-4d35-badf-3032bfb3e979" containerName="cleanup" Oct 30 02:00:20.944923 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.944922 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="dff6267c-a0a8-4d35-badf-3032bfb3e979" containerName="cleanup" Oct 30 02:00:20.944923 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.944931 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="2956c338-3b66-48aa-b3cc-521dbdef04ce" containerName="cleanup" Oct 30 02:00:20.946048 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.944941 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="2956c338-3b66-48aa-b3cc-521dbdef04ce" containerName="cleanup" Oct 30 02:00:20.946048 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.944959 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="dff6267c-a0a8-4d35-badf-3032bfb3e979" containerName="cleanup" Oct 30 02:00:20.946048 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.944973 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="32354403-56a7-4f01-a228-122805b44ba0" containerName="cleanup" Oct 30 02:00:20.967038 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.964944 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c12916d7-2cprr"] Oct 30 02:00:20.967038 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.965104 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c12916d7-2cprr" Oct 30 02:00:20.967038 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.965579 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s"] Oct 30 02:00:20.979984 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.979947 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:00:20.986446 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.986258 6572 scope.go:122] "RemoveContainer" containerID="d691202ed78cbfff7334feb918c2992df87ed55da968b6284c2ed7ac8e59d550" Oct 30 02:00:20.996939 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:20.996847 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s" Oct 30 02:00:21.036185 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.036086 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888760-j979j" Oct 30 02:00:21.088322 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.088177 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:21.090378 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.090228 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67qq7\" (UniqueName: \"kubernetes.io/projected/15e59e08-4ed8-4206-80c7-244b505243ae-kube-api-access-67qq7\") pod \"maas-api-revoke-sub-c12916d7-2cprr\" (UID: \"15e59e08-4ed8-4206-80c7-244b505243ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c12916d7-2cprr" Oct 30 02:00:21.090378 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.090369 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15e59e08-4ed8-4206-80c7-244b505243ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c12916d7-2cprr\" (UID: \"15e59e08-4ed8-4206-80c7-244b505243ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c12916d7-2cprr" Oct 30 02:00:21.090879 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.090627 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15e59e08-4ed8-4206-80c7-244b505243ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c12916d7-2cprr\" (UID: \"15e59e08-4ed8-4206-80c7-244b505243ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c12916d7-2cprr" Oct 30 02:00:21.159994 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.159867 6572 scope.go:122] "RemoveContainer" containerID="7945297f774a9cac967f628c6177e8e6610feb1ec5114975ee2e000ee361ac9c" Oct 30 02:00:21.189962 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.189935 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298" Oct 30 02:00:21.202898 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.202868 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888760-btnnt" Oct 30 02:00:21.205908 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.205883 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888760-m68hl" Oct 30 02:00:21.213381 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.213350 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc3321aa-698f-4c74-94ae-95e8729d9f27-kube-api-access-kmvqr\" (UniqueName: \"kubernetes.io/projected/bc3321aa-698f-4c74-94ae-95e8729d9f27-kube-api-access-kmvqr\") pod \"bc3321aa-698f-4c74-94ae-95e8729d9f27\" (UID: \"bc3321aa-698f-4c74-94ae-95e8729d9f27\") " Oct 30 02:00:21.213510 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.213474 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc3321aa-698f-4c74-94ae-95e8729d9f27-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc3321aa-698f-4c74-94ae-95e8729d9f27-maas-api-service-ca\") pod \"bc3321aa-698f-4c74-94ae-95e8729d9f27\" (UID: \"bc3321aa-698f-4c74-94ae-95e8729d9f27\") " Oct 30 02:00:21.213906 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.213880 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15e59e08-4ed8-4206-80c7-244b505243ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c12916d7-2cprr\" (UID: \"15e59e08-4ed8-4206-80c7-244b505243ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c12916d7-2cprr" Oct 30 02:00:21.214007 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.213881 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc3321aa-698f-4c74-94ae-95e8729d9f27-maas-api-service-ca" pod "bc3321aa-698f-4c74-94ae-95e8729d9f27" (UID: "bc3321aa-698f-4c74-94ae-95e8729d9f27"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:21.214063 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.214015 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67qq7\" (UniqueName: \"kubernetes.io/projected/15e59e08-4ed8-4206-80c7-244b505243ae-kube-api-access-67qq7\") pod \"maas-api-revoke-sub-c12916d7-2cprr\" (UID: \"15e59e08-4ed8-4206-80c7-244b505243ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c12916d7-2cprr" Oct 30 02:00:21.214161 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.214144 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15e59e08-4ed8-4206-80c7-244b505243ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c12916d7-2cprr\" (UID: \"15e59e08-4ed8-4206-80c7-244b505243ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c12916d7-2cprr" Oct 30 02:00:21.214440 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.214418 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc3321aa-698f-4c74-94ae-95e8729d9f27-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:21.214585 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.214567 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15e59e08-4ed8-4206-80c7-244b505243ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c12916d7-2cprr\" (UID: \"15e59e08-4ed8-4206-80c7-244b505243ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c12916d7-2cprr" Oct 30 02:00:21.215760 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.215720 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc3321aa-698f-4c74-94ae-95e8729d9f27-kube-api-access-kmvqr" pod "bc3321aa-698f-4c74-94ae-95e8729d9f27" (UID: "bc3321aa-698f-4c74-94ae-95e8729d9f27"). InnerVolumeSpecName "kube-api-access-kmvqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:21.216710 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.216687 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15e59e08-4ed8-4206-80c7-244b505243ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c12916d7-2cprr\" (UID: \"15e59e08-4ed8-4206-80c7-244b505243ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c12916d7-2cprr" Oct 30 02:00:21.252825 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.252701 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888760-k4ntx"] Oct 30 02:00:21.257146 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.257102 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67qq7\" (UniqueName: \"kubernetes.io/projected/15e59e08-4ed8-4206-80c7-244b505243ae-kube-api-access-67qq7\") pod \"maas-api-revoke-sub-c12916d7-2cprr\" (UID: \"15e59e08-4ed8-4206-80c7-244b505243ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c12916d7-2cprr" Oct 30 02:00:21.292900 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.292863 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888760-qczr8"] Oct 30 02:00:21.326062 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.326022 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad3ca000-d9f7-4d97-9a05-b39362550ba0-kube-api-access-6xmxt\" (UniqueName: \"kubernetes.io/projected/ad3ca000-d9f7-4d97-9a05-b39362550ba0-kube-api-access-6xmxt\") pod \"ad3ca000-d9f7-4d97-9a05-b39362550ba0\" (UID: \"ad3ca000-d9f7-4d97-9a05-b39362550ba0\") " Oct 30 02:00:21.326405 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.326382 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad3ca000-d9f7-4d97-9a05-b39362550ba0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad3ca000-d9f7-4d97-9a05-b39362550ba0-maas-api-service-ca\") pod \"ad3ca000-d9f7-4d97-9a05-b39362550ba0\" (UID: \"ad3ca000-d9f7-4d97-9a05-b39362550ba0\") " Oct 30 02:00:21.326612 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.326595 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/228bfbbd-21b3-4f89-b247-962811a659e6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/228bfbbd-21b3-4f89-b247-962811a659e6-maas-api-service-ca\") pod \"228bfbbd-21b3-4f89-b247-962811a659e6\" (UID: \"228bfbbd-21b3-4f89-b247-962811a659e6\") " Oct 30 02:00:21.326836 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.326817 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc58a71f-7f5f-4f68-aa97-a22aa15b854e-kube-api-access-tftl5\" (UniqueName: \"kubernetes.io/projected/dc58a71f-7f5f-4f68-aa97-a22aa15b854e-kube-api-access-tftl5\") pod \"dc58a71f-7f5f-4f68-aa97-a22aa15b854e\" (UID: \"dc58a71f-7f5f-4f68-aa97-a22aa15b854e\") " Oct 30 02:00:21.327021 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.327005 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/228bfbbd-21b3-4f89-b247-962811a659e6-kube-api-access-7mj5k\" (UniqueName: \"kubernetes.io/projected/228bfbbd-21b3-4f89-b247-962811a659e6-kube-api-access-7mj5k\") pod \"228bfbbd-21b3-4f89-b247-962811a659e6\" (UID: \"228bfbbd-21b3-4f89-b247-962811a659e6\") " Oct 30 02:00:21.327247 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.327229 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc58a71f-7f5f-4f68-aa97-a22aa15b854e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc58a71f-7f5f-4f68-aa97-a22aa15b854e-maas-api-service-ca\") pod \"dc58a71f-7f5f-4f68-aa97-a22aa15b854e\" (UID: \"dc58a71f-7f5f-4f68-aa97-a22aa15b854e\") " Oct 30 02:00:21.328660 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.328631 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmvqr\" (UniqueName: \"kubernetes.io/projected/bc3321aa-698f-4c74-94ae-95e8729d9f27-kube-api-access-kmvqr\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:21.329698 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.329664 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/228bfbbd-21b3-4f89-b247-962811a659e6-maas-api-service-ca" pod "228bfbbd-21b3-4f89-b247-962811a659e6" (UID: "228bfbbd-21b3-4f89-b247-962811a659e6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:21.330166 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.330142 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad3ca000-d9f7-4d97-9a05-b39362550ba0-maas-api-service-ca" pod "ad3ca000-d9f7-4d97-9a05-b39362550ba0" (UID: "ad3ca000-d9f7-4d97-9a05-b39362550ba0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:21.335289 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.335247 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/228bfbbd-21b3-4f89-b247-962811a659e6-kube-api-access-7mj5k" pod "228bfbbd-21b3-4f89-b247-962811a659e6" (UID: "228bfbbd-21b3-4f89-b247-962811a659e6"). InnerVolumeSpecName "kube-api-access-7mj5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:21.337600 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.337411 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc58a71f-7f5f-4f68-aa97-a22aa15b854e-maas-api-service-ca" pod "dc58a71f-7f5f-4f68-aa97-a22aa15b854e" (UID: "dc58a71f-7f5f-4f68-aa97-a22aa15b854e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:21.347897 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.347807 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad3ca000-d9f7-4d97-9a05-b39362550ba0-kube-api-access-6xmxt" pod "ad3ca000-d9f7-4d97-9a05-b39362550ba0" (UID: "ad3ca000-d9f7-4d97-9a05-b39362550ba0"). InnerVolumeSpecName "kube-api-access-6xmxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:21.353091 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.353047 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc58a71f-7f5f-4f68-aa97-a22aa15b854e-kube-api-access-tftl5" pod "dc58a71f-7f5f-4f68-aa97-a22aa15b854e" (UID: "dc58a71f-7f5f-4f68-aa97-a22aa15b854e"). InnerVolumeSpecName "kube-api-access-tftl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:21.356652 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.356619 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888760-5h4tv"] Oct 30 02:00:21.373472 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.373431 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888760-btnnt" event={"ID":"dc58a71f-7f5f-4f68-aa97-a22aa15b854e","Type":"ContainerDied","Data":"1e120c094ddf77870464de59f7029d72e1e7626451526bc1fa3ed904459eb6ca"} Oct 30 02:00:21.373472 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.373455 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888760-btnnt" Oct 30 02:00:21.373784 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.373497 6572 scope.go:122] "RemoveContainer" containerID="cd7a5a5205b70ca54435db09d40617d7b2b4c16ed1c7b52157aad55f5d137470" Oct 30 02:00:21.376231 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.376198 6572 generic.go:356] "Generic (PLEG): container finished" podID="d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b" containerID="b1c6d7e1349f97579b1746a25b8c9420fbce2becce1d4ee7c5f82f0efa97df6a" exitCode=60 Oct 30 02:00:21.376372 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.376319 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888760-6h8kw" event={"ID":"d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b","Type":"ContainerDied","Data":"b1c6d7e1349f97579b1746a25b8c9420fbce2becce1d4ee7c5f82f0efa97df6a"} Oct 30 02:00:21.376874 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.376850 6572 scope.go:122] "RemoveContainer" containerID="b1c6d7e1349f97579b1746a25b8c9420fbce2becce1d4ee7c5f82f0efa97df6a" Oct 30 02:00:21.377313 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:21.377270 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888760-6h8kw_odh-ai-gateway-infra(d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888760-6h8kw" podUID="d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b" Oct 30 02:00:21.383122 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.383093 6572 generic.go:356] "Generic (PLEG): container finished" podID="16ce2d50-2777-4827-80ec-025ff18072ed" containerID="b262c1ec3695e5daa8e3e5c89cee0d490e64475f94ea779d56f406e33508a398" exitCode=60 Oct 30 02:00:21.383484 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.383453 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888760-545xr" event={"ID":"16ce2d50-2777-4827-80ec-025ff18072ed","Type":"ContainerDied","Data":"b262c1ec3695e5daa8e3e5c89cee0d490e64475f94ea779d56f406e33508a398"} Oct 30 02:00:21.384286 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.384271 6572 scope.go:122] "RemoveContainer" containerID="b262c1ec3695e5daa8e3e5c89cee0d490e64475f94ea779d56f406e33508a398" Oct 30 02:00:21.384732 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:21.384715 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888760-545xr_odh-ai-gateway-infra(16ce2d50-2777-4827-80ec-025ff18072ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888760-545xr" podUID="16ce2d50-2777-4827-80ec-025ff18072ed" Oct 30 02:00:21.389240 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.389219 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298" Oct 30 02:00:21.390649 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.389436 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298" event={"ID":"ad3ca000-d9f7-4d97-9a05-b39362550ba0","Type":"ContainerDied","Data":"8b716c75d607574fef235f1a2604d0bc12726058674dc8dc26dc4930cb60c831"} Oct 30 02:00:21.390920 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.390730 6572 scope.go:122] "RemoveContainer" containerID="e26bfa123910ccde51549e589499fadc6a958c5ceae45035e29fd83115da0060" Oct 30 02:00:21.393455 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.393431 6572 generic.go:356] "Generic (PLEG): container finished" podID="45e5935e-478f-4fec-beae-0cae2ea14c1b" containerID="d0f7aae82673c5805fc09099177b4d02409e5147eba7f6347fdb94d966631ee7" exitCode=60 Oct 30 02:00:21.393677 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.393463 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888760-xhtnj" event={"ID":"45e5935e-478f-4fec-beae-0cae2ea14c1b","Type":"ContainerDied","Data":"d0f7aae82673c5805fc09099177b4d02409e5147eba7f6347fdb94d966631ee7"} Oct 30 02:00:21.394439 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.394418 6572 scope.go:122] "RemoveContainer" containerID="d0f7aae82673c5805fc09099177b4d02409e5147eba7f6347fdb94d966631ee7" Oct 30 02:00:21.395068 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:21.395013 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-76a19e-29888760-xhtnj_odh-ai-gateway-infra(45e5935e-478f-4fec-beae-0cae2ea14c1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888760-xhtnj" podUID="45e5935e-478f-4fec-beae-0cae2ea14c1b" Oct 30 02:00:21.398025 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.397996 6572 generic.go:356] "Generic (PLEG): container finished" podID="dbd7b9d4-eb19-4605-bdc9-21940dad8c92" containerID="2e77fe29b14324f4513950ed5c3a4a4a28bad9f88d4f02c6d284e98f039f0cd5" exitCode=60 Oct 30 02:00:21.398144 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.398028 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888760-dfspx" event={"ID":"dbd7b9d4-eb19-4605-bdc9-21940dad8c92","Type":"ContainerDied","Data":"2e77fe29b14324f4513950ed5c3a4a4a28bad9f88d4f02c6d284e98f039f0cd5"} Oct 30 02:00:21.399470 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.399441 6572 scope.go:122] "RemoveContainer" containerID="2e77fe29b14324f4513950ed5c3a4a4a28bad9f88d4f02c6d284e98f039f0cd5" Oct 30 02:00:21.399926 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:21.399895 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888760-dfspx_odh-ai-gateway-infra(dbd7b9d4-eb19-4605-bdc9-21940dad8c92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888760-dfspx" podUID="dbd7b9d4-eb19-4605-bdc9-21940dad8c92" Oct 30 02:00:21.407250 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.407220 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888760-j979j" event={"ID":"bc3321aa-698f-4c74-94ae-95e8729d9f27","Type":"ContainerDied","Data":"32823aabb63ee1ba008169b6f2af6dc2ce2f61d6c9c07fe75d719a2115befd89"} Oct 30 02:00:21.407250 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.407241 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888760-j979j" Oct 30 02:00:21.410736 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.410700 6572 generic.go:356] "Generic (PLEG): container finished" podID="05cb266a-e1fe-4b51-a18f-96d85a78ab16" containerID="148d1c60640f2d1e1d7d443ca6c2a1f813a58ac6ec04004c0dc7a2d56d86881a" exitCode=60 Oct 30 02:00:21.411021 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.410993 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888760-kwp4k" event={"ID":"05cb266a-e1fe-4b51-a18f-96d85a78ab16","Type":"ContainerDied","Data":"148d1c60640f2d1e1d7d443ca6c2a1f813a58ac6ec04004c0dc7a2d56d86881a"} Oct 30 02:00:21.415509 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.415478 6572 scope.go:122] "RemoveContainer" containerID="148d1c60640f2d1e1d7d443ca6c2a1f813a58ac6ec04004c0dc7a2d56d86881a" Oct 30 02:00:21.416102 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:21.415965 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888760-kwp4k_odh-ai-gateway-infra(05cb266a-e1fe-4b51-a18f-96d85a78ab16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888760-kwp4k" podUID="05cb266a-e1fe-4b51-a18f-96d85a78ab16" Oct 30 02:00:21.419535 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.419508 6572 scope.go:122] "RemoveContainer" containerID="5a27bddcd7b87b779e0da47909c1b6db34fe0d3f1bd22389da15918a6b9ebade" Oct 30 02:00:21.432795 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.432722 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/228bfbbd-21b3-4f89-b247-962811a659e6-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:21.432795 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.432781 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tftl5\" (UniqueName: \"kubernetes.io/projected/dc58a71f-7f5f-4f68-aa97-a22aa15b854e-kube-api-access-tftl5\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:21.432795 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.432800 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mj5k\" (UniqueName: \"kubernetes.io/projected/228bfbbd-21b3-4f89-b247-962811a659e6-kube-api-access-7mj5k\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:21.433093 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.432813 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc58a71f-7f5f-4f68-aa97-a22aa15b854e-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:21.433093 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.432828 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xmxt\" (UniqueName: \"kubernetes.io/projected/ad3ca000-d9f7-4d97-9a05-b39362550ba0-kube-api-access-6xmxt\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:21.433093 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.432841 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad3ca000-d9f7-4d97-9a05-b39362550ba0-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:21.434396 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.433840 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888760-hvkkt"] Oct 30 02:00:21.439619 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.439577 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888760-m68hl" event={"ID":"228bfbbd-21b3-4f89-b247-962811a659e6","Type":"ContainerDied","Data":"59a9b25a5a6067af42f28c3aaa6014335eec3037051baf002afbd08134b5b19d"} Oct 30 02:00:21.439821 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.439612 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888760-m68hl" Oct 30 02:00:21.460567 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.460545 6572 scope.go:122] "RemoveContainer" containerID="65f84e27d0953ceb2bcc066dcef193288d30e67322356ce32e7dc368ad911a97" Oct 30 02:00:21.482043 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.481966 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888760-btnnt"] Oct 30 02:00:21.487803 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.486852 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c12916d7-2cprr" Oct 30 02:00:21.493990 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.493955 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888760-btnnt"] Oct 30 02:00:21.497590 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.497476 6572 scope.go:122] "RemoveContainer" containerID="1e662ebe72d1fee4bc9b8ca9fa336dab1854a1d142a147cdf9ce430811d39534" Oct 30 02:00:21.525972 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.525207 6572 scope.go:122] "RemoveContainer" containerID="8939cae34cbc595709b337be49308d7a38faba8304ea3ed336874047e0a08770" Oct 30 02:00:21.583441 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.583416 6572 scope.go:122] "RemoveContainer" containerID="60e68186facd694ebacbcb41fdeb90a593ef4ff08eefc030711ce407d2ed8f08" Oct 30 02:00:21.609231 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.609201 6572 scope.go:122] "RemoveContainer" containerID="47c1389175e0ef1883f749a545a7b9e93b925273a24a210d8bef2456a195b6d3" Oct 30 02:00:21.628154 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.623893 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888760-m68hl"] Oct 30 02:00:21.640554 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.637485 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4560f025-tl8nm"] Oct 30 02:00:21.677230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.654608 6572 cpu_manager.go:414] "containerMap: removing container" podUID="bc3321aa-698f-4c74-94ae-95e8729d9f27" containerName="cleanup" Oct 30 02:00:21.677230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.654634 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc3321aa-698f-4c74-94ae-95e8729d9f27" containerName="cleanup" Oct 30 02:00:21.677230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.654664 6572 cpu_manager.go:414] "containerMap: removing container" podUID="ad3ca000-d9f7-4d97-9a05-b39362550ba0" containerName="cleanup" Oct 30 02:00:21.677230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.654671 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad3ca000-d9f7-4d97-9a05-b39362550ba0" containerName="cleanup" Oct 30 02:00:21.677230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.654688 6572 cpu_manager.go:414] "containerMap: removing container" podUID="bc3321aa-698f-4c74-94ae-95e8729d9f27" containerName="cleanup" Oct 30 02:00:21.677230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.654693 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc3321aa-698f-4c74-94ae-95e8729d9f27" containerName="cleanup" Oct 30 02:00:21.677230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.654734 6572 cpu_manager.go:414] "containerMap: removing container" podUID="228bfbbd-21b3-4f89-b247-962811a659e6" containerName="cleanup" Oct 30 02:00:21.677230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.654739 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="228bfbbd-21b3-4f89-b247-962811a659e6" containerName="cleanup" Oct 30 02:00:21.677230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.654774 6572 cpu_manager.go:414] "containerMap: removing container" podUID="228bfbbd-21b3-4f89-b247-962811a659e6" containerName="cleanup" Oct 30 02:00:21.677230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.654780 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="228bfbbd-21b3-4f89-b247-962811a659e6" containerName="cleanup" Oct 30 02:00:21.677230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.654792 6572 cpu_manager.go:414] "containerMap: removing container" podUID="ad3ca000-d9f7-4d97-9a05-b39362550ba0" containerName="cleanup" Oct 30 02:00:21.677230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.654800 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad3ca000-d9f7-4d97-9a05-b39362550ba0" containerName="cleanup" Oct 30 02:00:21.677230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.654810 6572 cpu_manager.go:414] "containerMap: removing container" podUID="dc58a71f-7f5f-4f68-aa97-a22aa15b854e" containerName="cleanup" Oct 30 02:00:21.677230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.654817 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc58a71f-7f5f-4f68-aa97-a22aa15b854e" containerName="cleanup" Oct 30 02:00:21.677230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.654865 6572 cpu_manager.go:414] "containerMap: removing container" podUID="228bfbbd-21b3-4f89-b247-962811a659e6" containerName="cleanup" Oct 30 02:00:21.677230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.654873 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="228bfbbd-21b3-4f89-b247-962811a659e6" containerName="cleanup" Oct 30 02:00:21.677230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.654899 6572 cpu_manager.go:414] "containerMap: removing container" podUID="dc58a71f-7f5f-4f68-aa97-a22aa15b854e" containerName="cleanup" Oct 30 02:00:21.677230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.654907 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc58a71f-7f5f-4f68-aa97-a22aa15b854e" containerName="cleanup" Oct 30 02:00:21.677230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.654966 6572 cpu_manager.go:414] "containerMap: removing container" podUID="dff6267c-a0a8-4d35-badf-3032bfb3e979" containerName="cleanup" Oct 30 02:00:21.677230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.654974 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dff6267c-a0a8-4d35-badf-3032bfb3e979" containerName="cleanup" Oct 30 02:00:21.677230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.655407 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="2956c338-3b66-48aa-b3cc-521dbdef04ce" containerName="cleanup" Oct 30 02:00:21.677230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.655431 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc3321aa-698f-4c74-94ae-95e8729d9f27" containerName="cleanup" Oct 30 02:00:21.677230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.655439 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad3ca000-d9f7-4d97-9a05-b39362550ba0" containerName="cleanup" Oct 30 02:00:21.677230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.655451 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc3321aa-698f-4c74-94ae-95e8729d9f27" containerName="cleanup" Oct 30 02:00:21.677230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.655461 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="228bfbbd-21b3-4f89-b247-962811a659e6" containerName="cleanup" Oct 30 02:00:21.677230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.655476 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="228bfbbd-21b3-4f89-b247-962811a659e6" containerName="cleanup" Oct 30 02:00:21.677230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.655482 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad3ca000-d9f7-4d97-9a05-b39362550ba0" containerName="cleanup" Oct 30 02:00:21.677230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.655491 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc58a71f-7f5f-4f68-aa97-a22aa15b854e" containerName="cleanup" Oct 30 02:00:21.677230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.655499 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="228bfbbd-21b3-4f89-b247-962811a659e6" containerName="cleanup" Oct 30 02:00:21.677230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.655507 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc58a71f-7f5f-4f68-aa97-a22aa15b854e" containerName="cleanup" Oct 30 02:00:21.677230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.655517 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="32354403-56a7-4f01-a228-122805b44ba0" containerName="cleanup" Oct 30 02:00:21.691345 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.691306 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888760-m68hl"] Oct 30 02:00:21.691542 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.691368 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4560f025-tl8nm"] Oct 30 02:00:21.691542 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.691492 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4560f025-tl8nm" Oct 30 02:00:21.709238 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.708874 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888760-qczr8" Oct 30 02:00:21.710796 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.710624 6572 scope.go:122] "RemoveContainer" containerID="6464156bb5fa6011f863d7851d003914b7a15e7f1341cfd85e0e2ca0ddfdba18" Oct 30 02:00:21.743155 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.743119 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f818d22e-27ff-4caa-a48d-352af6583de4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f818d22e-27ff-4caa-a48d-352af6583de4-maas-api-service-ca\") pod \"f818d22e-27ff-4caa-a48d-352af6583de4\" (UID: \"f818d22e-27ff-4caa-a48d-352af6583de4\") " Oct 30 02:00:21.743421 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.743400 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f818d22e-27ff-4caa-a48d-352af6583de4-kube-api-access-7gxdh\" (UniqueName: \"kubernetes.io/projected/f818d22e-27ff-4caa-a48d-352af6583de4-kube-api-access-7gxdh\") pod \"f818d22e-27ff-4caa-a48d-352af6583de4\" (UID: \"f818d22e-27ff-4caa-a48d-352af6583de4\") " Oct 30 02:00:21.744351 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.744320 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/250d2557-a543-4749-aa8c-6483de133989-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4560f025-tl8nm\" (UID: \"250d2557-a543-4749-aa8c-6483de133989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4560f025-tl8nm" Oct 30 02:00:21.744539 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.744522 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/250d2557-a543-4749-aa8c-6483de133989-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4560f025-tl8nm\" (UID: \"250d2557-a543-4749-aa8c-6483de133989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4560f025-tl8nm" Oct 30 02:00:21.745016 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.744997 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cfbg\" (UniqueName: \"kubernetes.io/projected/250d2557-a543-4749-aa8c-6483de133989-kube-api-access-2cfbg\") pod \"maas-api-revoke-sub-4560f025-tl8nm\" (UID: \"250d2557-a543-4749-aa8c-6483de133989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4560f025-tl8nm" Oct 30 02:00:21.746798 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.746775 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f818d22e-27ff-4caa-a48d-352af6583de4-maas-api-service-ca" pod "f818d22e-27ff-4caa-a48d-352af6583de4" (UID: "f818d22e-27ff-4caa-a48d-352af6583de4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:21.752999 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.752969 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f818d22e-27ff-4caa-a48d-352af6583de4-kube-api-access-7gxdh" pod "f818d22e-27ff-4caa-a48d-352af6583de4" (UID: "f818d22e-27ff-4caa-a48d-352af6583de4"). InnerVolumeSpecName "kube-api-access-7gxdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:21.776373 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.776349 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888760-hvkkt" Oct 30 02:00:21.781489 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.781466 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888760-k4ntx" Oct 30 02:00:21.796948 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.796915 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888760-j979j"] Oct 30 02:00:21.804659 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.804620 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888760-j979j"] Oct 30 02:00:21.847209 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.847175 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea84dc40-9569-4a87-a537-a8e0c0fb119d-kube-api-access-jft2l\" (UniqueName: \"kubernetes.io/projected/ea84dc40-9569-4a87-a537-a8e0c0fb119d-kube-api-access-jft2l\") pod \"ea84dc40-9569-4a87-a537-a8e0c0fb119d\" (UID: \"ea84dc40-9569-4a87-a537-a8e0c0fb119d\") " Oct 30 02:00:21.847435 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.847415 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea84dc40-9569-4a87-a537-a8e0c0fb119d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea84dc40-9569-4a87-a537-a8e0c0fb119d-maas-api-service-ca\") pod \"ea84dc40-9569-4a87-a537-a8e0c0fb119d\" (UID: \"ea84dc40-9569-4a87-a537-a8e0c0fb119d\") " Oct 30 02:00:21.847594 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.847545 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c-maas-api-service-ca\") pod \"ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c\" (UID: \"ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c\") " Oct 30 02:00:21.847689 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.847623 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c-kube-api-access-zw5bp\" (UniqueName: \"kubernetes.io/projected/ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c-kube-api-access-zw5bp\") pod \"ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c\" (UID: \"ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c\") " Oct 30 02:00:21.848060 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.848039 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cfbg\" (UniqueName: \"kubernetes.io/projected/250d2557-a543-4749-aa8c-6483de133989-kube-api-access-2cfbg\") pod \"maas-api-revoke-sub-4560f025-tl8nm\" (UID: \"250d2557-a543-4749-aa8c-6483de133989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4560f025-tl8nm" Oct 30 02:00:21.848195 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.848181 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/250d2557-a543-4749-aa8c-6483de133989-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4560f025-tl8nm\" (UID: \"250d2557-a543-4749-aa8c-6483de133989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4560f025-tl8nm" Oct 30 02:00:21.848257 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.848234 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/250d2557-a543-4749-aa8c-6483de133989-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4560f025-tl8nm\" (UID: \"250d2557-a543-4749-aa8c-6483de133989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4560f025-tl8nm" Oct 30 02:00:21.848360 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.848347 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f818d22e-27ff-4caa-a48d-352af6583de4-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:21.848432 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.848366 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gxdh\" (UniqueName: \"kubernetes.io/projected/f818d22e-27ff-4caa-a48d-352af6583de4-kube-api-access-7gxdh\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:21.852830 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.852784 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/250d2557-a543-4749-aa8c-6483de133989-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4560f025-tl8nm\" (UID: \"250d2557-a543-4749-aa8c-6483de133989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4560f025-tl8nm" Oct 30 02:00:21.854401 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.854231 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c-kube-api-access-zw5bp" pod "ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c" (UID: "ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c"). InnerVolumeSpecName "kube-api-access-zw5bp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:21.854543 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.854408 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/250d2557-a543-4749-aa8c-6483de133989-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4560f025-tl8nm\" (UID: \"250d2557-a543-4749-aa8c-6483de133989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4560f025-tl8nm" Oct 30 02:00:21.854896 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.854711 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c-maas-api-service-ca" pod "ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c" (UID: "ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:21.854896 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.854720 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea84dc40-9569-4a87-a537-a8e0c0fb119d-maas-api-service-ca" pod "ea84dc40-9569-4a87-a537-a8e0c0fb119d" (UID: "ea84dc40-9569-4a87-a537-a8e0c0fb119d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:21.854896 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.854874 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea84dc40-9569-4a87-a537-a8e0c0fb119d-kube-api-access-jft2l" pod "ea84dc40-9569-4a87-a537-a8e0c0fb119d" (UID: "ea84dc40-9569-4a87-a537-a8e0c0fb119d"). InnerVolumeSpecName "kube-api-access-jft2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:21.872868 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.872780 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cfbg\" (UniqueName: \"kubernetes.io/projected/250d2557-a543-4749-aa8c-6483de133989-kube-api-access-2cfbg\") pod \"maas-api-revoke-sub-4560f025-tl8nm\" (UID: \"250d2557-a543-4749-aa8c-6483de133989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4560f025-tl8nm" Oct 30 02:00:21.889980 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.889946 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888760-5h4tv" Oct 30 02:00:21.928586 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.928552 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c12916d7-2cprr"] Oct 30 02:00:21.929086 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:00:21.929060 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15e59e08_4ed8_4206_80c7_244b505243ae.slice/crio-3fa8fd3f49d52aa07fbc6a96a09aac9f6bb242cb0745e23a25666bdce035e9d7 WatchSource:0}: Error finding container 3fa8fd3f49d52aa07fbc6a96a09aac9f6bb242cb0745e23a25666bdce035e9d7: Status 404 returned error can't find the container with id 3fa8fd3f49d52aa07fbc6a96a09aac9f6bb242cb0745e23a25666bdce035e9d7 Oct 30 02:00:21.949590 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.949556 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/419770d0-51af-4843-9ab2-8cbbe8215357-kube-api-access-6m8sn\" (UniqueName: \"kubernetes.io/projected/419770d0-51af-4843-9ab2-8cbbe8215357-kube-api-access-6m8sn\") pod \"419770d0-51af-4843-9ab2-8cbbe8215357\" (UID: \"419770d0-51af-4843-9ab2-8cbbe8215357\") " Oct 30 02:00:21.949805 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.949638 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/419770d0-51af-4843-9ab2-8cbbe8215357-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/419770d0-51af-4843-9ab2-8cbbe8215357-maas-api-service-ca\") pod \"419770d0-51af-4843-9ab2-8cbbe8215357\" (UID: \"419770d0-51af-4843-9ab2-8cbbe8215357\") " Oct 30 02:00:21.950322 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.950295 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/419770d0-51af-4843-9ab2-8cbbe8215357-maas-api-service-ca" pod "419770d0-51af-4843-9ab2-8cbbe8215357" (UID: "419770d0-51af-4843-9ab2-8cbbe8215357"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:21.950551 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.950528 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jft2l\" (UniqueName: \"kubernetes.io/projected/ea84dc40-9569-4a87-a537-a8e0c0fb119d-kube-api-access-jft2l\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:21.950655 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.950555 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea84dc40-9569-4a87-a537-a8e0c0fb119d-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:21.950655 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.950571 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:21.950655 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.950582 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw5bp\" (UniqueName: \"kubernetes.io/projected/ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c-kube-api-access-zw5bp\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:21.950655 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.950591 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/419770d0-51af-4843-9ab2-8cbbe8215357-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:21.952506 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.952477 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/419770d0-51af-4843-9ab2-8cbbe8215357-kube-api-access-6m8sn" pod "419770d0-51af-4843-9ab2-8cbbe8215357" (UID: "419770d0-51af-4843-9ab2-8cbbe8215357"). InnerVolumeSpecName "kube-api-access-6m8sn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:21.961915 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.961884 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298"] Oct 30 02:00:21.969050 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:21.969016 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-817fd7-29888760-8b298"] Oct 30 02:00:22.049986 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.049927 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4560f025-tl8nm" Oct 30 02:00:22.052493 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.052457 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m8sn\" (UniqueName: \"kubernetes.io/projected/419770d0-51af-4843-9ab2-8cbbe8215357-kube-api-access-6m8sn\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:22.435009 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:00:22.434981 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod250d2557_a543_4749_aa8c_6483de133989.slice/crio-5ff12d1963ace89961cc2897abe7f48c7bd6f71657da40f5913b973bf9a47c84 WatchSource:0}: Error finding container 5ff12d1963ace89961cc2897abe7f48c7bd6f71657da40f5913b973bf9a47c84: Status 404 returned error can't find the container with id 5ff12d1963ace89961cc2897abe7f48c7bd6f71657da40f5913b973bf9a47c84 Oct 30 02:00:22.436296 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.436270 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4560f025-tl8nm"] Oct 30 02:00:22.444097 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.444071 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888760-6h8kw"] Oct 30 02:00:22.466027 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.466000 6572 generic.go:356] "Generic (PLEG): container finished" podID="9dc37d0b-7989-4437-9902-df0a34eb682a" containerID="6e0af7545ef3200cc8b919722eaa72982f749e9797e04c5b4aae3d4a4dea6940" exitCode=60 Oct 30 02:00:22.466156 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.466078 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2eae39-29888760-t6tz5" event={"ID":"9dc37d0b-7989-4437-9902-df0a34eb682a","Type":"ContainerDied","Data":"6e0af7545ef3200cc8b919722eaa72982f749e9797e04c5b4aae3d4a4dea6940"} Oct 30 02:00:22.466212 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.466155 6572 scope.go:122] "RemoveContainer" containerID="f73df7b446098ac9a80890f69a0a6a6942226e9a5e1c714291020f300ee6dbf9" Oct 30 02:00:22.466625 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.466604 6572 scope.go:122] "RemoveContainer" containerID="6e0af7545ef3200cc8b919722eaa72982f749e9797e04c5b4aae3d4a4dea6940" Oct 30 02:00:22.467015 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:22.466988 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2eae39-29888760-t6tz5_odh-ai-gateway-infra(9dc37d0b-7989-4437-9902-df0a34eb682a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2eae39-29888760-t6tz5" podUID="9dc37d0b-7989-4437-9902-df0a34eb682a" Oct 30 02:00:22.476251 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.476226 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888760-hvkkt" Oct 30 02:00:22.476366 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.476248 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888760-hvkkt" event={"ID":"ea84dc40-9569-4a87-a537-a8e0c0fb119d","Type":"ContainerDied","Data":"ec5c831b4661f2c60e6ce44377ac1de0e1ac8666f09d4a48daa332cc5a2529ac"} Oct 30 02:00:22.479237 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.479212 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4560f025-tl8nm" event={"ID":"250d2557-a543-4749-aa8c-6483de133989","Type":"ContainerStarted","Data":"5ff12d1963ace89961cc2897abe7f48c7bd6f71657da40f5913b973bf9a47c84"} Oct 30 02:00:22.483364 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.483338 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c12916d7-2cprr" event={"ID":"15e59e08-4ed8-4206-80c7-244b505243ae","Type":"ContainerStarted","Data":"a16cf0994fb06470a6c97f1c889e39c2d94b628d040081252a1c2ab55449209f"} Oct 30 02:00:22.483543 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.483388 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c12916d7-2cprr" event={"ID":"15e59e08-4ed8-4206-80c7-244b505243ae","Type":"ContainerStarted","Data":"3fa8fd3f49d52aa07fbc6a96a09aac9f6bb242cb0745e23a25666bdce035e9d7"} Oct 30 02:00:22.487511 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.487488 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888760-qczr8" event={"ID":"f818d22e-27ff-4caa-a48d-352af6583de4","Type":"ContainerDied","Data":"6b42e2e0100c1f49070c6b68b80c12949aa33533281852116058370f33e490d5"} Oct 30 02:00:22.487627 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.487491 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888760-qczr8" Oct 30 02:00:22.493517 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.493495 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888760-k4ntx" Oct 30 02:00:22.493517 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.493503 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888760-k4ntx" event={"ID":"ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c","Type":"ContainerDied","Data":"a2665ee7958ead93588c8fb2e74904526df5b41ee7806de5fe0f9d4f7bf67750"} Oct 30 02:00:22.495166 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.495146 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888760-5h4tv" Oct 30 02:00:22.495345 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.495170 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888760-5h4tv" event={"ID":"419770d0-51af-4843-9ab2-8cbbe8215357","Type":"ContainerDied","Data":"8fec76af3acc2551971fd9ca7e144653748ff1ed9bdb9ab3f330d2917ab6566a"} Oct 30 02:00:22.497433 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.497414 6572 scope.go:122] "RemoveContainer" containerID="448b6cf2427529aecb68ad76e2cb583a19f749ea5475f0c012084ea3f0d6ee24" Oct 30 02:00:22.512851 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.505574 6572 generic.go:356] "Generic (PLEG): container finished" podID="b8e5ee17-3f55-4488-aba3-2a81eceb15a4" containerID="568d9324083e89252227aaf3c0f46bb5a5d320df6fe85894a486e06f4366eec9" exitCode=60 Oct 30 02:00:22.512851 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.505648 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-5xgtv" event={"ID":"b8e5ee17-3f55-4488-aba3-2a81eceb15a4","Type":"ContainerDied","Data":"568d9324083e89252227aaf3c0f46bb5a5d320df6fe85894a486e06f4366eec9"} Oct 30 02:00:22.512851 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.506187 6572 scope.go:122] "RemoveContainer" containerID="568d9324083e89252227aaf3c0f46bb5a5d320df6fe85894a486e06f4366eec9" Oct 30 02:00:22.512851 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:22.506709 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888760-5xgtv_odh-ai-gateway-infra(b8e5ee17-3f55-4488-aba3-2a81eceb15a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-5xgtv" podUID="b8e5ee17-3f55-4488-aba3-2a81eceb15a4" Oct 30 02:00:22.522641 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.522615 6572 scope.go:122] "RemoveContainer" containerID="08737f497b2d983f762cd7df00c78c07c2014adaa0abd5706b21e9ff768e4c4a" Oct 30 02:00:22.525640 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.525615 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888760-545xr"] Oct 30 02:00:22.538987 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.538319 6572 scope.go:122] "RemoveContainer" containerID="c75763091396844c9163dceb400dcf5b6d38521579f2d18b1da45e8954d18638" Oct 30 02:00:22.558796 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.558711 6572 scope.go:122] "RemoveContainer" containerID="9a34470cbc34e4a2868d5cdeafe1052054730841d5309098780b32119f52b52b" Oct 30 02:00:22.561676 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.561646 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="228bfbbd-21b3-4f89-b247-962811a659e6" path="/var/lib/kubelet/pods/228bfbbd-21b3-4f89-b247-962811a659e6/volumes" Oct 30 02:00:22.562368 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.562330 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2956c338-3b66-48aa-b3cc-521dbdef04ce" path="/var/lib/kubelet/pods/2956c338-3b66-48aa-b3cc-521dbdef04ce/volumes" Oct 30 02:00:22.563071 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.563048 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad3ca000-d9f7-4d97-9a05-b39362550ba0" path="/var/lib/kubelet/pods/ad3ca000-d9f7-4d97-9a05-b39362550ba0/volumes" Oct 30 02:00:22.565504 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.565424 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc3321aa-698f-4c74-94ae-95e8729d9f27" path="/var/lib/kubelet/pods/bc3321aa-698f-4c74-94ae-95e8729d9f27/volumes" Oct 30 02:00:22.566474 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.566456 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc58a71f-7f5f-4f68-aa97-a22aa15b854e" path="/var/lib/kubelet/pods/dc58a71f-7f5f-4f68-aa97-a22aa15b854e/volumes" Oct 30 02:00:22.567294 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.567276 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dff6267c-a0a8-4d35-badf-3032bfb3e979" path="/var/lib/kubelet/pods/dff6267c-a0a8-4d35-badf-3032bfb3e979/volumes" Oct 30 02:00:22.574889 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.574868 6572 scope.go:122] "RemoveContainer" containerID="8801fe0095f947337dc65df5994bacac5f5e8762b8ad984070e0f7a19b3448bb" Oct 30 02:00:22.577432 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.577412 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888760-xhtnj"] Oct 30 02:00:22.625013 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.624952 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c12916d7-2cprr" podStartSLOduration=2.624934514 podStartE2EDuration="2.624934514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:22.624444557 +0000 UTC m=+2548.200869960" watchObservedRunningTime="2026-10-30 02:00:22.624934514 +0000 UTC m=+2548.201359914" Oct 30 02:00:22.626080 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.626047 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888760-6h8kw" Oct 30 02:00:22.664417 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.664381 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b-kube-api-access-cn6rk\" (UniqueName: \"kubernetes.io/projected/d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b-kube-api-access-cn6rk\") pod \"d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b\" (UID: \"d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b\") " Oct 30 02:00:22.664417 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.664432 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b-maas-api-service-ca\") pod \"d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b\" (UID: \"d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b\") " Oct 30 02:00:22.665232 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.665165 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b-maas-api-service-ca" pod "d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b" (UID: "d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:22.666992 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.666968 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b-kube-api-access-cn6rk" pod "d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b" (UID: "d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b"). InnerVolumeSpecName "kube-api-access-cn6rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:22.691427 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.691396 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888760-dfspx"] Oct 30 02:00:22.817453 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.781037 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn6rk\" (UniqueName: \"kubernetes.io/projected/d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b-kube-api-access-cn6rk\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:22.817453 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.781076 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:22.835605 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.830258 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:22.852116 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.852080 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888760-xhtnj" Oct 30 02:00:22.876934 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.876874 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888760-545xr" Oct 30 02:00:22.881349 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.881320 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888760-qczr8"] Oct 30 02:00:22.901961 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.901917 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888760-qczr8"] Oct 30 02:00:22.918273 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.917919 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888760-5h4tv"] Oct 30 02:00:22.928355 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.928325 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888760-5h4tv"] Oct 30 02:00:22.937583 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.937558 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888760-kwp4k"] Oct 30 02:00:22.957196 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.957089 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888760-dfspx" Oct 30 02:00:22.970608 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.970538 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888760-hvkkt"] Oct 30 02:00:22.976895 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.976865 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888760-hvkkt"] Oct 30 02:00:22.983510 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.983473 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16ce2d50-2777-4827-80ec-025ff18072ed-kube-api-access-cbkvl\" (UniqueName: \"kubernetes.io/projected/16ce2d50-2777-4827-80ec-025ff18072ed-kube-api-access-cbkvl\") pod \"16ce2d50-2777-4827-80ec-025ff18072ed\" (UID: \"16ce2d50-2777-4827-80ec-025ff18072ed\") " Oct 30 02:00:22.983670 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.983532 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16ce2d50-2777-4827-80ec-025ff18072ed-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16ce2d50-2777-4827-80ec-025ff18072ed-maas-api-service-ca\") pod \"16ce2d50-2777-4827-80ec-025ff18072ed\" (UID: \"16ce2d50-2777-4827-80ec-025ff18072ed\") " Oct 30 02:00:22.983873 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.983846 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45e5935e-478f-4fec-beae-0cae2ea14c1b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45e5935e-478f-4fec-beae-0cae2ea14c1b-maas-api-service-ca\") pod \"45e5935e-478f-4fec-beae-0cae2ea14c1b\" (UID: \"45e5935e-478f-4fec-beae-0cae2ea14c1b\") " Oct 30 02:00:22.984045 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.983941 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45e5935e-478f-4fec-beae-0cae2ea14c1b-kube-api-access-xb5qv\" (UniqueName: \"kubernetes.io/projected/45e5935e-478f-4fec-beae-0cae2ea14c1b-kube-api-access-xb5qv\") pod \"45e5935e-478f-4fec-beae-0cae2ea14c1b\" (UID: \"45e5935e-478f-4fec-beae-0cae2ea14c1b\") " Oct 30 02:00:22.984282 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.984258 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45e5935e-478f-4fec-beae-0cae2ea14c1b-maas-api-service-ca" pod "45e5935e-478f-4fec-beae-0cae2ea14c1b" (UID: "45e5935e-478f-4fec-beae-0cae2ea14c1b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:22.984476 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.984412 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16ce2d50-2777-4827-80ec-025ff18072ed-maas-api-service-ca" pod "16ce2d50-2777-4827-80ec-025ff18072ed" (UID: "16ce2d50-2777-4827-80ec-025ff18072ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:22.985044 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.985014 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16ce2d50-2777-4827-80ec-025ff18072ed-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:22.985190 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.985174 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45e5935e-478f-4fec-beae-0cae2ea14c1b-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:22.986713 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.986588 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16ce2d50-2777-4827-80ec-025ff18072ed-kube-api-access-cbkvl" pod "16ce2d50-2777-4827-80ec-025ff18072ed" (UID: "16ce2d50-2777-4827-80ec-025ff18072ed"). InnerVolumeSpecName "kube-api-access-cbkvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:22.987159 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:22.987135 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45e5935e-478f-4fec-beae-0cae2ea14c1b-kube-api-access-xb5qv" pod "45e5935e-478f-4fec-beae-0cae2ea14c1b" (UID: "45e5935e-478f-4fec-beae-0cae2ea14c1b"). InnerVolumeSpecName "kube-api-access-xb5qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:23.008730 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.008687 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888760-k4ntx"] Oct 30 02:00:23.015342 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.015304 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888760-k4ntx"] Oct 30 02:00:23.089239 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.089103 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbd7b9d4-eb19-4605-bdc9-21940dad8c92-kube-api-access-qd9rh\" (UniqueName: \"kubernetes.io/projected/dbd7b9d4-eb19-4605-bdc9-21940dad8c92-kube-api-access-qd9rh\") pod \"dbd7b9d4-eb19-4605-bdc9-21940dad8c92\" (UID: \"dbd7b9d4-eb19-4605-bdc9-21940dad8c92\") " Oct 30 02:00:23.089499 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.089320 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbd7b9d4-eb19-4605-bdc9-21940dad8c92-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbd7b9d4-eb19-4605-bdc9-21940dad8c92-maas-api-service-ca\") pod \"dbd7b9d4-eb19-4605-bdc9-21940dad8c92\" (UID: \"dbd7b9d4-eb19-4605-bdc9-21940dad8c92\") " Oct 30 02:00:23.089988 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.089839 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd7b9d4-eb19-4605-bdc9-21940dad8c92-maas-api-service-ca" pod "dbd7b9d4-eb19-4605-bdc9-21940dad8c92" (UID: "dbd7b9d4-eb19-4605-bdc9-21940dad8c92"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:23.096821 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.091476 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd7b9d4-eb19-4605-bdc9-21940dad8c92-kube-api-access-qd9rh" pod "dbd7b9d4-eb19-4605-bdc9-21940dad8c92" (UID: "dbd7b9d4-eb19-4605-bdc9-21940dad8c92"). InnerVolumeSpecName "kube-api-access-qd9rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:23.096821 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.095921 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbd7b9d4-eb19-4605-bdc9-21940dad8c92-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:23.096821 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.095949 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb5qv\" (UniqueName: \"kubernetes.io/projected/45e5935e-478f-4fec-beae-0cae2ea14c1b-kube-api-access-xb5qv\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:23.096821 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.095962 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbkvl\" (UniqueName: \"kubernetes.io/projected/16ce2d50-2777-4827-80ec-025ff18072ed-kube-api-access-cbkvl\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:23.096821 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.095976 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd9rh\" (UniqueName: \"kubernetes.io/projected/dbd7b9d4-eb19-4605-bdc9-21940dad8c92-kube-api-access-qd9rh\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:23.234041 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.233961 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888760-kwp4k" Oct 30 02:00:23.298373 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.298314 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05cb266a-e1fe-4b51-a18f-96d85a78ab16-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05cb266a-e1fe-4b51-a18f-96d85a78ab16-maas-api-service-ca\") pod \"05cb266a-e1fe-4b51-a18f-96d85a78ab16\" (UID: \"05cb266a-e1fe-4b51-a18f-96d85a78ab16\") " Oct 30 02:00:23.298531 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.298411 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05cb266a-e1fe-4b51-a18f-96d85a78ab16-kube-api-access-jqkbf\" (UniqueName: \"kubernetes.io/projected/05cb266a-e1fe-4b51-a18f-96d85a78ab16-kube-api-access-jqkbf\") pod \"05cb266a-e1fe-4b51-a18f-96d85a78ab16\" (UID: \"05cb266a-e1fe-4b51-a18f-96d85a78ab16\") " Oct 30 02:00:23.299049 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.299012 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05cb266a-e1fe-4b51-a18f-96d85a78ab16-maas-api-service-ca" pod "05cb266a-e1fe-4b51-a18f-96d85a78ab16" (UID: "05cb266a-e1fe-4b51-a18f-96d85a78ab16"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:23.299786 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.299737 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05cb266a-e1fe-4b51-a18f-96d85a78ab16-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:23.302972 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.302943 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05cb266a-e1fe-4b51-a18f-96d85a78ab16-kube-api-access-jqkbf" pod "05cb266a-e1fe-4b51-a18f-96d85a78ab16" (UID: "05cb266a-e1fe-4b51-a18f-96d85a78ab16"). InnerVolumeSpecName "kube-api-access-jqkbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:23.401068 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.401029 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqkbf\" (UniqueName: \"kubernetes.io/projected/05cb266a-e1fe-4b51-a18f-96d85a78ab16-kube-api-access-jqkbf\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:23.523554 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.523526 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888760-6h8kw" Oct 30 02:00:23.523784 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.523513 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888760-6h8kw" event={"ID":"d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b","Type":"ContainerDied","Data":"0628039ef6f3839f190386fe610deea13cd7c7d0786c6f982286a464699fd13d"} Oct 30 02:00:23.523784 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.523686 6572 scope.go:122] "RemoveContainer" containerID="b1c6d7e1349f97579b1746a25b8c9420fbce2becce1d4ee7c5f82f0efa97df6a" Oct 30 02:00:23.526905 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.526878 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888760-545xr" event={"ID":"16ce2d50-2777-4827-80ec-025ff18072ed","Type":"ContainerDied","Data":"f4da7ab21469b510e241e70cdc19c7b5f0c31f5fdd405937ba72a9f3a77bd561"} Oct 30 02:00:23.527034 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.526906 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888760-545xr" Oct 30 02:00:23.530323 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.530292 6572 generic.go:356] "Generic (PLEG): container finished" podID="250d2557-a543-4749-aa8c-6483de133989" containerID="c624cb2e4813b6439ba0ce52d8bf23d0ceeb9dc743e7a1fd6d02958bc71d7a65" exitCode=0 Oct 30 02:00:23.530458 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.530361 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4560f025-tl8nm" event={"ID":"250d2557-a543-4749-aa8c-6483de133989","Type":"ContainerDied","Data":"c624cb2e4813b6439ba0ce52d8bf23d0ceeb9dc743e7a1fd6d02958bc71d7a65"} Oct 30 02:00:23.531829 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.531809 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888760-xhtnj" event={"ID":"45e5935e-478f-4fec-beae-0cae2ea14c1b","Type":"ContainerDied","Data":"fee24ed70ae7528e112201ccb54a72293fa1ba817cfe0acc6a5c9ceca52b4e11"} Oct 30 02:00:23.531913 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.531828 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888760-xhtnj" Oct 30 02:00:23.533544 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.533523 6572 generic.go:356] "Generic (PLEG): container finished" podID="15e59e08-4ed8-4206-80c7-244b505243ae" containerID="a16cf0994fb06470a6c97f1c889e39c2d94b628d040081252a1c2ab55449209f" exitCode=0 Oct 30 02:00:23.533661 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.533603 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c12916d7-2cprr" event={"ID":"15e59e08-4ed8-4206-80c7-244b505243ae","Type":"ContainerDied","Data":"a16cf0994fb06470a6c97f1c889e39c2d94b628d040081252a1c2ab55449209f"} Oct 30 02:00:23.535083 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.535066 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888760-dfspx" Oct 30 02:00:23.535083 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.535073 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888760-dfspx" event={"ID":"dbd7b9d4-eb19-4605-bdc9-21940dad8c92","Type":"ContainerDied","Data":"babc29cf7fa79f9d2afe4f2c40bde16f4cef974cbcee257b78f7b35857c2942c"} Oct 30 02:00:23.537852 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.537828 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888760-kwp4k" event={"ID":"05cb266a-e1fe-4b51-a18f-96d85a78ab16","Type":"ContainerDied","Data":"4e6467d19876d2700ab0b46f04bf4e83599bb6537b25613d3514bc72c9ad4be5"} Oct 30 02:00:23.537852 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.537843 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888760-kwp4k" Oct 30 02:00:23.550052 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.550023 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2eae39-29888760-t6tz5"] Oct 30 02:00:23.567951 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.567913 6572 scope.go:122] "RemoveContainer" containerID="b262c1ec3695e5daa8e3e5c89cee0d490e64475f94ea779d56f406e33508a398" Oct 30 02:00:23.584057 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.584029 6572 scope.go:122] "RemoveContainer" containerID="d0f7aae82673c5805fc09099177b4d02409e5147eba7f6347fdb94d966631ee7" Oct 30 02:00:23.597381 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.597355 6572 scope.go:122] "RemoveContainer" containerID="2e77fe29b14324f4513950ed5c3a4a4a28bad9f88d4f02c6d284e98f039f0cd5" Oct 30 02:00:23.609330 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.609015 6572 scope.go:122] "RemoveContainer" containerID="148d1c60640f2d1e1d7d443ca6c2a1f813a58ac6ec04004c0dc7a2d56d86881a" Oct 30 02:00:23.690444 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.690411 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888760-5xgtv"] Oct 30 02:00:23.711579 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.711489 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2eae39-29888760-t6tz5" Oct 30 02:00:23.724365 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.724326 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888760-545xr"] Oct 30 02:00:23.729240 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.729207 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888760-545xr"] Oct 30 02:00:23.753454 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.753410 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888760-xhtnj"] Oct 30 02:00:23.758236 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.758166 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888760-xhtnj"] Oct 30 02:00:23.790936 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.790873 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888760-dfspx"] Oct 30 02:00:23.796569 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.796539 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888760-dfspx"] Oct 30 02:00:23.806326 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.806295 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dc37d0b-7989-4437-9902-df0a34eb682a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dc37d0b-7989-4437-9902-df0a34eb682a-maas-api-service-ca\") pod \"9dc37d0b-7989-4437-9902-df0a34eb682a\" (UID: \"9dc37d0b-7989-4437-9902-df0a34eb682a\") " Oct 30 02:00:23.806326 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.806339 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dc37d0b-7989-4437-9902-df0a34eb682a-kube-api-access-zfrph\" (UniqueName: \"kubernetes.io/projected/9dc37d0b-7989-4437-9902-df0a34eb682a-kube-api-access-zfrph\") pod \"9dc37d0b-7989-4437-9902-df0a34eb682a\" (UID: \"9dc37d0b-7989-4437-9902-df0a34eb682a\") " Oct 30 02:00:23.806870 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.806828 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dc37d0b-7989-4437-9902-df0a34eb682a-maas-api-service-ca" pod "9dc37d0b-7989-4437-9902-df0a34eb682a" (UID: "9dc37d0b-7989-4437-9902-df0a34eb682a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:23.806995 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.806968 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dc37d0b-7989-4437-9902-df0a34eb682a-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:23.810093 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.810070 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc37d0b-7989-4437-9902-df0a34eb682a-kube-api-access-zfrph" pod "9dc37d0b-7989-4437-9902-df0a34eb682a" (UID: "9dc37d0b-7989-4437-9902-df0a34eb682a"). InnerVolumeSpecName "kube-api-access-zfrph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:23.853781 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.853740 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-5xgtv" Oct 30 02:00:23.864455 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.864426 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888760-6h8kw"] Oct 30 02:00:23.868944 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.868920 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888760-6h8kw"] Oct 30 02:00:23.898710 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.898678 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888760-kwp4k"] Oct 30 02:00:23.906516 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.906489 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888760-kwp4k"] Oct 30 02:00:23.907992 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.907962 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8e5ee17-3f55-4488-aba3-2a81eceb15a4-kube-api-access-6txhh\" (UniqueName: \"kubernetes.io/projected/b8e5ee17-3f55-4488-aba3-2a81eceb15a4-kube-api-access-6txhh\") pod \"b8e5ee17-3f55-4488-aba3-2a81eceb15a4\" (UID: \"b8e5ee17-3f55-4488-aba3-2a81eceb15a4\") " Oct 30 02:00:23.908179 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.908157 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8e5ee17-3f55-4488-aba3-2a81eceb15a4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8e5ee17-3f55-4488-aba3-2a81eceb15a4-maas-api-service-ca\") pod \"b8e5ee17-3f55-4488-aba3-2a81eceb15a4\" (UID: \"b8e5ee17-3f55-4488-aba3-2a81eceb15a4\") " Oct 30 02:00:23.908532 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.908501 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8e5ee17-3f55-4488-aba3-2a81eceb15a4-maas-api-service-ca" pod "b8e5ee17-3f55-4488-aba3-2a81eceb15a4" (UID: "b8e5ee17-3f55-4488-aba3-2a81eceb15a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:23.908789 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.908742 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8e5ee17-3f55-4488-aba3-2a81eceb15a4-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:23.908871 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.908789 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfrph\" (UniqueName: \"kubernetes.io/projected/9dc37d0b-7989-4437-9902-df0a34eb682a-kube-api-access-zfrph\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:23.910290 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:23.910267 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8e5ee17-3f55-4488-aba3-2a81eceb15a4-kube-api-access-6txhh" pod "b8e5ee17-3f55-4488-aba3-2a81eceb15a4" (UID: "b8e5ee17-3f55-4488-aba3-2a81eceb15a4"). InnerVolumeSpecName "kube-api-access-6txhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:24.009817 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.009721 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6txhh\" (UniqueName: \"kubernetes.io/projected/b8e5ee17-3f55-4488-aba3-2a81eceb15a4-kube-api-access-6txhh\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:24.473229 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.473189 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:24.547587 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.547552 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05cb266a-e1fe-4b51-a18f-96d85a78ab16" path="/var/lib/kubelet/pods/05cb266a-e1fe-4b51-a18f-96d85a78ab16/volumes" Oct 30 02:00:24.548194 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.548171 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16ce2d50-2777-4827-80ec-025ff18072ed" path="/var/lib/kubelet/pods/16ce2d50-2777-4827-80ec-025ff18072ed/volumes" Oct 30 02:00:24.548730 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.548711 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="419770d0-51af-4843-9ab2-8cbbe8215357" path="/var/lib/kubelet/pods/419770d0-51af-4843-9ab2-8cbbe8215357/volumes" Oct 30 02:00:24.549471 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.549453 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45e5935e-478f-4fec-beae-0cae2ea14c1b" path="/var/lib/kubelet/pods/45e5935e-478f-4fec-beae-0cae2ea14c1b/volumes" Oct 30 02:00:24.549900 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.549884 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b" path="/var/lib/kubelet/pods/d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b/volumes" Oct 30 02:00:24.550292 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.550279 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbd7b9d4-eb19-4605-bdc9-21940dad8c92" path="/var/lib/kubelet/pods/dbd7b9d4-eb19-4605-bdc9-21940dad8c92/volumes" Oct 30 02:00:24.550998 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.550978 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea84dc40-9569-4a87-a537-a8e0c0fb119d" path="/var/lib/kubelet/pods/ea84dc40-9569-4a87-a537-a8e0c0fb119d/volumes" Oct 30 02:00:24.551386 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.551372 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c" path="/var/lib/kubelet/pods/ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c/volumes" Oct 30 02:00:24.551791 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.551777 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f818d22e-27ff-4caa-a48d-352af6583de4" path="/var/lib/kubelet/pods/f818d22e-27ff-4caa-a48d-352af6583de4/volumes" Oct 30 02:00:24.554470 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.554446 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2eae39-29888760-t6tz5" Oct 30 02:00:24.554619 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.554480 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2eae39-29888760-t6tz5" event={"ID":"9dc37d0b-7989-4437-9902-df0a34eb682a","Type":"ContainerDied","Data":"5a2df92cf71ae8c940ae45055c823287af02c86acf1d9ffed22a65a37573e6c2"} Oct 30 02:00:24.554619 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.554540 6572 scope.go:122] "RemoveContainer" containerID="6e0af7545ef3200cc8b919722eaa72982f749e9797e04c5b4aae3d4a4dea6940" Oct 30 02:00:24.563794 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.563231 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-5xgtv" Oct 30 02:00:24.563794 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.563272 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-5xgtv" event={"ID":"b8e5ee17-3f55-4488-aba3-2a81eceb15a4","Type":"ContainerDied","Data":"e63efb694621c6844de4606ed90a3e9723514ba6df6f1763af7707f8af5d6c6b"} Oct 30 02:00:24.570507 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.570405 6572 scope.go:122] "RemoveContainer" containerID="568d9324083e89252227aaf3c0f46bb5a5d320df6fe85894a486e06f4366eec9" Oct 30 02:00:24.645834 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.640822 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2eae39-29888760-t6tz5"] Oct 30 02:00:24.646618 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.646588 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2eae39-29888760-t6tz5"] Oct 30 02:00:24.680802 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.680766 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888760-5xgtv"] Oct 30 02:00:24.685237 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.685207 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888760-5xgtv"] Oct 30 02:00:24.750969 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.750939 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c12916d7-2cprr" Oct 30 02:00:24.819952 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.819924 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4560f025-tl8nm" Oct 30 02:00:24.833563 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.833531 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15e59e08-4ed8-4206-80c7-244b505243ae-kube-api-access-67qq7\" (UniqueName: \"kubernetes.io/projected/15e59e08-4ed8-4206-80c7-244b505243ae-kube-api-access-67qq7\") pod \"15e59e08-4ed8-4206-80c7-244b505243ae\" (UID: \"15e59e08-4ed8-4206-80c7-244b505243ae\") " Oct 30 02:00:24.833836 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.833674 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15e59e08-4ed8-4206-80c7-244b505243ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15e59e08-4ed8-4206-80c7-244b505243ae-maas-api-xks-ca\") pod \"15e59e08-4ed8-4206-80c7-244b505243ae\" (UID: \"15e59e08-4ed8-4206-80c7-244b505243ae\") " Oct 30 02:00:24.833836 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.833726 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15e59e08-4ed8-4206-80c7-244b505243ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15e59e08-4ed8-4206-80c7-244b505243ae-maas-api-openshift-ca\") pod \"15e59e08-4ed8-4206-80c7-244b505243ae\" (UID: \"15e59e08-4ed8-4206-80c7-244b505243ae\") " Oct 30 02:00:24.834139 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.834114 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15e59e08-4ed8-4206-80c7-244b505243ae-maas-api-openshift-ca" pod "15e59e08-4ed8-4206-80c7-244b505243ae" (UID: "15e59e08-4ed8-4206-80c7-244b505243ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:24.834476 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.834457 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15e59e08-4ed8-4206-80c7-244b505243ae-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:24.835945 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.835919 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15e59e08-4ed8-4206-80c7-244b505243ae-maas-api-xks-ca" pod "15e59e08-4ed8-4206-80c7-244b505243ae" (UID: "15e59e08-4ed8-4206-80c7-244b505243ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:00:24.836037 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.836019 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15e59e08-4ed8-4206-80c7-244b505243ae-kube-api-access-67qq7" pod "15e59e08-4ed8-4206-80c7-244b505243ae" (UID: "15e59e08-4ed8-4206-80c7-244b505243ae"). InnerVolumeSpecName "kube-api-access-67qq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:24.935739 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.935699 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/250d2557-a543-4749-aa8c-6483de133989-kube-api-access-2cfbg\" (UniqueName: \"kubernetes.io/projected/250d2557-a543-4749-aa8c-6483de133989-kube-api-access-2cfbg\") pod \"250d2557-a543-4749-aa8c-6483de133989\" (UID: \"250d2557-a543-4749-aa8c-6483de133989\") " Oct 30 02:00:24.935961 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.935864 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/250d2557-a543-4749-aa8c-6483de133989-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/250d2557-a543-4749-aa8c-6483de133989-maas-api-xks-ca\") pod \"250d2557-a543-4749-aa8c-6483de133989\" (UID: \"250d2557-a543-4749-aa8c-6483de133989\") " Oct 30 02:00:24.936051 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.936036 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/250d2557-a543-4749-aa8c-6483de133989-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/250d2557-a543-4749-aa8c-6483de133989-maas-api-openshift-ca\") pod \"250d2557-a543-4749-aa8c-6483de133989\" (UID: \"250d2557-a543-4749-aa8c-6483de133989\") " Oct 30 02:00:24.936391 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.936365 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/250d2557-a543-4749-aa8c-6483de133989-maas-api-openshift-ca" pod "250d2557-a543-4749-aa8c-6483de133989" (UID: "250d2557-a543-4749-aa8c-6483de133989"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:24.936514 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.936496 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/250d2557-a543-4749-aa8c-6483de133989-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:24.936559 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.936515 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67qq7\" (UniqueName: \"kubernetes.io/projected/15e59e08-4ed8-4206-80c7-244b505243ae-kube-api-access-67qq7\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:24.936559 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.936525 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15e59e08-4ed8-4206-80c7-244b505243ae-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:24.937938 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.937915 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/250d2557-a543-4749-aa8c-6483de133989-maas-api-xks-ca" pod "250d2557-a543-4749-aa8c-6483de133989" (UID: "250d2557-a543-4749-aa8c-6483de133989"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:00:24.938012 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:24.937985 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/250d2557-a543-4749-aa8c-6483de133989-kube-api-access-2cfbg" pod "250d2557-a543-4749-aa8c-6483de133989" (UID: "250d2557-a543-4749-aa8c-6483de133989"). InnerVolumeSpecName "kube-api-access-2cfbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:25.038519 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:25.038436 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cfbg\" (UniqueName: \"kubernetes.io/projected/250d2557-a543-4749-aa8c-6483de133989-kube-api-access-2cfbg\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:25.038519 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:25.038468 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/250d2557-a543-4749-aa8c-6483de133989-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:25.575247 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:25.575219 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4560f025-tl8nm" Oct 30 02:00:25.575445 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:25.575220 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4560f025-tl8nm" event={"ID":"250d2557-a543-4749-aa8c-6483de133989","Type":"ContainerDied","Data":"5ff12d1963ace89961cc2897abe7f48c7bd6f71657da40f5913b973bf9a47c84"} Oct 30 02:00:25.575445 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:25.575353 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ff12d1963ace89961cc2897abe7f48c7bd6f71657da40f5913b973bf9a47c84" Oct 30 02:00:25.576666 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:25.576646 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c12916d7-2cprr" Oct 30 02:00:25.576832 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:25.576677 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c12916d7-2cprr" event={"ID":"15e59e08-4ed8-4206-80c7-244b505243ae","Type":"ContainerDied","Data":"3fa8fd3f49d52aa07fbc6a96a09aac9f6bb242cb0745e23a25666bdce035e9d7"} Oct 30 02:00:25.576832 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:25.576722 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fa8fd3f49d52aa07fbc6a96a09aac9f6bb242cb0745e23a25666bdce035e9d7" Oct 30 02:00:25.981645 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:25.981595 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s" podUID="d818fec7-418c-447e-bd2e-b64e07fdbb86" containerName="payload-processing" containerID="cri-o://1fe3028850ee4cc59a0ed9533c4a01140b8ccf6577fec972b976e0588a015c15" gracePeriod=25 Oct 30 02:00:26.048258 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:26.048223 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:26.311169 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:26.311141 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s" Oct 30 02:00:26.473481 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:26.473451 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d818fec7-418c-447e-bd2e-b64e07fdbb86-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d818fec7-418c-447e-bd2e-b64e07fdbb86-routing-overlay-volume\") pod \"d818fec7-418c-447e-bd2e-b64e07fdbb86\" (UID: \"d818fec7-418c-447e-bd2e-b64e07fdbb86\") " Oct 30 02:00:26.473692 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:26.473514 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d818fec7-418c-447e-bd2e-b64e07fdbb86-kube-api-access-dcjv8\" (UniqueName: \"kubernetes.io/projected/d818fec7-418c-447e-bd2e-b64e07fdbb86-kube-api-access-dcjv8\") pod \"d818fec7-418c-447e-bd2e-b64e07fdbb86\" (UID: \"d818fec7-418c-447e-bd2e-b64e07fdbb86\") " Oct 30 02:00:26.473785 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:26.473717 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d818fec7-418c-447e-bd2e-b64e07fdbb86-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d818fec7-418c-447e-bd2e-b64e07fdbb86-plugins-config-volume\") pod \"d818fec7-418c-447e-bd2e-b64e07fdbb86\" (UID: \"d818fec7-418c-447e-bd2e-b64e07fdbb86\") " Oct 30 02:00:26.473864 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:26.473789 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d818fec7-418c-447e-bd2e-b64e07fdbb86-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d818fec7-418c-447e-bd2e-b64e07fdbb86-provider-credentials\") pod \"d818fec7-418c-447e-bd2e-b64e07fdbb86\" (UID: \"d818fec7-418c-447e-bd2e-b64e07fdbb86\") " Oct 30 02:00:26.473989 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:26.473921 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d818fec7-418c-447e-bd2e-b64e07fdbb86-routing-overlay-volume" pod "d818fec7-418c-447e-bd2e-b64e07fdbb86" (UID: "d818fec7-418c-447e-bd2e-b64e07fdbb86"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:26.474182 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:26.474155 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d818fec7-418c-447e-bd2e-b64e07fdbb86-plugins-config-volume" pod "d818fec7-418c-447e-bd2e-b64e07fdbb86" (UID: "d818fec7-418c-447e-bd2e-b64e07fdbb86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:26.474536 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:26.474515 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d818fec7-418c-447e-bd2e-b64e07fdbb86-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:26.474606 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:26.474545 6572 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d818fec7-418c-447e-bd2e-b64e07fdbb86-routing-overlay-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:26.476136 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:26.476109 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d818fec7-418c-447e-bd2e-b64e07fdbb86-provider-credentials" pod "d818fec7-418c-447e-bd2e-b64e07fdbb86" (UID: "d818fec7-418c-447e-bd2e-b64e07fdbb86"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:26.476220 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:26.476161 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d818fec7-418c-447e-bd2e-b64e07fdbb86-kube-api-access-dcjv8" pod "d818fec7-418c-447e-bd2e-b64e07fdbb86" (UID: "d818fec7-418c-447e-bd2e-b64e07fdbb86"). InnerVolumeSpecName "kube-api-access-dcjv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:26.543951 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:26.543873 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dc37d0b-7989-4437-9902-df0a34eb682a" path="/var/lib/kubelet/pods/9dc37d0b-7989-4437-9902-df0a34eb682a/volumes" Oct 30 02:00:26.544375 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:26.544361 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8e5ee17-3f55-4488-aba3-2a81eceb15a4" path="/var/lib/kubelet/pods/b8e5ee17-3f55-4488-aba3-2a81eceb15a4/volumes" Oct 30 02:00:26.576137 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:26.576110 6572 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d818fec7-418c-447e-bd2e-b64e07fdbb86-provider-credentials\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:26.576137 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:26.576136 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcjv8\" (UniqueName: \"kubernetes.io/projected/d818fec7-418c-447e-bd2e-b64e07fdbb86-kube-api-access-dcjv8\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:26.586679 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:26.586652 6572 generic.go:356] "Generic (PLEG): container finished" podID="d818fec7-418c-447e-bd2e-b64e07fdbb86" containerID="1fe3028850ee4cc59a0ed9533c4a01140b8ccf6577fec972b976e0588a015c15" exitCode=0 Oct 30 02:00:26.586830 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:26.586700 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s" Oct 30 02:00:26.586830 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:26.586740 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s" event={"ID":"d818fec7-418c-447e-bd2e-b64e07fdbb86","Type":"ContainerDied","Data":"1fe3028850ee4cc59a0ed9533c4a01140b8ccf6577fec972b976e0588a015c15"} Oct 30 02:00:26.586830 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:26.586809 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s" event={"ID":"d818fec7-418c-447e-bd2e-b64e07fdbb86","Type":"ContainerDied","Data":"75ba27ac60df1fd1f8806195bfbaa523d7c237d39563a812e55da185f3fdeae7"} Oct 30 02:00:26.586954 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:26.586840 6572 scope.go:122] "RemoveContainer" containerID="1fe3028850ee4cc59a0ed9533c4a01140b8ccf6577fec972b976e0588a015c15" Oct 30 02:00:26.598110 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:26.598092 6572 scope.go:122] "RemoveContainer" containerID="1fe3028850ee4cc59a0ed9533c4a01140b8ccf6577fec972b976e0588a015c15" Oct 30 02:00:26.598398 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:26.598371 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fe3028850ee4cc59a0ed9533c4a01140b8ccf6577fec972b976e0588a015c15\": container with ID starting with 1fe3028850ee4cc59a0ed9533c4a01140b8ccf6577fec972b976e0588a015c15 not found: ID does not exist" containerID="1fe3028850ee4cc59a0ed9533c4a01140b8ccf6577fec972b976e0588a015c15" Oct 30 02:00:26.598521 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:26.598406 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fe3028850ee4cc59a0ed9533c4a01140b8ccf6577fec972b976e0588a015c15"} err="failed to get container status \"1fe3028850ee4cc59a0ed9533c4a01140b8ccf6577fec972b976e0588a015c15\": rpc error: code = NotFound desc = could not find container \"1fe3028850ee4cc59a0ed9533c4a01140b8ccf6577fec972b976e0588a015c15\": container with ID starting with 1fe3028850ee4cc59a0ed9533c4a01140b8ccf6577fec972b976e0588a015c15 not found: ID does not exist" Oct 30 02:00:26.638982 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:26.638950 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s"] Oct 30 02:00:26.643083 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:26.643056 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-7857rf79s"] Oct 30 02:00:27.602406 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:27.602342 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:27.858784 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:27.858662 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:28.559184 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:28.559151 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d818fec7-418c-447e-bd2e-b64e07fdbb86" path="/var/lib/kubelet/pods/d818fec7-418c-447e-bd2e-b64e07fdbb86/volumes" Oct 30 02:00:29.085873 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:29.085839 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:30.563489 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:30.563455 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:32.021071 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.021021 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:32.224806 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.224770 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:32.385887 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.385790 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c02f572b-f5j2k"] Oct 30 02:00:32.386688 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.386663 6572 cpu_manager.go:414] "containerMap: removing container" podUID="d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b" containerName="cleanup" Oct 30 02:00:32.386688 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.386688 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b" containerName="cleanup" Oct 30 02:00:32.386932 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.386700 6572 cpu_manager.go:414] "containerMap: removing container" podUID="9dc37d0b-7989-4437-9902-df0a34eb682a" containerName="cleanup" Oct 30 02:00:32.386932 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.386706 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dc37d0b-7989-4437-9902-df0a34eb682a" containerName="cleanup" Oct 30 02:00:32.386932 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.386714 6572 cpu_manager.go:414] "containerMap: removing container" podUID="dbd7b9d4-eb19-4605-bdc9-21940dad8c92" containerName="cleanup" Oct 30 02:00:32.386932 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.386720 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbd7b9d4-eb19-4605-bdc9-21940dad8c92" containerName="cleanup" Oct 30 02:00:32.386932 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.386728 6572 cpu_manager.go:414] "containerMap: removing container" podUID="ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c" containerName="cleanup" Oct 30 02:00:32.386932 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.386733 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c" containerName="cleanup" Oct 30 02:00:32.386932 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.386757 6572 cpu_manager.go:414] "containerMap: removing container" podUID="d818fec7-418c-447e-bd2e-b64e07fdbb86" containerName="payload-processing" Oct 30 02:00:32.386932 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.386765 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d818fec7-418c-447e-bd2e-b64e07fdbb86" containerName="payload-processing" Oct 30 02:00:32.386932 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.386781 6572 cpu_manager.go:414] "containerMap: removing container" podUID="45e5935e-478f-4fec-beae-0cae2ea14c1b" containerName="cleanup" Oct 30 02:00:32.386932 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.386789 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45e5935e-478f-4fec-beae-0cae2ea14c1b" containerName="cleanup" Oct 30 02:00:32.386932 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.386798 6572 cpu_manager.go:414] "containerMap: removing container" podUID="b8e5ee17-3f55-4488-aba3-2a81eceb15a4" containerName="cleanup" Oct 30 02:00:32.386932 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.386805 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8e5ee17-3f55-4488-aba3-2a81eceb15a4" containerName="cleanup" Oct 30 02:00:32.386932 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.386824 6572 cpu_manager.go:414] "containerMap: removing container" podUID="16ce2d50-2777-4827-80ec-025ff18072ed" containerName="cleanup" Oct 30 02:00:32.386932 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.386831 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16ce2d50-2777-4827-80ec-025ff18072ed" containerName="cleanup" Oct 30 02:00:32.386932 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.386847 6572 cpu_manager.go:414] "containerMap: removing container" podUID="05cb266a-e1fe-4b51-a18f-96d85a78ab16" containerName="cleanup" Oct 30 02:00:32.386932 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.386854 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05cb266a-e1fe-4b51-a18f-96d85a78ab16" containerName="cleanup" Oct 30 02:00:32.386932 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.386865 6572 cpu_manager.go:414] "containerMap: removing container" podUID="ad3ca000-d9f7-4d97-9a05-b39362550ba0" containerName="cleanup" Oct 30 02:00:32.386932 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.386872 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad3ca000-d9f7-4d97-9a05-b39362550ba0" containerName="cleanup" Oct 30 02:00:32.386932 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.386887 6572 cpu_manager.go:414] "containerMap: removing container" podUID="bc3321aa-698f-4c74-94ae-95e8729d9f27" containerName="cleanup" Oct 30 02:00:32.386932 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.386894 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc3321aa-698f-4c74-94ae-95e8729d9f27" containerName="cleanup" Oct 30 02:00:32.386932 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.386925 6572 cpu_manager.go:414] "containerMap: removing container" podUID="ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c" containerName="cleanup" Oct 30 02:00:32.386932 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.386931 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c" containerName="cleanup" Oct 30 02:00:32.386932 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.386941 6572 cpu_manager.go:414] "containerMap: removing container" podUID="419770d0-51af-4843-9ab2-8cbbe8215357" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.386948 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="419770d0-51af-4843-9ab2-8cbbe8215357" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.386958 6572 cpu_manager.go:414] "containerMap: removing container" podUID="15e59e08-4ed8-4206-80c7-244b505243ae" containerName="revoke-keys" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.386964 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15e59e08-4ed8-4206-80c7-244b505243ae" containerName="revoke-keys" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.386978 6572 cpu_manager.go:414] "containerMap: removing container" podUID="16ce2d50-2777-4827-80ec-025ff18072ed" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.386984 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16ce2d50-2777-4827-80ec-025ff18072ed" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.386993 6572 cpu_manager.go:414] "containerMap: removing container" podUID="ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.386999 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387009 6572 cpu_manager.go:414] "containerMap: removing container" podUID="45e5935e-478f-4fec-beae-0cae2ea14c1b" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387015 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45e5935e-478f-4fec-beae-0cae2ea14c1b" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387029 6572 cpu_manager.go:414] "containerMap: removing container" podUID="f818d22e-27ff-4caa-a48d-352af6583de4" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387035 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f818d22e-27ff-4caa-a48d-352af6583de4" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387046 6572 cpu_manager.go:414] "containerMap: removing container" podUID="ea84dc40-9569-4a87-a537-a8e0c0fb119d" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387053 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea84dc40-9569-4a87-a537-a8e0c0fb119d" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387070 6572 cpu_manager.go:414] "containerMap: removing container" podUID="f818d22e-27ff-4caa-a48d-352af6583de4" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387076 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f818d22e-27ff-4caa-a48d-352af6583de4" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387092 6572 cpu_manager.go:414] "containerMap: removing container" podUID="f818d22e-27ff-4caa-a48d-352af6583de4" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387098 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f818d22e-27ff-4caa-a48d-352af6583de4" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387108 6572 cpu_manager.go:414] "containerMap: removing container" podUID="dc58a71f-7f5f-4f68-aa97-a22aa15b854e" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387114 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc58a71f-7f5f-4f68-aa97-a22aa15b854e" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387124 6572 cpu_manager.go:414] "containerMap: removing container" podUID="16ce2d50-2777-4827-80ec-025ff18072ed" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387130 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16ce2d50-2777-4827-80ec-025ff18072ed" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387142 6572 cpu_manager.go:414] "containerMap: removing container" podUID="ea84dc40-9569-4a87-a537-a8e0c0fb119d" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387150 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea84dc40-9569-4a87-a537-a8e0c0fb119d" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387159 6572 cpu_manager.go:414] "containerMap: removing container" podUID="05cb266a-e1fe-4b51-a18f-96d85a78ab16" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387166 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05cb266a-e1fe-4b51-a18f-96d85a78ab16" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387176 6572 cpu_manager.go:414] "containerMap: removing container" podUID="9dc37d0b-7989-4437-9902-df0a34eb682a" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387182 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dc37d0b-7989-4437-9902-df0a34eb682a" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387199 6572 cpu_manager.go:414] "containerMap: removing container" podUID="419770d0-51af-4843-9ab2-8cbbe8215357" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387205 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="419770d0-51af-4843-9ab2-8cbbe8215357" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387218 6572 cpu_manager.go:414] "containerMap: removing container" podUID="250d2557-a543-4749-aa8c-6483de133989" containerName="revoke-keys" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387225 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="250d2557-a543-4749-aa8c-6483de133989" containerName="revoke-keys" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387239 6572 cpu_manager.go:414] "containerMap: removing container" podUID="d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387246 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387261 6572 cpu_manager.go:414] "containerMap: removing container" podUID="dbd7b9d4-eb19-4605-bdc9-21940dad8c92" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387267 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbd7b9d4-eb19-4605-bdc9-21940dad8c92" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387278 6572 cpu_manager.go:414] "containerMap: removing container" podUID="b8e5ee17-3f55-4488-aba3-2a81eceb15a4" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387285 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8e5ee17-3f55-4488-aba3-2a81eceb15a4" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387484 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="05cb266a-e1fe-4b51-a18f-96d85a78ab16" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387500 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc3321aa-698f-4c74-94ae-95e8729d9f27" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387515 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dc37d0b-7989-4437-9902-df0a34eb682a" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387525 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387533 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="419770d0-51af-4843-9ab2-8cbbe8215357" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387541 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="15e59e08-4ed8-4206-80c7-244b505243ae" containerName="revoke-keys" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387549 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="16ce2d50-2777-4827-80ec-025ff18072ed" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387558 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="45e5935e-478f-4fec-beae-0cae2ea14c1b" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387567 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="45e5935e-478f-4fec-beae-0cae2ea14c1b" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387578 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="f818d22e-27ff-4caa-a48d-352af6583de4" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387586 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387594 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea84dc40-9569-4a87-a537-a8e0c0fb119d" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387603 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad3ca000-d9f7-4d97-9a05-b39362550ba0" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387614 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="f818d22e-27ff-4caa-a48d-352af6583de4" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387626 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="f818d22e-27ff-4caa-a48d-352af6583de4" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387636 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc58a71f-7f5f-4f68-aa97-a22aa15b854e" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387645 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="16ce2d50-2777-4827-80ec-025ff18072ed" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387655 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea84dc40-9569-4a87-a537-a8e0c0fb119d" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387666 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dc37d0b-7989-4437-9902-df0a34eb682a" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387676 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="419770d0-51af-4843-9ab2-8cbbe8215357" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387685 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="05cb266a-e1fe-4b51-a18f-96d85a78ab16" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387693 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="250d2557-a543-4749-aa8c-6483de133989" containerName="revoke-keys" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387703 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbd7b9d4-eb19-4605-bdc9-21940dad8c92" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387713 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8e5ee17-3f55-4488-aba3-2a81eceb15a4" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387719 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b" containerName="cleanup" Oct 30 02:00:32.388126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387728 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbd7b9d4-eb19-4605-bdc9-21940dad8c92" containerName="cleanup" Oct 30 02:00:32.391729 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.387739 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8e5ee17-3f55-4488-aba3-2a81eceb15a4" containerName="cleanup" Oct 30 02:00:32.391729 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.388558 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b" containerName="cleanup" Oct 30 02:00:32.391729 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.388579 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbd7b9d4-eb19-4605-bdc9-21940dad8c92" containerName="cleanup" Oct 30 02:00:32.391729 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.388587 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c" containerName="cleanup" Oct 30 02:00:32.391729 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.388596 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="d818fec7-418c-447e-bd2e-b64e07fdbb86" containerName="payload-processing" Oct 30 02:00:32.391729 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.388603 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8e5ee17-3f55-4488-aba3-2a81eceb15a4" containerName="cleanup" Oct 30 02:00:32.391729 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.388611 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="16ce2d50-2777-4827-80ec-025ff18072ed" containerName="cleanup" Oct 30 02:00:32.394562 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.394537 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02f572b-f5j2k" Oct 30 02:00:32.424821 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.421914 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:00:32.424821 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.422044 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:00:32.424821 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.422103 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:00:32.424821 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.422136 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:00:32.427818 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.427785 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c02f572b-f5j2k"] Oct 30 02:00:32.517995 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.517949 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7402575-2689-4b54-859f-7f02d662456f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c02f572b-f5j2k\" (UID: \"a7402575-2689-4b54-859f-7f02d662456f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02f572b-f5j2k" Oct 30 02:00:32.518204 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.518092 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7402575-2689-4b54-859f-7f02d662456f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c02f572b-f5j2k\" (UID: \"a7402575-2689-4b54-859f-7f02d662456f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02f572b-f5j2k" Oct 30 02:00:32.518204 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.518182 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9f4j\" (UniqueName: \"kubernetes.io/projected/a7402575-2689-4b54-859f-7f02d662456f-kube-api-access-z9f4j\") pod \"maas-api-revoke-sub-c02f572b-f5j2k\" (UID: \"a7402575-2689-4b54-859f-7f02d662456f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02f572b-f5j2k" Oct 30 02:00:32.539640 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.539596 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:32.619609 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.619564 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7402575-2689-4b54-859f-7f02d662456f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c02f572b-f5j2k\" (UID: \"a7402575-2689-4b54-859f-7f02d662456f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02f572b-f5j2k" Oct 30 02:00:32.619862 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.619667 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7402575-2689-4b54-859f-7f02d662456f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c02f572b-f5j2k\" (UID: \"a7402575-2689-4b54-859f-7f02d662456f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02f572b-f5j2k" Oct 30 02:00:32.619862 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.619741 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9f4j\" (UniqueName: \"kubernetes.io/projected/a7402575-2689-4b54-859f-7f02d662456f-kube-api-access-z9f4j\") pod \"maas-api-revoke-sub-c02f572b-f5j2k\" (UID: \"a7402575-2689-4b54-859f-7f02d662456f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02f572b-f5j2k" Oct 30 02:00:32.620349 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.620326 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7402575-2689-4b54-859f-7f02d662456f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c02f572b-f5j2k\" (UID: \"a7402575-2689-4b54-859f-7f02d662456f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02f572b-f5j2k" Oct 30 02:00:32.622205 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.622179 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7402575-2689-4b54-859f-7f02d662456f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c02f572b-f5j2k\" (UID: \"a7402575-2689-4b54-859f-7f02d662456f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02f572b-f5j2k" Oct 30 02:00:32.640184 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.640101 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9f4j\" (UniqueName: \"kubernetes.io/projected/a7402575-2689-4b54-859f-7f02d662456f-kube-api-access-z9f4j\") pod \"maas-api-revoke-sub-c02f572b-f5j2k\" (UID: \"a7402575-2689-4b54-859f-7f02d662456f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02f572b-f5j2k" Oct 30 02:00:32.730073 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.730030 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02f572b-f5j2k" Oct 30 02:00:32.796219 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:32.796186 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:33.167726 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:33.167696 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c02f572b-f5j2k"] Oct 30 02:00:33.168436 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:00:33.168410 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7402575_2689_4b54_859f_7f02d662456f.slice/crio-ded1575bba5f4e29588b8c948b15a1a2444d615ea49bca3e986cab3abb2b52a6 WatchSource:0}: Error finding container ded1575bba5f4e29588b8c948b15a1a2444d615ea49bca3e986cab3abb2b52a6: Status 404 returned error can't find the container with id ded1575bba5f4e29588b8c948b15a1a2444d615ea49bca3e986cab3abb2b52a6 Oct 30 02:00:33.723334 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:33.723215 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:33.725987 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:33.725932 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02f572b-f5j2k" event={"ID":"a7402575-2689-4b54-859f-7f02d662456f","Type":"ContainerStarted","Data":"fb52c385c51c09c4f3502cabd8790fc7a2bea5186ba64ed5dd071a2607ae0fca"} Oct 30 02:00:33.726334 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:33.726000 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02f572b-f5j2k" event={"ID":"a7402575-2689-4b54-859f-7f02d662456f","Type":"ContainerStarted","Data":"ded1575bba5f4e29588b8c948b15a1a2444d615ea49bca3e986cab3abb2b52a6"} Oct 30 02:00:33.769081 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:33.769019 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02f572b-f5j2k" podStartSLOduration=1.7690005279999998 podStartE2EDuration="1.769000528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:33.767005522 +0000 UTC m=+2559.343430923" watchObservedRunningTime="2026-10-30 02:00:33.769000528 +0000 UTC m=+2559.345425929" Oct 30 02:00:34.736841 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:34.736807 6572 generic.go:356] "Generic (PLEG): container finished" podID="a7402575-2689-4b54-859f-7f02d662456f" containerID="fb52c385c51c09c4f3502cabd8790fc7a2bea5186ba64ed5dd071a2607ae0fca" exitCode=0 Oct 30 02:00:34.737258 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:34.736885 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02f572b-f5j2k" event={"ID":"a7402575-2689-4b54-859f-7f02d662456f","Type":"ContainerDied","Data":"fb52c385c51c09c4f3502cabd8790fc7a2bea5186ba64ed5dd071a2607ae0fca"} Oct 30 02:00:35.374562 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:35.374525 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:35.922488 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:35.922458 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02f572b-f5j2k" Oct 30 02:00:35.937386 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:35.937354 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7402575-2689-4b54-859f-7f02d662456f-kube-api-access-z9f4j\" (UniqueName: \"kubernetes.io/projected/a7402575-2689-4b54-859f-7f02d662456f-kube-api-access-z9f4j\") pod \"a7402575-2689-4b54-859f-7f02d662456f\" (UID: \"a7402575-2689-4b54-859f-7f02d662456f\") " Oct 30 02:00:35.937525 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:35.937510 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7402575-2689-4b54-859f-7f02d662456f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7402575-2689-4b54-859f-7f02d662456f-maas-api-openshift-ca\") pod \"a7402575-2689-4b54-859f-7f02d662456f\" (UID: \"a7402575-2689-4b54-859f-7f02d662456f\") " Oct 30 02:00:35.937658 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:35.937643 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7402575-2689-4b54-859f-7f02d662456f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7402575-2689-4b54-859f-7f02d662456f-maas-api-xks-ca\") pod \"a7402575-2689-4b54-859f-7f02d662456f\" (UID: \"a7402575-2689-4b54-859f-7f02d662456f\") " Oct 30 02:00:35.937873 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:35.937846 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7402575-2689-4b54-859f-7f02d662456f-maas-api-openshift-ca" pod "a7402575-2689-4b54-859f-7f02d662456f" (UID: "a7402575-2689-4b54-859f-7f02d662456f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:35.938234 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:35.938216 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7402575-2689-4b54-859f-7f02d662456f-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:35.939606 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:35.939576 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7402575-2689-4b54-859f-7f02d662456f-kube-api-access-z9f4j" pod "a7402575-2689-4b54-859f-7f02d662456f" (UID: "a7402575-2689-4b54-859f-7f02d662456f"). InnerVolumeSpecName "kube-api-access-z9f4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:35.939719 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:35.939700 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7402575-2689-4b54-859f-7f02d662456f-maas-api-xks-ca" pod "a7402575-2689-4b54-859f-7f02d662456f" (UID: "a7402575-2689-4b54-859f-7f02d662456f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:00:36.032419 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:36.032339 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6033f115-d5rgn"] Oct 30 02:00:36.036486 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:36.036461 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6033f115-d5rgn"] Oct 30 02:00:36.040235 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:36.040210 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7402575-2689-4b54-859f-7f02d662456f-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:36.040235 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:36.040233 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9f4j\" (UniqueName: \"kubernetes.io/projected/a7402575-2689-4b54-859f-7f02d662456f-kube-api-access-z9f4j\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:36.544394 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:36.544361 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a63de485-78a7-4de4-b777-d2f3b085de13" path="/var/lib/kubelet/pods/a63de485-78a7-4de4-b777-d2f3b085de13/volumes" Oct 30 02:00:36.756168 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:36.756142 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02f572b-f5j2k" Oct 30 02:00:36.756366 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:36.756140 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c02f572b-f5j2k" event={"ID":"a7402575-2689-4b54-859f-7f02d662456f","Type":"ContainerDied","Data":"ded1575bba5f4e29588b8c948b15a1a2444d615ea49bca3e986cab3abb2b52a6"} Oct 30 02:00:36.756366 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:36.756265 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ded1575bba5f4e29588b8c948b15a1a2444d615ea49bca3e986cab3abb2b52a6" Oct 30 02:00:36.929058 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:36.928322 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:38.434770 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:38.418361 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:38.776962 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:38.776871 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:38.953967 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:38.953923 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:39.997376 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:39.997335 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:41.449520 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:41.449481 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:42.982078 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:42.982032 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:43.624413 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:43.624372 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:43.805045 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:43.805013 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:43.855066 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:43.855027 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq"] Oct 30 02:00:43.856205 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:43.856182 6572 cpu_manager.go:414] "containerMap: removing container" podUID="b8e5ee17-3f55-4488-aba3-2a81eceb15a4" containerName="cleanup" Oct 30 02:00:43.856266 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:43.856210 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8e5ee17-3f55-4488-aba3-2a81eceb15a4" containerName="cleanup" Oct 30 02:00:43.856266 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:43.856237 6572 cpu_manager.go:414] "containerMap: removing container" podUID="d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b" containerName="cleanup" Oct 30 02:00:43.856266 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:43.856244 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2b04e2c-ad5e-4433-a35f-5dd1d5b1523b" containerName="cleanup" Oct 30 02:00:43.856266 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:43.856259 6572 cpu_manager.go:414] "containerMap: removing container" podUID="9dc37d0b-7989-4437-9902-df0a34eb682a" containerName="cleanup" Oct 30 02:00:43.856410 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:43.856268 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dc37d0b-7989-4437-9902-df0a34eb682a" containerName="cleanup" Oct 30 02:00:43.856410 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:43.856281 6572 cpu_manager.go:414] "containerMap: removing container" podUID="dbd7b9d4-eb19-4605-bdc9-21940dad8c92" containerName="cleanup" Oct 30 02:00:43.856410 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:43.856289 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbd7b9d4-eb19-4605-bdc9-21940dad8c92" containerName="cleanup" Oct 30 02:00:43.856410 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:43.856347 6572 cpu_manager.go:414] "containerMap: removing container" podUID="45e5935e-478f-4fec-beae-0cae2ea14c1b" containerName="cleanup" Oct 30 02:00:43.856410 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:43.856354 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45e5935e-478f-4fec-beae-0cae2ea14c1b" containerName="cleanup" Oct 30 02:00:43.856410 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:43.856386 6572 cpu_manager.go:414] "containerMap: removing container" podUID="a7402575-2689-4b54-859f-7f02d662456f" containerName="revoke-keys" Oct 30 02:00:43.856410 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:43.856394 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7402575-2689-4b54-859f-7f02d662456f" containerName="revoke-keys" Oct 30 02:00:43.856410 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:43.856409 6572 cpu_manager.go:414] "containerMap: removing container" podUID="ea84dc40-9569-4a87-a537-a8e0c0fb119d" containerName="cleanup" Oct 30 02:00:43.856702 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:43.856417 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea84dc40-9569-4a87-a537-a8e0c0fb119d" containerName="cleanup" Oct 30 02:00:43.856702 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:43.856429 6572 cpu_manager.go:414] "containerMap: removing container" podUID="05cb266a-e1fe-4b51-a18f-96d85a78ab16" containerName="cleanup" Oct 30 02:00:43.856702 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:43.856436 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05cb266a-e1fe-4b51-a18f-96d85a78ab16" containerName="cleanup" Oct 30 02:00:43.856702 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:43.856453 6572 cpu_manager.go:414] "containerMap: removing container" podUID="419770d0-51af-4843-9ab2-8cbbe8215357" containerName="cleanup" Oct 30 02:00:43.856702 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:43.856460 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="419770d0-51af-4843-9ab2-8cbbe8215357" containerName="cleanup" Oct 30 02:00:43.856702 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:43.856632 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dc37d0b-7989-4437-9902-df0a34eb682a" containerName="cleanup" Oct 30 02:00:43.856702 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:43.856649 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="45e5935e-478f-4fec-beae-0cae2ea14c1b" containerName="cleanup" Oct 30 02:00:43.856702 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:43.856663 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7402575-2689-4b54-859f-7f02d662456f" containerName="revoke-keys" Oct 30 02:00:43.856702 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:43.856676 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea84dc40-9569-4a87-a537-a8e0c0fb119d" containerName="cleanup" Oct 30 02:00:43.856702 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:43.856687 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="419770d0-51af-4843-9ab2-8cbbe8215357" containerName="cleanup" Oct 30 02:00:43.856702 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:43.856698 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="05cb266a-e1fe-4b51-a18f-96d85a78ab16" containerName="cleanup" Oct 30 02:00:43.857183 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:43.856710 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed66d3d7-b7a2-4dbf-8ea6-5144e1cc879c" containerName="cleanup" Oct 30 02:00:43.862638 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:43.862613 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq" Oct 30 02:00:43.878490 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:43.878415 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq"] Oct 30 02:00:43.974703 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:43.974671 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bec9f91-e94e-47a5-9726-2e7632fe9718-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq\" (UID: \"9bec9f91-e94e-47a5-9726-2e7632fe9718\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq" Oct 30 02:00:43.974918 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:43.974893 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9bec9f91-e94e-47a5-9726-2e7632fe9718-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq\" (UID: \"9bec9f91-e94e-47a5-9726-2e7632fe9718\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq" Oct 30 02:00:43.975038 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:43.975019 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv27k\" (UniqueName: \"kubernetes.io/projected/9bec9f91-e94e-47a5-9726-2e7632fe9718-kube-api-access-tv27k\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq\" (UID: \"9bec9f91-e94e-47a5-9726-2e7632fe9718\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq" Oct 30 02:00:43.975084 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:43.975056 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9bec9f91-e94e-47a5-9726-2e7632fe9718-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq\" (UID: \"9bec9f91-e94e-47a5-9726-2e7632fe9718\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq" Oct 30 02:00:44.077208 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:44.077168 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv27k\" (UniqueName: \"kubernetes.io/projected/9bec9f91-e94e-47a5-9726-2e7632fe9718-kube-api-access-tv27k\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq\" (UID: \"9bec9f91-e94e-47a5-9726-2e7632fe9718\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq" Oct 30 02:00:44.077208 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:44.077211 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9bec9f91-e94e-47a5-9726-2e7632fe9718-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq\" (UID: \"9bec9f91-e94e-47a5-9726-2e7632fe9718\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq" Oct 30 02:00:44.077672 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:44.077385 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bec9f91-e94e-47a5-9726-2e7632fe9718-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq\" (UID: \"9bec9f91-e94e-47a5-9726-2e7632fe9718\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq" Oct 30 02:00:44.077672 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:44.077560 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9bec9f91-e94e-47a5-9726-2e7632fe9718-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq\" (UID: \"9bec9f91-e94e-47a5-9726-2e7632fe9718\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq" Oct 30 02:00:44.078165 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:44.078143 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bec9f91-e94e-47a5-9726-2e7632fe9718-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq\" (UID: \"9bec9f91-e94e-47a5-9726-2e7632fe9718\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq" Oct 30 02:00:44.078240 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:44.078175 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9bec9f91-e94e-47a5-9726-2e7632fe9718-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq\" (UID: \"9bec9f91-e94e-47a5-9726-2e7632fe9718\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq" Oct 30 02:00:44.079755 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:44.079728 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9bec9f91-e94e-47a5-9726-2e7632fe9718-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq\" (UID: \"9bec9f91-e94e-47a5-9726-2e7632fe9718\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq" Oct 30 02:00:44.095836 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:44.095807 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv27k\" (UniqueName: \"kubernetes.io/projected/9bec9f91-e94e-47a5-9726-2e7632fe9718-kube-api-access-tv27k\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq\" (UID: \"9bec9f91-e94e-47a5-9726-2e7632fe9718\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq" Oct 30 02:00:44.207730 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:44.207695 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq" Oct 30 02:00:44.527141 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:44.527103 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq"] Oct 30 02:00:44.591497 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:44.591463 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq"] Oct 30 02:00:44.591666 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:44.591536 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx"] Oct 30 02:00:44.666623 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:44.666592 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:44.689243 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:44.689213 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:00:44.831876 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:44.831740 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq" event={"ID":"9bec9f91-e94e-47a5-9726-2e7632fe9718","Type":"ContainerStarted","Data":"62471d04b4188111bcf37934c5fa3a44f74ceb464e293d32f09dc7bf42670ed1"} Oct 30 02:00:44.831876 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:44.831823 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq" event={"ID":"9bec9f91-e94e-47a5-9726-2e7632fe9718","Type":"ContainerStarted","Data":"7750cd83e4161871e1161be2005dcceebfd2ea23852e21af49ea9a430e0c495f"} Oct 30 02:00:44.831876 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:44.831856 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq" Oct 30 02:00:44.853993 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:44.853960 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:44.935778 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:44.929609 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq" podStartSLOduration=1.929587218 podStartE2EDuration="1.929587218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:44.892349911 +0000 UTC m=+2570.468775315" watchObservedRunningTime="2026-10-30 02:00:44.929587218 +0000 UTC m=+2570.506012621" Oct 30 02:00:45.185505 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:45.185466 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:45.376733 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:45.376701 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:46.128238 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:46.128197 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:46.372572 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:46.372535 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:46.677690 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:46.670926 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:47.231965 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:47.231926 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-d5b799485-zjfmq"] Oct 30 02:00:47.236815 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:47.236790 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d5b799485-zjfmq" Oct 30 02:00:47.252696 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:47.252669 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-d5b799485-zjfmq"] Oct 30 02:00:47.260996 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:47.260969 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-bf552440ea85\"" Oct 30 02:00:47.345272 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:47.345235 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98f6258a-b3e1-4510-92b4-d23b95a4545d-plugins-config-volume\") pod \"payload-processing-external-model-d5b799485-zjfmq\" (UID: \"98f6258a-b3e1-4510-92b4-d23b95a4545d\") " pod="models-as-a-service/payload-processing-external-model-d5b799485-zjfmq" Oct 30 02:00:47.345464 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:47.345294 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9fkz\" (UniqueName: \"kubernetes.io/projected/98f6258a-b3e1-4510-92b4-d23b95a4545d-kube-api-access-s9fkz\") pod \"payload-processing-external-model-d5b799485-zjfmq\" (UID: \"98f6258a-b3e1-4510-92b4-d23b95a4545d\") " pod="models-as-a-service/payload-processing-external-model-d5b799485-zjfmq" Oct 30 02:00:47.345464 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:47.345447 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/98f6258a-b3e1-4510-92b4-d23b95a4545d-provider-credentials\") pod \"payload-processing-external-model-d5b799485-zjfmq\" (UID: \"98f6258a-b3e1-4510-92b4-d23b95a4545d\") " pod="models-as-a-service/payload-processing-external-model-d5b799485-zjfmq" Oct 30 02:00:47.345583 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:47.345561 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/98f6258a-b3e1-4510-92b4-d23b95a4545d-routing-overlay-volume\") pod \"payload-processing-external-model-d5b799485-zjfmq\" (UID: \"98f6258a-b3e1-4510-92b4-d23b95a4545d\") " pod="models-as-a-service/payload-processing-external-model-d5b799485-zjfmq" Oct 30 02:00:47.446954 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:47.446920 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/98f6258a-b3e1-4510-92b4-d23b95a4545d-provider-credentials\") pod \"payload-processing-external-model-d5b799485-zjfmq\" (UID: \"98f6258a-b3e1-4510-92b4-d23b95a4545d\") " pod="models-as-a-service/payload-processing-external-model-d5b799485-zjfmq" Oct 30 02:00:47.447434 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:47.447004 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/98f6258a-b3e1-4510-92b4-d23b95a4545d-routing-overlay-volume\") pod \"payload-processing-external-model-d5b799485-zjfmq\" (UID: \"98f6258a-b3e1-4510-92b4-d23b95a4545d\") " pod="models-as-a-service/payload-processing-external-model-d5b799485-zjfmq" Oct 30 02:00:47.447434 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:47.447186 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98f6258a-b3e1-4510-92b4-d23b95a4545d-plugins-config-volume\") pod \"payload-processing-external-model-d5b799485-zjfmq\" (UID: \"98f6258a-b3e1-4510-92b4-d23b95a4545d\") " pod="models-as-a-service/payload-processing-external-model-d5b799485-zjfmq" Oct 30 02:00:47.447434 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:47.447250 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9fkz\" (UniqueName: \"kubernetes.io/projected/98f6258a-b3e1-4510-92b4-d23b95a4545d-kube-api-access-s9fkz\") pod \"payload-processing-external-model-d5b799485-zjfmq\" (UID: \"98f6258a-b3e1-4510-92b4-d23b95a4545d\") " pod="models-as-a-service/payload-processing-external-model-d5b799485-zjfmq" Oct 30 02:00:47.448003 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:47.447953 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/98f6258a-b3e1-4510-92b4-d23b95a4545d-routing-overlay-volume\") pod \"payload-processing-external-model-d5b799485-zjfmq\" (UID: \"98f6258a-b3e1-4510-92b4-d23b95a4545d\") " pod="models-as-a-service/payload-processing-external-model-d5b799485-zjfmq" Oct 30 02:00:47.448452 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:47.448430 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98f6258a-b3e1-4510-92b4-d23b95a4545d-plugins-config-volume\") pod \"payload-processing-external-model-d5b799485-zjfmq\" (UID: \"98f6258a-b3e1-4510-92b4-d23b95a4545d\") " pod="models-as-a-service/payload-processing-external-model-d5b799485-zjfmq" Oct 30 02:00:47.450143 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:47.450119 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/98f6258a-b3e1-4510-92b4-d23b95a4545d-provider-credentials\") pod \"payload-processing-external-model-d5b799485-zjfmq\" (UID: \"98f6258a-b3e1-4510-92b4-d23b95a4545d\") " pod="models-as-a-service/payload-processing-external-model-d5b799485-zjfmq" Oct 30 02:00:47.466178 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:47.466141 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9fkz\" (UniqueName: \"kubernetes.io/projected/98f6258a-b3e1-4510-92b4-d23b95a4545d-kube-api-access-s9fkz\") pod \"payload-processing-external-model-d5b799485-zjfmq\" (UID: \"98f6258a-b3e1-4510-92b4-d23b95a4545d\") " pod="models-as-a-service/payload-processing-external-model-d5b799485-zjfmq" Oct 30 02:00:47.554551 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:47.554450 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d5b799485-zjfmq" Oct 30 02:00:47.859921 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:47.859891 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-d5b799485-zjfmq"] Oct 30 02:00:47.861113 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:00:47.861088 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f6258a_b3e1_4510_92b4_d23b95a4545d.slice/crio-003f8ea1d7ee7945a35926794fc8085f3ea6581154c0be21b51e7d46da4262dc WatchSource:0}: Error finding container 003f8ea1d7ee7945a35926794fc8085f3ea6581154c0be21b51e7d46da4262dc: Status 404 returned error can't find the container with id 003f8ea1d7ee7945a35926794fc8085f3ea6581154c0be21b51e7d46da4262dc Oct 30 02:00:48.297472 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:48.296823 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:48.872102 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:48.872062 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d5b799485-zjfmq" event={"ID":"98f6258a-b3e1-4510-92b4-d23b95a4545d","Type":"ContainerStarted","Data":"36a9fe612ee488058b07b34e84e35841b83dbe4587d91ec446290ad1192281bf"} Oct 30 02:00:48.872552 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:48.872121 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d5b799485-zjfmq" event={"ID":"98f6258a-b3e1-4510-92b4-d23b95a4545d","Type":"ContainerStarted","Data":"003f8ea1d7ee7945a35926794fc8085f3ea6581154c0be21b51e7d46da4262dc"} Oct 30 02:00:48.872552 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:48.872149 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-d5b799485-zjfmq" Oct 30 02:00:48.912364 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:48.912308 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-d5b799485-zjfmq" podStartSLOduration=1.912291228 podStartE2EDuration="1.912291228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:48.910126714 +0000 UTC m=+2574.486552121" watchObservedRunningTime="2026-10-30 02:00:48.912291228 +0000 UTC m=+2574.488716631" Oct 30 02:00:49.591991 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:49.591939 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx" podUID="e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a" containerName="payload-processing" containerID="cri-o://644163eb4e418e8be8f8f796e5f6ac1f00562d2a4eca65ca3998b867dbf2e16d" gracePeriod=25 Oct 30 02:00:49.833287 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:49.832201 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq" podUID="9bec9f91-e94e-47a5-9726-2e7632fe9718" containerName="payload-processing" containerID="cri-o://62471d04b4188111bcf37934c5fa3a44f74ceb464e293d32f09dc7bf42670ed1" gracePeriod=25 Oct 30 02:00:49.885047 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:49.885011 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:49.887512 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:49.887481 6572 generic.go:356] "Generic (PLEG): container finished" podID="e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a" containerID="644163eb4e418e8be8f8f796e5f6ac1f00562d2a4eca65ca3998b867dbf2e16d" exitCode=0 Oct 30 02:00:49.887638 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:49.887552 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx" event={"ID":"e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a","Type":"ContainerDied","Data":"644163eb4e418e8be8f8f796e5f6ac1f00562d2a4eca65ca3998b867dbf2e16d"} Oct 30 02:00:49.964921 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:49.964896 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx" Oct 30 02:00:50.139771 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:50.132601 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a-routing-overlay-volume\") pod \"e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a\" (UID: \"e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a\") " Oct 30 02:00:50.201793 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:50.181500 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a-routing-overlay-volume" pod "e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a" (UID: "e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:50.201793 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:50.181577 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a-provider-credentials\") pod \"e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a\" (UID: \"e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a\") " Oct 30 02:00:50.201793 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:50.181702 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a-kube-api-access-btvll\" (UniqueName: \"kubernetes.io/projected/e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a-kube-api-access-btvll\") pod \"e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a\" (UID: \"e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a\") " Oct 30 02:00:50.201793 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:50.181734 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a-plugins-config-volume\") pod \"e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a\" (UID: \"e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a\") " Oct 30 02:00:50.201793 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:50.182605 6572 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a-routing-overlay-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:50.201793 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:50.194708 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a-plugins-config-volume" pod "e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a" (UID: "e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:50.271326 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:50.270920 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a-provider-credentials" pod "e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a" (UID: "e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:50.271326 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:50.271097 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a-kube-api-access-btvll" pod "e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a" (UID: "e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a"). InnerVolumeSpecName "kube-api-access-btvll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:50.285015 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:50.284899 6572 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a-provider-credentials\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:50.285015 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:50.284939 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btvll\" (UniqueName: \"kubernetes.io/projected/e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a-kube-api-access-btvll\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:50.285015 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:50.284955 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:50.501534 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:50.501498 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq" Oct 30 02:00:50.591989 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:50.591951 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bec9f91-e94e-47a5-9726-2e7632fe9718-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9bec9f91-e94e-47a5-9726-2e7632fe9718-routing-overlay-volume\") pod \"9bec9f91-e94e-47a5-9726-2e7632fe9718\" (UID: \"9bec9f91-e94e-47a5-9726-2e7632fe9718\") " Oct 30 02:00:50.592224 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:50.592008 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bec9f91-e94e-47a5-9726-2e7632fe9718-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9bec9f91-e94e-47a5-9726-2e7632fe9718-provider-credentials\") pod \"9bec9f91-e94e-47a5-9726-2e7632fe9718\" (UID: \"9bec9f91-e94e-47a5-9726-2e7632fe9718\") " Oct 30 02:00:50.592224 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:50.592059 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bec9f91-e94e-47a5-9726-2e7632fe9718-kube-api-access-tv27k\" (UniqueName: \"kubernetes.io/projected/9bec9f91-e94e-47a5-9726-2e7632fe9718-kube-api-access-tv27k\") pod \"9bec9f91-e94e-47a5-9726-2e7632fe9718\" (UID: \"9bec9f91-e94e-47a5-9726-2e7632fe9718\") " Oct 30 02:00:50.592224 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:50.592090 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bec9f91-e94e-47a5-9726-2e7632fe9718-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bec9f91-e94e-47a5-9726-2e7632fe9718-plugins-config-volume\") pod \"9bec9f91-e94e-47a5-9726-2e7632fe9718\" (UID: \"9bec9f91-e94e-47a5-9726-2e7632fe9718\") " Oct 30 02:00:50.592493 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:50.592463 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bec9f91-e94e-47a5-9726-2e7632fe9718-routing-overlay-volume" pod "9bec9f91-e94e-47a5-9726-2e7632fe9718" (UID: "9bec9f91-e94e-47a5-9726-2e7632fe9718"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:50.592990 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:50.592966 6572 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9bec9f91-e94e-47a5-9726-2e7632fe9718-routing-overlay-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:50.593119 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:50.593037 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bec9f91-e94e-47a5-9726-2e7632fe9718-plugins-config-volume" pod "9bec9f91-e94e-47a5-9726-2e7632fe9718" (UID: "9bec9f91-e94e-47a5-9726-2e7632fe9718"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:50.595153 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:50.595125 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bec9f91-e94e-47a5-9726-2e7632fe9718-kube-api-access-tv27k" pod "9bec9f91-e94e-47a5-9726-2e7632fe9718" (UID: "9bec9f91-e94e-47a5-9726-2e7632fe9718"). InnerVolumeSpecName "kube-api-access-tv27k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:50.595260 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:50.595213 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bec9f91-e94e-47a5-9726-2e7632fe9718-provider-credentials" pod "9bec9f91-e94e-47a5-9726-2e7632fe9718" (UID: "9bec9f91-e94e-47a5-9726-2e7632fe9718"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:50.694681 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:50.694639 6572 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9bec9f91-e94e-47a5-9726-2e7632fe9718-provider-credentials\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:50.694681 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:50.694676 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv27k\" (UniqueName: \"kubernetes.io/projected/9bec9f91-e94e-47a5-9726-2e7632fe9718-kube-api-access-tv27k\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:50.694681 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:50.694687 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bec9f91-e94e-47a5-9726-2e7632fe9718-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:50.899804 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:50.899673 6572 generic.go:356] "Generic (PLEG): container finished" podID="9bec9f91-e94e-47a5-9726-2e7632fe9718" containerID="62471d04b4188111bcf37934c5fa3a44f74ceb464e293d32f09dc7bf42670ed1" exitCode=0 Oct 30 02:00:50.899804 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:50.899723 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq" Oct 30 02:00:50.900359 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:50.899804 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq" event={"ID":"9bec9f91-e94e-47a5-9726-2e7632fe9718","Type":"ContainerDied","Data":"62471d04b4188111bcf37934c5fa3a44f74ceb464e293d32f09dc7bf42670ed1"} Oct 30 02:00:50.900359 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:50.899864 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq" event={"ID":"9bec9f91-e94e-47a5-9726-2e7632fe9718","Type":"ContainerDied","Data":"7750cd83e4161871e1161be2005dcceebfd2ea23852e21af49ea9a430e0c495f"} Oct 30 02:00:50.900359 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:50.899895 6572 scope.go:122] "RemoveContainer" containerID="62471d04b4188111bcf37934c5fa3a44f74ceb464e293d32f09dc7bf42670ed1" Oct 30 02:00:50.901729 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:50.901675 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx" event={"ID":"e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a","Type":"ContainerDied","Data":"b58fd9e11f1fe2d6fb3237095678cbf2102028d33c2b7f554a73ea4dd70bfc9b"} Oct 30 02:00:50.901822 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:50.901691 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx" Oct 30 02:00:50.913141 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:50.913119 6572 scope.go:122] "RemoveContainer" containerID="62471d04b4188111bcf37934c5fa3a44f74ceb464e293d32f09dc7bf42670ed1" Oct 30 02:00:50.913515 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:00:50.913494 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62471d04b4188111bcf37934c5fa3a44f74ceb464e293d32f09dc7bf42670ed1\": container with ID starting with 62471d04b4188111bcf37934c5fa3a44f74ceb464e293d32f09dc7bf42670ed1 not found: ID does not exist" containerID="62471d04b4188111bcf37934c5fa3a44f74ceb464e293d32f09dc7bf42670ed1" Oct 30 02:00:50.913575 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:50.913526 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62471d04b4188111bcf37934c5fa3a44f74ceb464e293d32f09dc7bf42670ed1"} err="failed to get container status \"62471d04b4188111bcf37934c5fa3a44f74ceb464e293d32f09dc7bf42670ed1\": rpc error: code = NotFound desc = could not find container \"62471d04b4188111bcf37934c5fa3a44f74ceb464e293d32f09dc7bf42670ed1\": container with ID starting with 62471d04b4188111bcf37934c5fa3a44f74ceb464e293d32f09dc7bf42670ed1 not found: ID does not exist" Oct 30 02:00:50.913575 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:50.913548 6572 scope.go:122] "RemoveContainer" containerID="644163eb4e418e8be8f8f796e5f6ac1f00562d2a4eca65ca3998b867dbf2e16d" Oct 30 02:00:50.964094 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:50.964048 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx"] Oct 30 02:00:50.968585 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:50.968547 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-974bdpjmx"] Oct 30 02:00:51.001467 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:51.001421 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq"] Oct 30 02:00:51.006044 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:51.006007 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-6775ncssq"] Oct 30 02:00:51.494773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:51.494284 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:51.704158 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:51.704122 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:52.544466 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:52.544432 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bec9f91-e94e-47a5-9726-2e7632fe9718" path="/var/lib/kubelet/pods/9bec9f91-e94e-47a5-9726-2e7632fe9718/volumes" Oct 30 02:00:52.544934 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:52.544919 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a" path="/var/lib/kubelet/pods/e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a/volumes" Oct 30 02:00:53.166521 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:53.166488 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:53.623152 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:53.623061 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3f8da4e-x6wx2"] Oct 30 02:00:53.624049 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:53.624026 6572 cpu_manager.go:414] "containerMap: removing container" podUID="e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a" containerName="payload-processing" Oct 30 02:00:53.624049 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:53.624049 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a" containerName="payload-processing" Oct 30 02:00:53.624245 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:53.624103 6572 cpu_manager.go:414] "containerMap: removing container" podUID="9bec9f91-e94e-47a5-9726-2e7632fe9718" containerName="payload-processing" Oct 30 02:00:53.624245 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:53.624115 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bec9f91-e94e-47a5-9726-2e7632fe9718" containerName="payload-processing" Oct 30 02:00:53.624379 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:53.624365 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="e73dcd7f-a5c3-4468-a15e-d98a9bf94c5a" containerName="payload-processing" Oct 30 02:00:53.624435 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:53.624392 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bec9f91-e94e-47a5-9726-2e7632fe9718" containerName="payload-processing" Oct 30 02:00:53.630907 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:53.630885 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f8da4e-x6wx2" Oct 30 02:00:53.645575 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:53.645547 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3bc7205-b501-4898-a3cb-222876418dbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3f8da4e-x6wx2\" (UID: \"f3bc7205-b501-4898-a3cb-222876418dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f8da4e-x6wx2" Oct 30 02:00:53.645710 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:53.645596 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lbc8\" (UniqueName: \"kubernetes.io/projected/f3bc7205-b501-4898-a3cb-222876418dbd-kube-api-access-4lbc8\") pod \"maas-api-revoke-sub-b3f8da4e-x6wx2\" (UID: \"f3bc7205-b501-4898-a3cb-222876418dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f8da4e-x6wx2" Oct 30 02:00:53.645710 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:53.645674 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3bc7205-b501-4898-a3cb-222876418dbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3f8da4e-x6wx2\" (UID: \"f3bc7205-b501-4898-a3cb-222876418dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f8da4e-x6wx2" Oct 30 02:00:53.647997 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:53.647970 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3f8da4e-x6wx2"] Oct 30 02:00:53.651864 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:53.651843 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:00:53.651958 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:53.651899 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:00:53.656816 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:53.656797 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:00:53.656940 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:53.656898 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:00:53.747400 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:53.747364 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3bc7205-b501-4898-a3cb-222876418dbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3f8da4e-x6wx2\" (UID: \"f3bc7205-b501-4898-a3cb-222876418dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f8da4e-x6wx2" Oct 30 02:00:53.747562 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:53.747413 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lbc8\" (UniqueName: \"kubernetes.io/projected/f3bc7205-b501-4898-a3cb-222876418dbd-kube-api-access-4lbc8\") pod \"maas-api-revoke-sub-b3f8da4e-x6wx2\" (UID: \"f3bc7205-b501-4898-a3cb-222876418dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f8da4e-x6wx2" Oct 30 02:00:53.747562 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:53.747454 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3bc7205-b501-4898-a3cb-222876418dbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3f8da4e-x6wx2\" (UID: \"f3bc7205-b501-4898-a3cb-222876418dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f8da4e-x6wx2" Oct 30 02:00:53.748110 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:53.748075 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3bc7205-b501-4898-a3cb-222876418dbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3f8da4e-x6wx2\" (UID: \"f3bc7205-b501-4898-a3cb-222876418dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f8da4e-x6wx2" Oct 30 02:00:53.749979 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:53.749960 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3bc7205-b501-4898-a3cb-222876418dbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3f8da4e-x6wx2\" (UID: \"f3bc7205-b501-4898-a3cb-222876418dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f8da4e-x6wx2" Oct 30 02:00:53.766414 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:53.766381 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lbc8\" (UniqueName: \"kubernetes.io/projected/f3bc7205-b501-4898-a3cb-222876418dbd-kube-api-access-4lbc8\") pod \"maas-api-revoke-sub-b3f8da4e-x6wx2\" (UID: \"f3bc7205-b501-4898-a3cb-222876418dbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f8da4e-x6wx2" Oct 30 02:00:53.950503 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:53.950468 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f8da4e-x6wx2" Oct 30 02:00:54.256625 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:00:54.256596 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3bc7205_b501_4898_a3cb_222876418dbd.slice/crio-a6c6a6a67faeb090f177706e66147de19c4f8f4d89dc6f171634217aa44eb7a3 WatchSource:0}: Error finding container a6c6a6a67faeb090f177706e66147de19c4f8f4d89dc6f171634217aa44eb7a3: Status 404 returned error can't find the container with id a6c6a6a67faeb090f177706e66147de19c4f8f4d89dc6f171634217aa44eb7a3 Oct 30 02:00:54.256886 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:54.256862 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3f8da4e-x6wx2"] Oct 30 02:00:54.772692 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:54.772591 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:54.944592 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:54.944557 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f8da4e-x6wx2" event={"ID":"f3bc7205-b501-4898-a3cb-222876418dbd","Type":"ContainerStarted","Data":"e03bd87c6bfb1bf25658f4fee7a82b071f670363bcbb2445ef42f0a8bfa731a9"} Oct 30 02:00:54.944804 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:54.944623 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f8da4e-x6wx2" event={"ID":"f3bc7205-b501-4898-a3cb-222876418dbd","Type":"ContainerStarted","Data":"a6c6a6a67faeb090f177706e66147de19c4f8f4d89dc6f171634217aa44eb7a3"} Oct 30 02:00:55.956179 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:55.956138 6572 generic.go:356] "Generic (PLEG): container finished" podID="f3bc7205-b501-4898-a3cb-222876418dbd" containerID="e03bd87c6bfb1bf25658f4fee7a82b071f670363bcbb2445ef42f0a8bfa731a9" exitCode=0 Oct 30 02:00:55.956690 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:55.956374 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f8da4e-x6wx2" event={"ID":"f3bc7205-b501-4898-a3cb-222876418dbd","Type":"ContainerDied","Data":"e03bd87c6bfb1bf25658f4fee7a82b071f670363bcbb2445ef42f0a8bfa731a9"} Oct 30 02:00:56.391257 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:56.391113 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:57.138483 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:57.138454 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f8da4e-x6wx2" Oct 30 02:00:57.202405 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:57.202371 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3bc7205-b501-4898-a3cb-222876418dbd-kube-api-access-4lbc8\" (UniqueName: \"kubernetes.io/projected/f3bc7205-b501-4898-a3cb-222876418dbd-kube-api-access-4lbc8\") pod \"f3bc7205-b501-4898-a3cb-222876418dbd\" (UID: \"f3bc7205-b501-4898-a3cb-222876418dbd\") " Oct 30 02:00:57.202663 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:57.202638 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3bc7205-b501-4898-a3cb-222876418dbd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3bc7205-b501-4898-a3cb-222876418dbd-maas-api-openshift-ca\") pod \"f3bc7205-b501-4898-a3cb-222876418dbd\" (UID: \"f3bc7205-b501-4898-a3cb-222876418dbd\") " Oct 30 02:00:57.202819 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:57.202778 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3bc7205-b501-4898-a3cb-222876418dbd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3bc7205-b501-4898-a3cb-222876418dbd-maas-api-xks-ca\") pod \"f3bc7205-b501-4898-a3cb-222876418dbd\" (UID: \"f3bc7205-b501-4898-a3cb-222876418dbd\") " Oct 30 02:00:57.203223 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:57.203191 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3bc7205-b501-4898-a3cb-222876418dbd-maas-api-openshift-ca" pod "f3bc7205-b501-4898-a3cb-222876418dbd" (UID: "f3bc7205-b501-4898-a3cb-222876418dbd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:57.203454 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:57.203434 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3bc7205-b501-4898-a3cb-222876418dbd-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:57.205128 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:57.205101 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3bc7205-b501-4898-a3cb-222876418dbd-kube-api-access-4lbc8" pod "f3bc7205-b501-4898-a3cb-222876418dbd" (UID: "f3bc7205-b501-4898-a3cb-222876418dbd"). InnerVolumeSpecName "kube-api-access-4lbc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:57.205341 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:57.205323 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3bc7205-b501-4898-a3cb-222876418dbd-maas-api-xks-ca" pod "f3bc7205-b501-4898-a3cb-222876418dbd" (UID: "f3bc7205-b501-4898-a3cb-222876418dbd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:00:57.305273 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:57.305236 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3bc7205-b501-4898-a3cb-222876418dbd-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:57.305273 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:57.305271 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lbc8\" (UniqueName: \"kubernetes.io/projected/f3bc7205-b501-4898-a3cb-222876418dbd-kube-api-access-4lbc8\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:00:57.852209 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:57.852168 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:57.996577 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:57.996512 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f8da4e-x6wx2" event={"ID":"f3bc7205-b501-4898-a3cb-222876418dbd","Type":"ContainerDied","Data":"a6c6a6a67faeb090f177706e66147de19c4f8f4d89dc6f171634217aa44eb7a3"} Oct 30 02:00:57.996827 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:57.996525 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f8da4e-x6wx2" Oct 30 02:00:57.996827 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:57.996632 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6c6a6a67faeb090f177706e66147de19c4f8f4d89dc6f171634217aa44eb7a3" Oct 30 02:00:58.032546 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:58.032512 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:58.842898 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:58.842865 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:59.034073 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:59.033996 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:59.890888 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:59.890845 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-d5b799485-zjfmq" Oct 30 02:00:59.966045 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:59.966005 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:59.986777 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:00:59.986231 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-546cdc55c7-d64gv"] Oct 30 02:01:00.192028 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:00.191994 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-55f6dc654-dtcnz"] Oct 30 02:01:00.192782 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:00.192730 6572 cpu_manager.go:414] "containerMap: removing container" podUID="f3bc7205-b501-4898-a3cb-222876418dbd" containerName="revoke-keys" Oct 30 02:01:00.192782 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:00.192768 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3bc7205-b501-4898-a3cb-222876418dbd" containerName="revoke-keys" Oct 30 02:01:00.192975 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:00.192964 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3bc7205-b501-4898-a3cb-222876418dbd" containerName="revoke-keys" Oct 30 02:01:00.198437 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:00.198416 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55f6dc654-dtcnz" Oct 30 02:01:00.212401 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:00.212372 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-8c415a273753\"" Oct 30 02:01:00.215971 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:00.215941 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-55f6dc654-dtcnz"] Oct 30 02:01:00.266714 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:00.266681 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d87xq\" (UniqueName: \"kubernetes.io/projected/20d4a043-ea77-43f9-979c-aeddef258caf-kube-api-access-d87xq\") pod \"payload-processing-external-model-55f6dc654-dtcnz\" (UID: \"20d4a043-ea77-43f9-979c-aeddef258caf\") " pod="models-as-a-service/payload-processing-external-model-55f6dc654-dtcnz" Oct 30 02:01:00.266913 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:00.266817 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/20d4a043-ea77-43f9-979c-aeddef258caf-provider-credentials\") pod \"payload-processing-external-model-55f6dc654-dtcnz\" (UID: \"20d4a043-ea77-43f9-979c-aeddef258caf\") " pod="models-as-a-service/payload-processing-external-model-55f6dc654-dtcnz" Oct 30 02:01:00.266975 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:00.266956 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20d4a043-ea77-43f9-979c-aeddef258caf-plugins-config-volume\") pod \"payload-processing-external-model-55f6dc654-dtcnz\" (UID: \"20d4a043-ea77-43f9-979c-aeddef258caf\") " pod="models-as-a-service/payload-processing-external-model-55f6dc654-dtcnz" Oct 30 02:01:00.267053 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:00.267027 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/20d4a043-ea77-43f9-979c-aeddef258caf-routing-overlay-volume\") pod \"payload-processing-external-model-55f6dc654-dtcnz\" (UID: \"20d4a043-ea77-43f9-979c-aeddef258caf\") " pod="models-as-a-service/payload-processing-external-model-55f6dc654-dtcnz" Oct 30 02:01:00.368259 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:00.368219 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d87xq\" (UniqueName: \"kubernetes.io/projected/20d4a043-ea77-43f9-979c-aeddef258caf-kube-api-access-d87xq\") pod \"payload-processing-external-model-55f6dc654-dtcnz\" (UID: \"20d4a043-ea77-43f9-979c-aeddef258caf\") " pod="models-as-a-service/payload-processing-external-model-55f6dc654-dtcnz" Oct 30 02:01:00.368432 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:00.368342 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/20d4a043-ea77-43f9-979c-aeddef258caf-provider-credentials\") pod \"payload-processing-external-model-55f6dc654-dtcnz\" (UID: \"20d4a043-ea77-43f9-979c-aeddef258caf\") " pod="models-as-a-service/payload-processing-external-model-55f6dc654-dtcnz" Oct 30 02:01:00.368510 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:00.368484 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20d4a043-ea77-43f9-979c-aeddef258caf-plugins-config-volume\") pod \"payload-processing-external-model-55f6dc654-dtcnz\" (UID: \"20d4a043-ea77-43f9-979c-aeddef258caf\") " pod="models-as-a-service/payload-processing-external-model-55f6dc654-dtcnz" Oct 30 02:01:00.368557 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:00.368533 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/20d4a043-ea77-43f9-979c-aeddef258caf-routing-overlay-volume\") pod \"payload-processing-external-model-55f6dc654-dtcnz\" (UID: \"20d4a043-ea77-43f9-979c-aeddef258caf\") " pod="models-as-a-service/payload-processing-external-model-55f6dc654-dtcnz" Oct 30 02:01:00.369313 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:00.369281 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/20d4a043-ea77-43f9-979c-aeddef258caf-routing-overlay-volume\") pod \"payload-processing-external-model-55f6dc654-dtcnz\" (UID: \"20d4a043-ea77-43f9-979c-aeddef258caf\") " pod="models-as-a-service/payload-processing-external-model-55f6dc654-dtcnz" Oct 30 02:01:00.369313 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:00.369305 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20d4a043-ea77-43f9-979c-aeddef258caf-plugins-config-volume\") pod \"payload-processing-external-model-55f6dc654-dtcnz\" (UID: \"20d4a043-ea77-43f9-979c-aeddef258caf\") " pod="models-as-a-service/payload-processing-external-model-55f6dc654-dtcnz" Oct 30 02:01:00.371051 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:00.371032 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/20d4a043-ea77-43f9-979c-aeddef258caf-provider-credentials\") pod \"payload-processing-external-model-55f6dc654-dtcnz\" (UID: \"20d4a043-ea77-43f9-979c-aeddef258caf\") " pod="models-as-a-service/payload-processing-external-model-55f6dc654-dtcnz" Oct 30 02:01:00.387076 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:00.387053 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d87xq\" (UniqueName: \"kubernetes.io/projected/20d4a043-ea77-43f9-979c-aeddef258caf-kube-api-access-d87xq\") pod \"payload-processing-external-model-55f6dc654-dtcnz\" (UID: \"20d4a043-ea77-43f9-979c-aeddef258caf\") " pod="models-as-a-service/payload-processing-external-model-55f6dc654-dtcnz" Oct 30 02:01:00.513425 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:00.513339 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55f6dc654-dtcnz" Oct 30 02:01:00.817234 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:01:00.817202 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d4a043_ea77_43f9_979c_aeddef258caf.slice/crio-0a0cabf226d17a4f1f216fec56d3a9409013bd87e22d31f07701153022bedca4 WatchSource:0}: Error finding container 0a0cabf226d17a4f1f216fec56d3a9409013bd87e22d31f07701153022bedca4: Status 404 returned error can't find the container with id 0a0cabf226d17a4f1f216fec56d3a9409013bd87e22d31f07701153022bedca4 Oct 30 02:01:00.819221 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:00.819199 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-55f6dc654-dtcnz"] Oct 30 02:01:01.038986 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:01.038948 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55f6dc654-dtcnz" event={"ID":"20d4a043-ea77-43f9-979c-aeddef258caf","Type":"ContainerStarted","Data":"dde60e2d88727ed8fd61056f4711ddc57e1ea2321df51d296eb0a7d03e19a062"} Oct 30 02:01:01.039430 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:01.039016 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-55f6dc654-dtcnz" Oct 30 02:01:01.039430 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:01.039039 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55f6dc654-dtcnz" event={"ID":"20d4a043-ea77-43f9-979c-aeddef258caf","Type":"ContainerStarted","Data":"0a0cabf226d17a4f1f216fec56d3a9409013bd87e22d31f07701153022bedca4"} Oct 30 02:01:01.079190 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:01.079099 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-55f6dc654-dtcnz" podStartSLOduration=1.07908518 podStartE2EDuration="1.07908518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:01:01.07579534 +0000 UTC m=+2586.652220741" watchObservedRunningTime="2026-10-30 02:01:01.07908518 +0000 UTC m=+2586.655510590" Oct 30 02:01:01.562403 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:01.560658 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:03.214290 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:03.214248 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:04.342224 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:04.342190 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5da09025-87cdv"] Oct 30 02:01:04.347060 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:04.347043 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da09025-87cdv" Oct 30 02:01:04.372202 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:04.372170 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5da09025-87cdv"] Oct 30 02:01:04.378600 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:04.378574 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:01:04.378600 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:04.378582 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:01:04.378822 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:04.378582 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:01:04.378822 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:04.378663 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:01:04.427374 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:04.427342 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c9af176-42be-4616-8164-7d70c34dad4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5da09025-87cdv\" (UID: \"2c9af176-42be-4616-8164-7d70c34dad4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da09025-87cdv" Oct 30 02:01:04.427547 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:04.427527 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrmk5\" (UniqueName: \"kubernetes.io/projected/2c9af176-42be-4616-8164-7d70c34dad4c-kube-api-access-qrmk5\") pod \"maas-api-revoke-sub-5da09025-87cdv\" (UID: \"2c9af176-42be-4616-8164-7d70c34dad4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da09025-87cdv" Oct 30 02:01:04.427832 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:04.427811 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c9af176-42be-4616-8164-7d70c34dad4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5da09025-87cdv\" (UID: \"2c9af176-42be-4616-8164-7d70c34dad4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da09025-87cdv" Oct 30 02:01:04.529064 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:04.529026 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c9af176-42be-4616-8164-7d70c34dad4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5da09025-87cdv\" (UID: \"2c9af176-42be-4616-8164-7d70c34dad4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da09025-87cdv" Oct 30 02:01:04.529236 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:04.529170 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrmk5\" (UniqueName: \"kubernetes.io/projected/2c9af176-42be-4616-8164-7d70c34dad4c-kube-api-access-qrmk5\") pod \"maas-api-revoke-sub-5da09025-87cdv\" (UID: \"2c9af176-42be-4616-8164-7d70c34dad4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da09025-87cdv" Oct 30 02:01:04.529236 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:04.529231 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c9af176-42be-4616-8164-7d70c34dad4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5da09025-87cdv\" (UID: \"2c9af176-42be-4616-8164-7d70c34dad4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da09025-87cdv" Oct 30 02:01:04.529910 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:04.529886 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c9af176-42be-4616-8164-7d70c34dad4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5da09025-87cdv\" (UID: \"2c9af176-42be-4616-8164-7d70c34dad4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da09025-87cdv" Oct 30 02:01:04.531543 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:04.531515 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c9af176-42be-4616-8164-7d70c34dad4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5da09025-87cdv\" (UID: \"2c9af176-42be-4616-8164-7d70c34dad4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da09025-87cdv" Oct 30 02:01:04.551961 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:04.551936 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrmk5\" (UniqueName: \"kubernetes.io/projected/2c9af176-42be-4616-8164-7d70c34dad4c-kube-api-access-qrmk5\") pod \"maas-api-revoke-sub-5da09025-87cdv\" (UID: \"2c9af176-42be-4616-8164-7d70c34dad4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da09025-87cdv" Oct 30 02:01:04.662210 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:04.662189 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da09025-87cdv" Oct 30 02:01:04.971478 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:01:04.971444 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c9af176_42be_4616_8164_7d70c34dad4c.slice/crio-01626043691df717a523d90556f98d49a1b418130994e571d03c9d0bc3a3e3f9 WatchSource:0}: Error finding container 01626043691df717a523d90556f98d49a1b418130994e571d03c9d0bc3a3e3f9: Status 404 returned error can't find the container with id 01626043691df717a523d90556f98d49a1b418130994e571d03c9d0bc3a3e3f9 Oct 30 02:01:04.971933 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:04.971910 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5da09025-87cdv"] Oct 30 02:01:04.987514 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:04.987480 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-546cdc55c7-d64gv" podUID="6130dbc6-b1f1-4672-90cc-7e3be07be0e1" containerName="payload-processing" containerID="cri-o://3cbd133dbe2bc8a965078eaac06c8f939bff5fa44198ff2b90fd47c1e3b9c518" gracePeriod=25 Oct 30 02:01:05.084853 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:05.084818 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:05.087234 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:05.087207 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da09025-87cdv" event={"ID":"2c9af176-42be-4616-8164-7d70c34dad4c","Type":"ContainerStarted","Data":"01626043691df717a523d90556f98d49a1b418130994e571d03c9d0bc3a3e3f9"} Oct 30 02:01:05.250662 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:05.250637 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-546cdc55c7-d64gv" Oct 30 02:01:05.341088 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:05.341058 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6130dbc6-b1f1-4672-90cc-7e3be07be0e1-kube-api-access-r286c\" (UniqueName: \"kubernetes.io/projected/6130dbc6-b1f1-4672-90cc-7e3be07be0e1-kube-api-access-r286c\") pod \"6130dbc6-b1f1-4672-90cc-7e3be07be0e1\" (UID: \"6130dbc6-b1f1-4672-90cc-7e3be07be0e1\") " Oct 30 02:01:05.341265 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:05.341160 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6130dbc6-b1f1-4672-90cc-7e3be07be0e1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6130dbc6-b1f1-4672-90cc-7e3be07be0e1-routing-overlay-volume\") pod \"6130dbc6-b1f1-4672-90cc-7e3be07be0e1\" (UID: \"6130dbc6-b1f1-4672-90cc-7e3be07be0e1\") " Oct 30 02:01:05.341265 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:05.341202 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6130dbc6-b1f1-4672-90cc-7e3be07be0e1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6130dbc6-b1f1-4672-90cc-7e3be07be0e1-provider-credentials\") pod \"6130dbc6-b1f1-4672-90cc-7e3be07be0e1\" (UID: \"6130dbc6-b1f1-4672-90cc-7e3be07be0e1\") " Oct 30 02:01:05.341356 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:05.341266 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6130dbc6-b1f1-4672-90cc-7e3be07be0e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6130dbc6-b1f1-4672-90cc-7e3be07be0e1-plugins-config-volume\") pod \"6130dbc6-b1f1-4672-90cc-7e3be07be0e1\" (UID: \"6130dbc6-b1f1-4672-90cc-7e3be07be0e1\") " Oct 30 02:01:05.341598 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:05.341517 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6130dbc6-b1f1-4672-90cc-7e3be07be0e1-routing-overlay-volume" pod "6130dbc6-b1f1-4672-90cc-7e3be07be0e1" (UID: "6130dbc6-b1f1-4672-90cc-7e3be07be0e1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:05.341742 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:05.341701 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6130dbc6-b1f1-4672-90cc-7e3be07be0e1-plugins-config-volume" pod "6130dbc6-b1f1-4672-90cc-7e3be07be0e1" (UID: "6130dbc6-b1f1-4672-90cc-7e3be07be0e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:05.341864 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:05.341846 6572 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6130dbc6-b1f1-4672-90cc-7e3be07be0e1-routing-overlay-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:01:05.341915 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:05.341870 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6130dbc6-b1f1-4672-90cc-7e3be07be0e1-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:01:05.343631 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:05.343611 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6130dbc6-b1f1-4672-90cc-7e3be07be0e1-kube-api-access-r286c" pod "6130dbc6-b1f1-4672-90cc-7e3be07be0e1" (UID: "6130dbc6-b1f1-4672-90cc-7e3be07be0e1"). InnerVolumeSpecName "kube-api-access-r286c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:05.343958 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:05.343641 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6130dbc6-b1f1-4672-90cc-7e3be07be0e1-provider-credentials" pod "6130dbc6-b1f1-4672-90cc-7e3be07be0e1" (UID: "6130dbc6-b1f1-4672-90cc-7e3be07be0e1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:05.392811 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:05.392781 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:05.461991 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:05.445072 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r286c\" (UniqueName: \"kubernetes.io/projected/6130dbc6-b1f1-4672-90cc-7e3be07be0e1-kube-api-access-r286c\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:01:05.461991 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:05.445108 6572 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6130dbc6-b1f1-4672-90cc-7e3be07be0e1-provider-credentials\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:01:05.629176 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:05.629139 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:05.634281 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:05.634258 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-55f6dc654-dtcnz"] Oct 30 02:01:05.637794 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:05.637771 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-55f6dc654-dtcnz" Oct 30 02:01:06.108799 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:06.108738 6572 generic.go:356] "Generic (PLEG): container finished" podID="6130dbc6-b1f1-4672-90cc-7e3be07be0e1" containerID="3cbd133dbe2bc8a965078eaac06c8f939bff5fa44198ff2b90fd47c1e3b9c518" exitCode=0 Oct 30 02:01:06.109007 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:06.108811 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-546cdc55c7-d64gv" Oct 30 02:01:06.109007 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:06.108810 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-546cdc55c7-d64gv" event={"ID":"6130dbc6-b1f1-4672-90cc-7e3be07be0e1","Type":"ContainerDied","Data":"3cbd133dbe2bc8a965078eaac06c8f939bff5fa44198ff2b90fd47c1e3b9c518"} Oct 30 02:01:06.109007 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:06.108876 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-546cdc55c7-d64gv" event={"ID":"6130dbc6-b1f1-4672-90cc-7e3be07be0e1","Type":"ContainerDied","Data":"faae4b324bb3e6e3f74d27da8fcc7668444c763c68be477da68f320bd242839c"} Oct 30 02:01:06.109007 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:06.108908 6572 scope.go:122] "RemoveContainer" containerID="3cbd133dbe2bc8a965078eaac06c8f939bff5fa44198ff2b90fd47c1e3b9c518" Oct 30 02:01:06.111788 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:06.111572 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da09025-87cdv" event={"ID":"2c9af176-42be-4616-8164-7d70c34dad4c","Type":"ContainerStarted","Data":"7296d4b88560babfcab03c1eb67e8fba9c9f8ff3ec6e84ddb528dcfee89f161e"} Oct 30 02:01:06.165743 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:06.165692 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da09025-87cdv" podStartSLOduration=2.165674855 podStartE2EDuration="2.165674855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:01:06.163784637 +0000 UTC m=+2591.740210040" watchObservedRunningTime="2026-10-30 02:01:06.165674855 +0000 UTC m=+2591.742100257" Oct 30 02:01:06.188435 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:06.188411 6572 scope.go:122] "RemoveContainer" containerID="3cbd133dbe2bc8a965078eaac06c8f939bff5fa44198ff2b90fd47c1e3b9c518" Oct 30 02:01:06.188788 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:06.188765 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cbd133dbe2bc8a965078eaac06c8f939bff5fa44198ff2b90fd47c1e3b9c518\": container with ID starting with 3cbd133dbe2bc8a965078eaac06c8f939bff5fa44198ff2b90fd47c1e3b9c518 not found: ID does not exist" containerID="3cbd133dbe2bc8a965078eaac06c8f939bff5fa44198ff2b90fd47c1e3b9c518" Oct 30 02:01:06.188885 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:06.188795 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cbd133dbe2bc8a965078eaac06c8f939bff5fa44198ff2b90fd47c1e3b9c518"} err="failed to get container status \"3cbd133dbe2bc8a965078eaac06c8f939bff5fa44198ff2b90fd47c1e3b9c518\": rpc error: code = NotFound desc = could not find container \"3cbd133dbe2bc8a965078eaac06c8f939bff5fa44198ff2b90fd47c1e3b9c518\": container with ID starting with 3cbd133dbe2bc8a965078eaac06c8f939bff5fa44198ff2b90fd47c1e3b9c518 not found: ID does not exist" Oct 30 02:01:06.221513 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:06.221471 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-546cdc55c7-d64gv"] Oct 30 02:01:06.225550 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:06.225520 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-546cdc55c7-d64gv"] Oct 30 02:01:06.544869 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:06.544836 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6130dbc6-b1f1-4672-90cc-7e3be07be0e1" path="/var/lib/kubelet/pods/6130dbc6-b1f1-4672-90cc-7e3be07be0e1/volumes" Oct 30 02:01:06.786441 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:06.786343 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:07.122829 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:07.122791 6572 generic.go:356] "Generic (PLEG): container finished" podID="2c9af176-42be-4616-8164-7d70c34dad4c" containerID="7296d4b88560babfcab03c1eb67e8fba9c9f8ff3ec6e84ddb528dcfee89f161e" exitCode=0 Oct 30 02:01:07.123042 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:07.122879 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da09025-87cdv" event={"ID":"2c9af176-42be-4616-8164-7d70c34dad4c","Type":"ContainerDied","Data":"7296d4b88560babfcab03c1eb67e8fba9c9f8ff3ec6e84ddb528dcfee89f161e"} Oct 30 02:01:07.146465 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:07.146430 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:08.089841 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:08.089805 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:08.288038 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:08.287999 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:08.295241 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:08.295215 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da09025-87cdv" Oct 30 02:01:08.412947 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:08.412826 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c9af176-42be-4616-8164-7d70c34dad4c-kube-api-access-qrmk5\" (UniqueName: \"kubernetes.io/projected/2c9af176-42be-4616-8164-7d70c34dad4c-kube-api-access-qrmk5\") pod \"2c9af176-42be-4616-8164-7d70c34dad4c\" (UID: \"2c9af176-42be-4616-8164-7d70c34dad4c\") " Oct 30 02:01:08.413299 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:08.413260 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c9af176-42be-4616-8164-7d70c34dad4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c9af176-42be-4616-8164-7d70c34dad4c-maas-api-openshift-ca\") pod \"2c9af176-42be-4616-8164-7d70c34dad4c\" (UID: \"2c9af176-42be-4616-8164-7d70c34dad4c\") " Oct 30 02:01:08.413808 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:08.413780 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c9af176-42be-4616-8164-7d70c34dad4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c9af176-42be-4616-8164-7d70c34dad4c-maas-api-xks-ca\") pod \"2c9af176-42be-4616-8164-7d70c34dad4c\" (UID: \"2c9af176-42be-4616-8164-7d70c34dad4c\") " Oct 30 02:01:08.416193 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:08.416167 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c9af176-42be-4616-8164-7d70c34dad4c-maas-api-openshift-ca" pod "2c9af176-42be-4616-8164-7d70c34dad4c" (UID: "2c9af176-42be-4616-8164-7d70c34dad4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:08.417725 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:08.417687 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c9af176-42be-4616-8164-7d70c34dad4c-maas-api-xks-ca" pod "2c9af176-42be-4616-8164-7d70c34dad4c" (UID: "2c9af176-42be-4616-8164-7d70c34dad4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:01:08.418281 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:08.418256 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c9af176-42be-4616-8164-7d70c34dad4c-kube-api-access-qrmk5" pod "2c9af176-42be-4616-8164-7d70c34dad4c" (UID: "2c9af176-42be-4616-8164-7d70c34dad4c"). InnerVolumeSpecName "kube-api-access-qrmk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:08.482154 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:08.482122 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:08.516067 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:08.516040 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrmk5\" (UniqueName: \"kubernetes.io/projected/2c9af176-42be-4616-8164-7d70c34dad4c-kube-api-access-qrmk5\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:01:08.516067 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:08.516063 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c9af176-42be-4616-8164-7d70c34dad4c-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:01:08.516243 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:08.516075 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c9af176-42be-4616-8164-7d70c34dad4c-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:01:08.689796 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:08.689673 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:09.144498 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:09.144418 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da09025-87cdv" Oct 30 02:01:09.144498 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:09.144441 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da09025-87cdv" event={"ID":"2c9af176-42be-4616-8164-7d70c34dad4c","Type":"ContainerDied","Data":"01626043691df717a523d90556f98d49a1b418130994e571d03c9d0bc3a3e3f9"} Oct 30 02:01:09.144498 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:09.144486 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01626043691df717a523d90556f98d49a1b418130994e571d03c9d0bc3a3e3f9" Oct 30 02:01:09.323714 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:09.323665 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:09.529460 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:09.529427 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:10.635865 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:10.635818 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-55f6dc654-dtcnz" podUID="20d4a043-ea77-43f9-979c-aeddef258caf" containerName="payload-processing" containerID="cri-o://dde60e2d88727ed8fd61056f4711ddc57e1ea2321df51d296eb0a7d03e19a062" gracePeriod=25 Oct 30 02:01:10.871628 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:10.871563 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:11.078783 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.077147 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:11.289863 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.289825 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg"] Oct 30 02:01:11.291958 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.291930 6572 cpu_manager.go:414] "containerMap: removing container" podUID="2c9af176-42be-4616-8164-7d70c34dad4c" containerName="revoke-keys" Oct 30 02:01:11.292136 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.292120 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c9af176-42be-4616-8164-7d70c34dad4c" containerName="revoke-keys" Oct 30 02:01:11.292302 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.292290 6572 cpu_manager.go:414] "containerMap: removing container" podUID="6130dbc6-b1f1-4672-90cc-7e3be07be0e1" containerName="payload-processing" Oct 30 02:01:11.292383 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.292373 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6130dbc6-b1f1-4672-90cc-7e3be07be0e1" containerName="payload-processing" Oct 30 02:01:11.292831 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.292811 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="6130dbc6-b1f1-4672-90cc-7e3be07be0e1" containerName="payload-processing" Oct 30 02:01:11.292989 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.292977 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c9af176-42be-4616-8164-7d70c34dad4c" containerName="revoke-keys" Oct 30 02:01:11.339774 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.331255 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg" Oct 30 02:01:11.390772 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.387179 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2e618f\"/\"agc-cred-ed67495fc34f\"" Oct 30 02:01:11.413774 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.408226 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2e618f\"/\"openshift-service-ca.crt\"" Oct 30 02:01:11.413774 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.408396 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2e618f\"/\"payload-processing-external-model-e2e--66567ea0-dockercfg-f5spw\"" Oct 30 02:01:11.413774 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.409238 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2e618f\"/\"routing-overlay\"" Oct 30 02:01:11.413774 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.409382 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2e618f\"/\"kube-root-ca.crt\"" Oct 30 02:01:11.413774 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.409475 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2e618f\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2e618f\"" Oct 30 02:01:11.433792 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.422594 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm47m\" (UniqueName: \"kubernetes.io/projected/a1a20ed5-0fcc-441e-a82d-5402e05472e0-kube-api-access-fm47m\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg\" (UID: \"a1a20ed5-0fcc-441e-a82d-5402e05472e0\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg" Oct 30 02:01:11.433792 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.422681 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a20ed5-0fcc-441e-a82d-5402e05472e0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg\" (UID: \"a1a20ed5-0fcc-441e-a82d-5402e05472e0\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg" Oct 30 02:01:11.433792 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.422826 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a1a20ed5-0fcc-441e-a82d-5402e05472e0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg\" (UID: \"a1a20ed5-0fcc-441e-a82d-5402e05472e0\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg" Oct 30 02:01:11.433792 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.422918 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a1a20ed5-0fcc-441e-a82d-5402e05472e0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg\" (UID: \"a1a20ed5-0fcc-441e-a82d-5402e05472e0\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg" Oct 30 02:01:11.511864 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.470372 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55f6dc654-dtcnz" Oct 30 02:01:11.531270 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.531238 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg"] Oct 30 02:01:11.540926 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.538427 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20d4a043-ea77-43f9-979c-aeddef258caf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20d4a043-ea77-43f9-979c-aeddef258caf-plugins-config-volume\") pod \"20d4a043-ea77-43f9-979c-aeddef258caf\" (UID: \"20d4a043-ea77-43f9-979c-aeddef258caf\") " Oct 30 02:01:11.540926 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.538578 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20d4a043-ea77-43f9-979c-aeddef258caf-provider-credentials\" (UniqueName: \"kubernetes.io/projected/20d4a043-ea77-43f9-979c-aeddef258caf-provider-credentials\") pod \"20d4a043-ea77-43f9-979c-aeddef258caf\" (UID: \"20d4a043-ea77-43f9-979c-aeddef258caf\") " Oct 30 02:01:11.540926 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.538620 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20d4a043-ea77-43f9-979c-aeddef258caf-kube-api-access-d87xq\" (UniqueName: \"kubernetes.io/projected/20d4a043-ea77-43f9-979c-aeddef258caf-kube-api-access-d87xq\") pod \"20d4a043-ea77-43f9-979c-aeddef258caf\" (UID: \"20d4a043-ea77-43f9-979c-aeddef258caf\") " Oct 30 02:01:11.540926 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.538891 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20d4a043-ea77-43f9-979c-aeddef258caf-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/20d4a043-ea77-43f9-979c-aeddef258caf-routing-overlay-volume\") pod \"20d4a043-ea77-43f9-979c-aeddef258caf\" (UID: \"20d4a043-ea77-43f9-979c-aeddef258caf\") " Oct 30 02:01:11.540926 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.539178 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a1a20ed5-0fcc-441e-a82d-5402e05472e0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg\" (UID: \"a1a20ed5-0fcc-441e-a82d-5402e05472e0\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg" Oct 30 02:01:11.540926 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.539278 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a1a20ed5-0fcc-441e-a82d-5402e05472e0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg\" (UID: \"a1a20ed5-0fcc-441e-a82d-5402e05472e0\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg" Oct 30 02:01:11.540926 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.539484 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm47m\" (UniqueName: \"kubernetes.io/projected/a1a20ed5-0fcc-441e-a82d-5402e05472e0-kube-api-access-fm47m\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg\" (UID: \"a1a20ed5-0fcc-441e-a82d-5402e05472e0\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg" Oct 30 02:01:11.540926 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.539542 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a20ed5-0fcc-441e-a82d-5402e05472e0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg\" (UID: \"a1a20ed5-0fcc-441e-a82d-5402e05472e0\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg" Oct 30 02:01:11.540926 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.540688 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a20ed5-0fcc-441e-a82d-5402e05472e0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg\" (UID: \"a1a20ed5-0fcc-441e-a82d-5402e05472e0\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg" Oct 30 02:01:11.541875 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.541820 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20d4a043-ea77-43f9-979c-aeddef258caf-plugins-config-volume" pod "20d4a043-ea77-43f9-979c-aeddef258caf" (UID: "20d4a043-ea77-43f9-979c-aeddef258caf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:11.582906 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.581205 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d4a043-ea77-43f9-979c-aeddef258caf-provider-credentials" pod "20d4a043-ea77-43f9-979c-aeddef258caf" (UID: "20d4a043-ea77-43f9-979c-aeddef258caf"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:11.582906 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.581929 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20d4a043-ea77-43f9-979c-aeddef258caf-routing-overlay-volume" pod "20d4a043-ea77-43f9-979c-aeddef258caf" (UID: "20d4a043-ea77-43f9-979c-aeddef258caf"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:11.585809 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.584123 6572 generic.go:356] "Generic (PLEG): container finished" podID="20d4a043-ea77-43f9-979c-aeddef258caf" containerID="dde60e2d88727ed8fd61056f4711ddc57e1ea2321df51d296eb0a7d03e19a062" exitCode=0 Oct 30 02:01:11.585809 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.584676 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55f6dc654-dtcnz" Oct 30 02:01:11.585809 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.585356 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55f6dc654-dtcnz" event={"ID":"20d4a043-ea77-43f9-979c-aeddef258caf","Type":"ContainerDied","Data":"dde60e2d88727ed8fd61056f4711ddc57e1ea2321df51d296eb0a7d03e19a062"} Oct 30 02:01:11.589853 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.589828 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a1a20ed5-0fcc-441e-a82d-5402e05472e0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg\" (UID: \"a1a20ed5-0fcc-441e-a82d-5402e05472e0\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg" Oct 30 02:01:11.590936 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.590357 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55f6dc654-dtcnz" event={"ID":"20d4a043-ea77-43f9-979c-aeddef258caf","Type":"ContainerDied","Data":"0a0cabf226d17a4f1f216fec56d3a9409013bd87e22d31f07701153022bedca4"} Oct 30 02:01:11.595782 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.590372 6572 scope.go:122] "RemoveContainer" containerID="dde60e2d88727ed8fd61056f4711ddc57e1ea2321df51d296eb0a7d03e19a062" Oct 30 02:01:11.595782 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.592832 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a1a20ed5-0fcc-441e-a82d-5402e05472e0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg\" (UID: \"a1a20ed5-0fcc-441e-a82d-5402e05472e0\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg" Oct 30 02:01:11.598608 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.598581 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d4a043-ea77-43f9-979c-aeddef258caf-kube-api-access-d87xq" pod "20d4a043-ea77-43f9-979c-aeddef258caf" (UID: "20d4a043-ea77-43f9-979c-aeddef258caf"). InnerVolumeSpecName "kube-api-access-d87xq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:11.599533 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.599481 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm47m\" (UniqueName: \"kubernetes.io/projected/a1a20ed5-0fcc-441e-a82d-5402e05472e0-kube-api-access-fm47m\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg\" (UID: \"a1a20ed5-0fcc-441e-a82d-5402e05472e0\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg" Oct 30 02:01:11.634976 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.631203 6572 scope.go:122] "RemoveContainer" containerID="dde60e2d88727ed8fd61056f4711ddc57e1ea2321df51d296eb0a7d03e19a062" Oct 30 02:01:11.634976 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:11.631637 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dde60e2d88727ed8fd61056f4711ddc57e1ea2321df51d296eb0a7d03e19a062\": container with ID starting with dde60e2d88727ed8fd61056f4711ddc57e1ea2321df51d296eb0a7d03e19a062 not found: ID does not exist" containerID="dde60e2d88727ed8fd61056f4711ddc57e1ea2321df51d296eb0a7d03e19a062" Oct 30 02:01:11.634976 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.631674 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dde60e2d88727ed8fd61056f4711ddc57e1ea2321df51d296eb0a7d03e19a062"} err="failed to get container status \"dde60e2d88727ed8fd61056f4711ddc57e1ea2321df51d296eb0a7d03e19a062\": rpc error: code = NotFound desc = could not find container \"dde60e2d88727ed8fd61056f4711ddc57e1ea2321df51d296eb0a7d03e19a062\": container with ID starting with dde60e2d88727ed8fd61056f4711ddc57e1ea2321df51d296eb0a7d03e19a062 not found: ID does not exist" Oct 30 02:01:11.641195 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.641111 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20d4a043-ea77-43f9-979c-aeddef258caf-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:01:11.641195 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.641143 6572 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/20d4a043-ea77-43f9-979c-aeddef258caf-provider-credentials\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:01:11.641195 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.641157 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d87xq\" (UniqueName: \"kubernetes.io/projected/20d4a043-ea77-43f9-979c-aeddef258caf-kube-api-access-d87xq\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:01:11.641195 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.641171 6572 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/20d4a043-ea77-43f9-979c-aeddef258caf-routing-overlay-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:01:11.884473 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.884350 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg" Oct 30 02:01:11.925381 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.925353 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:11.988862 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.988814 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-55f6dc654-dtcnz"] Oct 30 02:01:11.992329 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:11.992302 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-55f6dc654-dtcnz"] Oct 30 02:01:12.120775 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:12.120542 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv"] Oct 30 02:01:12.124728 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:12.121843 6572 cpu_manager.go:414] "containerMap: removing container" podUID="20d4a043-ea77-43f9-979c-aeddef258caf" containerName="payload-processing" Oct 30 02:01:12.124728 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:12.121872 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20d4a043-ea77-43f9-979c-aeddef258caf" containerName="payload-processing" Oct 30 02:01:12.124728 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:12.122072 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="20d4a043-ea77-43f9-979c-aeddef258caf" containerName="payload-processing" Oct 30 02:01:12.127895 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:12.127868 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv" Oct 30 02:01:12.133445 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:12.132939 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv"] Oct 30 02:01:12.151172 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:12.143685 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-76a19e\"/\"payload-processing-external-model-e2e--70e20018-dockercfg-cvwmc\"" Oct 30 02:01:12.151172 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:12.143788 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-76a19e\"/\"agc-cred-139046a94a22\"" Oct 30 02:01:12.151172 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:12.143885 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-76a19e\"/\"payload-processing-external-model-plugins-e2e-extmod-a-76a19e\"" Oct 30 02:01:12.151172 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:12.143945 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-76a19e\"/\"openshift-service-ca.crt\"" Oct 30 02:01:12.151172 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:12.144627 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-76a19e\"/\"kube-root-ca.crt\"" Oct 30 02:01:12.151172 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:12.144723 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-76a19e\"/\"routing-overlay\"" Oct 30 02:01:12.485374 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:12.485336 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecba6182-7fb8-4aef-81eb-18851f944bec-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv\" (UID: \"ecba6182-7fb8-4aef-81eb-18851f944bec\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv" Oct 30 02:01:12.485659 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:12.485637 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ecba6182-7fb8-4aef-81eb-18851f944bec-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv\" (UID: \"ecba6182-7fb8-4aef-81eb-18851f944bec\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv" Oct 30 02:01:12.485926 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:12.485905 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ecba6182-7fb8-4aef-81eb-18851f944bec-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv\" (UID: \"ecba6182-7fb8-4aef-81eb-18851f944bec\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv" Oct 30 02:01:12.486224 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:12.486204 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhzd7\" (UniqueName: \"kubernetes.io/projected/ecba6182-7fb8-4aef-81eb-18851f944bec-kube-api-access-jhzd7\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv\" (UID: \"ecba6182-7fb8-4aef-81eb-18851f944bec\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv" Oct 30 02:01:12.533485 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:12.533457 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg"] Oct 30 02:01:12.613922 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:12.587131 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20d4a043-ea77-43f9-979c-aeddef258caf" path="/var/lib/kubelet/pods/20d4a043-ea77-43f9-979c-aeddef258caf/volumes" Oct 30 02:01:12.613922 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:12.589489 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ecba6182-7fb8-4aef-81eb-18851f944bec-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv\" (UID: \"ecba6182-7fb8-4aef-81eb-18851f944bec\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv" Oct 30 02:01:12.613922 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:12.589696 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhzd7\" (UniqueName: \"kubernetes.io/projected/ecba6182-7fb8-4aef-81eb-18851f944bec-kube-api-access-jhzd7\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv\" (UID: \"ecba6182-7fb8-4aef-81eb-18851f944bec\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv" Oct 30 02:01:12.613922 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:12.589734 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecba6182-7fb8-4aef-81eb-18851f944bec-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv\" (UID: \"ecba6182-7fb8-4aef-81eb-18851f944bec\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv" Oct 30 02:01:12.613922 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:12.589794 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ecba6182-7fb8-4aef-81eb-18851f944bec-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv\" (UID: \"ecba6182-7fb8-4aef-81eb-18851f944bec\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv" Oct 30 02:01:12.613922 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:12.590600 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ecba6182-7fb8-4aef-81eb-18851f944bec-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv\" (UID: \"ecba6182-7fb8-4aef-81eb-18851f944bec\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv" Oct 30 02:01:12.613922 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:12.593871 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ecba6182-7fb8-4aef-81eb-18851f944bec-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv\" (UID: \"ecba6182-7fb8-4aef-81eb-18851f944bec\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv" Oct 30 02:01:12.613922 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:12.595013 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecba6182-7fb8-4aef-81eb-18851f944bec-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv\" (UID: \"ecba6182-7fb8-4aef-81eb-18851f944bec\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv" Oct 30 02:01:12.635120 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:12.635084 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg" event={"ID":"a1a20ed5-0fcc-441e-a82d-5402e05472e0","Type":"ContainerStarted","Data":"ce681d1942522483e576a51f41c6ca3e1caf3fe2434b8a801d30cb79e6756f14"} Oct 30 02:01:12.658166 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:12.658079 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhzd7\" (UniqueName: \"kubernetes.io/projected/ecba6182-7fb8-4aef-81eb-18851f944bec-kube-api-access-jhzd7\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv\" (UID: \"ecba6182-7fb8-4aef-81eb-18851f944bec\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv" Oct 30 02:01:12.758312 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:12.758066 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv" Oct 30 02:01:13.086659 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:13.086514 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:14.084809 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:14.082193 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg" event={"ID":"a1a20ed5-0fcc-441e-a82d-5402e05472e0","Type":"ContainerStarted","Data":"fe1784f20d1ac368419981a1ca6875fd6e214bba3863faa19ab31ad93b838e2a"} Oct 30 02:01:14.111950 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:14.105541 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg" Oct 30 02:01:14.204638 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:14.204596 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv"] Oct 30 02:01:14.214031 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:01:14.211804 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecba6182_7fb8_4aef_81eb_18851f944bec.slice/crio-ea5a2107d62a70738a20882a6821b9ec8ba45ebc1a2fcadf3772d7eb75283e4a WatchSource:0}: Error finding container ea5a2107d62a70738a20882a6821b9ec8ba45ebc1a2fcadf3772d7eb75283e4a: Status 404 returned error can't find the container with id ea5a2107d62a70738a20882a6821b9ec8ba45ebc1a2fcadf3772d7eb75283e4a Oct 30 02:01:14.322771 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:14.319079 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg" podStartSLOduration=3.31905532 podStartE2EDuration="3.31905532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:01:14.269985865 +0000 UTC m=+2599.846411279" watchObservedRunningTime="2026-10-30 02:01:14.31905532 +0000 UTC m=+2599.895480724" Oct 30 02:01:14.785856 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:14.785815 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:01:15.093648 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:15.093609 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv" event={"ID":"ecba6182-7fb8-4aef-81eb-18851f944bec","Type":"ContainerStarted","Data":"208a112dff4e4073af313179050ff209b7613007098c0b7f5ecdff19491fc6d5"} Oct 30 02:01:15.094255 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:15.093704 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv" event={"ID":"ecba6182-7fb8-4aef-81eb-18851f944bec","Type":"ContainerStarted","Data":"ea5a2107d62a70738a20882a6821b9ec8ba45ebc1a2fcadf3772d7eb75283e4a"} Oct 30 02:01:15.094255 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:15.093842 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv" Oct 30 02:01:15.188730 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:15.188673 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:15.596037 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:15.596004 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:15.771499 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:15.771466 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:15.885247 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:15.885128 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv" podStartSLOduration=3.8851125140000002 podStartE2EDuration="3.885112514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:01:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:01:15.184766235 +0000 UTC m=+2600.761191635" watchObservedRunningTime="2026-10-30 02:01:15.885112514 +0000 UTC m=+2601.461537906" Oct 30 02:01:15.886805 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:15.886774 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq"] Oct 30 02:01:15.892325 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:15.892304 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq" Oct 30 02:01:15.909323 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:15.909297 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq"] Oct 30 02:01:15.974490 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:15.974459 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b2de26f6-a4fd-4c53-9591-30b3cbbef009-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq\" (UID: \"b2de26f6-a4fd-4c53-9591-30b3cbbef009\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq" Oct 30 02:01:15.974490 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:15.974496 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrppl\" (UniqueName: \"kubernetes.io/projected/b2de26f6-a4fd-4c53-9591-30b3cbbef009-kube-api-access-rrppl\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq\" (UID: \"b2de26f6-a4fd-4c53-9591-30b3cbbef009\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq" Oct 30 02:01:15.974733 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:15.974641 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b2de26f6-a4fd-4c53-9591-30b3cbbef009-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq\" (UID: \"b2de26f6-a4fd-4c53-9591-30b3cbbef009\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq" Oct 30 02:01:15.974733 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:15.974690 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2de26f6-a4fd-4c53-9591-30b3cbbef009-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq\" (UID: \"b2de26f6-a4fd-4c53-9591-30b3cbbef009\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq" Oct 30 02:01:16.075641 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:16.075603 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b2de26f6-a4fd-4c53-9591-30b3cbbef009-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq\" (UID: \"b2de26f6-a4fd-4c53-9591-30b3cbbef009\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq" Oct 30 02:01:16.075847 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:16.075673 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2de26f6-a4fd-4c53-9591-30b3cbbef009-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq\" (UID: \"b2de26f6-a4fd-4c53-9591-30b3cbbef009\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq" Oct 30 02:01:16.075847 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:16.075814 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b2de26f6-a4fd-4c53-9591-30b3cbbef009-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq\" (UID: \"b2de26f6-a4fd-4c53-9591-30b3cbbef009\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq" Oct 30 02:01:16.075847 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:16.075834 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrppl\" (UniqueName: \"kubernetes.io/projected/b2de26f6-a4fd-4c53-9591-30b3cbbef009-kube-api-access-rrppl\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq\" (UID: \"b2de26f6-a4fd-4c53-9591-30b3cbbef009\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq" Oct 30 02:01:16.076430 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:16.076389 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b2de26f6-a4fd-4c53-9591-30b3cbbef009-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq\" (UID: \"b2de26f6-a4fd-4c53-9591-30b3cbbef009\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq" Oct 30 02:01:16.076560 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:16.076456 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2de26f6-a4fd-4c53-9591-30b3cbbef009-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq\" (UID: \"b2de26f6-a4fd-4c53-9591-30b3cbbef009\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq" Oct 30 02:01:16.078365 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:16.078348 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b2de26f6-a4fd-4c53-9591-30b3cbbef009-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq\" (UID: \"b2de26f6-a4fd-4c53-9591-30b3cbbef009\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq" Oct 30 02:01:16.095331 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:16.095294 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrppl\" (UniqueName: \"kubernetes.io/projected/b2de26f6-a4fd-4c53-9591-30b3cbbef009-kube-api-access-rrppl\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq\" (UID: \"b2de26f6-a4fd-4c53-9591-30b3cbbef009\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq" Oct 30 02:01:16.209147 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:16.209035 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq" Oct 30 02:01:16.623230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:16.623201 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq"] Oct 30 02:01:16.623814 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:01:16.623784 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2de26f6_a4fd_4c53_9591_30b3cbbef009.slice/crio-473e9fcf591f86b63459cebdde3dcf463a30206cb9b6d54b7b325920d99c5431 WatchSource:0}: Error finding container 473e9fcf591f86b63459cebdde3dcf463a30206cb9b6d54b7b325920d99c5431: Status 404 returned error can't find the container with id 473e9fcf591f86b63459cebdde3dcf463a30206cb9b6d54b7b325920d99c5431 Oct 30 02:01:16.804774 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:16.800045 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:17.053772 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:17.048593 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:17.142775 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:17.141509 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq" event={"ID":"b2de26f6-a4fd-4c53-9591-30b3cbbef009","Type":"ContainerStarted","Data":"7f5441dcf88d07f086c45bbfad2acca958f257f4b0b5af9592629aa1ddc692fc"} Oct 30 02:01:17.142775 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:17.141590 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq" event={"ID":"b2de26f6-a4fd-4c53-9591-30b3cbbef009","Type":"ContainerStarted","Data":"473e9fcf591f86b63459cebdde3dcf463a30206cb9b6d54b7b325920d99c5431"} Oct 30 02:01:17.143343 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:17.143228 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq" Oct 30 02:01:17.217771 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:17.217126 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq" podStartSLOduration=2.217103743 podStartE2EDuration="2.217103743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:01:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:01:17.193207583 +0000 UTC m=+2602.769632986" watchObservedRunningTime="2026-10-30 02:01:17.217103743 +0000 UTC m=+2602.793529146" Oct 30 02:01:17.291724 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:17.291592 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:17.304000 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:17.303918 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-496830df-mj9ls"] Oct 30 02:01:17.309283 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:17.309259 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-496830df-mj9ls" Oct 30 02:01:17.323530 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:17.323502 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:01:17.323666 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:17.323551 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:01:17.323666 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:17.323584 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:01:17.323789 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:17.323664 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:01:17.325153 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:17.325131 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-496830df-mj9ls"] Oct 30 02:01:17.338243 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:17.338214 6572 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-92f0cc/agc-cred-ce62c5464e20: secret "agc-cred-ce62c5464e20" not found Oct 30 02:01:17.338243 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:17.338244 6572 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq: secret "agc-cred-ce62c5464e20" not found Oct 30 02:01:17.339418 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:17.339400 6572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2de26f6-a4fd-4c53-9591-30b3cbbef009-provider-credentials podName:b2de26f6-a4fd-4c53-9591-30b3cbbef009 nodeName:}" failed. No retries permitted until 2026-10-30 02:01:17.839378819 +0000 UTC m=+2603.415804216 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b2de26f6-a4fd-4c53-9591-30b3cbbef009-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq" (UID: "b2de26f6-a4fd-4c53-9591-30b3cbbef009") : secret "agc-cred-ce62c5464e20" not found Oct 30 02:01:17.440064 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:17.440033 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64c62270-5436-4905-a965-ee29d002850a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-496830df-mj9ls\" (UID: \"64c62270-5436-4905-a965-ee29d002850a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-496830df-mj9ls" Oct 30 02:01:17.440226 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:17.440158 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdlsd\" (UniqueName: \"kubernetes.io/projected/64c62270-5436-4905-a965-ee29d002850a-kube-api-access-tdlsd\") pod \"maas-api-revoke-sub-496830df-mj9ls\" (UID: \"64c62270-5436-4905-a965-ee29d002850a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-496830df-mj9ls" Oct 30 02:01:17.440302 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:17.440287 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64c62270-5436-4905-a965-ee29d002850a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-496830df-mj9ls\" (UID: \"64c62270-5436-4905-a965-ee29d002850a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-496830df-mj9ls" Oct 30 02:01:17.541787 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:17.541742 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64c62270-5436-4905-a965-ee29d002850a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-496830df-mj9ls\" (UID: \"64c62270-5436-4905-a965-ee29d002850a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-496830df-mj9ls" Oct 30 02:01:17.541970 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:17.541847 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdlsd\" (UniqueName: \"kubernetes.io/projected/64c62270-5436-4905-a965-ee29d002850a-kube-api-access-tdlsd\") pod \"maas-api-revoke-sub-496830df-mj9ls\" (UID: \"64c62270-5436-4905-a965-ee29d002850a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-496830df-mj9ls" Oct 30 02:01:17.541970 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:17.541954 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64c62270-5436-4905-a965-ee29d002850a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-496830df-mj9ls\" (UID: \"64c62270-5436-4905-a965-ee29d002850a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-496830df-mj9ls" Oct 30 02:01:17.542904 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:17.542870 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64c62270-5436-4905-a965-ee29d002850a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-496830df-mj9ls\" (UID: \"64c62270-5436-4905-a965-ee29d002850a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-496830df-mj9ls" Oct 30 02:01:17.544392 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:17.544365 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64c62270-5436-4905-a965-ee29d002850a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-496830df-mj9ls\" (UID: \"64c62270-5436-4905-a965-ee29d002850a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-496830df-mj9ls" Oct 30 02:01:17.561021 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:17.560961 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdlsd\" (UniqueName: \"kubernetes.io/projected/64c62270-5436-4905-a965-ee29d002850a-kube-api-access-tdlsd\") pod \"maas-api-revoke-sub-496830df-mj9ls\" (UID: \"64c62270-5436-4905-a965-ee29d002850a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-496830df-mj9ls" Oct 30 02:01:17.627280 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:17.627242 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-496830df-mj9ls" Oct 30 02:01:17.636673 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:17.636647 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq"] Oct 30 02:01:17.852180 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:17.852107 6572 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-92f0cc/agc-cred-ce62c5464e20: secret "agc-cred-ce62c5464e20" not found Oct 30 02:01:17.852180 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:17.852133 6572 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq: secret "agc-cred-ce62c5464e20" not found Oct 30 02:01:17.852180 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:17.852185 6572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2de26f6-a4fd-4c53-9591-30b3cbbef009-provider-credentials podName:b2de26f6-a4fd-4c53-9591-30b3cbbef009 nodeName:}" failed. No retries permitted until 2026-10-30 02:01:18.852168659 +0000 UTC m=+2604.428594039 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b2de26f6-a4fd-4c53-9591-30b3cbbef009-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq" (UID: "b2de26f6-a4fd-4c53-9591-30b3cbbef009") : secret "agc-cred-ce62c5464e20" not found Oct 30 02:01:17.946366 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:01:17.946323 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64c62270_5436_4905_a965_ee29d002850a.slice/crio-f55e0e003c460235ee9f0f59c54b395ea4ea7609b487882ca2b5d75ced3fb9b6 WatchSource:0}: Error finding container f55e0e003c460235ee9f0f59c54b395ea4ea7609b487882ca2b5d75ced3fb9b6: Status 404 returned error can't find the container with id f55e0e003c460235ee9f0f59c54b395ea4ea7609b487882ca2b5d75ced3fb9b6 Oct 30 02:01:17.947991 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:17.947772 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-496830df-mj9ls"] Oct 30 02:01:18.152492 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:18.152463 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-496830df-mj9ls" event={"ID":"64c62270-5436-4905-a965-ee29d002850a","Type":"ContainerStarted","Data":"f55e0e003c460235ee9f0f59c54b395ea4ea7609b487882ca2b5d75ced3fb9b6"} Oct 30 02:01:18.798892 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:18.795560 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:18.873230 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:18.873198 6572 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-92f0cc/agc-cred-ce62c5464e20: secret "agc-cred-ce62c5464e20" not found Oct 30 02:01:18.873230 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:18.873221 6572 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq: secret "agc-cred-ce62c5464e20" not found Oct 30 02:01:18.873425 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:18.873301 6572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2de26f6-a4fd-4c53-9591-30b3cbbef009-provider-credentials podName:b2de26f6-a4fd-4c53-9591-30b3cbbef009 nodeName:}" failed. No retries permitted until 2026-10-30 02:01:20.873287377 +0000 UTC m=+2606.449712757 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b2de26f6-a4fd-4c53-9591-30b3cbbef009-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq" (UID: "b2de26f6-a4fd-4c53-9591-30b3cbbef009") : secret "agc-cred-ce62c5464e20" not found Oct 30 02:01:19.162591 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:19.162559 6572 generic.go:356] "Generic (PLEG): container finished" podID="64c62270-5436-4905-a965-ee29d002850a" containerID="a33c844c7622991d252f438cbb9b069895ef8c33ef29c79086e68696f777de02" exitCode=0 Oct 30 02:01:19.163009 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:19.162635 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-496830df-mj9ls" event={"ID":"64c62270-5436-4905-a965-ee29d002850a","Type":"ContainerDied","Data":"a33c844c7622991d252f438cbb9b069895ef8c33ef29c79086e68696f777de02"} Oct 30 02:01:20.333472 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:20.320310 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:20.413603 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:20.413580 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-496830df-mj9ls" Oct 30 02:01:20.511258 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:20.511178 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64c62270-5436-4905-a965-ee29d002850a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64c62270-5436-4905-a965-ee29d002850a-maas-api-openshift-ca\") pod \"64c62270-5436-4905-a965-ee29d002850a\" (UID: \"64c62270-5436-4905-a965-ee29d002850a\") " Oct 30 02:01:20.511258 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:20.511253 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64c62270-5436-4905-a965-ee29d002850a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64c62270-5436-4905-a965-ee29d002850a-maas-api-xks-ca\") pod \"64c62270-5436-4905-a965-ee29d002850a\" (UID: \"64c62270-5436-4905-a965-ee29d002850a\") " Oct 30 02:01:20.511484 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:20.511395 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64c62270-5436-4905-a965-ee29d002850a-kube-api-access-tdlsd\" (UniqueName: \"kubernetes.io/projected/64c62270-5436-4905-a965-ee29d002850a-kube-api-access-tdlsd\") pod \"64c62270-5436-4905-a965-ee29d002850a\" (UID: \"64c62270-5436-4905-a965-ee29d002850a\") " Oct 30 02:01:20.512415 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:20.512389 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64c62270-5436-4905-a965-ee29d002850a-maas-api-openshift-ca" pod "64c62270-5436-4905-a965-ee29d002850a" (UID: "64c62270-5436-4905-a965-ee29d002850a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:20.513963 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:20.513934 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64c62270-5436-4905-a965-ee29d002850a-kube-api-access-tdlsd" pod "64c62270-5436-4905-a965-ee29d002850a" (UID: "64c62270-5436-4905-a965-ee29d002850a"). InnerVolumeSpecName "kube-api-access-tdlsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:20.514180 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:20.514156 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64c62270-5436-4905-a965-ee29d002850a-maas-api-xks-ca" pod "64c62270-5436-4905-a965-ee29d002850a" (UID: "64c62270-5436-4905-a965-ee29d002850a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:01:20.613643 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:20.613387 6572 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-92f0cc/agc-cred-ce62c5464e20: secret "agc-cred-ce62c5464e20" not found Oct 30 02:01:20.613643 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:20.613416 6572 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-92f0cc/agc-cred-edfce4f81270: secret "agc-cred-edfce4f81270" not found Oct 30 02:01:20.613643 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:20.613429 6572 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7: [secret "agc-cred-ce62c5464e20" not found, secret "agc-cred-edfce4f81270" not found] Oct 30 02:01:20.613643 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:20.613531 6572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0c61263-a7c1-4792-8970-6fe6388f11bf-provider-credentials podName:b0c61263-a7c1-4792-8970-6fe6388f11bf nodeName:}" failed. No retries permitted until 2026-10-30 02:01:21.113509914 +0000 UTC m=+2606.689935308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b0c61263-a7c1-4792-8970-6fe6388f11bf-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7" (UID: "b0c61263-a7c1-4792-8970-6fe6388f11bf") : [secret "agc-cred-ce62c5464e20" not found, secret "agc-cred-edfce4f81270" not found] Oct 30 02:01:20.613643 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:20.613540 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64c62270-5436-4905-a965-ee29d002850a-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.613643 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:20.613562 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64c62270-5436-4905-a965-ee29d002850a-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.613643 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:20.613578 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdlsd\" (UniqueName: \"kubernetes.io/projected/64c62270-5436-4905-a965-ee29d002850a-kube-api-access-tdlsd\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.918872 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:20.918834 6572 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-92f0cc/agc-cred-ce62c5464e20: secret "agc-cred-ce62c5464e20" not found Oct 30 02:01:20.918872 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:20.918859 6572 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq: secret "agc-cred-ce62c5464e20" not found Oct 30 02:01:20.919177 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:20.918928 6572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2de26f6-a4fd-4c53-9591-30b3cbbef009-provider-credentials podName:b2de26f6-a4fd-4c53-9591-30b3cbbef009 nodeName:}" failed. No retries permitted until 2026-10-30 02:01:24.918910355 +0000 UTC m=+2610.495335754 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b2de26f6-a4fd-4c53-9591-30b3cbbef009-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq" (UID: "b2de26f6-a4fd-4c53-9591-30b3cbbef009") : secret "agc-cred-ce62c5464e20" not found Oct 30 02:01:21.121283 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:21.121247 6572 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-92f0cc/agc-cred-ce62c5464e20: secret "agc-cred-ce62c5464e20" not found Oct 30 02:01:21.121283 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:21.121277 6572 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-92f0cc/agc-cred-edfce4f81270: secret "agc-cred-edfce4f81270" not found Oct 30 02:01:21.121283 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:21.121286 6572 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7: [secret "agc-cred-ce62c5464e20" not found, secret "agc-cred-edfce4f81270" not found] Oct 30 02:01:21.121542 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:21.121344 6572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0c61263-a7c1-4792-8970-6fe6388f11bf-provider-credentials podName:b0c61263-a7c1-4792-8970-6fe6388f11bf nodeName:}" failed. No retries permitted until 2026-10-30 02:01:22.121330478 +0000 UTC m=+2607.697755861 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b0c61263-a7c1-4792-8970-6fe6388f11bf-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7" (UID: "b0c61263-a7c1-4792-8970-6fe6388f11bf") : [secret "agc-cred-ce62c5464e20" not found, secret "agc-cred-edfce4f81270" not found] Oct 30 02:01:21.181706 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:21.181626 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-496830df-mj9ls" Oct 30 02:01:21.181706 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:21.181657 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-496830df-mj9ls" event={"ID":"64c62270-5436-4905-a965-ee29d002850a","Type":"ContainerDied","Data":"f55e0e003c460235ee9f0f59c54b395ea4ea7609b487882ca2b5d75ced3fb9b6"} Oct 30 02:01:21.181932 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:21.181726 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f55e0e003c460235ee9f0f59c54b395ea4ea7609b487882ca2b5d75ced3fb9b6" Oct 30 02:01:21.884303 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:21.880473 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:22.149312 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:22.149234 6572 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-92f0cc/agc-cred-ce62c5464e20: secret "agc-cred-ce62c5464e20" not found Oct 30 02:01:22.149312 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:22.149263 6572 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-92f0cc/agc-cred-edfce4f81270: secret "agc-cred-edfce4f81270" not found Oct 30 02:01:22.149312 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:22.149273 6572 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7: [secret "agc-cred-ce62c5464e20" not found, secret "agc-cred-edfce4f81270" not found] Oct 30 02:01:22.149543 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:22.149348 6572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0c61263-a7c1-4792-8970-6fe6388f11bf-provider-credentials podName:b0c61263-a7c1-4792-8970-6fe6388f11bf nodeName:}" failed. No retries permitted until 2026-10-30 02:01:24.149331506 +0000 UTC m=+2609.725756886 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b0c61263-a7c1-4792-8970-6fe6388f11bf-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7" (UID: "b0c61263-a7c1-4792-8970-6fe6388f11bf") : [secret "agc-cred-ce62c5464e20" not found, secret "agc-cred-edfce4f81270" not found] Oct 30 02:01:23.451952 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:23.451916 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:24.163460 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:24.163419 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq" podUID="b2de26f6-a4fd-4c53-9591-30b3cbbef009" containerName="payload-processing" containerID="cri-o://7f5441dcf88d07f086c45bbfad2acca958f257f4b0b5af9592629aa1ddc692fc" gracePeriod=25 Oct 30 02:01:24.186171 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:24.186144 6572 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-92f0cc/agc-cred-ce62c5464e20: secret "agc-cred-ce62c5464e20" not found Oct 30 02:01:24.186171 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:24.186171 6572 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-92f0cc/agc-cred-edfce4f81270: secret "agc-cred-edfce4f81270" not found Oct 30 02:01:24.186343 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:24.186180 6572 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7: [secret "agc-cred-ce62c5464e20" not found, secret "agc-cred-edfce4f81270" not found] Oct 30 02:01:24.186343 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:24.186238 6572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0c61263-a7c1-4792-8970-6fe6388f11bf-provider-credentials podName:b0c61263-a7c1-4792-8970-6fe6388f11bf nodeName:}" failed. No retries permitted until 2026-10-30 02:01:28.186220965 +0000 UTC m=+2613.762646346 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b0c61263-a7c1-4792-8970-6fe6388f11bf-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7" (UID: "b0c61263-a7c1-4792-8970-6fe6388f11bf") : [secret "agc-cred-ce62c5464e20" not found, secret "agc-cred-edfce4f81270" not found] Oct 30 02:01:24.439108 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:24.439074 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq" Oct 30 02:01:24.490847 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:24.490814 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2de26f6-a4fd-4c53-9591-30b3cbbef009-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b2de26f6-a4fd-4c53-9591-30b3cbbef009-provider-credentials\") pod \"b2de26f6-a4fd-4c53-9591-30b3cbbef009\" (UID: \"b2de26f6-a4fd-4c53-9591-30b3cbbef009\") " Oct 30 02:01:24.490847 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:24.490853 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2de26f6-a4fd-4c53-9591-30b3cbbef009-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b2de26f6-a4fd-4c53-9591-30b3cbbef009-routing-overlay-volume\") pod \"b2de26f6-a4fd-4c53-9591-30b3cbbef009\" (UID: \"b2de26f6-a4fd-4c53-9591-30b3cbbef009\") " Oct 30 02:01:24.491403 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:24.490960 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2de26f6-a4fd-4c53-9591-30b3cbbef009-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2de26f6-a4fd-4c53-9591-30b3cbbef009-plugins-config-volume\") pod \"b2de26f6-a4fd-4c53-9591-30b3cbbef009\" (UID: \"b2de26f6-a4fd-4c53-9591-30b3cbbef009\") " Oct 30 02:01:24.491403 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:24.491043 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2de26f6-a4fd-4c53-9591-30b3cbbef009-kube-api-access-rrppl\" (UniqueName: \"kubernetes.io/projected/b2de26f6-a4fd-4c53-9591-30b3cbbef009-kube-api-access-rrppl\") pod \"b2de26f6-a4fd-4c53-9591-30b3cbbef009\" (UID: \"b2de26f6-a4fd-4c53-9591-30b3cbbef009\") " Oct 30 02:01:24.491560 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:24.491533 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2de26f6-a4fd-4c53-9591-30b3cbbef009-plugins-config-volume" pod "b2de26f6-a4fd-4c53-9591-30b3cbbef009" (UID: "b2de26f6-a4fd-4c53-9591-30b3cbbef009"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:24.491930 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:24.491905 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2de26f6-a4fd-4c53-9591-30b3cbbef009-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:01:24.492037 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:24.491927 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2de26f6-a4fd-4c53-9591-30b3cbbef009-routing-overlay-volume" pod "b2de26f6-a4fd-4c53-9591-30b3cbbef009" (UID: "b2de26f6-a4fd-4c53-9591-30b3cbbef009"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:24.493350 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:24.493329 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2de26f6-a4fd-4c53-9591-30b3cbbef009-kube-api-access-rrppl" pod "b2de26f6-a4fd-4c53-9591-30b3cbbef009" (UID: "b2de26f6-a4fd-4c53-9591-30b3cbbef009"). InnerVolumeSpecName "kube-api-access-rrppl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:24.493606 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:24.493580 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2de26f6-a4fd-4c53-9591-30b3cbbef009-provider-credentials" pod "b2de26f6-a4fd-4c53-9591-30b3cbbef009" (UID: "b2de26f6-a4fd-4c53-9591-30b3cbbef009"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:24.595227 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:24.595142 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrppl\" (UniqueName: \"kubernetes.io/projected/b2de26f6-a4fd-4c53-9591-30b3cbbef009-kube-api-access-rrppl\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:01:24.595227 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:24.595181 6572 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b2de26f6-a4fd-4c53-9591-30b3cbbef009-provider-credentials\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:01:24.595227 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:24.595197 6572 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b2de26f6-a4fd-4c53-9591-30b3cbbef009-routing-overlay-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:01:24.991742 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:24.991478 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:25.096902 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:25.096873 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg" Oct 30 02:01:25.221241 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:25.221206 6572 generic.go:356] "Generic (PLEG): container finished" podID="b2de26f6-a4fd-4c53-9591-30b3cbbef009" containerID="7f5441dcf88d07f086c45bbfad2acca958f257f4b0b5af9592629aa1ddc692fc" exitCode=0 Oct 30 02:01:25.221434 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:25.221271 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq" Oct 30 02:01:25.221434 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:25.221281 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq" event={"ID":"b2de26f6-a4fd-4c53-9591-30b3cbbef009","Type":"ContainerDied","Data":"7f5441dcf88d07f086c45bbfad2acca958f257f4b0b5af9592629aa1ddc692fc"} Oct 30 02:01:25.221434 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:25.221342 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq" event={"ID":"b2de26f6-a4fd-4c53-9591-30b3cbbef009","Type":"ContainerDied","Data":"473e9fcf591f86b63459cebdde3dcf463a30206cb9b6d54b7b325920d99c5431"} Oct 30 02:01:25.221434 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:25.221372 6572 scope.go:122] "RemoveContainer" containerID="7f5441dcf88d07f086c45bbfad2acca958f257f4b0b5af9592629aa1ddc692fc" Oct 30 02:01:25.234457 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:25.234351 6572 scope.go:122] "RemoveContainer" containerID="7f5441dcf88d07f086c45bbfad2acca958f257f4b0b5af9592629aa1ddc692fc" Oct 30 02:01:25.235534 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:25.234773 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f5441dcf88d07f086c45bbfad2acca958f257f4b0b5af9592629aa1ddc692fc\": container with ID starting with 7f5441dcf88d07f086c45bbfad2acca958f257f4b0b5af9592629aa1ddc692fc not found: ID does not exist" containerID="7f5441dcf88d07f086c45bbfad2acca958f257f4b0b5af9592629aa1ddc692fc" Oct 30 02:01:25.235534 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:25.234810 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f5441dcf88d07f086c45bbfad2acca958f257f4b0b5af9592629aa1ddc692fc"} err="failed to get container status \"7f5441dcf88d07f086c45bbfad2acca958f257f4b0b5af9592629aa1ddc692fc\": rpc error: code = NotFound desc = could not find container \"7f5441dcf88d07f086c45bbfad2acca958f257f4b0b5af9592629aa1ddc692fc\": container with ID starting with 7f5441dcf88d07f086c45bbfad2acca958f257f4b0b5af9592629aa1ddc692fc not found: ID does not exist" Oct 30 02:01:25.285245 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:25.285159 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq"] Oct 30 02:01:25.292393 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:25.292361 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67cchqq"] Oct 30 02:01:26.106980 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:26.106949 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv" Oct 30 02:01:26.567822 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:26.567740 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:26.575598 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:26.575564 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2de26f6-a4fd-4c53-9591-30b3cbbef009" path="/var/lib/kubelet/pods/b2de26f6-a4fd-4c53-9591-30b3cbbef009/volumes" Oct 30 02:01:28.197526 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:28.197063 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:28.253196 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:28.253163 6572 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-92f0cc/agc-cred-ce62c5464e20: secret "agc-cred-ce62c5464e20" not found Oct 30 02:01:28.253196 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:28.253188 6572 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-92f0cc/agc-cred-edfce4f81270: secret "agc-cred-edfce4f81270" not found Oct 30 02:01:28.253196 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:28.253197 6572 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7: [secret "agc-cred-ce62c5464e20" not found, secret "agc-cred-edfce4f81270" not found] Oct 30 02:01:28.253458 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:28.253247 6572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0c61263-a7c1-4792-8970-6fe6388f11bf-provider-credentials podName:b0c61263-a7c1-4792-8970-6fe6388f11bf nodeName:}" failed. No retries permitted until 2026-10-30 02:01:36.253232262 +0000 UTC m=+2621.829657646 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b0c61263-a7c1-4792-8970-6fe6388f11bf-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7" (UID: "b0c61263-a7c1-4792-8970-6fe6388f11bf") : [secret "agc-cred-ce62c5464e20" not found, secret "agc-cred-edfce4f81270" not found] Oct 30 02:01:29.531111 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:29.531076 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7"] Oct 30 02:01:29.809613 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:29.809513 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:29.991954 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:29.991919 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:30.491332 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:30.491294 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:31.474903 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:31.474868 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:33.167971 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:33.167939 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:34.531560 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:34.531524 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7" podUID="b0c61263-a7c1-4792-8970-6fe6388f11bf" containerName="payload-processing" containerID="cri-o://c879fb15956d07e3ba835063190599f74656dea2ee33241c2854154d20dd3cb5" gracePeriod=25 Oct 30 02:01:34.829602 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:34.829470 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:34.865297 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:34.865273 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7" Oct 30 02:01:34.946542 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:34.946512 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0c61263-a7c1-4792-8970-6fe6388f11bf-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b0c61263-a7c1-4792-8970-6fe6388f11bf-routing-overlay-volume\") pod \"b0c61263-a7c1-4792-8970-6fe6388f11bf\" (UID: \"b0c61263-a7c1-4792-8970-6fe6388f11bf\") " Oct 30 02:01:34.946732 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:34.946565 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0c61263-a7c1-4792-8970-6fe6388f11bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0c61263-a7c1-4792-8970-6fe6388f11bf-plugins-config-volume\") pod \"b0c61263-a7c1-4792-8970-6fe6388f11bf\" (UID: \"b0c61263-a7c1-4792-8970-6fe6388f11bf\") " Oct 30 02:01:34.946837 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:34.946722 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0c61263-a7c1-4792-8970-6fe6388f11bf-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b0c61263-a7c1-4792-8970-6fe6388f11bf-provider-credentials\") pod \"b0c61263-a7c1-4792-8970-6fe6388f11bf\" (UID: \"b0c61263-a7c1-4792-8970-6fe6388f11bf\") " Oct 30 02:01:34.946837 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:34.946814 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0c61263-a7c1-4792-8970-6fe6388f11bf-kube-api-access-m5dzj\" (UniqueName: \"kubernetes.io/projected/b0c61263-a7c1-4792-8970-6fe6388f11bf-kube-api-access-m5dzj\") pod \"b0c61263-a7c1-4792-8970-6fe6388f11bf\" (UID: \"b0c61263-a7c1-4792-8970-6fe6388f11bf\") " Oct 30 02:01:34.947054 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:34.947025 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0c61263-a7c1-4792-8970-6fe6388f11bf-routing-overlay-volume" pod "b0c61263-a7c1-4792-8970-6fe6388f11bf" (UID: "b0c61263-a7c1-4792-8970-6fe6388f11bf"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:34.947114 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:34.947038 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0c61263-a7c1-4792-8970-6fe6388f11bf-plugins-config-volume" pod "b0c61263-a7c1-4792-8970-6fe6388f11bf" (UID: "b0c61263-a7c1-4792-8970-6fe6388f11bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:34.947561 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:34.947536 6572 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b0c61263-a7c1-4792-8970-6fe6388f11bf-routing-overlay-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:01:34.947671 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:34.947565 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0c61263-a7c1-4792-8970-6fe6388f11bf-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:01:34.949143 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:34.949118 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c61263-a7c1-4792-8970-6fe6388f11bf-provider-credentials" pod "b0c61263-a7c1-4792-8970-6fe6388f11bf" (UID: "b0c61263-a7c1-4792-8970-6fe6388f11bf"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:34.949606 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:34.949585 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c61263-a7c1-4792-8970-6fe6388f11bf-kube-api-access-m5dzj" pod "b0c61263-a7c1-4792-8970-6fe6388f11bf" (UID: "b0c61263-a7c1-4792-8970-6fe6388f11bf"). InnerVolumeSpecName "kube-api-access-m5dzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:35.049421 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:35.049392 6572 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b0c61263-a7c1-4792-8970-6fe6388f11bf-provider-credentials\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:01:35.049421 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:35.049418 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5dzj\" (UniqueName: \"kubernetes.io/projected/b0c61263-a7c1-4792-8970-6fe6388f11bf-kube-api-access-m5dzj\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:01:35.350573 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:35.350542 6572 generic.go:356] "Generic (PLEG): container finished" podID="b0c61263-a7c1-4792-8970-6fe6388f11bf" containerID="c879fb15956d07e3ba835063190599f74656dea2ee33241c2854154d20dd3cb5" exitCode=0 Oct 30 02:01:35.350741 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:35.350589 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7" Oct 30 02:01:35.350741 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:35.350620 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7" event={"ID":"b0c61263-a7c1-4792-8970-6fe6388f11bf","Type":"ContainerDied","Data":"c879fb15956d07e3ba835063190599f74656dea2ee33241c2854154d20dd3cb5"} Oct 30 02:01:35.350741 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:35.350683 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7" event={"ID":"b0c61263-a7c1-4792-8970-6fe6388f11bf","Type":"ContainerDied","Data":"39ad8178957e7880c0562d51ec4d998387c592aa56d8945bc7992399998d075e"} Oct 30 02:01:35.350741 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:35.350716 6572 scope.go:122] "RemoveContainer" containerID="c879fb15956d07e3ba835063190599f74656dea2ee33241c2854154d20dd3cb5" Oct 30 02:01:35.361671 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:35.361649 6572 scope.go:122] "RemoveContainer" containerID="c879fb15956d07e3ba835063190599f74656dea2ee33241c2854154d20dd3cb5" Oct 30 02:01:35.361977 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:35.361955 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c879fb15956d07e3ba835063190599f74656dea2ee33241c2854154d20dd3cb5\": container with ID starting with c879fb15956d07e3ba835063190599f74656dea2ee33241c2854154d20dd3cb5 not found: ID does not exist" containerID="c879fb15956d07e3ba835063190599f74656dea2ee33241c2854154d20dd3cb5" Oct 30 02:01:35.362048 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:35.361985 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c879fb15956d07e3ba835063190599f74656dea2ee33241c2854154d20dd3cb5"} err="failed to get container status \"c879fb15956d07e3ba835063190599f74656dea2ee33241c2854154d20dd3cb5\": rpc error: code = NotFound desc = could not find container \"c879fb15956d07e3ba835063190599f74656dea2ee33241c2854154d20dd3cb5\": container with ID starting with c879fb15956d07e3ba835063190599f74656dea2ee33241c2854154d20dd3cb5 not found: ID does not exist" Oct 30 02:01:35.402363 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:35.402336 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7"] Oct 30 02:01:35.413474 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:35.413454 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-84ddbp7k7"] Oct 30 02:01:36.378771 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:36.378490 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:36.546513 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:36.546480 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0c61263-a7c1-4792-8970-6fe6388f11bf" path="/var/lib/kubelet/pods/b0c61263-a7c1-4792-8970-6fe6388f11bf/volumes" Oct 30 02:01:36.789779 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:36.788120 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:37.244209 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:37.244177 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:38.050492 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:38.050454 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:39.790771 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:39.790725 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:41.254773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:41.252495 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:42.755571 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:42.755535 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:42.937543 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:42.937505 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:43.752133 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:43.752092 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:43.963718 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:43.963677 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:44.688419 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:01:44.688382 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:01:44.952878 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:44.952783 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:45.089143 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:45.089098 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-86d69bd698-dhmzl"] Oct 30 02:01:45.089916 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:45.089897 6572 cpu_manager.go:414] "containerMap: removing container" podUID="b2de26f6-a4fd-4c53-9591-30b3cbbef009" containerName="payload-processing" Oct 30 02:01:45.089916 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:45.089914 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2de26f6-a4fd-4c53-9591-30b3cbbef009" containerName="payload-processing" Oct 30 02:01:45.090116 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:45.089939 6572 cpu_manager.go:414] "containerMap: removing container" podUID="64c62270-5436-4905-a965-ee29d002850a" containerName="revoke-keys" Oct 30 02:01:45.090116 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:45.089949 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64c62270-5436-4905-a965-ee29d002850a" containerName="revoke-keys" Oct 30 02:01:45.090116 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:45.089960 6572 cpu_manager.go:414] "containerMap: removing container" podUID="b0c61263-a7c1-4792-8970-6fe6388f11bf" containerName="payload-processing" Oct 30 02:01:45.090116 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:45.089966 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0c61263-a7c1-4792-8970-6fe6388f11bf" containerName="payload-processing" Oct 30 02:01:45.090116 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:45.090113 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="64c62270-5436-4905-a965-ee29d002850a" containerName="revoke-keys" Oct 30 02:01:45.090404 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:45.090125 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0c61263-a7c1-4792-8970-6fe6388f11bf" containerName="payload-processing" Oct 30 02:01:45.090404 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:45.090142 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2de26f6-a4fd-4c53-9591-30b3cbbef009" containerName="payload-processing" Oct 30 02:01:45.095703 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:45.095680 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-86d69bd698-dhmzl" Oct 30 02:01:45.112668 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:45.112640 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-86d69bd698-dhmzl"] Oct 30 02:01:45.118665 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:45.118617 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-7ecb73ededa6\"" Oct 30 02:01:45.152453 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:45.152418 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad-provider-credentials\") pod \"payload-processing-external-model-86d69bd698-dhmzl\" (UID: \"9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad\") " pod="models-as-a-service/payload-processing-external-model-86d69bd698-dhmzl" Oct 30 02:01:45.152658 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:45.152574 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad-plugins-config-volume\") pod \"payload-processing-external-model-86d69bd698-dhmzl\" (UID: \"9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad\") " pod="models-as-a-service/payload-processing-external-model-86d69bd698-dhmzl" Oct 30 02:01:45.152658 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:45.152649 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8b5h\" (UniqueName: \"kubernetes.io/projected/9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad-kube-api-access-z8b5h\") pod \"payload-processing-external-model-86d69bd698-dhmzl\" (UID: \"9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad\") " pod="models-as-a-service/payload-processing-external-model-86d69bd698-dhmzl" Oct 30 02:01:45.152817 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:45.152676 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad-routing-overlay-volume\") pod \"payload-processing-external-model-86d69bd698-dhmzl\" (UID: \"9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad\") " pod="models-as-a-service/payload-processing-external-model-86d69bd698-dhmzl" Oct 30 02:01:45.254028 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:45.253939 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad-provider-credentials\") pod \"payload-processing-external-model-86d69bd698-dhmzl\" (UID: \"9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad\") " pod="models-as-a-service/payload-processing-external-model-86d69bd698-dhmzl" Oct 30 02:01:45.254193 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:45.254087 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad-plugins-config-volume\") pod \"payload-processing-external-model-86d69bd698-dhmzl\" (UID: \"9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad\") " pod="models-as-a-service/payload-processing-external-model-86d69bd698-dhmzl" Oct 30 02:01:45.254193 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:45.254145 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8b5h\" (UniqueName: \"kubernetes.io/projected/9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad-kube-api-access-z8b5h\") pod \"payload-processing-external-model-86d69bd698-dhmzl\" (UID: \"9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad\") " pod="models-as-a-service/payload-processing-external-model-86d69bd698-dhmzl" Oct 30 02:01:45.254193 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:45.254168 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad-routing-overlay-volume\") pod \"payload-processing-external-model-86d69bd698-dhmzl\" (UID: \"9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad\") " pod="models-as-a-service/payload-processing-external-model-86d69bd698-dhmzl" Oct 30 02:01:45.254929 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:45.254902 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad-routing-overlay-volume\") pod \"payload-processing-external-model-86d69bd698-dhmzl\" (UID: \"9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad\") " pod="models-as-a-service/payload-processing-external-model-86d69bd698-dhmzl" Oct 30 02:01:45.255242 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:45.255191 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad-plugins-config-volume\") pod \"payload-processing-external-model-86d69bd698-dhmzl\" (UID: \"9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad\") " pod="models-as-a-service/payload-processing-external-model-86d69bd698-dhmzl" Oct 30 02:01:45.256839 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:45.256820 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad-provider-credentials\") pod \"payload-processing-external-model-86d69bd698-dhmzl\" (UID: \"9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad\") " pod="models-as-a-service/payload-processing-external-model-86d69bd698-dhmzl" Oct 30 02:01:45.275400 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:45.275373 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8b5h\" (UniqueName: \"kubernetes.io/projected/9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad-kube-api-access-z8b5h\") pod \"payload-processing-external-model-86d69bd698-dhmzl\" (UID: \"9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad\") " pod="models-as-a-service/payload-processing-external-model-86d69bd698-dhmzl" Oct 30 02:01:45.416363 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:45.416315 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-86d69bd698-dhmzl" Oct 30 02:01:45.774434 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:45.774354 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-86d69bd698-dhmzl"] Oct 30 02:01:46.611279 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:46.611237 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-86d69bd698-dhmzl" event={"ID":"9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad","Type":"ContainerStarted","Data":"e5a43dcb6721a7914d53317856c94d85cc456ba67723f98c0cf3ee9b44d7269a"} Oct 30 02:01:46.611755 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:46.611306 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-86d69bd698-dhmzl" event={"ID":"9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad","Type":"ContainerStarted","Data":"edfb0759b9708229649cad56d9691db767eb8aa2d53cc73f4ac15732597915a1"} Oct 30 02:01:46.611755 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:46.611479 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-86d69bd698-dhmzl" Oct 30 02:01:46.663346 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:46.663286 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-86d69bd698-dhmzl" podStartSLOduration=1.663270871 podStartE2EDuration="1.663270871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:01:46.657542501 +0000 UTC m=+2632.233967904" watchObservedRunningTime="2026-10-30 02:01:46.663270871 +0000 UTC m=+2632.239696273" Oct 30 02:01:46.860593 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:46.860552 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:48.515772 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:48.514812 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:49.399197 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:49.399161 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70cbc47d-7v8z4"] Oct 30 02:01:49.404023 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:49.404004 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cbc47d-7v8z4" Oct 30 02:01:49.422289 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:49.422258 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70cbc47d-7v8z4"] Oct 30 02:01:49.427412 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:49.427347 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:01:49.427412 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:49.427364 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:01:49.427412 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:49.427405 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:01:49.429394 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:49.429370 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:01:49.519838 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:49.519807 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm68p\" (UniqueName: \"kubernetes.io/projected/dfe3389b-960e-45cc-a22a-634e4a64bbcc-kube-api-access-zm68p\") pod \"maas-api-revoke-sub-70cbc47d-7v8z4\" (UID: \"dfe3389b-960e-45cc-a22a-634e4a64bbcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cbc47d-7v8z4" Oct 30 02:01:49.520239 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:49.519869 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfe3389b-960e-45cc-a22a-634e4a64bbcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70cbc47d-7v8z4\" (UID: \"dfe3389b-960e-45cc-a22a-634e4a64bbcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cbc47d-7v8z4" Oct 30 02:01:49.520239 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:49.520035 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfe3389b-960e-45cc-a22a-634e4a64bbcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70cbc47d-7v8z4\" (UID: \"dfe3389b-960e-45cc-a22a-634e4a64bbcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cbc47d-7v8z4" Oct 30 02:01:49.621628 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:49.621588 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfe3389b-960e-45cc-a22a-634e4a64bbcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70cbc47d-7v8z4\" (UID: \"dfe3389b-960e-45cc-a22a-634e4a64bbcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cbc47d-7v8z4" Oct 30 02:01:49.621875 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:49.621824 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm68p\" (UniqueName: \"kubernetes.io/projected/dfe3389b-960e-45cc-a22a-634e4a64bbcc-kube-api-access-zm68p\") pod \"maas-api-revoke-sub-70cbc47d-7v8z4\" (UID: \"dfe3389b-960e-45cc-a22a-634e4a64bbcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cbc47d-7v8z4" Oct 30 02:01:49.621934 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:49.621901 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfe3389b-960e-45cc-a22a-634e4a64bbcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70cbc47d-7v8z4\" (UID: \"dfe3389b-960e-45cc-a22a-634e4a64bbcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cbc47d-7v8z4" Oct 30 02:01:49.622396 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:49.622372 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfe3389b-960e-45cc-a22a-634e4a64bbcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70cbc47d-7v8z4\" (UID: \"dfe3389b-960e-45cc-a22a-634e4a64bbcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cbc47d-7v8z4" Oct 30 02:01:49.624289 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:49.624268 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfe3389b-960e-45cc-a22a-634e4a64bbcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70cbc47d-7v8z4\" (UID: \"dfe3389b-960e-45cc-a22a-634e4a64bbcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cbc47d-7v8z4" Oct 30 02:01:49.641392 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:49.641363 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm68p\" (UniqueName: \"kubernetes.io/projected/dfe3389b-960e-45cc-a22a-634e4a64bbcc-kube-api-access-zm68p\") pod \"maas-api-revoke-sub-70cbc47d-7v8z4\" (UID: \"dfe3389b-960e-45cc-a22a-634e4a64bbcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cbc47d-7v8z4" Oct 30 02:01:49.723168 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:49.723126 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cbc47d-7v8z4" Oct 30 02:01:50.030077 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:50.029957 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:50.074516 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:50.074488 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70cbc47d-7v8z4"] Oct 30 02:01:50.075016 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:01:50.074993 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfe3389b_960e_45cc_a22a_634e4a64bbcc.slice/crio-caac974f161996d17b3a7f249b87a783c81f6dd393b7e7cb26100f651dcbb6ff WatchSource:0}: Error finding container caac974f161996d17b3a7f249b87a783c81f6dd393b7e7cb26100f651dcbb6ff: Status 404 returned error can't find the container with id caac974f161996d17b3a7f249b87a783c81f6dd393b7e7cb26100f651dcbb6ff Oct 30 02:01:50.652959 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:50.652922 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cbc47d-7v8z4" event={"ID":"dfe3389b-960e-45cc-a22a-634e4a64bbcc","Type":"ContainerStarted","Data":"155596eb43aee35ea78b451d49cec556135657f20aa015270f254909dceee0d2"} Oct 30 02:01:50.653376 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:50.652976 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cbc47d-7v8z4" event={"ID":"dfe3389b-960e-45cc-a22a-634e4a64bbcc","Type":"ContainerStarted","Data":"caac974f161996d17b3a7f249b87a783c81f6dd393b7e7cb26100f651dcbb6ff"} Oct 30 02:01:50.703182 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:50.703069 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cbc47d-7v8z4" podStartSLOduration=1.703052934 podStartE2EDuration="1.703052934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:01:50.690352271 +0000 UTC m=+2636.266777675" watchObservedRunningTime="2026-10-30 02:01:50.703052934 +0000 UTC m=+2636.279478374" Oct 30 02:01:51.571822 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:51.571729 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:51.663944 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:51.663913 6572 generic.go:356] "Generic (PLEG): container finished" podID="dfe3389b-960e-45cc-a22a-634e4a64bbcc" containerID="155596eb43aee35ea78b451d49cec556135657f20aa015270f254909dceee0d2" exitCode=0 Oct 30 02:01:51.664352 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:51.663995 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cbc47d-7v8z4" event={"ID":"dfe3389b-960e-45cc-a22a-634e4a64bbcc","Type":"ContainerDied","Data":"155596eb43aee35ea78b451d49cec556135657f20aa015270f254909dceee0d2"} Oct 30 02:01:52.848302 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:52.848270 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cbc47d-7v8z4" Oct 30 02:01:52.884256 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:52.884219 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfe3389b-960e-45cc-a22a-634e4a64bbcc-kube-api-access-zm68p\" (UniqueName: \"kubernetes.io/projected/dfe3389b-960e-45cc-a22a-634e4a64bbcc-kube-api-access-zm68p\") pod \"dfe3389b-960e-45cc-a22a-634e4a64bbcc\" (UID: \"dfe3389b-960e-45cc-a22a-634e4a64bbcc\") " Oct 30 02:01:52.884431 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:52.884267 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfe3389b-960e-45cc-a22a-634e4a64bbcc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfe3389b-960e-45cc-a22a-634e4a64bbcc-maas-api-openshift-ca\") pod \"dfe3389b-960e-45cc-a22a-634e4a64bbcc\" (UID: \"dfe3389b-960e-45cc-a22a-634e4a64bbcc\") " Oct 30 02:01:52.884431 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:52.884336 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfe3389b-960e-45cc-a22a-634e4a64bbcc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfe3389b-960e-45cc-a22a-634e4a64bbcc-maas-api-xks-ca\") pod \"dfe3389b-960e-45cc-a22a-634e4a64bbcc\" (UID: \"dfe3389b-960e-45cc-a22a-634e4a64bbcc\") " Oct 30 02:01:52.884707 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:52.884679 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfe3389b-960e-45cc-a22a-634e4a64bbcc-maas-api-openshift-ca" pod "dfe3389b-960e-45cc-a22a-634e4a64bbcc" (UID: "dfe3389b-960e-45cc-a22a-634e4a64bbcc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:52.884859 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:52.884840 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfe3389b-960e-45cc-a22a-634e4a64bbcc-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:01:52.886537 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:52.886507 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfe3389b-960e-45cc-a22a-634e4a64bbcc-maas-api-xks-ca" pod "dfe3389b-960e-45cc-a22a-634e4a64bbcc" (UID: "dfe3389b-960e-45cc-a22a-634e4a64bbcc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:01:52.886622 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:52.886583 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfe3389b-960e-45cc-a22a-634e4a64bbcc-kube-api-access-zm68p" pod "dfe3389b-960e-45cc-a22a-634e4a64bbcc" (UID: "dfe3389b-960e-45cc-a22a-634e4a64bbcc"). InnerVolumeSpecName "kube-api-access-zm68p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:52.986459 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:52.986370 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm68p\" (UniqueName: \"kubernetes.io/projected/dfe3389b-960e-45cc-a22a-634e4a64bbcc-kube-api-access-zm68p\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:01:52.986459 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:52.986399 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfe3389b-960e-45cc-a22a-634e4a64bbcc-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:01:53.291362 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:53.290784 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:53.684923 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:53.684889 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cbc47d-7v8z4" Oct 30 02:01:53.684923 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:53.684906 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70cbc47d-7v8z4" event={"ID":"dfe3389b-960e-45cc-a22a-634e4a64bbcc","Type":"ContainerDied","Data":"caac974f161996d17b3a7f249b87a783c81f6dd393b7e7cb26100f651dcbb6ff"} Oct 30 02:01:53.685166 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:53.684957 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="caac974f161996d17b3a7f249b87a783c81f6dd393b7e7cb26100f651dcbb6ff" Oct 30 02:01:54.868765 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:54.868686 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:56.395182 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:56.395146 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:57.626937 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:57.626902 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-86d69bd698-dhmzl" Oct 30 02:01:57.700838 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:57.700795 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-d5b799485-zjfmq"] Oct 30 02:01:57.875609 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:57.874364 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:58.323808 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:58.323773 6572 scope.go:122] "RemoveContainer" containerID="2b6217efdd0c833fbea16563ef24e7ce09ac5bded090de2375fef42897c71705" Oct 30 02:01:58.335967 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:58.335941 6572 scope.go:122] "RemoveContainer" containerID="26deb63e4e9da60299a5441ba926eed4619104eadd8938a96a65cde4ab737be9" Oct 30 02:01:59.450235 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:01:59.450101 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:00.625149 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:00.625110 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m"] Oct 30 02:02:00.789223 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:00.789168 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:00.987700 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:00.987660 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:01.174082 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:01.174046 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:02.106774 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:02.101432 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:02.241687 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:02.241602 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz"] Oct 30 02:02:02.243184 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:02.243153 6572 cpu_manager.go:414] "containerMap: removing container" podUID="dfe3389b-960e-45cc-a22a-634e4a64bbcc" containerName="revoke-keys" Oct 30 02:02:02.243377 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:02.243362 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfe3389b-960e-45cc-a22a-634e4a64bbcc" containerName="revoke-keys" Oct 30 02:02:02.243884 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:02.243858 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfe3389b-960e-45cc-a22a-634e4a64bbcc" containerName="revoke-keys" Oct 30 02:02:02.252650 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:02.251982 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz"] Oct 30 02:02:02.252650 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:02.252082 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz" Oct 30 02:02:02.277891 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:02.277864 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-817fd7\"/\"agc-cred-08a8e8dfe09b\"" Oct 30 02:02:02.278066 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:02.277951 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-817fd7\"/\"payload-processing-external-model-e2e--6dbff1fe-dockercfg-rswzx\"" Oct 30 02:02:02.286203 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:02.286171 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:02.336703 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:02.336655 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tcd8\" (UniqueName: \"kubernetes.io/projected/ee6b2fa6-f2bc-4692-9273-f26f74133cfb-kube-api-access-8tcd8\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz\" (UID: \"ee6b2fa6-f2bc-4692-9273-f26f74133cfb\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz" Oct 30 02:02:02.336941 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:02.336719 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ee6b2fa6-f2bc-4692-9273-f26f74133cfb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz\" (UID: \"ee6b2fa6-f2bc-4692-9273-f26f74133cfb\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz" Oct 30 02:02:02.337014 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:02.336974 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ee6b2fa6-f2bc-4692-9273-f26f74133cfb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz\" (UID: \"ee6b2fa6-f2bc-4692-9273-f26f74133cfb\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz" Oct 30 02:02:02.337075 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:02.337038 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee6b2fa6-f2bc-4692-9273-f26f74133cfb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz\" (UID: \"ee6b2fa6-f2bc-4692-9273-f26f74133cfb\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz" Oct 30 02:02:02.445768 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:02.440811 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ee6b2fa6-f2bc-4692-9273-f26f74133cfb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz\" (UID: \"ee6b2fa6-f2bc-4692-9273-f26f74133cfb\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz" Oct 30 02:02:02.445768 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:02.440884 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee6b2fa6-f2bc-4692-9273-f26f74133cfb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz\" (UID: \"ee6b2fa6-f2bc-4692-9273-f26f74133cfb\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz" Oct 30 02:02:02.445768 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:02.441010 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tcd8\" (UniqueName: \"kubernetes.io/projected/ee6b2fa6-f2bc-4692-9273-f26f74133cfb-kube-api-access-8tcd8\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz\" (UID: \"ee6b2fa6-f2bc-4692-9273-f26f74133cfb\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz" Oct 30 02:02:02.445768 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:02.441080 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ee6b2fa6-f2bc-4692-9273-f26f74133cfb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz\" (UID: \"ee6b2fa6-f2bc-4692-9273-f26f74133cfb\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz" Oct 30 02:02:02.445768 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:02:02.441248 6572 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-817fd7/agc-cred-08a8e8dfe09b: secret "agc-cred-08a8e8dfe09b" not found Oct 30 02:02:02.445768 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:02:02.441263 6572 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz: secret "agc-cred-08a8e8dfe09b" not found Oct 30 02:02:02.445768 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:02:02.441325 6572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee6b2fa6-f2bc-4692-9273-f26f74133cfb-provider-credentials podName:ee6b2fa6-f2bc-4692-9273-f26f74133cfb nodeName:}" failed. No retries permitted until 2026-10-30 02:02:02.94130316 +0000 UTC m=+2648.517728545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ee6b2fa6-f2bc-4692-9273-f26f74133cfb-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz" (UID: "ee6b2fa6-f2bc-4692-9273-f26f74133cfb") : secret "agc-cred-08a8e8dfe09b" not found Oct 30 02:02:02.445768 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:02.442321 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ee6b2fa6-f2bc-4692-9273-f26f74133cfb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz\" (UID: \"ee6b2fa6-f2bc-4692-9273-f26f74133cfb\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz" Oct 30 02:02:02.445768 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:02.443038 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee6b2fa6-f2bc-4692-9273-f26f74133cfb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz\" (UID: \"ee6b2fa6-f2bc-4692-9273-f26f74133cfb\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz" Oct 30 02:02:02.470189 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:02.470108 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tcd8\" (UniqueName: \"kubernetes.io/projected/ee6b2fa6-f2bc-4692-9273-f26f74133cfb-kube-api-access-8tcd8\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz\" (UID: \"ee6b2fa6-f2bc-4692-9273-f26f74133cfb\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz" Oct 30 02:02:02.702245 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:02.702154 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-d5b799485-zjfmq" podUID="98f6258a-b3e1-4510-92b4-d23b95a4545d" containerName="payload-processing" containerID="cri-o://36a9fe612ee488058b07b34e84e35841b83dbe4587d91ec446290ad1192281bf" gracePeriod=25 Oct 30 02:02:02.713633 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:02.713597 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:02.955537 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:02.954213 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ee6b2fa6-f2bc-4692-9273-f26f74133cfb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz\" (UID: \"ee6b2fa6-f2bc-4692-9273-f26f74133cfb\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz" Oct 30 02:02:02.955537 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:02:02.954569 6572 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-817fd7/agc-cred-08a8e8dfe09b: secret "agc-cred-08a8e8dfe09b" not found Oct 30 02:02:02.955537 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:02:02.954589 6572 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz: secret "agc-cred-08a8e8dfe09b" not found Oct 30 02:02:02.955537 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:02:02.954667 6572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee6b2fa6-f2bc-4692-9273-f26f74133cfb-provider-credentials podName:ee6b2fa6-f2bc-4692-9273-f26f74133cfb nodeName:}" failed. No retries permitted until 2026-10-30 02:02:03.954646893 +0000 UTC m=+2649.531072297 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ee6b2fa6-f2bc-4692-9273-f26f74133cfb-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz" (UID: "ee6b2fa6-f2bc-4692-9273-f26f74133cfb") : secret "agc-cred-08a8e8dfe09b" not found Oct 30 02:02:02.995480 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:02.995431 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:03.037868 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:03.037842 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d5b799485-zjfmq" Oct 30 02:02:03.148547 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:03.148512 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz"] Oct 30 02:02:03.149021 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:02:03.148999 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz" podUID="ee6b2fa6-f2bc-4692-9273-f26f74133cfb" Oct 30 02:02:03.156612 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:03.156585 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98f6258a-b3e1-4510-92b4-d23b95a4545d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98f6258a-b3e1-4510-92b4-d23b95a4545d-plugins-config-volume\") pod \"98f6258a-b3e1-4510-92b4-d23b95a4545d\" (UID: \"98f6258a-b3e1-4510-92b4-d23b95a4545d\") " Oct 30 02:02:03.156836 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:03.156819 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98f6258a-b3e1-4510-92b4-d23b95a4545d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/98f6258a-b3e1-4510-92b4-d23b95a4545d-routing-overlay-volume\") pod \"98f6258a-b3e1-4510-92b4-d23b95a4545d\" (UID: \"98f6258a-b3e1-4510-92b4-d23b95a4545d\") " Oct 30 02:02:03.156925 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:03.156875 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98f6258a-b3e1-4510-92b4-d23b95a4545d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/98f6258a-b3e1-4510-92b4-d23b95a4545d-provider-credentials\") pod \"98f6258a-b3e1-4510-92b4-d23b95a4545d\" (UID: \"98f6258a-b3e1-4510-92b4-d23b95a4545d\") " Oct 30 02:02:03.156925 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:03.156914 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98f6258a-b3e1-4510-92b4-d23b95a4545d-kube-api-access-s9fkz\" (UniqueName: \"kubernetes.io/projected/98f6258a-b3e1-4510-92b4-d23b95a4545d-kube-api-access-s9fkz\") pod \"98f6258a-b3e1-4510-92b4-d23b95a4545d\" (UID: \"98f6258a-b3e1-4510-92b4-d23b95a4545d\") " Oct 30 02:02:03.157072 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:03.157047 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98f6258a-b3e1-4510-92b4-d23b95a4545d-plugins-config-volume" pod "98f6258a-b3e1-4510-92b4-d23b95a4545d" (UID: "98f6258a-b3e1-4510-92b4-d23b95a4545d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:02:03.157536 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:03.157510 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98f6258a-b3e1-4510-92b4-d23b95a4545d-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:02:03.157663 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:03.157556 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98f6258a-b3e1-4510-92b4-d23b95a4545d-routing-overlay-volume" pod "98f6258a-b3e1-4510-92b4-d23b95a4545d" (UID: "98f6258a-b3e1-4510-92b4-d23b95a4545d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:02:03.163900 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:03.163817 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98f6258a-b3e1-4510-92b4-d23b95a4545d-provider-credentials" pod "98f6258a-b3e1-4510-92b4-d23b95a4545d" (UID: "98f6258a-b3e1-4510-92b4-d23b95a4545d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:02:03.164043 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:03.164003 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98f6258a-b3e1-4510-92b4-d23b95a4545d-kube-api-access-s9fkz" pod "98f6258a-b3e1-4510-92b4-d23b95a4545d" (UID: "98f6258a-b3e1-4510-92b4-d23b95a4545d"). InnerVolumeSpecName "kube-api-access-s9fkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:02:03.258538 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:03.258456 6572 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/98f6258a-b3e1-4510-92b4-d23b95a4545d-provider-credentials\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:02:03.258538 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:03.258486 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9fkz\" (UniqueName: \"kubernetes.io/projected/98f6258a-b3e1-4510-92b4-d23b95a4545d-kube-api-access-s9fkz\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:02:03.258538 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:03.258500 6572 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/98f6258a-b3e1-4510-92b4-d23b95a4545d-routing-overlay-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:02:03.303352 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:03.303322 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:03.793790 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:03.793739 6572 generic.go:356] "Generic (PLEG): container finished" podID="98f6258a-b3e1-4510-92b4-d23b95a4545d" containerID="36a9fe612ee488058b07b34e84e35841b83dbe4587d91ec446290ad1192281bf" exitCode=0 Oct 30 02:02:03.793968 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:03.793817 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d5b799485-zjfmq" Oct 30 02:02:03.793968 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:03.793862 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d5b799485-zjfmq" event={"ID":"98f6258a-b3e1-4510-92b4-d23b95a4545d","Type":"ContainerDied","Data":"36a9fe612ee488058b07b34e84e35841b83dbe4587d91ec446290ad1192281bf"} Oct 30 02:02:03.793968 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:03.793931 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d5b799485-zjfmq" event={"ID":"98f6258a-b3e1-4510-92b4-d23b95a4545d","Type":"ContainerDied","Data":"003f8ea1d7ee7945a35926794fc8085f3ea6581154c0be21b51e7d46da4262dc"} Oct 30 02:02:03.794159 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:03.793994 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz" Oct 30 02:02:03.794159 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:03.793963 6572 scope.go:122] "RemoveContainer" containerID="36a9fe612ee488058b07b34e84e35841b83dbe4587d91ec446290ad1192281bf" Oct 30 02:02:03.804329 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:03.804307 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz" Oct 30 02:02:03.805472 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:03.805454 6572 scope.go:122] "RemoveContainer" containerID="36a9fe612ee488058b07b34e84e35841b83dbe4587d91ec446290ad1192281bf" Oct 30 02:02:03.805867 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:02:03.805739 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36a9fe612ee488058b07b34e84e35841b83dbe4587d91ec446290ad1192281bf\": container with ID starting with 36a9fe612ee488058b07b34e84e35841b83dbe4587d91ec446290ad1192281bf not found: ID does not exist" containerID="36a9fe612ee488058b07b34e84e35841b83dbe4587d91ec446290ad1192281bf" Oct 30 02:02:03.805867 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:03.805803 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36a9fe612ee488058b07b34e84e35841b83dbe4587d91ec446290ad1192281bf"} err="failed to get container status \"36a9fe612ee488058b07b34e84e35841b83dbe4587d91ec446290ad1192281bf\": rpc error: code = NotFound desc = could not find container \"36a9fe612ee488058b07b34e84e35841b83dbe4587d91ec446290ad1192281bf\": container with ID starting with 36a9fe612ee488058b07b34e84e35841b83dbe4587d91ec446290ad1192281bf not found: ID does not exist" Oct 30 02:02:03.883991 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:03.883958 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-d5b799485-zjfmq"] Oct 30 02:02:03.889019 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:03.888992 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-d5b799485-zjfmq"] Oct 30 02:02:03.983449 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:03.983412 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee6b2fa6-f2bc-4692-9273-f26f74133cfb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ee6b2fa6-f2bc-4692-9273-f26f74133cfb-routing-overlay-volume\") pod \"ee6b2fa6-f2bc-4692-9273-f26f74133cfb\" (UID: \"ee6b2fa6-f2bc-4692-9273-f26f74133cfb\") " Oct 30 02:02:03.983651 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:03.983634 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee6b2fa6-f2bc-4692-9273-f26f74133cfb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee6b2fa6-f2bc-4692-9273-f26f74133cfb-plugins-config-volume\") pod \"ee6b2fa6-f2bc-4692-9273-f26f74133cfb\" (UID: \"ee6b2fa6-f2bc-4692-9273-f26f74133cfb\") " Oct 30 02:02:03.983705 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:03.983672 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee6b2fa6-f2bc-4692-9273-f26f74133cfb-kube-api-access-8tcd8\" (UniqueName: \"kubernetes.io/projected/ee6b2fa6-f2bc-4692-9273-f26f74133cfb-kube-api-access-8tcd8\") pod \"ee6b2fa6-f2bc-4692-9273-f26f74133cfb\" (UID: \"ee6b2fa6-f2bc-4692-9273-f26f74133cfb\") " Oct 30 02:02:03.983850 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:03.983825 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee6b2fa6-f2bc-4692-9273-f26f74133cfb-routing-overlay-volume" pod "ee6b2fa6-f2bc-4692-9273-f26f74133cfb" (UID: "ee6b2fa6-f2bc-4692-9273-f26f74133cfb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:02:03.984095 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:03.984073 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee6b2fa6-f2bc-4692-9273-f26f74133cfb-plugins-config-volume" pod "ee6b2fa6-f2bc-4692-9273-f26f74133cfb" (UID: "ee6b2fa6-f2bc-4692-9273-f26f74133cfb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:02:03.984180 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:03.984091 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ee6b2fa6-f2bc-4692-9273-f26f74133cfb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz\" (UID: \"ee6b2fa6-f2bc-4692-9273-f26f74133cfb\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz" Oct 30 02:02:03.984242 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:02:03.984190 6572 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-817fd7/agc-cred-08a8e8dfe09b: secret "agc-cred-08a8e8dfe09b" not found Oct 30 02:02:03.984242 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:02:03.984208 6572 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz: secret "agc-cred-08a8e8dfe09b" not found Oct 30 02:02:03.984356 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:03.984261 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee6b2fa6-f2bc-4692-9273-f26f74133cfb-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:02:03.984356 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:02:03.984279 6572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee6b2fa6-f2bc-4692-9273-f26f74133cfb-provider-credentials podName:ee6b2fa6-f2bc-4692-9273-f26f74133cfb nodeName:}" failed. No retries permitted until 2026-10-30 02:02:05.984255743 +0000 UTC m=+2651.560681136 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ee6b2fa6-f2bc-4692-9273-f26f74133cfb-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz" (UID: "ee6b2fa6-f2bc-4692-9273-f26f74133cfb") : secret "agc-cred-08a8e8dfe09b" not found Oct 30 02:02:03.984356 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:03.984301 6572 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ee6b2fa6-f2bc-4692-9273-f26f74133cfb-routing-overlay-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:02:03.986055 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:03.986029 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee6b2fa6-f2bc-4692-9273-f26f74133cfb-kube-api-access-8tcd8" pod "ee6b2fa6-f2bc-4692-9273-f26f74133cfb" (UID: "ee6b2fa6-f2bc-4692-9273-f26f74133cfb"). InnerVolumeSpecName "kube-api-access-8tcd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:02:04.085538 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:04.085447 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tcd8\" (UniqueName: \"kubernetes.io/projected/ee6b2fa6-f2bc-4692-9273-f26f74133cfb-kube-api-access-8tcd8\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:02:04.581769 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:04.581376 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:04.593511 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:04.593459 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98f6258a-b3e1-4510-92b4-d23b95a4545d" path="/var/lib/kubelet/pods/98f6258a-b3e1-4510-92b4-d23b95a4545d/volumes" Oct 30 02:02:04.805066 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:04.805038 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz" Oct 30 02:02:04.879870 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:04.879774 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz"] Oct 30 02:02:04.887199 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:04.887164 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-65f8rqkkz"] Oct 30 02:02:04.999967 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:04.999934 6572 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ee6b2fa6-f2bc-4692-9273-f26f74133cfb-provider-credentials\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:02:05.625856 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:05.625815 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m" podUID="17e90574-f0e0-40dd-8ada-bf19c3839a6d" containerName="payload-processing" containerID="cri-o://c65589b423f010aa4404a0bdc3d09d4c8a12c3619ee1a988dbcc49a638d68916" gracePeriod=25 Oct 30 02:02:05.820233 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:05.820106 6572 generic.go:356] "Generic (PLEG): container finished" podID="17e90574-f0e0-40dd-8ada-bf19c3839a6d" containerID="c65589b423f010aa4404a0bdc3d09d4c8a12c3619ee1a988dbcc49a638d68916" exitCode=0 Oct 30 02:02:05.820233 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:05.820160 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m" event={"ID":"17e90574-f0e0-40dd-8ada-bf19c3839a6d","Type":"ContainerDied","Data":"c65589b423f010aa4404a0bdc3d09d4c8a12c3619ee1a988dbcc49a638d68916"} Oct 30 02:02:05.906456 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:05.906433 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m" Oct 30 02:02:06.013197 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:06.013163 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17e90574-f0e0-40dd-8ada-bf19c3839a6d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/17e90574-f0e0-40dd-8ada-bf19c3839a6d-routing-overlay-volume\") pod \"17e90574-f0e0-40dd-8ada-bf19c3839a6d\" (UID: \"17e90574-f0e0-40dd-8ada-bf19c3839a6d\") " Oct 30 02:02:06.013406 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:06.013205 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17e90574-f0e0-40dd-8ada-bf19c3839a6d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/17e90574-f0e0-40dd-8ada-bf19c3839a6d-provider-credentials\") pod \"17e90574-f0e0-40dd-8ada-bf19c3839a6d\" (UID: \"17e90574-f0e0-40dd-8ada-bf19c3839a6d\") " Oct 30 02:02:06.013406 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:06.013241 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17e90574-f0e0-40dd-8ada-bf19c3839a6d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e90574-f0e0-40dd-8ada-bf19c3839a6d-plugins-config-volume\") pod \"17e90574-f0e0-40dd-8ada-bf19c3839a6d\" (UID: \"17e90574-f0e0-40dd-8ada-bf19c3839a6d\") " Oct 30 02:02:06.013406 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:06.013353 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17e90574-f0e0-40dd-8ada-bf19c3839a6d-kube-api-access-nvcmv\" (UniqueName: \"kubernetes.io/projected/17e90574-f0e0-40dd-8ada-bf19c3839a6d-kube-api-access-nvcmv\") pod \"17e90574-f0e0-40dd-8ada-bf19c3839a6d\" (UID: \"17e90574-f0e0-40dd-8ada-bf19c3839a6d\") " Oct 30 02:02:06.013617 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:06.013590 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17e90574-f0e0-40dd-8ada-bf19c3839a6d-routing-overlay-volume" pod "17e90574-f0e0-40dd-8ada-bf19c3839a6d" (UID: "17e90574-f0e0-40dd-8ada-bf19c3839a6d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:02:06.013706 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:06.013686 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17e90574-f0e0-40dd-8ada-bf19c3839a6d-plugins-config-volume" pod "17e90574-f0e0-40dd-8ada-bf19c3839a6d" (UID: "17e90574-f0e0-40dd-8ada-bf19c3839a6d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:02:06.014079 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:06.014059 6572 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/17e90574-f0e0-40dd-8ada-bf19c3839a6d-routing-overlay-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:02:06.014166 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:06.014081 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e90574-f0e0-40dd-8ada-bf19c3839a6d-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:02:06.015504 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:06.015480 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e90574-f0e0-40dd-8ada-bf19c3839a6d-kube-api-access-nvcmv" pod "17e90574-f0e0-40dd-8ada-bf19c3839a6d" (UID: "17e90574-f0e0-40dd-8ada-bf19c3839a6d"). InnerVolumeSpecName "kube-api-access-nvcmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:02:06.015592 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:06.015517 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e90574-f0e0-40dd-8ada-bf19c3839a6d-provider-credentials" pod "17e90574-f0e0-40dd-8ada-bf19c3839a6d" (UID: "17e90574-f0e0-40dd-8ada-bf19c3839a6d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:02:06.115523 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:06.115490 6572 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/17e90574-f0e0-40dd-8ada-bf19c3839a6d-provider-credentials\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:02:06.115523 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:06.115518 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvcmv\" (UniqueName: \"kubernetes.io/projected/17e90574-f0e0-40dd-8ada-bf19c3839a6d-kube-api-access-nvcmv\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:02:06.316330 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:06.316295 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:06.544581 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:06.544546 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee6b2fa6-f2bc-4692-9273-f26f74133cfb" path="/var/lib/kubelet/pods/ee6b2fa6-f2bc-4692-9273-f26f74133cfb/volumes" Oct 30 02:02:06.829780 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:06.829722 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m" Oct 30 02:02:06.830276 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:06.829782 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m" event={"ID":"17e90574-f0e0-40dd-8ada-bf19c3839a6d","Type":"ContainerDied","Data":"ebf40f31e661ad6feb76b38a42bf895d800ef5c70d6692c50d6b09cb2efce9c8"} Oct 30 02:02:06.830276 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:06.829864 6572 scope.go:122] "RemoveContainer" containerID="c65589b423f010aa4404a0bdc3d09d4c8a12c3619ee1a988dbcc49a638d68916" Oct 30 02:02:06.886605 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:06.886569 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m"] Oct 30 02:02:06.898425 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:06.898389 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-84d5k2m2m"] Oct 30 02:02:07.249309 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:07.249275 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:07.685943 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:07.685879 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:08.546261 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:08.545491 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17e90574-f0e0-40dd-8ada-bf19c3839a6d" path="/var/lib/kubelet/pods/17e90574-f0e0-40dd-8ada-bf19c3839a6d/volumes" Oct 30 02:02:08.804523 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:08.804428 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:08.993970 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:08.993937 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:09.307771 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:09.306726 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:10.922771 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:10.921603 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:12.029068 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:12.029029 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f81dd7e1-4d5g6"] Oct 30 02:02:12.040595 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:12.040562 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f81dd7e1-4d5g6"] Oct 30 02:02:12.587074 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:12.587032 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4397e72b-fc07-423d-9682-7cb11feb5d5d" path="/var/lib/kubelet/pods/4397e72b-fc07-423d-9682-7cb11feb5d5d/volumes" Oct 30 02:02:12.623505 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:12.623470 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:14.132181 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:14.132141 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:14.690373 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:02:14.690331 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:02:15.420781 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:15.420667 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:15.604459 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:15.604426 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:15.795935 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:15.795829 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:16.414523 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:16.414489 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:16.612089 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:16.605253 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:17.886194 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:17.881024 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:17.911060 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:17.910989 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-69d99678c7-nhwvh"] Oct 30 02:02:17.912557 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:17.912530 6572 cpu_manager.go:414] "containerMap: removing container" podUID="98f6258a-b3e1-4510-92b4-d23b95a4545d" containerName="payload-processing" Oct 30 02:02:17.912741 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:17.912727 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98f6258a-b3e1-4510-92b4-d23b95a4545d" containerName="payload-processing" Oct 30 02:02:17.912917 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:17.912902 6572 cpu_manager.go:414] "containerMap: removing container" podUID="17e90574-f0e0-40dd-8ada-bf19c3839a6d" containerName="payload-processing" Oct 30 02:02:17.913004 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:17.912993 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17e90574-f0e0-40dd-8ada-bf19c3839a6d" containerName="payload-processing" Oct 30 02:02:17.913347 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:17.913282 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="17e90574-f0e0-40dd-8ada-bf19c3839a6d" containerName="payload-processing" Oct 30 02:02:17.913467 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:17.913455 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="98f6258a-b3e1-4510-92b4-d23b95a4545d" containerName="payload-processing" Oct 30 02:02:17.923304 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:17.923277 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-69d99678c7-nhwvh"] Oct 30 02:02:17.924599 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:17.924137 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-69d99678c7-nhwvh" Oct 30 02:02:17.942422 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:17.942397 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-84d872aa7a07\"" Oct 30 02:02:18.026206 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:18.026161 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcjz4\" (UniqueName: \"kubernetes.io/projected/602ab155-5413-4fe5-9929-9369acca84aa-kube-api-access-dcjz4\") pod \"payload-processing-external-model-69d99678c7-nhwvh\" (UID: \"602ab155-5413-4fe5-9929-9369acca84aa\") " pod="models-as-a-service/payload-processing-external-model-69d99678c7-nhwvh" Oct 30 02:02:18.026437 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:18.026349 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/602ab155-5413-4fe5-9929-9369acca84aa-provider-credentials\") pod \"payload-processing-external-model-69d99678c7-nhwvh\" (UID: \"602ab155-5413-4fe5-9929-9369acca84aa\") " pod="models-as-a-service/payload-processing-external-model-69d99678c7-nhwvh" Oct 30 02:02:18.026437 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:18.026388 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/602ab155-5413-4fe5-9929-9369acca84aa-routing-overlay-volume\") pod \"payload-processing-external-model-69d99678c7-nhwvh\" (UID: \"602ab155-5413-4fe5-9929-9369acca84aa\") " pod="models-as-a-service/payload-processing-external-model-69d99678c7-nhwvh" Oct 30 02:02:18.026576 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:18.026554 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/602ab155-5413-4fe5-9929-9369acca84aa-plugins-config-volume\") pod \"payload-processing-external-model-69d99678c7-nhwvh\" (UID: \"602ab155-5413-4fe5-9929-9369acca84aa\") " pod="models-as-a-service/payload-processing-external-model-69d99678c7-nhwvh" Oct 30 02:02:18.127995 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:18.127950 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/602ab155-5413-4fe5-9929-9369acca84aa-provider-credentials\") pod \"payload-processing-external-model-69d99678c7-nhwvh\" (UID: \"602ab155-5413-4fe5-9929-9369acca84aa\") " pod="models-as-a-service/payload-processing-external-model-69d99678c7-nhwvh" Oct 30 02:02:18.127995 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:18.128001 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/602ab155-5413-4fe5-9929-9369acca84aa-routing-overlay-volume\") pod \"payload-processing-external-model-69d99678c7-nhwvh\" (UID: \"602ab155-5413-4fe5-9929-9369acca84aa\") " pod="models-as-a-service/payload-processing-external-model-69d99678c7-nhwvh" Oct 30 02:02:18.128271 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:18.128089 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/602ab155-5413-4fe5-9929-9369acca84aa-plugins-config-volume\") pod \"payload-processing-external-model-69d99678c7-nhwvh\" (UID: \"602ab155-5413-4fe5-9929-9369acca84aa\") " pod="models-as-a-service/payload-processing-external-model-69d99678c7-nhwvh" Oct 30 02:02:18.128271 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:18.128207 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcjz4\" (UniqueName: \"kubernetes.io/projected/602ab155-5413-4fe5-9929-9369acca84aa-kube-api-access-dcjz4\") pod \"payload-processing-external-model-69d99678c7-nhwvh\" (UID: \"602ab155-5413-4fe5-9929-9369acca84aa\") " pod="models-as-a-service/payload-processing-external-model-69d99678c7-nhwvh" Oct 30 02:02:18.128927 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:18.128903 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/602ab155-5413-4fe5-9929-9369acca84aa-plugins-config-volume\") pod \"payload-processing-external-model-69d99678c7-nhwvh\" (UID: \"602ab155-5413-4fe5-9929-9369acca84aa\") " pod="models-as-a-service/payload-processing-external-model-69d99678c7-nhwvh" Oct 30 02:02:18.129073 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:18.128928 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/602ab155-5413-4fe5-9929-9369acca84aa-routing-overlay-volume\") pod \"payload-processing-external-model-69d99678c7-nhwvh\" (UID: \"602ab155-5413-4fe5-9929-9369acca84aa\") " pod="models-as-a-service/payload-processing-external-model-69d99678c7-nhwvh" Oct 30 02:02:18.131059 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:18.131032 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/602ab155-5413-4fe5-9929-9369acca84aa-provider-credentials\") pod \"payload-processing-external-model-69d99678c7-nhwvh\" (UID: \"602ab155-5413-4fe5-9929-9369acca84aa\") " pod="models-as-a-service/payload-processing-external-model-69d99678c7-nhwvh" Oct 30 02:02:18.148222 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:18.148139 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcjz4\" (UniqueName: \"kubernetes.io/projected/602ab155-5413-4fe5-9929-9369acca84aa-kube-api-access-dcjz4\") pod \"payload-processing-external-model-69d99678c7-nhwvh\" (UID: \"602ab155-5413-4fe5-9929-9369acca84aa\") " pod="models-as-a-service/payload-processing-external-model-69d99678c7-nhwvh" Oct 30 02:02:18.234028 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:18.233986 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:18.242693 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:18.242649 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-69d99678c7-nhwvh" Oct 30 02:02:18.421120 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:18.421009 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:18.678715 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:18.678686 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-69d99678c7-nhwvh"] Oct 30 02:02:19.025829 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:19.024719 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-69d99678c7-nhwvh" event={"ID":"602ab155-5413-4fe5-9929-9369acca84aa","Type":"ContainerStarted","Data":"d5c10a1c412dd12a9eff0f2082b74e23acf9467a582889b33ce4ff3cee4b8ce3"} Oct 30 02:02:19.025829 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:19.024815 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-69d99678c7-nhwvh" event={"ID":"602ab155-5413-4fe5-9929-9369acca84aa","Type":"ContainerStarted","Data":"a2a11c510bbae50fcc51675a39449ece8e0a803fa9a8929ec3e12cb3bd38cf0b"} Oct 30 02:02:19.026381 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:19.026049 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-69d99678c7-nhwvh" Oct 30 02:02:19.067047 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:19.066985 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-69d99678c7-nhwvh" podStartSLOduration=2.066963606 podStartE2EDuration="2.066963606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:02:19.063130536 +0000 UTC m=+2664.639555939" watchObservedRunningTime="2026-10-30 02:02:19.066963606 +0000 UTC m=+2664.643389008" Oct 30 02:02:19.606769 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:19.605571 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:21.756522 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:21.756257 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:23.390772 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:23.387359 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:24.835730 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:24.835691 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:26.385785 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:26.384085 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:27.929302 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:27.929263 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:29.485462 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:29.485424 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:31.024424 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:31.024375 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:31.127686 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:31.127658 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-69d99678c7-nhwvh" Oct 30 02:02:31.198057 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:31.198022 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-86d69bd698-dhmzl"] Oct 30 02:02:32.613478 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:32.613436 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:34.193318 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:34.193277 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:35.718503 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:35.717949 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:36.199083 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:36.199032 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-86d69bd698-dhmzl" podUID="9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad" containerName="payload-processing" containerID="cri-o://e5a43dcb6721a7914d53317856c94d85cc456ba67723f98c0cf3ee9b44d7269a" gracePeriod=25 Oct 30 02:02:36.507887 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:36.507855 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-86d69bd698-dhmzl" Oct 30 02:02:36.600327 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:36.600276 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad-plugins-config-volume\") pod \"9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad\" (UID: \"9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad\") " Oct 30 02:02:36.600658 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:36.600334 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad-kube-api-access-z8b5h\" (UniqueName: \"kubernetes.io/projected/9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad-kube-api-access-z8b5h\") pod \"9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad\" (UID: \"9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad\") " Oct 30 02:02:36.600658 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:36.600589 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad-routing-overlay-volume\") pod \"9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad\" (UID: \"9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad\") " Oct 30 02:02:36.600658 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:36.600630 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad-provider-credentials\") pod \"9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad\" (UID: \"9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad\") " Oct 30 02:02:36.601183 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:36.600816 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad-plugins-config-volume" pod "9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad" (UID: "9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:02:36.601986 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:36.601955 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad-routing-overlay-volume" pod "9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad" (UID: "9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:02:36.602231 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:36.602038 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:02:36.604321 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:36.604280 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad-kube-api-access-z8b5h" pod "9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad" (UID: "9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad"). InnerVolumeSpecName "kube-api-access-z8b5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:02:36.611703 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:36.611641 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad-provider-credentials" pod "9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad" (UID: "9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:02:36.704837 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:36.704794 6572 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad-routing-overlay-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:02:36.704837 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:36.704834 6572 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad-provider-credentials\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:02:36.705080 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:36.704851 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8b5h\" (UniqueName: \"kubernetes.io/projected/9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad-kube-api-access-z8b5h\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:02:37.304027 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:37.303988 6572 generic.go:356] "Generic (PLEG): container finished" podID="9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad" containerID="e5a43dcb6721a7914d53317856c94d85cc456ba67723f98c0cf3ee9b44d7269a" exitCode=0 Oct 30 02:02:37.304518 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:37.304071 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-86d69bd698-dhmzl" Oct 30 02:02:37.304518 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:37.304073 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-86d69bd698-dhmzl" event={"ID":"9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad","Type":"ContainerDied","Data":"e5a43dcb6721a7914d53317856c94d85cc456ba67723f98c0cf3ee9b44d7269a"} Oct 30 02:02:37.304518 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:37.304257 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-86d69bd698-dhmzl" event={"ID":"9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad","Type":"ContainerDied","Data":"edfb0759b9708229649cad56d9691db767eb8aa2d53cc73f4ac15732597915a1"} Oct 30 02:02:37.304518 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:37.304288 6572 scope.go:122] "RemoveContainer" containerID="e5a43dcb6721a7914d53317856c94d85cc456ba67723f98c0cf3ee9b44d7269a" Oct 30 02:02:37.317465 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:37.317442 6572 scope.go:122] "RemoveContainer" containerID="e5a43dcb6721a7914d53317856c94d85cc456ba67723f98c0cf3ee9b44d7269a" Oct 30 02:02:37.317833 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:02:37.317811 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5a43dcb6721a7914d53317856c94d85cc456ba67723f98c0cf3ee9b44d7269a\": container with ID starting with e5a43dcb6721a7914d53317856c94d85cc456ba67723f98c0cf3ee9b44d7269a not found: ID does not exist" containerID="e5a43dcb6721a7914d53317856c94d85cc456ba67723f98c0cf3ee9b44d7269a" Oct 30 02:02:37.317895 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:37.317845 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5a43dcb6721a7914d53317856c94d85cc456ba67723f98c0cf3ee9b44d7269a"} err="failed to get container status \"e5a43dcb6721a7914d53317856c94d85cc456ba67723f98c0cf3ee9b44d7269a\": rpc error: code = NotFound desc = could not find container \"e5a43dcb6721a7914d53317856c94d85cc456ba67723f98c0cf3ee9b44d7269a\": container with ID starting with e5a43dcb6721a7914d53317856c94d85cc456ba67723f98c0cf3ee9b44d7269a not found: ID does not exist" Oct 30 02:02:37.366807 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:37.366769 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-86d69bd698-dhmzl"] Oct 30 02:02:37.374284 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:37.374248 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-86d69bd698-dhmzl"] Oct 30 02:02:37.560220 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:37.560121 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:38.548033 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:38.547991 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad" path="/var/lib/kubelet/pods/9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad/volumes" Oct 30 02:02:39.236645 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:39.236594 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:41.273615 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:41.273585 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:42.733599 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:42.733494 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:44.278219 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:44.278095 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:44.690390 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:02:44.690352 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:02:45.940779 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:45.938966 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:47.516426 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:47.516387 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:49.090908 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:49.090869 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:50.829583 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:50.829545 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:52.487717 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:52.487675 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:54.039250 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:54.039212 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:55.526966 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:55.526920 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:57.124578 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:57.124521 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:58.578174 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:58.578135 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:58.764641 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:02:58.764603 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:00.019865 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:00.019832 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:01.494209 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:01.494165 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:03.045400 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:03.045364 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:04.706781 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:04.705832 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:06.310089 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:06.310045 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:07.392240 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:07.392202 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:07.885852 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:07.885816 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:08.976521 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:08.976483 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:09.167228 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:09.167193 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:09.402957 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:09.402859 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:10.987564 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:10.987521 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:12.578515 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:12.578475 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:14.113510 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:14.113472 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:14.689920 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:03:14.689879 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:03:16.037780 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:16.037711 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:17.456578 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:17.454515 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:19.041796 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:19.041735 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b765065e-524x2"] Oct 30 02:03:19.062945 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:19.062903 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b765065e-524x2"] Oct 30 02:03:19.528444 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:19.528410 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:20.544687 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:20.544649 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5175c99-df96-4d90-90ea-c2274424a8a5" path="/var/lib/kubelet/pods/e5175c99-df96-4d90-90ea-c2274424a8a5/volumes" Oct 30 02:03:21.298308 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:21.298259 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:22.913769 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:22.908597 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:24.437916 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:24.437881 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:26.024443 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:26.024411 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:26.033522 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:26.033489 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-171c395f-9mw5n"] Oct 30 02:03:26.044049 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:26.044019 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-171c395f-9mw5n"] Oct 30 02:03:26.545431 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:26.545393 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11f367bb-5a71-4dad-8f72-88ef98f109f9" path="/var/lib/kubelet/pods/11f367bb-5a71-4dad-8f72-88ef98f109f9/volumes" Oct 30 02:03:27.702513 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:27.702479 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:29.258707 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:29.258654 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:30.779276 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:30.779241 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:32.448886 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:32.447277 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:34.004124 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:34.004079 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:35.483574 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:35.479872 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:36.958947 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:36.958794 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:38.688840 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:38.688803 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:40.286389 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:40.286290 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:41.874552 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:41.874488 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:43.461557 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:43.461514 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:44.689616 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:03:44.689576 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:03:44.926833 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:44.926792 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:46.497383 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:46.497344 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:48.092360 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:48.092103 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:49.670040 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:49.669995 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:51.212450 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:51.212412 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:52.901444 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:52.901406 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:54.378067 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:54.378014 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:55.996465 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:55.996415 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:57.603357 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:57.600822 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:58.431459 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:58.431425 6572 scope.go:122] "RemoveContainer" containerID="7a22d530689c97a3c544f1ad8ea6e9000d5396fb57832a6b74528ade2b7d839e" Oct 30 02:03:59.053921 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:03:59.053885 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:00.602327 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:00.602292 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:02.311429 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:02.311386 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:03.904004 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:03.903970 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:05.617362 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:05.617324 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:07.377013 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:07.376963 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:07.619809 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:07.619734 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:09.011256 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:09.011218 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:10.664900 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:10.664860 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:12.352977 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:12.352930 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:13.992906 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:13.992862 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:14.688448 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:04:14.688401 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:04:15.646773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:15.642437 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:17.374268 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:17.374228 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:18.998805 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:18.998023 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:19.752935 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:19.752899 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:19.962773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:19.961853 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:21.109284 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.109242 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:21.208063 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.208014 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4"] Oct 30 02:04:21.209064 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.209038 6572 cpu_manager.go:414] "containerMap: removing container" podUID="9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad" containerName="payload-processing" Oct 30 02:04:21.209064 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.209063 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad" containerName="payload-processing" Oct 30 02:04:21.209279 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.209203 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fa5f01c-e6fd-42b8-ba49-5b9efd29d2ad" containerName="payload-processing" Oct 30 02:04:21.213368 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.213343 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4" Oct 30 02:04:21.227490 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.227458 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2e618f\"/\"agc-cred-53d81d8dd4f4\"" Oct 30 02:04:21.250948 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.250904 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4"] Oct 30 02:04:21.274060 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.274008 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca478104-e26b-4bd8-8e86-684c2807e4aa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4\" (UID: \"ca478104-e26b-4bd8-8e86-684c2807e4aa\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4" Oct 30 02:04:21.274268 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.274165 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca478104-e26b-4bd8-8e86-684c2807e4aa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4\" (UID: \"ca478104-e26b-4bd8-8e86-684c2807e4aa\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4" Oct 30 02:04:21.274338 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.274283 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca478104-e26b-4bd8-8e86-684c2807e4aa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4\" (UID: \"ca478104-e26b-4bd8-8e86-684c2807e4aa\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4" Oct 30 02:04:21.274399 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.274342 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbl5g\" (UniqueName: \"kubernetes.io/projected/ca478104-e26b-4bd8-8e86-684c2807e4aa-kube-api-access-cbl5g\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4\" (UID: \"ca478104-e26b-4bd8-8e86-684c2807e4aa\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4" Oct 30 02:04:21.375684 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.375580 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca478104-e26b-4bd8-8e86-684c2807e4aa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4\" (UID: \"ca478104-e26b-4bd8-8e86-684c2807e4aa\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4" Oct 30 02:04:21.375684 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.375665 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca478104-e26b-4bd8-8e86-684c2807e4aa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4\" (UID: \"ca478104-e26b-4bd8-8e86-684c2807e4aa\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4" Oct 30 02:04:21.375956 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.375709 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca478104-e26b-4bd8-8e86-684c2807e4aa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4\" (UID: \"ca478104-e26b-4bd8-8e86-684c2807e4aa\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4" Oct 30 02:04:21.375956 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.375739 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbl5g\" (UniqueName: \"kubernetes.io/projected/ca478104-e26b-4bd8-8e86-684c2807e4aa-kube-api-access-cbl5g\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4\" (UID: \"ca478104-e26b-4bd8-8e86-684c2807e4aa\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4" Oct 30 02:04:21.376458 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.376428 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca478104-e26b-4bd8-8e86-684c2807e4aa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4\" (UID: \"ca478104-e26b-4bd8-8e86-684c2807e4aa\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4" Oct 30 02:04:21.376573 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.376548 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca478104-e26b-4bd8-8e86-684c2807e4aa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4\" (UID: \"ca478104-e26b-4bd8-8e86-684c2807e4aa\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4" Oct 30 02:04:21.379125 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.379093 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca478104-e26b-4bd8-8e86-684c2807e4aa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4\" (UID: \"ca478104-e26b-4bd8-8e86-684c2807e4aa\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4" Oct 30 02:04:21.395123 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.395083 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbl5g\" (UniqueName: \"kubernetes.io/projected/ca478104-e26b-4bd8-8e86-684c2807e4aa-kube-api-access-cbl5g\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4\" (UID: \"ca478104-e26b-4bd8-8e86-684c2807e4aa\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4" Oct 30 02:04:21.534802 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.534740 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4" Oct 30 02:04:21.836560 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.836520 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a75e7c1-6pfh4"] Oct 30 02:04:21.841997 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.841967 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a75e7c1-6pfh4" Oct 30 02:04:21.859462 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.859411 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:04:21.859799 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.859781 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:04:21.862084 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.862057 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a75e7c1-6pfh4"] Oct 30 02:04:21.867675 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.867642 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:04:21.867835 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.867685 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:04:21.869355 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:04:21.869327 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca478104_e26b_4bd8_8e86_684c2807e4aa.slice/crio-06674034d1417ebf0b0a8cc62fd504e18a6ebef5f7128f47e488aefd6b60423e WatchSource:0}: Error finding container 06674034d1417ebf0b0a8cc62fd504e18a6ebef5f7128f47e488aefd6b60423e: Status 404 returned error can't find the container with id 06674034d1417ebf0b0a8cc62fd504e18a6ebef5f7128f47e488aefd6b60423e Oct 30 02:04:21.879055 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.877316 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4"] Oct 30 02:04:21.884972 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.884940 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5a194d9-ae2d-43ba-8241-91b03c367658-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a75e7c1-6pfh4\" (UID: \"c5a194d9-ae2d-43ba-8241-91b03c367658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a75e7c1-6pfh4" Oct 30 02:04:21.885115 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.885087 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5a194d9-ae2d-43ba-8241-91b03c367658-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a75e7c1-6pfh4\" (UID: \"c5a194d9-ae2d-43ba-8241-91b03c367658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a75e7c1-6pfh4" Oct 30 02:04:21.887363 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.887337 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvgvf\" (UniqueName: \"kubernetes.io/projected/c5a194d9-ae2d-43ba-8241-91b03c367658-kube-api-access-pvgvf\") pod \"maas-api-revoke-sub-9a75e7c1-6pfh4\" (UID: \"c5a194d9-ae2d-43ba-8241-91b03c367658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a75e7c1-6pfh4" Oct 30 02:04:21.989462 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.988731 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvgvf\" (UniqueName: \"kubernetes.io/projected/c5a194d9-ae2d-43ba-8241-91b03c367658-kube-api-access-pvgvf\") pod \"maas-api-revoke-sub-9a75e7c1-6pfh4\" (UID: \"c5a194d9-ae2d-43ba-8241-91b03c367658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a75e7c1-6pfh4" Oct 30 02:04:21.989462 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.988982 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5a194d9-ae2d-43ba-8241-91b03c367658-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a75e7c1-6pfh4\" (UID: \"c5a194d9-ae2d-43ba-8241-91b03c367658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a75e7c1-6pfh4" Oct 30 02:04:21.991211 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.989819 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5a194d9-ae2d-43ba-8241-91b03c367658-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a75e7c1-6pfh4\" (UID: \"c5a194d9-ae2d-43ba-8241-91b03c367658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a75e7c1-6pfh4" Oct 30 02:04:21.991450 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.991420 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5a194d9-ae2d-43ba-8241-91b03c367658-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a75e7c1-6pfh4\" (UID: \"c5a194d9-ae2d-43ba-8241-91b03c367658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a75e7c1-6pfh4" Oct 30 02:04:21.996951 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.996922 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45"] Oct 30 02:04:21.999299 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:21.999276 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5a194d9-ae2d-43ba-8241-91b03c367658-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a75e7c1-6pfh4\" (UID: \"c5a194d9-ae2d-43ba-8241-91b03c367658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a75e7c1-6pfh4" Oct 30 02:04:22.003063 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:22.003043 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45" Oct 30 02:04:22.008139 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:22.008113 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvgvf\" (UniqueName: \"kubernetes.io/projected/c5a194d9-ae2d-43ba-8241-91b03c367658-kube-api-access-pvgvf\") pod \"maas-api-revoke-sub-9a75e7c1-6pfh4\" (UID: \"c5a194d9-ae2d-43ba-8241-91b03c367658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a75e7c1-6pfh4" Oct 30 02:04:22.017816 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:22.017788 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-76a19e\"/\"agc-cred-d4c1ba4bb37f\"" Oct 30 02:04:22.021541 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:22.021515 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45"] Oct 30 02:04:22.091126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:22.091033 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/285a2ade-2eaa-4b72-8a06-26e9a9395442-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45\" (UID: \"285a2ade-2eaa-4b72-8a06-26e9a9395442\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45" Oct 30 02:04:22.091126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:22.091100 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/285a2ade-2eaa-4b72-8a06-26e9a9395442-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45\" (UID: \"285a2ade-2eaa-4b72-8a06-26e9a9395442\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45" Oct 30 02:04:22.091337 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:22.091224 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q4q8\" (UniqueName: \"kubernetes.io/projected/285a2ade-2eaa-4b72-8a06-26e9a9395442-kube-api-access-7q4q8\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45\" (UID: \"285a2ade-2eaa-4b72-8a06-26e9a9395442\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45" Oct 30 02:04:22.091377 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:22.091347 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/285a2ade-2eaa-4b72-8a06-26e9a9395442-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45\" (UID: \"285a2ade-2eaa-4b72-8a06-26e9a9395442\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45" Oct 30 02:04:22.160281 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:22.160247 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a75e7c1-6pfh4" Oct 30 02:04:22.193140 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:22.193103 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/285a2ade-2eaa-4b72-8a06-26e9a9395442-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45\" (UID: \"285a2ade-2eaa-4b72-8a06-26e9a9395442\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45" Oct 30 02:04:22.193323 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:22.193213 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/285a2ade-2eaa-4b72-8a06-26e9a9395442-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45\" (UID: \"285a2ade-2eaa-4b72-8a06-26e9a9395442\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45" Oct 30 02:04:22.193323 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:22.193307 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q4q8\" (UniqueName: \"kubernetes.io/projected/285a2ade-2eaa-4b72-8a06-26e9a9395442-kube-api-access-7q4q8\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45\" (UID: \"285a2ade-2eaa-4b72-8a06-26e9a9395442\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45" Oct 30 02:04:22.193442 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:22.193425 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/285a2ade-2eaa-4b72-8a06-26e9a9395442-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45\" (UID: \"285a2ade-2eaa-4b72-8a06-26e9a9395442\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45" Oct 30 02:04:22.194188 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:22.194160 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/285a2ade-2eaa-4b72-8a06-26e9a9395442-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45\" (UID: \"285a2ade-2eaa-4b72-8a06-26e9a9395442\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45" Oct 30 02:04:22.194310 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:22.194202 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/285a2ade-2eaa-4b72-8a06-26e9a9395442-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45\" (UID: \"285a2ade-2eaa-4b72-8a06-26e9a9395442\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45" Oct 30 02:04:22.195873 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:22.195852 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/285a2ade-2eaa-4b72-8a06-26e9a9395442-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45\" (UID: \"285a2ade-2eaa-4b72-8a06-26e9a9395442\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45" Oct 30 02:04:22.213871 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:22.213835 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q4q8\" (UniqueName: \"kubernetes.io/projected/285a2ade-2eaa-4b72-8a06-26e9a9395442-kube-api-access-7q4q8\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45\" (UID: \"285a2ade-2eaa-4b72-8a06-26e9a9395442\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45" Oct 30 02:04:22.322425 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:22.322387 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45" Oct 30 02:04:22.466300 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:22.466265 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a75e7c1-6pfh4"] Oct 30 02:04:22.547003 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:22.546955 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a75e7c1-6pfh4" event={"ID":"c5a194d9-ae2d-43ba-8241-91b03c367658","Type":"ContainerStarted","Data":"1ed70687b62bc424632daf74179999ddf58911781fd0d697be8bbcaa92d22828"} Oct 30 02:04:22.549387 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:22.549353 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4" event={"ID":"ca478104-e26b-4bd8-8e86-684c2807e4aa","Type":"ContainerStarted","Data":"f8fa7aa42375178d7395200f1506c25ee03f10da39cbe8e87647edefa51750d9"} Oct 30 02:04:22.549546 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:22.549416 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4" event={"ID":"ca478104-e26b-4bd8-8e86-684c2807e4aa","Type":"ContainerStarted","Data":"06674034d1417ebf0b0a8cc62fd504e18a6ebef5f7128f47e488aefd6b60423e"} Oct 30 02:04:22.549546 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:22.549521 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4" Oct 30 02:04:22.594823 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:22.594760 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4" podStartSLOduration=1.594726367 podStartE2EDuration="1.594726367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:04:22.589980326 +0000 UTC m=+2788.166405741" watchObservedRunningTime="2026-10-30 02:04:22.594726367 +0000 UTC m=+2788.171151799" Oct 30 02:04:22.630383 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:22.630305 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45"] Oct 30 02:04:22.631063 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:04:22.631039 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod285a2ade_2eaa_4b72_8a06_26e9a9395442.slice/crio-2ebce2c60fd0e636d72b97dbdbe05d5d50284723a3484a311f72a6caf15247de WatchSource:0}: Error finding container 2ebce2c60fd0e636d72b97dbdbe05d5d50284723a3484a311f72a6caf15247de: Status 404 returned error can't find the container with id 2ebce2c60fd0e636d72b97dbdbe05d5d50284723a3484a311f72a6caf15247de Oct 30 02:04:22.679917 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:22.669804 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:23.560806 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:23.560737 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45" event={"ID":"285a2ade-2eaa-4b72-8a06-26e9a9395442","Type":"ContainerStarted","Data":"4237b59d59945979fec4facd4138e66bf3408540e9698d4d97318be7f9f1b702"} Oct 30 02:04:23.561274 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:23.560834 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45" event={"ID":"285a2ade-2eaa-4b72-8a06-26e9a9395442","Type":"ContainerStarted","Data":"2ebce2c60fd0e636d72b97dbdbe05d5d50284723a3484a311f72a6caf15247de"} Oct 30 02:04:23.561274 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:23.561024 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45" Oct 30 02:04:23.562610 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:23.562587 6572 generic.go:356] "Generic (PLEG): container finished" podID="c5a194d9-ae2d-43ba-8241-91b03c367658" containerID="f1ed0d42d984a5df6191b633744d8a2870c9e5f0bd2c647c881b0d8d961e0ec1" exitCode=0 Oct 30 02:04:23.562819 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:23.562679 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a75e7c1-6pfh4" event={"ID":"c5a194d9-ae2d-43ba-8241-91b03c367658","Type":"ContainerDied","Data":"f1ed0d42d984a5df6191b633744d8a2870c9e5f0bd2c647c881b0d8d961e0ec1"} Oct 30 02:04:23.617167 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:23.617112 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45" podStartSLOduration=2.6170971549999997 podStartE2EDuration="2.617097155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:04:23.601958162 +0000 UTC m=+2789.178383566" watchObservedRunningTime="2026-10-30 02:04:23.617097155 +0000 UTC m=+2789.193522553" Oct 30 02:04:24.308018 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:24.305052 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:24.747927 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:24.747902 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a75e7c1-6pfh4" Oct 30 02:04:24.840238 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:24.840138 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5a194d9-ae2d-43ba-8241-91b03c367658-kube-api-access-pvgvf\" (UniqueName: \"kubernetes.io/projected/c5a194d9-ae2d-43ba-8241-91b03c367658-kube-api-access-pvgvf\") pod \"c5a194d9-ae2d-43ba-8241-91b03c367658\" (UID: \"c5a194d9-ae2d-43ba-8241-91b03c367658\") " Oct 30 02:04:24.840238 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:24.840177 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5a194d9-ae2d-43ba-8241-91b03c367658-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5a194d9-ae2d-43ba-8241-91b03c367658-maas-api-openshift-ca\") pod \"c5a194d9-ae2d-43ba-8241-91b03c367658\" (UID: \"c5a194d9-ae2d-43ba-8241-91b03c367658\") " Oct 30 02:04:24.840238 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:24.840217 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5a194d9-ae2d-43ba-8241-91b03c367658-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5a194d9-ae2d-43ba-8241-91b03c367658-maas-api-xks-ca\") pod \"c5a194d9-ae2d-43ba-8241-91b03c367658\" (UID: \"c5a194d9-ae2d-43ba-8241-91b03c367658\") " Oct 30 02:04:24.840845 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:24.840807 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a194d9-ae2d-43ba-8241-91b03c367658-maas-api-openshift-ca" pod "c5a194d9-ae2d-43ba-8241-91b03c367658" (UID: "c5a194d9-ae2d-43ba-8241-91b03c367658"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:04:24.841254 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:24.841204 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5a194d9-ae2d-43ba-8241-91b03c367658-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:04:24.843222 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:24.843193 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5a194d9-ae2d-43ba-8241-91b03c367658-maas-api-xks-ca" pod "c5a194d9-ae2d-43ba-8241-91b03c367658" (UID: "c5a194d9-ae2d-43ba-8241-91b03c367658"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:04:24.843495 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:24.843471 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a194d9-ae2d-43ba-8241-91b03c367658-kube-api-access-pvgvf" pod "c5a194d9-ae2d-43ba-8241-91b03c367658" (UID: "c5a194d9-ae2d-43ba-8241-91b03c367658"). InnerVolumeSpecName "kube-api-access-pvgvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:24.943266 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:24.943229 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvgvf\" (UniqueName: \"kubernetes.io/projected/c5a194d9-ae2d-43ba-8241-91b03c367658-kube-api-access-pvgvf\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:04:24.943266 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:24.943263 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5a194d9-ae2d-43ba-8241-91b03c367658-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:04:25.581994 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:25.581965 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a75e7c1-6pfh4" Oct 30 02:04:25.582176 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:25.582000 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a75e7c1-6pfh4" event={"ID":"c5a194d9-ae2d-43ba-8241-91b03c367658","Type":"ContainerDied","Data":"1ed70687b62bc424632daf74179999ddf58911781fd0d697be8bbcaa92d22828"} Oct 30 02:04:25.582176 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:25.582046 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ed70687b62bc424632daf74179999ddf58911781fd0d697be8bbcaa92d22828" Oct 30 02:04:25.883865 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:25.882925 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:27.371719 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:27.371619 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:28.967209 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:28.966667 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:30.492454 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:30.492412 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:32.036060 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:32.036026 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:33.235874 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:33.235837 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:33.494959 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:33.494843 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-56bd9db94-6l572"] Oct 30 02:04:33.495675 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:33.495660 6572 cpu_manager.go:414] "containerMap: removing container" podUID="c5a194d9-ae2d-43ba-8241-91b03c367658" containerName="revoke-keys" Oct 30 02:04:33.495725 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:33.495676 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5a194d9-ae2d-43ba-8241-91b03c367658" containerName="revoke-keys" Oct 30 02:04:33.495871 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:33.495861 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5a194d9-ae2d-43ba-8241-91b03c367658" containerName="revoke-keys" Oct 30 02:04:33.499014 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:33.498992 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56bd9db94-6l572" Oct 30 02:04:33.506092 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:33.506043 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:33.530240 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:33.530204 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-56bd9db94-6l572"] Oct 30 02:04:33.566589 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:33.566560 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4" Oct 30 02:04:33.592104 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:33.592058 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/584d49a8-b8f8-4012-9740-763c8e930126-plugins-config-volume\") pod \"payload-processing-external-model-56bd9db94-6l572\" (UID: \"584d49a8-b8f8-4012-9740-763c8e930126\") " pod="models-as-a-service/payload-processing-external-model-56bd9db94-6l572" Oct 30 02:04:33.592423 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:33.592401 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jfwg\" (UniqueName: \"kubernetes.io/projected/584d49a8-b8f8-4012-9740-763c8e930126-kube-api-access-8jfwg\") pod \"payload-processing-external-model-56bd9db94-6l572\" (UID: \"584d49a8-b8f8-4012-9740-763c8e930126\") " pod="models-as-a-service/payload-processing-external-model-56bd9db94-6l572" Oct 30 02:04:33.592562 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:33.592499 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/584d49a8-b8f8-4012-9740-763c8e930126-routing-overlay-volume\") pod \"payload-processing-external-model-56bd9db94-6l572\" (UID: \"584d49a8-b8f8-4012-9740-763c8e930126\") " pod="models-as-a-service/payload-processing-external-model-56bd9db94-6l572" Oct 30 02:04:33.592562 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:33.592523 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/584d49a8-b8f8-4012-9740-763c8e930126-provider-credentials\") pod \"payload-processing-external-model-56bd9db94-6l572\" (UID: \"584d49a8-b8f8-4012-9740-763c8e930126\") " pod="models-as-a-service/payload-processing-external-model-56bd9db94-6l572" Oct 30 02:04:33.646098 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:33.646055 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg"] Oct 30 02:04:33.693699 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:33.693666 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jfwg\" (UniqueName: \"kubernetes.io/projected/584d49a8-b8f8-4012-9740-763c8e930126-kube-api-access-8jfwg\") pod \"payload-processing-external-model-56bd9db94-6l572\" (UID: \"584d49a8-b8f8-4012-9740-763c8e930126\") " pod="models-as-a-service/payload-processing-external-model-56bd9db94-6l572" Oct 30 02:04:33.693963 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:33.693730 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/584d49a8-b8f8-4012-9740-763c8e930126-routing-overlay-volume\") pod \"payload-processing-external-model-56bd9db94-6l572\" (UID: \"584d49a8-b8f8-4012-9740-763c8e930126\") " pod="models-as-a-service/payload-processing-external-model-56bd9db94-6l572" Oct 30 02:04:33.693963 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:33.693763 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/584d49a8-b8f8-4012-9740-763c8e930126-provider-credentials\") pod \"payload-processing-external-model-56bd9db94-6l572\" (UID: \"584d49a8-b8f8-4012-9740-763c8e930126\") " pod="models-as-a-service/payload-processing-external-model-56bd9db94-6l572" Oct 30 02:04:33.693963 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:33.693808 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/584d49a8-b8f8-4012-9740-763c8e930126-plugins-config-volume\") pod \"payload-processing-external-model-56bd9db94-6l572\" (UID: \"584d49a8-b8f8-4012-9740-763c8e930126\") " pod="models-as-a-service/payload-processing-external-model-56bd9db94-6l572" Oct 30 02:04:33.694848 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:33.694601 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/584d49a8-b8f8-4012-9740-763c8e930126-routing-overlay-volume\") pod \"payload-processing-external-model-56bd9db94-6l572\" (UID: \"584d49a8-b8f8-4012-9740-763c8e930126\") " pod="models-as-a-service/payload-processing-external-model-56bd9db94-6l572" Oct 30 02:04:33.695510 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:33.695479 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/584d49a8-b8f8-4012-9740-763c8e930126-plugins-config-volume\") pod \"payload-processing-external-model-56bd9db94-6l572\" (UID: \"584d49a8-b8f8-4012-9740-763c8e930126\") " pod="models-as-a-service/payload-processing-external-model-56bd9db94-6l572" Oct 30 02:04:33.696995 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:33.696971 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/584d49a8-b8f8-4012-9740-763c8e930126-provider-credentials\") pod \"payload-processing-external-model-56bd9db94-6l572\" (UID: \"584d49a8-b8f8-4012-9740-763c8e930126\") " pod="models-as-a-service/payload-processing-external-model-56bd9db94-6l572" Oct 30 02:04:33.713511 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:33.713474 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jfwg\" (UniqueName: \"kubernetes.io/projected/584d49a8-b8f8-4012-9740-763c8e930126-kube-api-access-8jfwg\") pod \"payload-processing-external-model-56bd9db94-6l572\" (UID: \"584d49a8-b8f8-4012-9740-763c8e930126\") " pod="models-as-a-service/payload-processing-external-model-56bd9db94-6l572" Oct 30 02:04:33.815488 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:33.815387 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56bd9db94-6l572" Oct 30 02:04:34.044779 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:34.033614 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:34.442798 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:34.436839 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-56bd9db94-6l572"] Oct 30 02:04:34.575516 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:34.575489 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45" Oct 30 02:04:34.653487 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:34.653400 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv"] Oct 30 02:04:34.671938 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:34.671895 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56bd9db94-6l572" event={"ID":"584d49a8-b8f8-4012-9740-763c8e930126","Type":"ContainerStarted","Data":"01922007307b03523cae1737abdfbb242a524c757e38783c635f9869cc37f763"} Oct 30 02:04:34.672144 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:34.671965 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56bd9db94-6l572" event={"ID":"584d49a8-b8f8-4012-9740-763c8e930126","Type":"ContainerStarted","Data":"c78894e8c63e810f028ad8ff54530275f95909ca191420640089bcb817106784"} Oct 30 02:04:34.672144 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:34.672001 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-56bd9db94-6l572" Oct 30 02:04:34.735588 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:34.735532 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-56bd9db94-6l572" podStartSLOduration=1.735514839 podStartE2EDuration="1.735514839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:04:34.730114512 +0000 UTC m=+2800.306539913" watchObservedRunningTime="2026-10-30 02:04:34.735514839 +0000 UTC m=+2800.311940242" Oct 30 02:04:35.390379 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:35.390348 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:35.690453 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:35.690418 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:35.784876 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:35.784838 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq"] Oct 30 02:04:35.791154 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:35.791122 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:04:35.812001 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:35.811962 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq"] Oct 30 02:04:35.815763 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:35.815710 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-717065-openshift-default-dockercfg-6l8vj\"" Oct 30 02:04:35.866457 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:35.866412 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-istio-token\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq\" (UID: \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:04:35.867257 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:35.867231 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-istio-podinfo\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq\" (UID: \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:04:35.867431 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:35.867376 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-credential-socket\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq\" (UID: \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:04:35.867991 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:35.867964 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp929\" (UniqueName: \"kubernetes.io/projected/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-kube-api-access-gp929\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq\" (UID: \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:04:35.868108 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:35.868030 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-istiod-ca-cert\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq\" (UID: \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:04:35.868108 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:35.868079 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-workload-socket\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq\" (UID: \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:04:35.868217 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:35.868143 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-workload-certs\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq\" (UID: \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:04:35.868217 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:35.868177 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-istio-envoy\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq\" (UID: \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:04:35.868217 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:35.868207 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-istio-data\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq\" (UID: \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:04:35.988845 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:35.977720 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-istiod-ca-cert\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq\" (UID: \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:04:35.989192 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:35.989157 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-workload-socket\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq\" (UID: \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:04:35.989406 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:35.989389 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-workload-certs\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq\" (UID: \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:04:35.989538 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:35.989523 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-istio-envoy\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq\" (UID: \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:04:35.989658 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:35.989641 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-istio-data\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq\" (UID: \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:04:35.995883 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:35.995847 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-istio-token\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq\" (UID: \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:04:35.996216 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:35.978875 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-istiod-ca-cert\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq\" (UID: \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:04:35.996459 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:35.996435 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-istio-podinfo\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq\" (UID: \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:04:36.001937 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:35.996631 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-credential-socket\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq\" (UID: \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:04:36.002284 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:36.002256 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp929\" (UniqueName: \"kubernetes.io/projected/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-kube-api-access-gp929\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq\" (UID: \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:04:36.002618 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:36.000121 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-workload-socket\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq\" (UID: \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:04:36.002813 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:36.000564 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-istio-data\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq\" (UID: \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:04:36.002976 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:36.000798 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-credential-socket\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq\" (UID: \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:04:36.003088 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:36.002091 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:36.003246 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:35.999718 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-workload-certs\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq\" (UID: \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:04:36.038883 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:36.037511 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp929\" (UniqueName: \"kubernetes.io/projected/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-kube-api-access-gp929\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq\" (UID: \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:04:36.076227 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:36.066221 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-istio-podinfo\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq\" (UID: \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:04:36.076227 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:36.070512 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-istio-token\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq\" (UID: \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:04:36.110769 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:36.097959 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-istio-envoy\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq\" (UID: \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:04:36.114321 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:36.114022 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:04:36.372361 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:36.372266 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:36.450206 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:36.450170 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq"] Oct 30 02:04:36.450387 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:04:36.450357 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03f91ca9_d4d4_4e24_b535_05f521c0b3b1.slice/crio-cc5ca1ae192c2901e7170a38fdaf3499dae5ab9f8ebbaae35fe2886f0f229cf4 WatchSource:0}: Error finding container cc5ca1ae192c2901e7170a38fdaf3499dae5ab9f8ebbaae35fe2886f0f229cf4: Status 404 returned error can't find the container with id cc5ca1ae192c2901e7170a38fdaf3499dae5ab9f8ebbaae35fe2886f0f229cf4 Oct 30 02:04:36.453783 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:36.453713 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:04:36.453886 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:36.453809 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:04:36.453886 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:36.453877 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:04:36.691316 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:36.691266 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" event={"ID":"03f91ca9-d4d4-4e24-b535-05f521c0b3b1","Type":"ContainerStarted","Data":"5ab944474ca5be3bb0733170ecb446ebd8ab1ccbfac2272c3a8d2cbf84656ca5"} Oct 30 02:04:36.691908 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:36.691330 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" event={"ID":"03f91ca9-d4d4-4e24-b535-05f521c0b3b1","Type":"ContainerStarted","Data":"cc5ca1ae192c2901e7170a38fdaf3499dae5ab9f8ebbaae35fe2886f0f229cf4"} Oct 30 02:04:36.738699 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:36.738648 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" podStartSLOduration=1.738632975 podStartE2EDuration="1.738632975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:04:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:04:36.733377524 +0000 UTC m=+2802.309802925" watchObservedRunningTime="2026-10-30 02:04:36.738632975 +0000 UTC m=+2802.315058379" Oct 30 02:04:37.114901 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:37.114812 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:04:37.121629 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:37.121605 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:04:37.570726 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:37.570685 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:37.700650 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:37.700618 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:04:37.701804 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:37.701782 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:04:38.646975 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:38.646930 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg" podUID="a1a20ed5-0fcc-441e-a82d-5402e05472e0" containerName="payload-processing" containerID="cri-o://fe1784f20d1ac368419981a1ca6875fd6e214bba3863faa19ab31ad93b838e2a" gracePeriod=25 Oct 30 02:04:38.935111 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:38.935083 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg" Oct 30 02:04:38.974568 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:38.974531 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1a20ed5-0fcc-441e-a82d-5402e05472e0-kube-api-access-fm47m\" (UniqueName: \"kubernetes.io/projected/a1a20ed5-0fcc-441e-a82d-5402e05472e0-kube-api-access-fm47m\") pod \"a1a20ed5-0fcc-441e-a82d-5402e05472e0\" (UID: \"a1a20ed5-0fcc-441e-a82d-5402e05472e0\") " Oct 30 02:04:38.974798 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:38.974643 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1a20ed5-0fcc-441e-a82d-5402e05472e0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a1a20ed5-0fcc-441e-a82d-5402e05472e0-routing-overlay-volume\") pod \"a1a20ed5-0fcc-441e-a82d-5402e05472e0\" (UID: \"a1a20ed5-0fcc-441e-a82d-5402e05472e0\") " Oct 30 02:04:38.975006 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:38.974988 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1a20ed5-0fcc-441e-a82d-5402e05472e0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a1a20ed5-0fcc-441e-a82d-5402e05472e0-provider-credentials\") pod \"a1a20ed5-0fcc-441e-a82d-5402e05472e0\" (UID: \"a1a20ed5-0fcc-441e-a82d-5402e05472e0\") " Oct 30 02:04:38.975107 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:38.975013 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a20ed5-0fcc-441e-a82d-5402e05472e0-routing-overlay-volume" pod "a1a20ed5-0fcc-441e-a82d-5402e05472e0" (UID: "a1a20ed5-0fcc-441e-a82d-5402e05472e0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:04:38.975107 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:38.975048 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1a20ed5-0fcc-441e-a82d-5402e05472e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a20ed5-0fcc-441e-a82d-5402e05472e0-plugins-config-volume\") pod \"a1a20ed5-0fcc-441e-a82d-5402e05472e0\" (UID: \"a1a20ed5-0fcc-441e-a82d-5402e05472e0\") " Oct 30 02:04:38.976360 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:38.975524 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a20ed5-0fcc-441e-a82d-5402e05472e0-plugins-config-volume" pod "a1a20ed5-0fcc-441e-a82d-5402e05472e0" (UID: "a1a20ed5-0fcc-441e-a82d-5402e05472e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:04:38.976360 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:38.976199 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a20ed5-0fcc-441e-a82d-5402e05472e0-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:04:38.976360 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:38.976226 6572 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a1a20ed5-0fcc-441e-a82d-5402e05472e0-routing-overlay-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:04:38.978273 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:38.978223 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a20ed5-0fcc-441e-a82d-5402e05472e0-provider-credentials" pod "a1a20ed5-0fcc-441e-a82d-5402e05472e0" (UID: "a1a20ed5-0fcc-441e-a82d-5402e05472e0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:38.978273 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:38.978255 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a20ed5-0fcc-441e-a82d-5402e05472e0-kube-api-access-fm47m" pod "a1a20ed5-0fcc-441e-a82d-5402e05472e0" (UID: "a1a20ed5-0fcc-441e-a82d-5402e05472e0"). InnerVolumeSpecName "kube-api-access-fm47m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:39.078088 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:39.078055 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm47m\" (UniqueName: \"kubernetes.io/projected/a1a20ed5-0fcc-441e-a82d-5402e05472e0-kube-api-access-fm47m\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:04:39.078088 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:39.078085 6572 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a1a20ed5-0fcc-441e-a82d-5402e05472e0-provider-credentials\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:04:39.257760 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:39.257714 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:39.490358 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:39.490108 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:39.654815 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:39.654688 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv" podUID="ecba6182-7fb8-4aef-81eb-18851f944bec" containerName="payload-processing" containerID="cri-o://208a112dff4e4073af313179050ff209b7613007098c0b7f5ecdff19491fc6d5" gracePeriod=25 Oct 30 02:04:39.721606 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:39.721571 6572 generic.go:356] "Generic (PLEG): container finished" podID="a1a20ed5-0fcc-441e-a82d-5402e05472e0" containerID="fe1784f20d1ac368419981a1ca6875fd6e214bba3863faa19ab31ad93b838e2a" exitCode=0 Oct 30 02:04:39.721801 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:39.721624 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg" Oct 30 02:04:39.721801 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:39.721659 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg" event={"ID":"a1a20ed5-0fcc-441e-a82d-5402e05472e0","Type":"ContainerDied","Data":"fe1784f20d1ac368419981a1ca6875fd6e214bba3863faa19ab31ad93b838e2a"} Oct 30 02:04:39.721801 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:39.721718 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg" event={"ID":"a1a20ed5-0fcc-441e-a82d-5402e05472e0","Type":"ContainerDied","Data":"ce681d1942522483e576a51f41c6ca3e1caf3fe2434b8a801d30cb79e6756f14"} Oct 30 02:04:39.721801 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:39.721760 6572 scope.go:122] "RemoveContainer" containerID="fe1784f20d1ac368419981a1ca6875fd6e214bba3863faa19ab31ad93b838e2a" Oct 30 02:04:39.770377 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:39.770353 6572 scope.go:122] "RemoveContainer" containerID="fe1784f20d1ac368419981a1ca6875fd6e214bba3863faa19ab31ad93b838e2a" Oct 30 02:04:39.770722 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:04:39.770700 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe1784f20d1ac368419981a1ca6875fd6e214bba3863faa19ab31ad93b838e2a\": container with ID starting with fe1784f20d1ac368419981a1ca6875fd6e214bba3863faa19ab31ad93b838e2a not found: ID does not exist" containerID="fe1784f20d1ac368419981a1ca6875fd6e214bba3863faa19ab31ad93b838e2a" Oct 30 02:04:39.770820 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:39.770736 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe1784f20d1ac368419981a1ca6875fd6e214bba3863faa19ab31ad93b838e2a"} err="failed to get container status \"fe1784f20d1ac368419981a1ca6875fd6e214bba3863faa19ab31ad93b838e2a\": rpc error: code = NotFound desc = could not find container \"fe1784f20d1ac368419981a1ca6875fd6e214bba3863faa19ab31ad93b838e2a\": container with ID starting with fe1784f20d1ac368419981a1ca6875fd6e214bba3863faa19ab31ad93b838e2a not found: ID does not exist" Oct 30 02:04:39.792251 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:39.792216 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg"] Oct 30 02:04:39.804313 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:39.804017 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5wngbg"] Oct 30 02:04:39.946154 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:39.946119 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv" Oct 30 02:04:39.998460 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:39.998427 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecba6182-7fb8-4aef-81eb-18851f944bec-kube-api-access-jhzd7\" (UniqueName: \"kubernetes.io/projected/ecba6182-7fb8-4aef-81eb-18851f944bec-kube-api-access-jhzd7\") pod \"ecba6182-7fb8-4aef-81eb-18851f944bec\" (UID: \"ecba6182-7fb8-4aef-81eb-18851f944bec\") " Oct 30 02:04:39.998660 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:39.998647 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecba6182-7fb8-4aef-81eb-18851f944bec-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ecba6182-7fb8-4aef-81eb-18851f944bec-provider-credentials\") pod \"ecba6182-7fb8-4aef-81eb-18851f944bec\" (UID: \"ecba6182-7fb8-4aef-81eb-18851f944bec\") " Oct 30 02:04:39.998729 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:39.998718 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecba6182-7fb8-4aef-81eb-18851f944bec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecba6182-7fb8-4aef-81eb-18851f944bec-plugins-config-volume\") pod \"ecba6182-7fb8-4aef-81eb-18851f944bec\" (UID: \"ecba6182-7fb8-4aef-81eb-18851f944bec\") " Oct 30 02:04:39.998800 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:39.998776 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecba6182-7fb8-4aef-81eb-18851f944bec-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ecba6182-7fb8-4aef-81eb-18851f944bec-routing-overlay-volume\") pod \"ecba6182-7fb8-4aef-81eb-18851f944bec\" (UID: \"ecba6182-7fb8-4aef-81eb-18851f944bec\") " Oct 30 02:04:39.999210 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:39.999179 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecba6182-7fb8-4aef-81eb-18851f944bec-routing-overlay-volume" pod "ecba6182-7fb8-4aef-81eb-18851f944bec" (UID: "ecba6182-7fb8-4aef-81eb-18851f944bec"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:04:39.999210 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:39.999228 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecba6182-7fb8-4aef-81eb-18851f944bec-plugins-config-volume" pod "ecba6182-7fb8-4aef-81eb-18851f944bec" (UID: "ecba6182-7fb8-4aef-81eb-18851f944bec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:04:39.999676 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:39.999651 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecba6182-7fb8-4aef-81eb-18851f944bec-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:04:39.999823 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:39.999683 6572 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ecba6182-7fb8-4aef-81eb-18851f944bec-routing-overlay-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:04:40.001456 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:40.001425 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecba6182-7fb8-4aef-81eb-18851f944bec-provider-credentials" pod "ecba6182-7fb8-4aef-81eb-18851f944bec" (UID: "ecba6182-7fb8-4aef-81eb-18851f944bec"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:40.001661 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:40.001639 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecba6182-7fb8-4aef-81eb-18851f944bec-kube-api-access-jhzd7" pod "ecba6182-7fb8-4aef-81eb-18851f944bec" (UID: "ecba6182-7fb8-4aef-81eb-18851f944bec"). InnerVolumeSpecName "kube-api-access-jhzd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:40.101353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:40.101311 6572 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ecba6182-7fb8-4aef-81eb-18851f944bec-provider-credentials\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:04:40.101353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:40.101346 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhzd7\" (UniqueName: \"kubernetes.io/projected/ecba6182-7fb8-4aef-81eb-18851f944bec-kube-api-access-jhzd7\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:04:40.544256 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:40.544217 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1a20ed5-0fcc-441e-a82d-5402e05472e0" path="/var/lib/kubelet/pods/a1a20ed5-0fcc-441e-a82d-5402e05472e0/volumes" Oct 30 02:04:40.734807 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:40.734733 6572 generic.go:356] "Generic (PLEG): container finished" podID="ecba6182-7fb8-4aef-81eb-18851f944bec" containerID="208a112dff4e4073af313179050ff209b7613007098c0b7f5ecdff19491fc6d5" exitCode=0 Oct 30 02:04:40.735152 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:40.735131 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv" event={"ID":"ecba6182-7fb8-4aef-81eb-18851f944bec","Type":"ContainerDied","Data":"208a112dff4e4073af313179050ff209b7613007098c0b7f5ecdff19491fc6d5"} Oct 30 02:04:40.735262 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:40.735190 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv" event={"ID":"ecba6182-7fb8-4aef-81eb-18851f944bec","Type":"ContainerDied","Data":"ea5a2107d62a70738a20882a6821b9ec8ba45ebc1a2fcadf3772d7eb75283e4a"} Oct 30 02:04:40.735262 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:40.735221 6572 scope.go:122] "RemoveContainer" containerID="208a112dff4e4073af313179050ff209b7613007098c0b7f5ecdff19491fc6d5" Oct 30 02:04:40.735386 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:40.735373 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv" Oct 30 02:04:40.748975 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:40.748948 6572 scope.go:122] "RemoveContainer" containerID="208a112dff4e4073af313179050ff209b7613007098c0b7f5ecdff19491fc6d5" Oct 30 02:04:40.749371 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:04:40.749342 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"208a112dff4e4073af313179050ff209b7613007098c0b7f5ecdff19491fc6d5\": container with ID starting with 208a112dff4e4073af313179050ff209b7613007098c0b7f5ecdff19491fc6d5 not found: ID does not exist" containerID="208a112dff4e4073af313179050ff209b7613007098c0b7f5ecdff19491fc6d5" Oct 30 02:04:40.749483 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:40.749397 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"208a112dff4e4073af313179050ff209b7613007098c0b7f5ecdff19491fc6d5"} err="failed to get container status \"208a112dff4e4073af313179050ff209b7613007098c0b7f5ecdff19491fc6d5\": rpc error: code = NotFound desc = could not find container \"208a112dff4e4073af313179050ff209b7613007098c0b7f5ecdff19491fc6d5\": container with ID starting with 208a112dff4e4073af313179050ff209b7613007098c0b7f5ecdff19491fc6d5 not found: ID does not exist" Oct 30 02:04:40.792513 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:40.792480 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv"] Oct 30 02:04:40.804691 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:40.804603 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f569llv"] Oct 30 02:04:40.901444 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:40.901405 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:41.170836 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:41.170800 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-717065-656f9c7c8d-mv46m"] Oct 30 02:04:41.172617 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:41.172587 6572 cpu_manager.go:414] "containerMap: removing container" podUID="ecba6182-7fb8-4aef-81eb-18851f944bec" containerName="payload-processing" Oct 30 02:04:41.185822 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:41.185787 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecba6182-7fb8-4aef-81eb-18851f944bec" containerName="payload-processing" Oct 30 02:04:41.186104 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:41.186083 6572 cpu_manager.go:414] "containerMap: removing container" podUID="a1a20ed5-0fcc-441e-a82d-5402e05472e0" containerName="payload-processing" Oct 30 02:04:41.186231 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:41.186216 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1a20ed5-0fcc-441e-a82d-5402e05472e0" containerName="payload-processing" Oct 30 02:04:41.186600 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:41.186578 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecba6182-7fb8-4aef-81eb-18851f944bec" containerName="payload-processing" Oct 30 02:04:41.186741 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:41.186728 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1a20ed5-0fcc-441e-a82d-5402e05472e0" containerName="payload-processing" Oct 30 02:04:41.192894 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:41.192868 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-717065-656f9c7c8d-mv46m" Oct 30 02:04:41.221808 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:41.220134 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-717065-dockercfg-g9sg9\"" Oct 30 02:04:41.221808 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:41.220297 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-717065\"" Oct 30 02:04:41.221808 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:41.221688 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:41.235918 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:41.234311 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdbb5\" (UniqueName: \"kubernetes.io/projected/5e718bf0-89f5-487a-a823-cdd6081e7431-kube-api-access-kdbb5\") pod \"payload-processing-e2e-inf-a-717065-656f9c7c8d-mv46m\" (UID: \"5e718bf0-89f5-487a-a823-cdd6081e7431\") " pod="openshift-ingress/payload-processing-e2e-inf-a-717065-656f9c7c8d-mv46m" Oct 30 02:04:41.235918 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:41.234487 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e718bf0-89f5-487a-a823-cdd6081e7431-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-717065-656f9c7c8d-mv46m\" (UID: \"5e718bf0-89f5-487a-a823-cdd6081e7431\") " pod="openshift-ingress/payload-processing-e2e-inf-a-717065-656f9c7c8d-mv46m" Oct 30 02:04:41.237700 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:41.237665 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-717065-656f9c7c8d-mv46m"] Oct 30 02:04:41.344057 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:41.344020 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e718bf0-89f5-487a-a823-cdd6081e7431-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-717065-656f9c7c8d-mv46m\" (UID: \"5e718bf0-89f5-487a-a823-cdd6081e7431\") " pod="openshift-ingress/payload-processing-e2e-inf-a-717065-656f9c7c8d-mv46m" Oct 30 02:04:41.344354 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:41.344329 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdbb5\" (UniqueName: \"kubernetes.io/projected/5e718bf0-89f5-487a-a823-cdd6081e7431-kube-api-access-kdbb5\") pod \"payload-processing-e2e-inf-a-717065-656f9c7c8d-mv46m\" (UID: \"5e718bf0-89f5-487a-a823-cdd6081e7431\") " pod="openshift-ingress/payload-processing-e2e-inf-a-717065-656f9c7c8d-mv46m" Oct 30 02:04:41.345050 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:41.345020 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e718bf0-89f5-487a-a823-cdd6081e7431-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-717065-656f9c7c8d-mv46m\" (UID: \"5e718bf0-89f5-487a-a823-cdd6081e7431\") " pod="openshift-ingress/payload-processing-e2e-inf-a-717065-656f9c7c8d-mv46m" Oct 30 02:04:41.364607 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:41.364577 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdbb5\" (UniqueName: \"kubernetes.io/projected/5e718bf0-89f5-487a-a823-cdd6081e7431-kube-api-access-kdbb5\") pod \"payload-processing-e2e-inf-a-717065-656f9c7c8d-mv46m\" (UID: \"5e718bf0-89f5-487a-a823-cdd6081e7431\") " pod="openshift-ingress/payload-processing-e2e-inf-a-717065-656f9c7c8d-mv46m" Oct 30 02:04:41.583485 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:41.583381 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-717065-656f9c7c8d-mv46m" Oct 30 02:04:41.905053 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:04:41.905017 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e718bf0_89f5_487a_a823_cdd6081e7431.slice/crio-0d02b579652bb4b105b65af9327e16c0195f0db3c7638edf3f1287de4fd3758c WatchSource:0}: Error finding container 0d02b579652bb4b105b65af9327e16c0195f0db3c7638edf3f1287de4fd3758c: Status 404 returned error can't find the container with id 0d02b579652bb4b105b65af9327e16c0195f0db3c7638edf3f1287de4fd3758c Oct 30 02:04:41.906316 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:41.906291 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-717065-656f9c7c8d-mv46m"] Oct 30 02:04:42.545389 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:42.545351 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecba6182-7fb8-4aef-81eb-18851f944bec" path="/var/lib/kubelet/pods/ecba6182-7fb8-4aef-81eb-18851f944bec/volumes" Oct 30 02:04:42.699317 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:42.698613 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:42.761534 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:42.761491 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-717065-656f9c7c8d-mv46m" event={"ID":"5e718bf0-89f5-487a-a823-cdd6081e7431","Type":"ContainerStarted","Data":"e80013f7a054d07f891b138509c9d9b49e1f9e53ffd696640443f3a8bb1e3164"} Oct 30 02:04:42.761715 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:42.761561 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-717065-656f9c7c8d-mv46m" event={"ID":"5e718bf0-89f5-487a-a823-cdd6081e7431","Type":"ContainerStarted","Data":"0d02b579652bb4b105b65af9327e16c0195f0db3c7638edf3f1287de4fd3758c"} Oct 30 02:04:42.761715 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:42.761591 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-717065-656f9c7c8d-mv46m" Oct 30 02:04:42.800714 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:42.800592 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-717065-656f9c7c8d-mv46m" podStartSLOduration=1.800571666 podStartE2EDuration="1.800571666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:04:42.799724088 +0000 UTC m=+2808.376149490" watchObservedRunningTime="2026-10-30 02:04:42.800571666 +0000 UTC m=+2808.376997069" Oct 30 02:04:44.238249 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:44.238213 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:44.616293 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:04:44.616187 6572 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-284e3d9c18c5: secret "agc-cred-284e3d9c18c5" not found Oct 30 02:04:44.616293 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:04:44.616238 6572 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-69d99678c7-nhwvh: secret "agc-cred-284e3d9c18c5" not found Oct 30 02:04:44.616563 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:04:44.616314 6572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/602ab155-5413-4fe5-9929-9369acca84aa-provider-credentials podName:602ab155-5413-4fe5-9929-9369acca84aa nodeName:}" failed. No retries permitted until 2026-10-30 02:04:45.116291419 +0000 UTC m=+2810.692716813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/602ab155-5413-4fe5-9929-9369acca84aa-provider-credentials") pod "payload-processing-external-model-69d99678c7-nhwvh" (UID: "602ab155-5413-4fe5-9929-9369acca84aa") : secret "agc-cred-284e3d9c18c5" not found Oct 30 02:04:44.688191 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:04:44.688147 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:04:45.125422 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:04:45.125389 6572 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-284e3d9c18c5: secret "agc-cred-284e3d9c18c5" not found Oct 30 02:04:45.125422 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:04:45.125422 6572 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-69d99678c7-nhwvh: secret "agc-cred-284e3d9c18c5" not found Oct 30 02:04:45.125674 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:04:45.125477 6572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/602ab155-5413-4fe5-9929-9369acca84aa-provider-credentials podName:602ab155-5413-4fe5-9929-9369acca84aa nodeName:}" failed. No retries permitted until 2026-10-30 02:04:46.125461745 +0000 UTC m=+2811.701887143 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/602ab155-5413-4fe5-9929-9369acca84aa-provider-credentials") pod "payload-processing-external-model-69d99678c7-nhwvh" (UID: "602ab155-5413-4fe5-9929-9369acca84aa") : secret "agc-cred-284e3d9c18c5" not found Oct 30 02:04:45.685091 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:45.685062 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-56bd9db94-6l572" Oct 30 02:04:45.758709 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:45.758667 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-69d99678c7-nhwvh"] Oct 30 02:04:45.803526 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:45.803486 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:46.141848 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:04:46.141759 6572 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-284e3d9c18c5: secret "agc-cred-284e3d9c18c5" not found Oct 30 02:04:46.141848 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:04:46.141799 6572 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-69d99678c7-nhwvh: secret "agc-cred-284e3d9c18c5" not found Oct 30 02:04:46.142096 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:04:46.141882 6572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/602ab155-5413-4fe5-9929-9369acca84aa-provider-credentials podName:602ab155-5413-4fe5-9929-9369acca84aa nodeName:}" failed. No retries permitted until 2026-10-30 02:04:48.141843232 +0000 UTC m=+2813.718268626 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/602ab155-5413-4fe5-9929-9369acca84aa-provider-credentials") pod "payload-processing-external-model-69d99678c7-nhwvh" (UID: "602ab155-5413-4fe5-9929-9369acca84aa") : secret "agc-cred-284e3d9c18c5" not found Oct 30 02:04:47.389408 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:47.388426 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:48.176580 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:04:48.176538 6572 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-284e3d9c18c5: secret "agc-cred-284e3d9c18c5" not found Oct 30 02:04:48.176580 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:04:48.176576 6572 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-69d99678c7-nhwvh: secret "agc-cred-284e3d9c18c5" not found Oct 30 02:04:48.176914 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:04:48.176642 6572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/602ab155-5413-4fe5-9929-9369acca84aa-provider-credentials podName:602ab155-5413-4fe5-9929-9369acca84aa nodeName:}" failed. No retries permitted until 2026-10-30 02:04:52.176624053 +0000 UTC m=+2817.753049434 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/602ab155-5413-4fe5-9929-9369acca84aa-provider-credentials") pod "payload-processing-external-model-69d99678c7-nhwvh" (UID: "602ab155-5413-4fe5-9929-9369acca84aa") : secret "agc-cred-284e3d9c18c5" not found Oct 30 02:04:48.907774 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:48.907431 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:50.497340 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:50.497304 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:50.759047 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:50.758926 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-69d99678c7-nhwvh" podUID="602ab155-5413-4fe5-9929-9369acca84aa" containerName="payload-processing" containerID="cri-o://d5c10a1c412dd12a9eff0f2082b74e23acf9467a582889b33ce4ff3cee4b8ce3" gracePeriod=25 Oct 30 02:04:51.046837 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:51.046811 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-69d99678c7-nhwvh" Oct 30 02:04:51.129117 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:51.129071 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/602ab155-5413-4fe5-9929-9369acca84aa-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/602ab155-5413-4fe5-9929-9369acca84aa-routing-overlay-volume\") pod \"602ab155-5413-4fe5-9929-9369acca84aa\" (UID: \"602ab155-5413-4fe5-9929-9369acca84aa\") " Oct 30 02:04:51.129336 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:51.129203 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/602ab155-5413-4fe5-9929-9369acca84aa-provider-credentials\" (UniqueName: \"kubernetes.io/projected/602ab155-5413-4fe5-9929-9369acca84aa-provider-credentials\") pod \"602ab155-5413-4fe5-9929-9369acca84aa\" (UID: \"602ab155-5413-4fe5-9929-9369acca84aa\") " Oct 30 02:04:51.129438 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:51.129418 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/602ab155-5413-4fe5-9929-9369acca84aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/602ab155-5413-4fe5-9929-9369acca84aa-plugins-config-volume\") pod \"602ab155-5413-4fe5-9929-9369acca84aa\" (UID: \"602ab155-5413-4fe5-9929-9369acca84aa\") " Oct 30 02:04:51.129572 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:51.129553 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/602ab155-5413-4fe5-9929-9369acca84aa-kube-api-access-dcjz4\" (UniqueName: \"kubernetes.io/projected/602ab155-5413-4fe5-9929-9369acca84aa-kube-api-access-dcjz4\") pod \"602ab155-5413-4fe5-9929-9369acca84aa\" (UID: \"602ab155-5413-4fe5-9929-9369acca84aa\") " Oct 30 02:04:51.131193 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:51.131126 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/602ab155-5413-4fe5-9929-9369acca84aa-routing-overlay-volume" pod "602ab155-5413-4fe5-9929-9369acca84aa" (UID: "602ab155-5413-4fe5-9929-9369acca84aa"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:04:51.131537 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:51.131513 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/602ab155-5413-4fe5-9929-9369acca84aa-plugins-config-volume" pod "602ab155-5413-4fe5-9929-9369acca84aa" (UID: "602ab155-5413-4fe5-9929-9369acca84aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:04:51.133548 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:51.133524 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/602ab155-5413-4fe5-9929-9369acca84aa-provider-credentials" pod "602ab155-5413-4fe5-9929-9369acca84aa" (UID: "602ab155-5413-4fe5-9929-9369acca84aa"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:51.133760 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:51.133721 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/602ab155-5413-4fe5-9929-9369acca84aa-kube-api-access-dcjz4" pod "602ab155-5413-4fe5-9929-9369acca84aa" (UID: "602ab155-5413-4fe5-9929-9369acca84aa"). InnerVolumeSpecName "kube-api-access-dcjz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:51.232195 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:51.232146 6572 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/602ab155-5413-4fe5-9929-9369acca84aa-routing-overlay-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:04:51.232195 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:51.232181 6572 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/602ab155-5413-4fe5-9929-9369acca84aa-provider-credentials\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:04:51.232195 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:51.232191 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/602ab155-5413-4fe5-9929-9369acca84aa-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:04:51.232195 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:51.232200 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcjz4\" (UniqueName: \"kubernetes.io/projected/602ab155-5413-4fe5-9929-9369acca84aa-kube-api-access-dcjz4\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:04:51.854549 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:51.854507 6572 generic.go:356] "Generic (PLEG): container finished" podID="602ab155-5413-4fe5-9929-9369acca84aa" containerID="d5c10a1c412dd12a9eff0f2082b74e23acf9467a582889b33ce4ff3cee4b8ce3" exitCode=0 Oct 30 02:04:51.855089 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:51.854563 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-69d99678c7-nhwvh" Oct 30 02:04:51.855089 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:51.854739 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-69d99678c7-nhwvh" event={"ID":"602ab155-5413-4fe5-9929-9369acca84aa","Type":"ContainerDied","Data":"d5c10a1c412dd12a9eff0f2082b74e23acf9467a582889b33ce4ff3cee4b8ce3"} Oct 30 02:04:51.855089 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:51.854836 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-69d99678c7-nhwvh" event={"ID":"602ab155-5413-4fe5-9929-9369acca84aa","Type":"ContainerDied","Data":"a2a11c510bbae50fcc51675a39449ece8e0a803fa9a8929ec3e12cb3bd38cf0b"} Oct 30 02:04:51.855089 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:51.854873 6572 scope.go:122] "RemoveContainer" containerID="d5c10a1c412dd12a9eff0f2082b74e23acf9467a582889b33ce4ff3cee4b8ce3" Oct 30 02:04:51.868514 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:51.868488 6572 scope.go:122] "RemoveContainer" containerID="d5c10a1c412dd12a9eff0f2082b74e23acf9467a582889b33ce4ff3cee4b8ce3" Oct 30 02:04:51.868873 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:04:51.868848 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5c10a1c412dd12a9eff0f2082b74e23acf9467a582889b33ce4ff3cee4b8ce3\": container with ID starting with d5c10a1c412dd12a9eff0f2082b74e23acf9467a582889b33ce4ff3cee4b8ce3 not found: ID does not exist" containerID="d5c10a1c412dd12a9eff0f2082b74e23acf9467a582889b33ce4ff3cee4b8ce3" Oct 30 02:04:51.869004 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:51.868878 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c10a1c412dd12a9eff0f2082b74e23acf9467a582889b33ce4ff3cee4b8ce3"} err="failed to get container status \"d5c10a1c412dd12a9eff0f2082b74e23acf9467a582889b33ce4ff3cee4b8ce3\": rpc error: code = NotFound desc = could not find container \"d5c10a1c412dd12a9eff0f2082b74e23acf9467a582889b33ce4ff3cee4b8ce3\": container with ID starting with d5c10a1c412dd12a9eff0f2082b74e23acf9467a582889b33ce4ff3cee4b8ce3 not found: ID does not exist" Oct 30 02:04:51.903653 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:04:51.903613 6572 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod602ab155_5413_4fe5_9929_9369acca84aa.slice/crio-a2a11c510bbae50fcc51675a39449ece8e0a803fa9a8929ec3e12cb3bd38cf0b\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod602ab155_5413_4fe5_9929_9369acca84aa.slice\": RecentStats: unable to find data in memory cache]" Oct 30 02:04:51.908676 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:51.908636 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-69d99678c7-nhwvh"] Oct 30 02:04:51.919198 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:51.919164 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-69d99678c7-nhwvh"] Oct 30 02:04:51.994767 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:51.994678 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:52.544959 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:52.544922 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="602ab155-5413-4fe5-9929-9369acca84aa" path="/var/lib/kubelet/pods/602ab155-5413-4fe5-9929-9369acca84aa/volumes" Oct 30 02:04:53.601219 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:53.601184 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:53.773374 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:53.773346 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-717065-656f9c7c8d-mv46m" Oct 30 02:04:55.182773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:55.181894 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:56.691884 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:56.691849 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:58.311881 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:58.311841 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:58.463225 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:58.463195 6572 scope.go:122] "RemoveContainer" containerID="a10dbcef00425120267d35a4ea67600d2ba257465768385c55f8c3bdbacaa3bd" Oct 30 02:04:58.474801 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:58.474779 6572 scope.go:122] "RemoveContainer" containerID="11137f13516f3739c874a1f44790c96e36e8174840bf1b7821c262e6e7955d17" Oct 30 02:04:59.978413 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:04:59.978375 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:00.029482 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:00.029419 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38bc62b8-zrngw"] Oct 30 02:05:00.034254 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:00.034222 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38bc62b8-zrngw"] Oct 30 02:05:00.544983 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:00.544948 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31878162-1468-4639-826b-a725564a1d41" path="/var/lib/kubelet/pods/31878162-1468-4639-826b-a725564a1d41/volumes" Oct 30 02:05:01.573951 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:01.573904 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:03.226684 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:03.226645 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:04.781770 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:04.780514 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:06.388970 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:06.388937 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:07.646458 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:07.646413 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:08.137572 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:08.137531 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:10.100574 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:10.100535 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:11.698708 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:11.698647 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:13.176447 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:13.176408 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:13.372797 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:13.372734 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:14.696918 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:14.696862 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:14.733689 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:05:14.733647 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:05:16.293774 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:16.292853 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:17.883048 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:17.883014 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:19.395613 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:19.395579 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:20.950621 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:20.950580 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:21.154060 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:21.154018 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:21.341081 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:21.340974 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:22.594783 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:22.588300 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:24.041335 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:24.041292 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c12916d7-2cprr"] Oct 30 02:05:24.087153 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:24.069183 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c12916d7-2cprr"] Oct 30 02:05:24.093707 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:24.093662 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:24.546723 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:24.546677 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15e59e08-4ed8-4206-80c7-244b505243ae" path="/var/lib/kubelet/pods/15e59e08-4ed8-4206-80c7-244b505243ae/volumes" Oct 30 02:05:25.035208 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:25.035176 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4560f025-tl8nm"] Oct 30 02:05:25.046019 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:25.045970 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4560f025-tl8nm"] Oct 30 02:05:25.624962 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:25.624901 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:26.544801 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:26.544762 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="250d2557-a543-4749-aa8c-6483de133989" path="/var/lib/kubelet/pods/250d2557-a543-4749-aa8c-6483de133989/volumes" Oct 30 02:05:27.277358 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:27.277319 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:28.842776 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:28.842442 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:30.511213 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:30.511175 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:32.082808 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:32.079295 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:33.653884 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:33.653832 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:35.289387 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:35.281658 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:36.029307 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:36.029275 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c02f572b-f5j2k"] Oct 30 02:05:36.043647 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:36.041944 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c02f572b-f5j2k"] Oct 30 02:05:36.544386 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:36.544351 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7402575-2689-4b54-859f-7f02d662456f" path="/var/lib/kubelet/pods/a7402575-2689-4b54-859f-7f02d662456f/volumes" Oct 30 02:05:36.904049 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:36.903952 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:38.365299 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:38.365266 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:39.532823 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:39.532790 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:39.973276 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:39.973239 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:41.513440 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:41.513392 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:42.999940 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:42.999904 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:44.474771 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:44.474230 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:44.690068 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:05:44.690028 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:05:46.301931 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:46.301890 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:47.890222 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:47.889832 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:48.034633 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:48.034596 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87f0af23-82nbd"] Oct 30 02:05:48.035511 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:48.035484 6572 cpu_manager.go:414] "containerMap: removing container" podUID="602ab155-5413-4fe5-9929-9369acca84aa" containerName="payload-processing" Oct 30 02:05:48.035511 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:48.035511 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="602ab155-5413-4fe5-9929-9369acca84aa" containerName="payload-processing" Oct 30 02:05:48.035808 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:48.035793 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="602ab155-5413-4fe5-9929-9369acca84aa" containerName="payload-processing" Oct 30 02:05:48.040938 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:48.040914 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f0af23-82nbd" Oct 30 02:05:48.047914 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:48.047888 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87f0af23-82nbd"] Oct 30 02:05:48.058094 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:48.058060 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:05:48.068990 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:48.068955 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:05:48.068990 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:48.068969 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:05:48.069186 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:48.069015 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:05:48.159176 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:48.159083 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4159ee6d-021b-4328-8481-e426234b034c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87f0af23-82nbd\" (UID: \"4159ee6d-021b-4328-8481-e426234b034c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f0af23-82nbd" Oct 30 02:05:48.159366 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:48.159270 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4159ee6d-021b-4328-8481-e426234b034c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87f0af23-82nbd\" (UID: \"4159ee6d-021b-4328-8481-e426234b034c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f0af23-82nbd" Oct 30 02:05:48.159551 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:48.159533 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js4lw\" (UniqueName: \"kubernetes.io/projected/4159ee6d-021b-4328-8481-e426234b034c-kube-api-access-js4lw\") pod \"maas-api-revoke-sub-87f0af23-82nbd\" (UID: \"4159ee6d-021b-4328-8481-e426234b034c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f0af23-82nbd" Oct 30 02:05:48.261100 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:48.261063 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4159ee6d-021b-4328-8481-e426234b034c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87f0af23-82nbd\" (UID: \"4159ee6d-021b-4328-8481-e426234b034c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f0af23-82nbd" Oct 30 02:05:48.261311 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:48.261170 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4159ee6d-021b-4328-8481-e426234b034c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87f0af23-82nbd\" (UID: \"4159ee6d-021b-4328-8481-e426234b034c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f0af23-82nbd" Oct 30 02:05:48.261389 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:48.261335 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js4lw\" (UniqueName: \"kubernetes.io/projected/4159ee6d-021b-4328-8481-e426234b034c-kube-api-access-js4lw\") pod \"maas-api-revoke-sub-87f0af23-82nbd\" (UID: \"4159ee6d-021b-4328-8481-e426234b034c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f0af23-82nbd" Oct 30 02:05:48.261778 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:48.261723 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4159ee6d-021b-4328-8481-e426234b034c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87f0af23-82nbd\" (UID: \"4159ee6d-021b-4328-8481-e426234b034c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f0af23-82nbd" Oct 30 02:05:48.263577 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:48.263556 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4159ee6d-021b-4328-8481-e426234b034c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87f0af23-82nbd\" (UID: \"4159ee6d-021b-4328-8481-e426234b034c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f0af23-82nbd" Oct 30 02:05:48.279245 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:48.279214 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js4lw\" (UniqueName: \"kubernetes.io/projected/4159ee6d-021b-4328-8481-e426234b034c-kube-api-access-js4lw\") pod \"maas-api-revoke-sub-87f0af23-82nbd\" (UID: \"4159ee6d-021b-4328-8481-e426234b034c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f0af23-82nbd" Oct 30 02:05:48.359483 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:48.359441 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f0af23-82nbd" Oct 30 02:05:48.694659 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:48.694627 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87f0af23-82nbd"] Oct 30 02:05:48.697216 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:05:48.697150 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4159ee6d_021b_4328_8481_e426234b034c.slice/crio-1c6dfe1705d6268649976f05c390ff0b7f418b6b347321dc193f20a3eccea168 WatchSource:0}: Error finding container 1c6dfe1705d6268649976f05c390ff0b7f418b6b347321dc193f20a3eccea168: Status 404 returned error can't find the container with id 1c6dfe1705d6268649976f05c390ff0b7f418b6b347321dc193f20a3eccea168 Oct 30 02:05:48.753267 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:48.753237 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f0af23-82nbd" event={"ID":"4159ee6d-021b-4328-8481-e426234b034c","Type":"ContainerStarted","Data":"1c6dfe1705d6268649976f05c390ff0b7f418b6b347321dc193f20a3eccea168"} Oct 30 02:05:49.517772 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:49.517545 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:49.764259 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:49.764226 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f0af23-82nbd" event={"ID":"4159ee6d-021b-4328-8481-e426234b034c","Type":"ContainerStarted","Data":"7cb0d34bf6ec8a80d917a26b231ce19f2ead0e374febd3931cc39d9c405f3577"} Oct 30 02:05:49.805372 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:49.805278 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f0af23-82nbd" podStartSLOduration=2.805264158 podStartE2EDuration="2.805264158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:05:49.801052348 +0000 UTC m=+2875.377477751" watchObservedRunningTime="2026-10-30 02:05:49.805264158 +0000 UTC m=+2875.381689559" Oct 30 02:05:50.775368 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:50.775329 6572 generic.go:356] "Generic (PLEG): container finished" podID="4159ee6d-021b-4328-8481-e426234b034c" containerID="7cb0d34bf6ec8a80d917a26b231ce19f2ead0e374febd3931cc39d9c405f3577" exitCode=0 Oct 30 02:05:50.775865 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:50.775407 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f0af23-82nbd" event={"ID":"4159ee6d-021b-4328-8481-e426234b034c","Type":"ContainerDied","Data":"7cb0d34bf6ec8a80d917a26b231ce19f2ead0e374febd3931cc39d9c405f3577"} Oct 30 02:05:51.105399 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:51.105292 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:51.996325 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:51.996302 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f0af23-82nbd" Oct 30 02:05:52.035959 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:52.035802 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4159ee6d-021b-4328-8481-e426234b034c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4159ee6d-021b-4328-8481-e426234b034c-maas-api-openshift-ca\") pod \"4159ee6d-021b-4328-8481-e426234b034c\" (UID: \"4159ee6d-021b-4328-8481-e426234b034c\") " Oct 30 02:05:52.035959 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:52.035884 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4159ee6d-021b-4328-8481-e426234b034c-kube-api-access-js4lw\" (UniqueName: \"kubernetes.io/projected/4159ee6d-021b-4328-8481-e426234b034c-kube-api-access-js4lw\") pod \"4159ee6d-021b-4328-8481-e426234b034c\" (UID: \"4159ee6d-021b-4328-8481-e426234b034c\") " Oct 30 02:05:52.036236 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:52.035982 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4159ee6d-021b-4328-8481-e426234b034c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4159ee6d-021b-4328-8481-e426234b034c-maas-api-xks-ca\") pod \"4159ee6d-021b-4328-8481-e426234b034c\" (UID: \"4159ee6d-021b-4328-8481-e426234b034c\") " Oct 30 02:05:52.037411 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:52.037371 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4159ee6d-021b-4328-8481-e426234b034c-maas-api-openshift-ca" pod "4159ee6d-021b-4328-8481-e426234b034c" (UID: "4159ee6d-021b-4328-8481-e426234b034c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:05:52.040068 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:52.040026 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4159ee6d-021b-4328-8481-e426234b034c-maas-api-xks-ca" pod "4159ee6d-021b-4328-8481-e426234b034c" (UID: "4159ee6d-021b-4328-8481-e426234b034c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:05:52.042140 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:52.042113 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4159ee6d-021b-4328-8481-e426234b034c-kube-api-access-js4lw" pod "4159ee6d-021b-4328-8481-e426234b034c" (UID: "4159ee6d-021b-4328-8481-e426234b034c"). InnerVolumeSpecName "kube-api-access-js4lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:05:52.138439 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:52.138400 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4159ee6d-021b-4328-8481-e426234b034c-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:05:52.138439 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:52.138428 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js4lw\" (UniqueName: \"kubernetes.io/projected/4159ee6d-021b-4328-8481-e426234b034c-kube-api-access-js4lw\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:05:52.138439 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:52.138446 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4159ee6d-021b-4328-8481-e426234b034c-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:05:52.596573 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:52.596527 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:52.794818 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:52.794774 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f0af23-82nbd" Oct 30 02:05:52.794818 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:52.794808 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87f0af23-82nbd" event={"ID":"4159ee6d-021b-4328-8481-e426234b034c","Type":"ContainerDied","Data":"1c6dfe1705d6268649976f05c390ff0b7f418b6b347321dc193f20a3eccea168"} Oct 30 02:05:52.795098 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:52.794869 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c6dfe1705d6268649976f05c390ff0b7f418b6b347321dc193f20a3eccea168" Oct 30 02:05:53.742925 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:53.742888 6572 patch_prober.go:29] interesting pod/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:05:53.743488 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:53.742983 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" podUID="03f91ca9-d4d4-4e24-b535-05f521c0b3b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:05:54.180040 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:54.179859 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:55.851871 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:55.851833 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:56.843060 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:56.843028 6572 generic.go:356] "Generic (PLEG): container finished" podID="03f91ca9-d4d4-4e24-b535-05f521c0b3b1" containerID="5ab944474ca5be3bb0733170ecb446ebd8ab1ccbfac2272c3a8d2cbf84656ca5" exitCode=0 Oct 30 02:05:56.843297 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:56.843100 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" event={"ID":"03f91ca9-d4d4-4e24-b535-05f521c0b3b1","Type":"ContainerDied","Data":"5ab944474ca5be3bb0733170ecb446ebd8ab1ccbfac2272c3a8d2cbf84656ca5"} Oct 30 02:05:56.843617 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:56.843603 6572 scope.go:122] "RemoveContainer" containerID="5ab944474ca5be3bb0733170ecb446ebd8ab1ccbfac2272c3a8d2cbf84656ca5" Oct 30 02:05:56.845564 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:56.845520 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:05:56.845653 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:56.845594 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:05:56.845653 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:56.845629 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:05:57.502725 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:57.502689 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:57.853545 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:57.853456 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" event={"ID":"03f91ca9-d4d4-4e24-b535-05f521c0b3b1","Type":"ContainerStarted","Data":"d493a0f67d59ae56e1fb01dacef256ea7d0b93f23a858a8ede485cf49c1a682d"} Oct 30 02:05:58.037459 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:58.037429 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3f8da4e-x6wx2"] Oct 30 02:05:58.041414 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:58.041390 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3f8da4e-x6wx2"] Oct 30 02:05:58.130902 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:58.126129 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:05:58.130902 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:58.126262 6572 patch_prober.go:29] interesting pod/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" start-of-body= Oct 30 02:05:58.130902 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:58.126324 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" podUID="03f91ca9-d4d4-4e24-b535-05f521c0b3b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" Oct 30 02:05:58.534224 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:58.534194 6572 scope.go:122] "RemoveContainer" containerID="cb423b65b755a1494b02905b7372fab803ac8e37df949e6b7ee7df95ccff9655" Oct 30 02:05:58.546415 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:58.546378 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3bc7205-b501-4898-a3cb-222876418dbd" path="/var/lib/kubelet/pods/f3bc7205-b501-4898-a3cb-222876418dbd/volumes" Oct 30 02:05:58.783923 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:58.783889 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcc8afe2-2l9cz"] Oct 30 02:05:58.784700 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:58.784659 6572 cpu_manager.go:414] "containerMap: removing container" podUID="4159ee6d-021b-4328-8481-e426234b034c" containerName="revoke-keys" Oct 30 02:05:58.784700 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:58.784674 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4159ee6d-021b-4328-8481-e426234b034c" containerName="revoke-keys" Oct 30 02:05:58.784859 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:58.784848 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="4159ee6d-021b-4328-8481-e426234b034c" containerName="revoke-keys" Oct 30 02:05:58.788346 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:58.788329 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc8afe2-2l9cz" Oct 30 02:05:58.817772 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:58.815069 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:05:58.817772 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:58.815204 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:05:58.817772 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:58.815267 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:05:58.817772 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:58.815379 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:05:58.833789 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:58.831185 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcc8afe2-2l9cz"] Oct 30 02:05:58.884626 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:58.884596 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b1b3330-e331-4b2f-b300-13c65a309666-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcc8afe2-2l9cz\" (UID: \"5b1b3330-e331-4b2f-b300-13c65a309666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc8afe2-2l9cz" Oct 30 02:05:58.884799 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:58.884737 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b1b3330-e331-4b2f-b300-13c65a309666-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcc8afe2-2l9cz\" (UID: \"5b1b3330-e331-4b2f-b300-13c65a309666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc8afe2-2l9cz" Oct 30 02:05:58.884944 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:58.884927 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5nq7\" (UniqueName: \"kubernetes.io/projected/5b1b3330-e331-4b2f-b300-13c65a309666-kube-api-access-g5nq7\") pod \"maas-api-revoke-sub-bcc8afe2-2l9cz\" (UID: \"5b1b3330-e331-4b2f-b300-13c65a309666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc8afe2-2l9cz" Oct 30 02:05:58.987874 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:58.987834 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b1b3330-e331-4b2f-b300-13c65a309666-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcc8afe2-2l9cz\" (UID: \"5b1b3330-e331-4b2f-b300-13c65a309666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc8afe2-2l9cz" Oct 30 02:05:58.988142 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:58.988045 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b1b3330-e331-4b2f-b300-13c65a309666-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcc8afe2-2l9cz\" (UID: \"5b1b3330-e331-4b2f-b300-13c65a309666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc8afe2-2l9cz" Oct 30 02:05:58.988270 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:58.988249 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5nq7\" (UniqueName: \"kubernetes.io/projected/5b1b3330-e331-4b2f-b300-13c65a309666-kube-api-access-g5nq7\") pod \"maas-api-revoke-sub-bcc8afe2-2l9cz\" (UID: \"5b1b3330-e331-4b2f-b300-13c65a309666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc8afe2-2l9cz" Oct 30 02:05:58.989354 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:58.989325 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b1b3330-e331-4b2f-b300-13c65a309666-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcc8afe2-2l9cz\" (UID: \"5b1b3330-e331-4b2f-b300-13c65a309666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc8afe2-2l9cz" Oct 30 02:05:58.990084 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:58.990053 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b1b3330-e331-4b2f-b300-13c65a309666-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcc8afe2-2l9cz\" (UID: \"5b1b3330-e331-4b2f-b300-13c65a309666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc8afe2-2l9cz" Oct 30 02:05:59.035079 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:59.034936 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5nq7\" (UniqueName: \"kubernetes.io/projected/5b1b3330-e331-4b2f-b300-13c65a309666-kube-api-access-g5nq7\") pod \"maas-api-revoke-sub-bcc8afe2-2l9cz\" (UID: \"5b1b3330-e331-4b2f-b300-13c65a309666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc8afe2-2l9cz" Oct 30 02:05:59.102373 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:59.102331 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc8afe2-2l9cz" Oct 30 02:05:59.120337 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:59.119877 6572 patch_prober.go:29] interesting pod/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" start-of-body= Oct 30 02:05:59.120337 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:59.119949 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" podUID="03f91ca9-d4d4-4e24-b535-05f521c0b3b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" Oct 30 02:05:59.180779 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:59.172734 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:59.536237 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:05:59.536208 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b1b3330_e331_4b2f_b300_13c65a309666.slice/crio-a57a411380ee6b2d4675f1513475cc038295aa6071dba56953592111170354e1 WatchSource:0}: Error finding container a57a411380ee6b2d4675f1513475cc038295aa6071dba56953592111170354e1: Status 404 returned error can't find the container with id a57a411380ee6b2d4675f1513475cc038295aa6071dba56953592111170354e1 Oct 30 02:05:59.538699 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:59.538663 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcc8afe2-2l9cz"] Oct 30 02:05:59.874033 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:05:59.873944 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc8afe2-2l9cz" event={"ID":"5b1b3330-e331-4b2f-b300-13c65a309666","Type":"ContainerStarted","Data":"a57a411380ee6b2d4675f1513475cc038295aa6071dba56953592111170354e1"} Oct 30 02:06:00.130346 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:00.129846 6572 patch_prober.go:29] interesting pod/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" start-of-body= Oct 30 02:06:00.130346 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:00.129910 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" podUID="03f91ca9-d4d4-4e24-b535-05f521c0b3b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" Oct 30 02:06:00.690687 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:00.690637 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:00.883852 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:00.883817 6572 generic.go:356] "Generic (PLEG): container finished" podID="5b1b3330-e331-4b2f-b300-13c65a309666" containerID="afe86eac8cad4c67362c7cfad93dc58cacab72be0862bce63c888c3fd711dffb" exitCode=0 Oct 30 02:06:00.884041 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:00.883919 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc8afe2-2l9cz" event={"ID":"5b1b3330-e331-4b2f-b300-13c65a309666","Type":"ContainerDied","Data":"afe86eac8cad4c67362c7cfad93dc58cacab72be0862bce63c888c3fd711dffb"} Oct 30 02:06:01.127834 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:01.118095 6572 patch_prober.go:29] interesting pod/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" start-of-body= Oct 30 02:06:01.127834 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:01.118152 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" podUID="03f91ca9-d4d4-4e24-b535-05f521c0b3b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" Oct 30 02:06:02.132204 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:02.131091 6572 patch_prober.go:29] interesting pod/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" start-of-body= Oct 30 02:06:02.132204 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:02.131154 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" podUID="03f91ca9-d4d4-4e24-b535-05f521c0b3b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" Oct 30 02:06:02.138478 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:02.138457 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc8afe2-2l9cz" Oct 30 02:06:02.168548 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:02.168515 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b1b3330-e331-4b2f-b300-13c65a309666-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b1b3330-e331-4b2f-b300-13c65a309666-maas-api-openshift-ca\") pod \"5b1b3330-e331-4b2f-b300-13c65a309666\" (UID: \"5b1b3330-e331-4b2f-b300-13c65a309666\") " Oct 30 02:06:02.168717 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:02.168571 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b1b3330-e331-4b2f-b300-13c65a309666-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b1b3330-e331-4b2f-b300-13c65a309666-maas-api-xks-ca\") pod \"5b1b3330-e331-4b2f-b300-13c65a309666\" (UID: \"5b1b3330-e331-4b2f-b300-13c65a309666\") " Oct 30 02:06:02.168830 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:02.168810 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b1b3330-e331-4b2f-b300-13c65a309666-kube-api-access-g5nq7\" (UniqueName: \"kubernetes.io/projected/5b1b3330-e331-4b2f-b300-13c65a309666-kube-api-access-g5nq7\") pod \"5b1b3330-e331-4b2f-b300-13c65a309666\" (UID: \"5b1b3330-e331-4b2f-b300-13c65a309666\") " Oct 30 02:06:02.168994 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:02.168974 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b1b3330-e331-4b2f-b300-13c65a309666-maas-api-openshift-ca" pod "5b1b3330-e331-4b2f-b300-13c65a309666" (UID: "5b1b3330-e331-4b2f-b300-13c65a309666"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:02.169423 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:02.169402 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b1b3330-e331-4b2f-b300-13c65a309666-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:02.170717 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:02.170690 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b1b3330-e331-4b2f-b300-13c65a309666-maas-api-xks-ca" pod "5b1b3330-e331-4b2f-b300-13c65a309666" (UID: "5b1b3330-e331-4b2f-b300-13c65a309666"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:06:02.170973 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:02.170950 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b1b3330-e331-4b2f-b300-13c65a309666-kube-api-access-g5nq7" pod "5b1b3330-e331-4b2f-b300-13c65a309666" (UID: "5b1b3330-e331-4b2f-b300-13c65a309666"). InnerVolumeSpecName "kube-api-access-g5nq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:02.343771 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:02.310282 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:02.343771 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:02.317197 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b1b3330-e331-4b2f-b300-13c65a309666-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:02.343771 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:02.317230 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5nq7\" (UniqueName: \"kubernetes.io/projected/5b1b3330-e331-4b2f-b300-13c65a309666-kube-api-access-g5nq7\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:02.904993 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:02.904910 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc8afe2-2l9cz" Oct 30 02:06:02.904993 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:02.904938 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc8afe2-2l9cz" event={"ID":"5b1b3330-e331-4b2f-b300-13c65a309666","Type":"ContainerDied","Data":"a57a411380ee6b2d4675f1513475cc038295aa6071dba56953592111170354e1"} Oct 30 02:06:02.904993 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:02.904987 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a57a411380ee6b2d4675f1513475cc038295aa6071dba56953592111170354e1" Oct 30 02:06:03.130927 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:03.130889 6572 patch_prober.go:29] interesting pod/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" start-of-body= Oct 30 02:06:03.131147 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:03.130946 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" podUID="03f91ca9-d4d4-4e24-b535-05f521c0b3b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" Oct 30 02:06:03.898004 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:03.897809 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:04.119906 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:04.119878 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:06:04.120319 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:04.120297 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:06:04.120844 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:04.120819 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:06:05.524028 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:05.523994 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:06.287977 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:06.287937 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:06.828674 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:06.828642 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:07.301652 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.301618 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst"] Oct 30 02:06:07.303063 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.303027 6572 cpu_manager.go:414] "containerMap: removing container" podUID="5b1b3330-e331-4b2f-b300-13c65a309666" containerName="revoke-keys" Oct 30 02:06:07.303289 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.303270 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b1b3330-e331-4b2f-b300-13c65a309666" containerName="revoke-keys" Oct 30 02:06:07.303665 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.303646 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b1b3330-e331-4b2f-b300-13c65a309666" containerName="revoke-keys" Oct 30 02:06:07.332775 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.307922 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:06:07.335677 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.334992 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l4p9\" (UniqueName: \"kubernetes.io/projected/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-kube-api-access-7l4p9\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst\" (UID: \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:06:07.335677 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.335041 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-istio-envoy\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst\" (UID: \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:06:07.335677 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.335079 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-workload-certs\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst\" (UID: \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:06:07.335677 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.335167 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-workload-socket\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst\" (UID: \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:06:07.335677 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.335216 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-istio-data\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst\" (UID: \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:06:07.335677 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.335251 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-istio-token\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst\" (UID: \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:06:07.335677 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.335286 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-istiod-ca-cert\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst\" (UID: \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:06:07.335677 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.335350 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-istio-podinfo\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst\" (UID: \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:06:07.335677 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.335418 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-credential-socket\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst\" (UID: \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:06:07.375770 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.375607 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst"] Oct 30 02:06:07.407178 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.407138 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f81908-openshift-default-dockercfg-8ck82\"" Oct 30 02:06:07.419066 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.418196 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:07.438813 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.437741 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-istio-data\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst\" (UID: \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:06:07.438813 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.437815 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-istio-token\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst\" (UID: \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:06:07.438813 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.437859 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-istiod-ca-cert\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst\" (UID: \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:06:07.438813 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.437927 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-istio-podinfo\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst\" (UID: \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:06:07.438813 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.437994 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-credential-socket\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst\" (UID: \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:06:07.438813 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.438147 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l4p9\" (UniqueName: \"kubernetes.io/projected/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-kube-api-access-7l4p9\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst\" (UID: \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:06:07.438813 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.438183 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-istio-envoy\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst\" (UID: \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:06:07.438813 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.438227 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-workload-certs\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst\" (UID: \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:06:07.438813 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.438332 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-workload-socket\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst\" (UID: \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:06:07.438813 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.438714 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-workload-socket\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst\" (UID: \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:06:07.438813 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.438799 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-istiod-ca-cert\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst\" (UID: \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:06:07.439563 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.438143 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-istio-data\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst\" (UID: \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:06:07.439563 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.439374 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-credential-socket\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst\" (UID: \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:06:07.446250 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.440065 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-workload-certs\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst\" (UID: \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:06:07.562324 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.560504 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l4p9\" (UniqueName: \"kubernetes.io/projected/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-kube-api-access-7l4p9\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst\" (UID: \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:06:07.581890 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.572780 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-istio-token\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst\" (UID: \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:06:07.592696 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.591911 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-istio-envoy\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst\" (UID: \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:06:07.596130 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.596106 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-istio-podinfo\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst\" (UID: \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:06:07.623403 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.623367 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:07.735798 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.735221 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:06:07.833525 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:07.833441 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:08.083388 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:06:08.083361 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee995a27_1255_4c9c_88c4_c6e4eb58a77d.slice/crio-bf711748e1edb4ac0df162a153deffecfe286b5d63cd82f773ec5636fa717f36 WatchSource:0}: Error finding container bf711748e1edb4ac0df162a153deffecfe286b5d63cd82f773ec5636fa717f36: Status 404 returned error can't find the container with id bf711748e1edb4ac0df162a153deffecfe286b5d63cd82f773ec5636fa717f36 Oct 30 02:06:08.083989 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:08.083967 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst"] Oct 30 02:06:08.086470 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:08.086440 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:06:08.086564 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:08.086500 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:06:08.086564 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:08.086532 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:06:09.013780 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:09.013699 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:09.017097 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:09.017056 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" event={"ID":"ee995a27-1255-4c9c-88c4-c6e4eb58a77d","Type":"ContainerStarted","Data":"e12add2d220062db183af8b624107420e07785b347a77a42019f4da8c03eff4c"} Oct 30 02:06:09.017345 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:09.017129 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" event={"ID":"ee995a27-1255-4c9c-88c4-c6e4eb58a77d","Type":"ContainerStarted","Data":"bf711748e1edb4ac0df162a153deffecfe286b5d63cd82f773ec5636fa717f36"} Oct 30 02:06:09.031334 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:09.031300 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5da09025-87cdv"] Oct 30 02:06:09.037113 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:09.037084 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5da09025-87cdv"] Oct 30 02:06:09.073321 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:09.073252 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" podStartSLOduration=2.073233315 podStartE2EDuration="2.073233315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:06:09.067961127 +0000 UTC m=+2894.644386530" watchObservedRunningTime="2026-10-30 02:06:09.073233315 +0000 UTC m=+2894.649658718" Oct 30 02:06:09.736245 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:09.736204 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:06:09.742694 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:09.742662 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:06:10.025315 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:10.025224 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:06:10.026441 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:10.026422 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:06:10.550400 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:10.550363 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c9af176-42be-4616-8164-7d70c34dad4c" path="/var/lib/kubelet/pods/2c9af176-42be-4616-8164-7d70c34dad4c/volumes" Oct 30 02:06:10.709488 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:10.709451 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:10.943865 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:10.943816 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:11.159471 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:11.159442 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:11.294905 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:11.290797 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e7e449c-wg98f"] Oct 30 02:06:11.325773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:11.325000 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e7e449c-wg98f"] Oct 30 02:06:11.325773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:11.325139 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7e449c-wg98f" Oct 30 02:06:11.348837 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:11.343068 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:06:11.360214 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:11.360182 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:06:11.360566 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:11.360545 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:06:11.360867 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:11.360846 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:06:11.400436 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:11.400392 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9058c6dd-6552-4d20-830e-14e0e2c90745-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e7e449c-wg98f\" (UID: \"9058c6dd-6552-4d20-830e-14e0e2c90745\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7e449c-wg98f" Oct 30 02:06:11.404279 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:11.404251 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dm95\" (UniqueName: \"kubernetes.io/projected/9058c6dd-6552-4d20-830e-14e0e2c90745-kube-api-access-6dm95\") pod \"maas-api-revoke-sub-7e7e449c-wg98f\" (UID: \"9058c6dd-6552-4d20-830e-14e0e2c90745\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7e449c-wg98f" Oct 30 02:06:11.404728 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:11.404697 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9058c6dd-6552-4d20-830e-14e0e2c90745-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e7e449c-wg98f\" (UID: \"9058c6dd-6552-4d20-830e-14e0e2c90745\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7e449c-wg98f" Oct 30 02:06:11.507086 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:11.507049 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9058c6dd-6552-4d20-830e-14e0e2c90745-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e7e449c-wg98f\" (UID: \"9058c6dd-6552-4d20-830e-14e0e2c90745\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7e449c-wg98f" Oct 30 02:06:11.507273 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:11.507098 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dm95\" (UniqueName: \"kubernetes.io/projected/9058c6dd-6552-4d20-830e-14e0e2c90745-kube-api-access-6dm95\") pod \"maas-api-revoke-sub-7e7e449c-wg98f\" (UID: \"9058c6dd-6552-4d20-830e-14e0e2c90745\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7e449c-wg98f" Oct 30 02:06:11.507273 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:11.507178 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9058c6dd-6552-4d20-830e-14e0e2c90745-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e7e449c-wg98f\" (UID: \"9058c6dd-6552-4d20-830e-14e0e2c90745\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7e449c-wg98f" Oct 30 02:06:11.507801 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:11.507774 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9058c6dd-6552-4d20-830e-14e0e2c90745-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e7e449c-wg98f\" (UID: \"9058c6dd-6552-4d20-830e-14e0e2c90745\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7e449c-wg98f" Oct 30 02:06:11.509595 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:11.509569 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9058c6dd-6552-4d20-830e-14e0e2c90745-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e7e449c-wg98f\" (UID: \"9058c6dd-6552-4d20-830e-14e0e2c90745\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7e449c-wg98f" Oct 30 02:06:11.533109 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:11.533076 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dm95\" (UniqueName: \"kubernetes.io/projected/9058c6dd-6552-4d20-830e-14e0e2c90745-kube-api-access-6dm95\") pod \"maas-api-revoke-sub-7e7e449c-wg98f\" (UID: \"9058c6dd-6552-4d20-830e-14e0e2c90745\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7e449c-wg98f" Oct 30 02:06:11.661037 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:11.661006 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:11.777369 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:11.777051 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7e449c-wg98f" Oct 30 02:06:12.008521 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:12.008420 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:12.114175 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:12.114136 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv"] Oct 30 02:06:12.122008 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:12.121974 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv" Oct 30 02:06:12.161973 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:12.161934 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv"] Oct 30 02:06:12.282715 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:12.282618 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dpx4\" (UniqueName: \"kubernetes.io/projected/048aae92-605f-4b4b-bfc3-b9f8e87a3a10-kube-api-access-9dpx4\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv\" (UID: \"048aae92-605f-4b4b-bfc3-b9f8e87a3a10\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv" Oct 30 02:06:12.282920 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:12.282775 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/048aae92-605f-4b4b-bfc3-b9f8e87a3a10-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv\" (UID: \"048aae92-605f-4b4b-bfc3-b9f8e87a3a10\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv" Oct 30 02:06:12.282920 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:12.282818 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/048aae92-605f-4b4b-bfc3-b9f8e87a3a10-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv\" (UID: \"048aae92-605f-4b4b-bfc3-b9f8e87a3a10\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv" Oct 30 02:06:12.282920 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:12.282853 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048aae92-605f-4b4b-bfc3-b9f8e87a3a10-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv\" (UID: \"048aae92-605f-4b4b-bfc3-b9f8e87a3a10\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv" Oct 30 02:06:12.291011 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:12.290986 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e7e449c-wg98f"] Oct 30 02:06:12.292583 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:06:12.292555 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9058c6dd_6552_4d20_830e_14e0e2c90745.slice/crio-e5ae444c47daf1b147e81f4bf4baf9f7cf4447187b81badddea0c88ccb1e9438 WatchSource:0}: Error finding container e5ae444c47daf1b147e81f4bf4baf9f7cf4447187b81badddea0c88ccb1e9438: Status 404 returned error can't find the container with id e5ae444c47daf1b147e81f4bf4baf9f7cf4447187b81badddea0c88ccb1e9438 Oct 30 02:06:12.384058 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:12.384023 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dpx4\" (UniqueName: \"kubernetes.io/projected/048aae92-605f-4b4b-bfc3-b9f8e87a3a10-kube-api-access-9dpx4\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv\" (UID: \"048aae92-605f-4b4b-bfc3-b9f8e87a3a10\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv" Oct 30 02:06:12.384260 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:12.384121 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/048aae92-605f-4b4b-bfc3-b9f8e87a3a10-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv\" (UID: \"048aae92-605f-4b4b-bfc3-b9f8e87a3a10\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv" Oct 30 02:06:12.384260 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:12.384144 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/048aae92-605f-4b4b-bfc3-b9f8e87a3a10-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv\" (UID: \"048aae92-605f-4b4b-bfc3-b9f8e87a3a10\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv" Oct 30 02:06:12.384260 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:12.384170 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048aae92-605f-4b4b-bfc3-b9f8e87a3a10-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv\" (UID: \"048aae92-605f-4b4b-bfc3-b9f8e87a3a10\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv" Oct 30 02:06:12.385078 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:12.385037 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/048aae92-605f-4b4b-bfc3-b9f8e87a3a10-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv\" (UID: \"048aae92-605f-4b4b-bfc3-b9f8e87a3a10\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv" Oct 30 02:06:12.385216 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:12.385147 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048aae92-605f-4b4b-bfc3-b9f8e87a3a10-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv\" (UID: \"048aae92-605f-4b4b-bfc3-b9f8e87a3a10\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv" Oct 30 02:06:12.387372 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:12.387331 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/048aae92-605f-4b4b-bfc3-b9f8e87a3a10-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv\" (UID: \"048aae92-605f-4b4b-bfc3-b9f8e87a3a10\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv" Oct 30 02:06:12.404397 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:12.404368 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dpx4\" (UniqueName: \"kubernetes.io/projected/048aae92-605f-4b4b-bfc3-b9f8e87a3a10-kube-api-access-9dpx4\") pod \"payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv\" (UID: \"048aae92-605f-4b4b-bfc3-b9f8e87a3a10\") " pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv" Oct 30 02:06:12.443983 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:12.443951 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv" Oct 30 02:06:12.760870 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:12.760825 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv"] Oct 30 02:06:13.108162 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:13.108051 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv" event={"ID":"048aae92-605f-4b4b-bfc3-b9f8e87a3a10","Type":"ContainerStarted","Data":"bb9ce9c7ce6658bf2bdc6fc8967ef625eb057b17d60351a3db8d0664d682a545"} Oct 30 02:06:13.108162 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:13.108131 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv" event={"ID":"048aae92-605f-4b4b-bfc3-b9f8e87a3a10","Type":"ContainerStarted","Data":"c2548e7cddb7c28985a96cc6cad4ea591bab99ff976c9dfc780183a7d5997f96"} Oct 30 02:06:13.108426 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:13.108166 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv" Oct 30 02:06:13.110266 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:13.110237 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7e449c-wg98f" event={"ID":"9058c6dd-6552-4d20-830e-14e0e2c90745","Type":"ContainerStarted","Data":"38ac7d5ebe3d022137f5d26045b815f12733c60475b3c4d600cbad9ef6dd1614"} Oct 30 02:06:13.110406 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:13.110284 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7e449c-wg98f" event={"ID":"9058c6dd-6552-4d20-830e-14e0e2c90745","Type":"ContainerStarted","Data":"e5ae444c47daf1b147e81f4bf4baf9f7cf4447187b81badddea0c88ccb1e9438"} Oct 30 02:06:13.125040 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:13.125001 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:13.154935 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:13.154879 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv" podStartSLOduration=1.154862534 podStartE2EDuration="1.154862534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:06:13.151933245 +0000 UTC m=+2898.728358648" watchObservedRunningTime="2026-10-30 02:06:13.154862534 +0000 UTC m=+2898.731287939" Oct 30 02:06:13.196217 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:13.196172 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7e449c-wg98f" podStartSLOduration=2.19615898 podStartE2EDuration="2.19615898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:06:13.192015845 +0000 UTC m=+2898.768441246" watchObservedRunningTime="2026-10-30 02:06:13.19615898 +0000 UTC m=+2898.772584381" Oct 30 02:06:13.963247 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:13.963210 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:14.058901 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:14.058865 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f81908-5b744797f9-5bkkh"] Oct 30 02:06:14.063686 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:14.063663 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f81908-5b744797f9-5bkkh" Oct 30 02:06:14.083887 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:14.083859 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f81908-dockercfg-brsc5\"" Oct 30 02:06:14.091011 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:14.090983 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f81908-5b744797f9-5bkkh"] Oct 30 02:06:14.095241 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:14.095207 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f81908\"" Oct 30 02:06:14.163772 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:14.118842 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f81908-5b744797f9-5bkkh\" (UID: \"e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f81908-5b744797f9-5bkkh" Oct 30 02:06:14.163772 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:14.149803 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rshgw\" (UniqueName: \"kubernetes.io/projected/e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497-kube-api-access-rshgw\") pod \"payload-pre-processing-e2e-inf-b-f81908-5b744797f9-5bkkh\" (UID: \"e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f81908-5b744797f9-5bkkh" Oct 30 02:06:14.324341 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:14.304647 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f81908-5b744797f9-5bkkh\" (UID: \"e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f81908-5b744797f9-5bkkh" Oct 30 02:06:14.324341 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:14.305255 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rshgw\" (UniqueName: \"kubernetes.io/projected/e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497-kube-api-access-rshgw\") pod \"payload-pre-processing-e2e-inf-b-f81908-5b744797f9-5bkkh\" (UID: \"e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f81908-5b744797f9-5bkkh" Oct 30 02:06:14.329490 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:14.328833 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f81908-5b744797f9-5bkkh\" (UID: \"e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f81908-5b744797f9-5bkkh" Oct 30 02:06:14.375206 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:14.375165 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:14.380938 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:14.380893 6572 generic.go:356] "Generic (PLEG): container finished" podID="9058c6dd-6552-4d20-830e-14e0e2c90745" containerID="38ac7d5ebe3d022137f5d26045b815f12733c60475b3c4d600cbad9ef6dd1614" exitCode=0 Oct 30 02:06:14.381298 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:14.381155 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7e449c-wg98f" event={"ID":"9058c6dd-6552-4d20-830e-14e0e2c90745","Type":"ContainerDied","Data":"38ac7d5ebe3d022137f5d26045b815f12733c60475b3c4d600cbad9ef6dd1614"} Oct 30 02:06:14.387856 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:14.387797 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rshgw\" (UniqueName: \"kubernetes.io/projected/e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497-kube-api-access-rshgw\") pod \"payload-pre-processing-e2e-inf-b-f81908-5b744797f9-5bkkh\" (UID: \"e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f81908-5b744797f9-5bkkh" Oct 30 02:06:14.678777 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:14.678730 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f81908-5b744797f9-5bkkh" Oct 30 02:06:14.689774 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:06:14.689721 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:06:14.918297 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:14.918261 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:14.987187 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:14.987155 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f81908-5b744797f9-5bkkh"] Oct 30 02:06:14.987508 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:06:14.987484 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode29b33d1_d0ec_4ad3_a2ac_7c27b90c2497.slice/crio-bef634cde777e1c2126892ab251fcb41bdb74ba4685b24f58609e830344916c8 WatchSource:0}: Error finding container bef634cde777e1c2126892ab251fcb41bdb74ba4685b24f58609e830344916c8: Status 404 returned error can't find the container with id bef634cde777e1c2126892ab251fcb41bdb74ba4685b24f58609e830344916c8 Oct 30 02:06:15.396676 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:15.396470 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f81908-5b744797f9-5bkkh" event={"ID":"e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497","Type":"ContainerStarted","Data":"3890e0823fde6ad3557cf92e1e3904f3e32b522c5138a700ea5c432c60d9bf56"} Oct 30 02:06:15.396676 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:15.396529 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f81908-5b744797f9-5bkkh" event={"ID":"e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497","Type":"ContainerStarted","Data":"bef634cde777e1c2126892ab251fcb41bdb74ba4685b24f58609e830344916c8"} Oct 30 02:06:15.396676 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:15.396668 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f81908-5b744797f9-5bkkh" Oct 30 02:06:15.445765 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:15.445687 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f81908-5b744797f9-5bkkh" podStartSLOduration=1.445669431 podStartE2EDuration="1.445669431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:06:15.44327368 +0000 UTC m=+2901.019699076" watchObservedRunningTime="2026-10-30 02:06:15.445669431 +0000 UTC m=+2901.022094835" Oct 30 02:06:15.552996 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:15.552971 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7e449c-wg98f" Oct 30 02:06:15.740572 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:15.740536 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9058c6dd-6552-4d20-830e-14e0e2c90745-kube-api-access-6dm95\" (UniqueName: \"kubernetes.io/projected/9058c6dd-6552-4d20-830e-14e0e2c90745-kube-api-access-6dm95\") pod \"9058c6dd-6552-4d20-830e-14e0e2c90745\" (UID: \"9058c6dd-6552-4d20-830e-14e0e2c90745\") " Oct 30 02:06:15.740572 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:15.740576 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9058c6dd-6552-4d20-830e-14e0e2c90745-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9058c6dd-6552-4d20-830e-14e0e2c90745-maas-api-xks-ca\") pod \"9058c6dd-6552-4d20-830e-14e0e2c90745\" (UID: \"9058c6dd-6552-4d20-830e-14e0e2c90745\") " Oct 30 02:06:15.741003 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:15.740617 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9058c6dd-6552-4d20-830e-14e0e2c90745-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9058c6dd-6552-4d20-830e-14e0e2c90745-maas-api-openshift-ca\") pod \"9058c6dd-6552-4d20-830e-14e0e2c90745\" (UID: \"9058c6dd-6552-4d20-830e-14e0e2c90745\") " Oct 30 02:06:15.741218 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:15.741185 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9058c6dd-6552-4d20-830e-14e0e2c90745-maas-api-openshift-ca" pod "9058c6dd-6552-4d20-830e-14e0e2c90745" (UID: "9058c6dd-6552-4d20-830e-14e0e2c90745"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:15.741568 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:15.741546 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9058c6dd-6552-4d20-830e-14e0e2c90745-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:15.742994 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:15.742966 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9058c6dd-6552-4d20-830e-14e0e2c90745-kube-api-access-6dm95" pod "9058c6dd-6552-4d20-830e-14e0e2c90745" (UID: "9058c6dd-6552-4d20-830e-14e0e2c90745"). InnerVolumeSpecName "kube-api-access-6dm95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:15.743103 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:15.743051 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9058c6dd-6552-4d20-830e-14e0e2c90745-maas-api-xks-ca" pod "9058c6dd-6552-4d20-830e-14e0e2c90745" (UID: "9058c6dd-6552-4d20-830e-14e0e2c90745"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:06:15.843356 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:15.843326 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dm95\" (UniqueName: \"kubernetes.io/projected/9058c6dd-6552-4d20-830e-14e0e2c90745-kube-api-access-6dm95\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:15.843356 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:15.843354 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9058c6dd-6552-4d20-830e-14e0e2c90745-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:16.410568 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:16.410538 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7e449c-wg98f" Oct 30 02:06:16.411140 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:16.410571 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7e449c-wg98f" event={"ID":"9058c6dd-6552-4d20-830e-14e0e2c90745","Type":"ContainerDied","Data":"e5ae444c47daf1b147e81f4bf4baf9f7cf4447187b81badddea0c88ccb1e9438"} Oct 30 02:06:16.411140 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:16.410617 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5ae444c47daf1b147e81f4bf4baf9f7cf4447187b81badddea0c88ccb1e9438" Oct 30 02:06:16.614879 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:16.614838 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:17.529780 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:17.529705 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e64858ed-tljsz"] Oct 30 02:06:17.530574 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:17.530554 6572 cpu_manager.go:414] "containerMap: removing container" podUID="9058c6dd-6552-4d20-830e-14e0e2c90745" containerName="revoke-keys" Oct 30 02:06:17.530574 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:17.530574 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9058c6dd-6552-4d20-830e-14e0e2c90745" containerName="revoke-keys" Oct 30 02:06:17.530723 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:17.530713 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="9058c6dd-6552-4d20-830e-14e0e2c90745" containerName="revoke-keys" Oct 30 02:06:17.534650 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:17.534627 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64858ed-tljsz" Oct 30 02:06:17.554041 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:17.554013 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e64858ed-tljsz"] Oct 30 02:06:17.559940 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:17.559916 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:06:17.560690 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:17.560669 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:06:17.560825 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:17.560676 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:06:17.560825 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:17.560697 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:06:17.673944 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:17.673910 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4de42b0f-0785-47fc-9b4a-7a58b82e2934-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e64858ed-tljsz\" (UID: \"4de42b0f-0785-47fc-9b4a-7a58b82e2934\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64858ed-tljsz" Oct 30 02:06:17.674141 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:17.674086 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccffn\" (UniqueName: \"kubernetes.io/projected/4de42b0f-0785-47fc-9b4a-7a58b82e2934-kube-api-access-ccffn\") pod \"maas-api-revoke-sub-e64858ed-tljsz\" (UID: \"4de42b0f-0785-47fc-9b4a-7a58b82e2934\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64858ed-tljsz" Oct 30 02:06:17.674192 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:17.674131 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4de42b0f-0785-47fc-9b4a-7a58b82e2934-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e64858ed-tljsz\" (UID: \"4de42b0f-0785-47fc-9b4a-7a58b82e2934\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64858ed-tljsz" Oct 30 02:06:17.775842 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:17.775808 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccffn\" (UniqueName: \"kubernetes.io/projected/4de42b0f-0785-47fc-9b4a-7a58b82e2934-kube-api-access-ccffn\") pod \"maas-api-revoke-sub-e64858ed-tljsz\" (UID: \"4de42b0f-0785-47fc-9b4a-7a58b82e2934\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64858ed-tljsz" Oct 30 02:06:17.776015 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:17.775854 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4de42b0f-0785-47fc-9b4a-7a58b82e2934-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e64858ed-tljsz\" (UID: \"4de42b0f-0785-47fc-9b4a-7a58b82e2934\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64858ed-tljsz" Oct 30 02:06:17.776072 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:17.776053 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4de42b0f-0785-47fc-9b4a-7a58b82e2934-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e64858ed-tljsz\" (UID: \"4de42b0f-0785-47fc-9b4a-7a58b82e2934\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64858ed-tljsz" Oct 30 02:06:17.776673 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:17.776643 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4de42b0f-0785-47fc-9b4a-7a58b82e2934-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e64858ed-tljsz\" (UID: \"4de42b0f-0785-47fc-9b4a-7a58b82e2934\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64858ed-tljsz" Oct 30 02:06:17.778353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:17.778321 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4de42b0f-0785-47fc-9b4a-7a58b82e2934-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e64858ed-tljsz\" (UID: \"4de42b0f-0785-47fc-9b4a-7a58b82e2934\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64858ed-tljsz" Oct 30 02:06:17.795198 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:17.795141 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccffn\" (UniqueName: \"kubernetes.io/projected/4de42b0f-0785-47fc-9b4a-7a58b82e2934-kube-api-access-ccffn\") pod \"maas-api-revoke-sub-e64858ed-tljsz\" (UID: \"4de42b0f-0785-47fc-9b4a-7a58b82e2934\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64858ed-tljsz" Oct 30 02:06:17.849421 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:17.849398 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64858ed-tljsz" Oct 30 02:06:18.156294 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:06:18.156267 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4de42b0f_0785_47fc_9b4a_7a58b82e2934.slice/crio-89df44f61b8e8cb9aea6776ed1d940187bd5b2025d651896e9d3e4c1786be8d8 WatchSource:0}: Error finding container 89df44f61b8e8cb9aea6776ed1d940187bd5b2025d651896e9d3e4c1786be8d8: Status 404 returned error can't find the container with id 89df44f61b8e8cb9aea6776ed1d940187bd5b2025d651896e9d3e4c1786be8d8 Oct 30 02:06:18.157825 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:18.157802 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e64858ed-tljsz"] Oct 30 02:06:18.300784 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:18.297724 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:18.430810 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:18.430772 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64858ed-tljsz" event={"ID":"4de42b0f-0785-47fc-9b4a-7a58b82e2934","Type":"ContainerStarted","Data":"89df44f61b8e8cb9aea6776ed1d940187bd5b2025d651896e9d3e4c1786be8d8"} Oct 30 02:06:19.443239 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:19.443198 6572 generic.go:356] "Generic (PLEG): container finished" podID="4de42b0f-0785-47fc-9b4a-7a58b82e2934" containerID="7d2e49c06f2dfeec9f872f00dda6fdd7abf3f8119f537c6a0f7d739ae44284c2" exitCode=0 Oct 30 02:06:19.443736 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:19.443332 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64858ed-tljsz" event={"ID":"4de42b0f-0785-47fc-9b4a-7a58b82e2934","Type":"ContainerDied","Data":"7d2e49c06f2dfeec9f872f00dda6fdd7abf3f8119f537c6a0f7d739ae44284c2"} Oct 30 02:06:19.927285 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:19.927252 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:20.627669 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:20.627642 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64858ed-tljsz" Oct 30 02:06:20.726643 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:20.726562 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4de42b0f-0785-47fc-9b4a-7a58b82e2934-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4de42b0f-0785-47fc-9b4a-7a58b82e2934-maas-api-openshift-ca\") pod \"4de42b0f-0785-47fc-9b4a-7a58b82e2934\" (UID: \"4de42b0f-0785-47fc-9b4a-7a58b82e2934\") " Oct 30 02:06:20.726643 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:20.726641 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4de42b0f-0785-47fc-9b4a-7a58b82e2934-kube-api-access-ccffn\" (UniqueName: \"kubernetes.io/projected/4de42b0f-0785-47fc-9b4a-7a58b82e2934-kube-api-access-ccffn\") pod \"4de42b0f-0785-47fc-9b4a-7a58b82e2934\" (UID: \"4de42b0f-0785-47fc-9b4a-7a58b82e2934\") " Oct 30 02:06:20.727001 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:20.726975 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4de42b0f-0785-47fc-9b4a-7a58b82e2934-maas-api-openshift-ca" pod "4de42b0f-0785-47fc-9b4a-7a58b82e2934" (UID: "4de42b0f-0785-47fc-9b4a-7a58b82e2934"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:20.727085 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:20.727022 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4de42b0f-0785-47fc-9b4a-7a58b82e2934-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4de42b0f-0785-47fc-9b4a-7a58b82e2934-maas-api-xks-ca\") pod \"4de42b0f-0785-47fc-9b4a-7a58b82e2934\" (UID: \"4de42b0f-0785-47fc-9b4a-7a58b82e2934\") " Oct 30 02:06:20.727634 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:20.727613 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4de42b0f-0785-47fc-9b4a-7a58b82e2934-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:20.729003 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:20.728978 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4de42b0f-0785-47fc-9b4a-7a58b82e2934-maas-api-xks-ca" pod "4de42b0f-0785-47fc-9b4a-7a58b82e2934" (UID: "4de42b0f-0785-47fc-9b4a-7a58b82e2934"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:06:20.729115 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:20.729021 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4de42b0f-0785-47fc-9b4a-7a58b82e2934-kube-api-access-ccffn" pod "4de42b0f-0785-47fc-9b4a-7a58b82e2934" (UID: "4de42b0f-0785-47fc-9b4a-7a58b82e2934"). InnerVolumeSpecName "kube-api-access-ccffn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:20.829254 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:20.829224 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4de42b0f-0785-47fc-9b4a-7a58b82e2934-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:20.829254 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:20.829250 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccffn\" (UniqueName: \"kubernetes.io/projected/4de42b0f-0785-47fc-9b4a-7a58b82e2934-kube-api-access-ccffn\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:21.031182 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:21.031090 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-496830df-mj9ls"] Oct 30 02:06:21.035286 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:21.035258 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-496830df-mj9ls"] Oct 30 02:06:21.468716 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:21.468682 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64858ed-tljsz" Oct 30 02:06:21.468963 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:21.468677 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64858ed-tljsz" event={"ID":"4de42b0f-0785-47fc-9b4a-7a58b82e2934","Type":"ContainerDied","Data":"89df44f61b8e8cb9aea6776ed1d940187bd5b2025d651896e9d3e4c1786be8d8"} Oct 30 02:06:21.468963 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:21.468906 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89df44f61b8e8cb9aea6776ed1d940187bd5b2025d651896e9d3e4c1786be8d8" Oct 30 02:06:21.597801 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:21.597765 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:22.050600 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:22.050562 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2560735-2cn2b"] Oct 30 02:06:22.051914 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:22.051893 6572 cpu_manager.go:414] "containerMap: removing container" podUID="4de42b0f-0785-47fc-9b4a-7a58b82e2934" containerName="revoke-keys" Oct 30 02:06:22.051986 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:22.051917 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4de42b0f-0785-47fc-9b4a-7a58b82e2934" containerName="revoke-keys" Oct 30 02:06:22.052211 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:22.052196 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="4de42b0f-0785-47fc-9b4a-7a58b82e2934" containerName="revoke-keys" Oct 30 02:06:22.056786 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:22.056732 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2560735-2cn2b" Oct 30 02:06:22.071417 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:22.071384 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:06:22.074674 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:22.074652 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2560735-2cn2b"] Oct 30 02:06:22.078513 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:22.078488 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:06:22.078647 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:22.078550 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:06:22.082844 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:22.082812 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:06:22.201859 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:22.201820 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8d84a7-9256-4836-b428-26e184dd11f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2560735-2cn2b\" (UID: \"2c8d84a7-9256-4836-b428-26e184dd11f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2560735-2cn2b" Oct 30 02:06:22.202079 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:22.202058 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8d84a7-9256-4836-b428-26e184dd11f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2560735-2cn2b\" (UID: \"2c8d84a7-9256-4836-b428-26e184dd11f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2560735-2cn2b" Oct 30 02:06:22.202169 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:22.202147 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv25z\" (UniqueName: \"kubernetes.io/projected/2c8d84a7-9256-4836-b428-26e184dd11f7-kube-api-access-qv25z\") pod \"maas-api-revoke-sub-b2560735-2cn2b\" (UID: \"2c8d84a7-9256-4836-b428-26e184dd11f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2560735-2cn2b" Oct 30 02:06:22.304473 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:22.304380 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8d84a7-9256-4836-b428-26e184dd11f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2560735-2cn2b\" (UID: \"2c8d84a7-9256-4836-b428-26e184dd11f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2560735-2cn2b" Oct 30 02:06:22.304662 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:22.304562 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8d84a7-9256-4836-b428-26e184dd11f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2560735-2cn2b\" (UID: \"2c8d84a7-9256-4836-b428-26e184dd11f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2560735-2cn2b" Oct 30 02:06:22.304662 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:22.304608 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv25z\" (UniqueName: \"kubernetes.io/projected/2c8d84a7-9256-4836-b428-26e184dd11f7-kube-api-access-qv25z\") pod \"maas-api-revoke-sub-b2560735-2cn2b\" (UID: \"2c8d84a7-9256-4836-b428-26e184dd11f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2560735-2cn2b" Oct 30 02:06:22.305418 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:22.305386 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8d84a7-9256-4836-b428-26e184dd11f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2560735-2cn2b\" (UID: \"2c8d84a7-9256-4836-b428-26e184dd11f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2560735-2cn2b" Oct 30 02:06:22.307087 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:22.307057 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8d84a7-9256-4836-b428-26e184dd11f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2560735-2cn2b\" (UID: \"2c8d84a7-9256-4836-b428-26e184dd11f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2560735-2cn2b" Oct 30 02:06:22.325609 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:22.325579 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv25z\" (UniqueName: \"kubernetes.io/projected/2c8d84a7-9256-4836-b428-26e184dd11f7-kube-api-access-qv25z\") pod \"maas-api-revoke-sub-b2560735-2cn2b\" (UID: \"2c8d84a7-9256-4836-b428-26e184dd11f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2560735-2cn2b" Oct 30 02:06:22.381778 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:22.378728 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2560735-2cn2b" Oct 30 02:06:22.551084 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:22.551042 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64c62270-5436-4905-a965-ee29d002850a" path="/var/lib/kubelet/pods/64c62270-5436-4905-a965-ee29d002850a/volumes" Oct 30 02:06:22.699435 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:06:22.699397 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c8d84a7_9256_4836_b428_26e184dd11f7.slice/crio-490ac8236569a65d9baf7ef19b12dc2128fed2b23c38e0e7caa90bc0341c6c81 WatchSource:0}: Error finding container 490ac8236569a65d9baf7ef19b12dc2128fed2b23c38e0e7caa90bc0341c6c81: Status 404 returned error can't find the container with id 490ac8236569a65d9baf7ef19b12dc2128fed2b23c38e0e7caa90bc0341c6c81 Oct 30 02:06:22.699594 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:22.699498 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2560735-2cn2b"] Oct 30 02:06:23.211343 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:23.211297 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45"] Oct 30 02:06:23.240692 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:23.240663 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:23.502820 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:23.502778 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2560735-2cn2b" event={"ID":"2c8d84a7-9256-4836-b428-26e184dd11f7","Type":"ContainerStarted","Data":"770cd5282723848a36b727228ceb39cd00b3d9794ecb693c87fb22f138bfa1c3"} Oct 30 02:06:23.502990 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:23.502845 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2560735-2cn2b" event={"ID":"2c8d84a7-9256-4836-b428-26e184dd11f7","Type":"ContainerStarted","Data":"490ac8236569a65d9baf7ef19b12dc2128fed2b23c38e0e7caa90bc0341c6c81"} Oct 30 02:06:23.543521 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:23.543447 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2560735-2cn2b" podStartSLOduration=1.543431312 podStartE2EDuration="1.543431312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:06:23.542779205 +0000 UTC m=+2909.119204604" watchObservedRunningTime="2026-10-30 02:06:23.543431312 +0000 UTC m=+2909.119856714" Oct 30 02:06:24.384462 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:24.384429 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv" Oct 30 02:06:24.461054 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:24.461018 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4"] Oct 30 02:06:24.514714 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:24.514678 6572 generic.go:356] "Generic (PLEG): container finished" podID="2c8d84a7-9256-4836-b428-26e184dd11f7" containerID="770cd5282723848a36b727228ceb39cd00b3d9794ecb693c87fb22f138bfa1c3" exitCode=0 Oct 30 02:06:24.514969 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:24.514790 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2560735-2cn2b" event={"ID":"2c8d84a7-9256-4836-b428-26e184dd11f7","Type":"ContainerDied","Data":"770cd5282723848a36b727228ceb39cd00b3d9794ecb693c87fb22f138bfa1c3"} Oct 30 02:06:24.901720 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:24.901674 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:25.681857 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:25.681829 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2560735-2cn2b" Oct 30 02:06:25.708045 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:25.708015 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c8d84a7-9256-4836-b428-26e184dd11f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8d84a7-9256-4836-b428-26e184dd11f7-maas-api-openshift-ca\") pod \"2c8d84a7-9256-4836-b428-26e184dd11f7\" (UID: \"2c8d84a7-9256-4836-b428-26e184dd11f7\") " Oct 30 02:06:25.708260 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:25.708245 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c8d84a7-9256-4836-b428-26e184dd11f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8d84a7-9256-4836-b428-26e184dd11f7-maas-api-xks-ca\") pod \"2c8d84a7-9256-4836-b428-26e184dd11f7\" (UID: \"2c8d84a7-9256-4836-b428-26e184dd11f7\") " Oct 30 02:06:25.708322 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:25.708301 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c8d84a7-9256-4836-b428-26e184dd11f7-kube-api-access-qv25z\" (UniqueName: \"kubernetes.io/projected/2c8d84a7-9256-4836-b428-26e184dd11f7-kube-api-access-qv25z\") pod \"2c8d84a7-9256-4836-b428-26e184dd11f7\" (UID: \"2c8d84a7-9256-4836-b428-26e184dd11f7\") " Oct 30 02:06:25.708410 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:25.708384 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8d84a7-9256-4836-b428-26e184dd11f7-maas-api-openshift-ca" pod "2c8d84a7-9256-4836-b428-26e184dd11f7" (UID: "2c8d84a7-9256-4836-b428-26e184dd11f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:25.708909 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:25.708888 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8d84a7-9256-4836-b428-26e184dd11f7-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:25.710397 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:25.710372 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c8d84a7-9256-4836-b428-26e184dd11f7-maas-api-xks-ca" pod "2c8d84a7-9256-4836-b428-26e184dd11f7" (UID: "2c8d84a7-9256-4836-b428-26e184dd11f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:06:25.710489 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:25.710465 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8d84a7-9256-4836-b428-26e184dd11f7-kube-api-access-qv25z" pod "2c8d84a7-9256-4836-b428-26e184dd11f7" (UID: "2c8d84a7-9256-4836-b428-26e184dd11f7"). InnerVolumeSpecName "kube-api-access-qv25z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:25.811089 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:25.810999 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8d84a7-9256-4836-b428-26e184dd11f7-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:25.811089 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:25.811039 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv25z\" (UniqueName: \"kubernetes.io/projected/2c8d84a7-9256-4836-b428-26e184dd11f7-kube-api-access-qv25z\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:26.414243 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:26.414210 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f81908-5b744797f9-5bkkh" Oct 30 02:06:26.511573 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:26.507816 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:26.548924 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:26.548877 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2560735-2cn2b" Oct 30 02:06:26.556168 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:26.556114 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2560735-2cn2b" event={"ID":"2c8d84a7-9256-4836-b428-26e184dd11f7","Type":"ContainerDied","Data":"490ac8236569a65d9baf7ef19b12dc2128fed2b23c38e0e7caa90bc0341c6c81"} Oct 30 02:06:26.556363 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:26.556215 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="490ac8236569a65d9baf7ef19b12dc2128fed2b23c38e0e7caa90bc0341c6c81" Oct 30 02:06:28.085955 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:28.085914 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:28.212158 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:28.212115 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45" podUID="285a2ade-2eaa-4b72-8a06-26e9a9395442" containerName="payload-processing" containerID="cri-o://4237b59d59945979fec4facd4138e66bf3408540e9698d4d97318be7f9f1b702" gracePeriod=25 Oct 30 02:06:28.494262 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:28.494235 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45" Oct 30 02:06:28.562653 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:28.562599 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/285a2ade-2eaa-4b72-8a06-26e9a9395442-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/285a2ade-2eaa-4b72-8a06-26e9a9395442-routing-overlay-volume\") pod \"285a2ade-2eaa-4b72-8a06-26e9a9395442\" (UID: \"285a2ade-2eaa-4b72-8a06-26e9a9395442\") " Oct 30 02:06:28.562868 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:28.562682 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/285a2ade-2eaa-4b72-8a06-26e9a9395442-provider-credentials\" (UniqueName: \"kubernetes.io/projected/285a2ade-2eaa-4b72-8a06-26e9a9395442-provider-credentials\") pod \"285a2ade-2eaa-4b72-8a06-26e9a9395442\" (UID: \"285a2ade-2eaa-4b72-8a06-26e9a9395442\") " Oct 30 02:06:28.562967 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:28.562868 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/285a2ade-2eaa-4b72-8a06-26e9a9395442-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/285a2ade-2eaa-4b72-8a06-26e9a9395442-plugins-config-volume\") pod \"285a2ade-2eaa-4b72-8a06-26e9a9395442\" (UID: \"285a2ade-2eaa-4b72-8a06-26e9a9395442\") " Oct 30 02:06:28.562967 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:28.562908 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/285a2ade-2eaa-4b72-8a06-26e9a9395442-kube-api-access-7q4q8\" (UniqueName: \"kubernetes.io/projected/285a2ade-2eaa-4b72-8a06-26e9a9395442-kube-api-access-7q4q8\") pod \"285a2ade-2eaa-4b72-8a06-26e9a9395442\" (UID: \"285a2ade-2eaa-4b72-8a06-26e9a9395442\") " Oct 30 02:06:28.565178 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:28.565144 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/285a2ade-2eaa-4b72-8a06-26e9a9395442-plugins-config-volume" pod "285a2ade-2eaa-4b72-8a06-26e9a9395442" (UID: "285a2ade-2eaa-4b72-8a06-26e9a9395442"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:28.565328 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:28.565221 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/285a2ade-2eaa-4b72-8a06-26e9a9395442-routing-overlay-volume" pod "285a2ade-2eaa-4b72-8a06-26e9a9395442" (UID: "285a2ade-2eaa-4b72-8a06-26e9a9395442"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:28.568388 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:28.568350 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/285a2ade-2eaa-4b72-8a06-26e9a9395442-kube-api-access-7q4q8" pod "285a2ade-2eaa-4b72-8a06-26e9a9395442" (UID: "285a2ade-2eaa-4b72-8a06-26e9a9395442"). InnerVolumeSpecName "kube-api-access-7q4q8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:28.571354 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:28.571323 6572 generic.go:356] "Generic (PLEG): container finished" podID="285a2ade-2eaa-4b72-8a06-26e9a9395442" containerID="4237b59d59945979fec4facd4138e66bf3408540e9698d4d97318be7f9f1b702" exitCode=0 Oct 30 02:06:28.571595 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:28.571569 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45" event={"ID":"285a2ade-2eaa-4b72-8a06-26e9a9395442","Type":"ContainerDied","Data":"4237b59d59945979fec4facd4138e66bf3408540e9698d4d97318be7f9f1b702"} Oct 30 02:06:28.571702 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:28.571632 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45" event={"ID":"285a2ade-2eaa-4b72-8a06-26e9a9395442","Type":"ContainerDied","Data":"2ebce2c60fd0e636d72b97dbdbe05d5d50284723a3484a311f72a6caf15247de"} Oct 30 02:06:28.571702 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:28.571665 6572 scope.go:122] "RemoveContainer" containerID="4237b59d59945979fec4facd4138e66bf3408540e9698d4d97318be7f9f1b702" Oct 30 02:06:28.571919 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:28.571899 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45" Oct 30 02:06:28.574452 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:28.574430 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/285a2ade-2eaa-4b72-8a06-26e9a9395442-provider-credentials" pod "285a2ade-2eaa-4b72-8a06-26e9a9395442" (UID: "285a2ade-2eaa-4b72-8a06-26e9a9395442"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:28.607553 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:28.607528 6572 scope.go:122] "RemoveContainer" containerID="4237b59d59945979fec4facd4138e66bf3408540e9698d4d97318be7f9f1b702" Oct 30 02:06:28.607960 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:06:28.607932 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4237b59d59945979fec4facd4138e66bf3408540e9698d4d97318be7f9f1b702\": container with ID starting with 4237b59d59945979fec4facd4138e66bf3408540e9698d4d97318be7f9f1b702 not found: ID does not exist" containerID="4237b59d59945979fec4facd4138e66bf3408540e9698d4d97318be7f9f1b702" Oct 30 02:06:28.608015 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:28.607972 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4237b59d59945979fec4facd4138e66bf3408540e9698d4d97318be7f9f1b702"} err="failed to get container status \"4237b59d59945979fec4facd4138e66bf3408540e9698d4d97318be7f9f1b702\": rpc error: code = NotFound desc = could not find container \"4237b59d59945979fec4facd4138e66bf3408540e9698d4d97318be7f9f1b702\": container with ID starting with 4237b59d59945979fec4facd4138e66bf3408540e9698d4d97318be7f9f1b702 not found: ID does not exist" Oct 30 02:06:28.704813 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:28.694089 6572 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/285a2ade-2eaa-4b72-8a06-26e9a9395442-routing-overlay-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:28.704813 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:28.694127 6572 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/285a2ade-2eaa-4b72-8a06-26e9a9395442-provider-credentials\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:28.704813 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:28.694143 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/285a2ade-2eaa-4b72-8a06-26e9a9395442-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:28.704813 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:28.694159 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q4q8\" (UniqueName: \"kubernetes.io/projected/285a2ade-2eaa-4b72-8a06-26e9a9395442-kube-api-access-7q4q8\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:28.717578 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:28.717490 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:28.927831 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:28.927800 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45"] Oct 30 02:06:28.939009 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:28.938977 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-57cdbwv45"] Oct 30 02:06:28.939775 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:28.939733 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:28.948598 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:28.948574 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-666655759c-nj68p"] Oct 30 02:06:28.949386 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:28.949368 6572 cpu_manager.go:414] "containerMap: removing container" podUID="285a2ade-2eaa-4b72-8a06-26e9a9395442" containerName="payload-processing" Oct 30 02:06:28.949446 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:28.949387 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="285a2ade-2eaa-4b72-8a06-26e9a9395442" containerName="payload-processing" Oct 30 02:06:28.949446 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:28.949407 6572 cpu_manager.go:414] "containerMap: removing container" podUID="2c8d84a7-9256-4836-b428-26e184dd11f7" containerName="revoke-keys" Oct 30 02:06:28.949446 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:28.949412 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c8d84a7-9256-4836-b428-26e184dd11f7" containerName="revoke-keys" Oct 30 02:06:28.949568 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:28.949545 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="285a2ade-2eaa-4b72-8a06-26e9a9395442" containerName="payload-processing" Oct 30 02:06:28.949568 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:28.949558 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c8d84a7-9256-4836-b428-26e184dd11f7" containerName="revoke-keys" Oct 30 02:06:28.955167 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:28.955118 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-666655759c-nj68p" Oct 30 02:06:28.974005 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:28.973977 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-666655759c-nj68p"] Oct 30 02:06:29.100390 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:29.100358 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5rts\" (UniqueName: \"kubernetes.io/projected/19de5ac4-66b5-4440-bb74-b945358bbc21-kube-api-access-l5rts\") pod \"payload-processing-external-model-666655759c-nj68p\" (UID: \"19de5ac4-66b5-4440-bb74-b945358bbc21\") " pod="models-as-a-service/payload-processing-external-model-666655759c-nj68p" Oct 30 02:06:29.100868 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:29.100424 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19de5ac4-66b5-4440-bb74-b945358bbc21-provider-credentials\") pod \"payload-processing-external-model-666655759c-nj68p\" (UID: \"19de5ac4-66b5-4440-bb74-b945358bbc21\") " pod="models-as-a-service/payload-processing-external-model-666655759c-nj68p" Oct 30 02:06:29.100868 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:29.100581 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19de5ac4-66b5-4440-bb74-b945358bbc21-routing-overlay-volume\") pod \"payload-processing-external-model-666655759c-nj68p\" (UID: \"19de5ac4-66b5-4440-bb74-b945358bbc21\") " pod="models-as-a-service/payload-processing-external-model-666655759c-nj68p" Oct 30 02:06:29.100868 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:29.100636 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19de5ac4-66b5-4440-bb74-b945358bbc21-plugins-config-volume\") pod \"payload-processing-external-model-666655759c-nj68p\" (UID: \"19de5ac4-66b5-4440-bb74-b945358bbc21\") " pod="models-as-a-service/payload-processing-external-model-666655759c-nj68p" Oct 30 02:06:29.201986 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:29.201957 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5rts\" (UniqueName: \"kubernetes.io/projected/19de5ac4-66b5-4440-bb74-b945358bbc21-kube-api-access-l5rts\") pod \"payload-processing-external-model-666655759c-nj68p\" (UID: \"19de5ac4-66b5-4440-bb74-b945358bbc21\") " pod="models-as-a-service/payload-processing-external-model-666655759c-nj68p" Oct 30 02:06:29.202161 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:29.202009 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19de5ac4-66b5-4440-bb74-b945358bbc21-provider-credentials\") pod \"payload-processing-external-model-666655759c-nj68p\" (UID: \"19de5ac4-66b5-4440-bb74-b945358bbc21\") " pod="models-as-a-service/payload-processing-external-model-666655759c-nj68p" Oct 30 02:06:29.202161 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:29.202117 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19de5ac4-66b5-4440-bb74-b945358bbc21-routing-overlay-volume\") pod \"payload-processing-external-model-666655759c-nj68p\" (UID: \"19de5ac4-66b5-4440-bb74-b945358bbc21\") " pod="models-as-a-service/payload-processing-external-model-666655759c-nj68p" Oct 30 02:06:29.202305 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:29.202162 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19de5ac4-66b5-4440-bb74-b945358bbc21-plugins-config-volume\") pod \"payload-processing-external-model-666655759c-nj68p\" (UID: \"19de5ac4-66b5-4440-bb74-b945358bbc21\") " pod="models-as-a-service/payload-processing-external-model-666655759c-nj68p" Oct 30 02:06:29.202864 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:29.202838 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19de5ac4-66b5-4440-bb74-b945358bbc21-routing-overlay-volume\") pod \"payload-processing-external-model-666655759c-nj68p\" (UID: \"19de5ac4-66b5-4440-bb74-b945358bbc21\") " pod="models-as-a-service/payload-processing-external-model-666655759c-nj68p" Oct 30 02:06:29.202976 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:29.202925 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19de5ac4-66b5-4440-bb74-b945358bbc21-plugins-config-volume\") pod \"payload-processing-external-model-666655759c-nj68p\" (UID: \"19de5ac4-66b5-4440-bb74-b945358bbc21\") " pod="models-as-a-service/payload-processing-external-model-666655759c-nj68p" Oct 30 02:06:29.204600 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:29.204581 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19de5ac4-66b5-4440-bb74-b945358bbc21-provider-credentials\") pod \"payload-processing-external-model-666655759c-nj68p\" (UID: \"19de5ac4-66b5-4440-bb74-b945358bbc21\") " pod="models-as-a-service/payload-processing-external-model-666655759c-nj68p" Oct 30 02:06:29.221536 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:29.221465 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5rts\" (UniqueName: \"kubernetes.io/projected/19de5ac4-66b5-4440-bb74-b945358bbc21-kube-api-access-l5rts\") pod \"payload-processing-external-model-666655759c-nj68p\" (UID: \"19de5ac4-66b5-4440-bb74-b945358bbc21\") " pod="models-as-a-service/payload-processing-external-model-666655759c-nj68p" Oct 30 02:06:29.274390 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:29.274349 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-666655759c-nj68p" Oct 30 02:06:29.462917 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:29.462829 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4" podUID="ca478104-e26b-4bd8-8e86-684c2807e4aa" containerName="payload-processing" containerID="cri-o://f8fa7aa42375178d7395200f1506c25ee03f10da39cbe8e87647edefa51750d9" gracePeriod=25 Oct 30 02:06:29.572921 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:06:29.572882 6572 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca478104_e26b_4bd8_8e86_684c2807e4aa.slice/crio-f8fa7aa42375178d7395200f1506c25ee03f10da39cbe8e87647edefa51750d9.scope\": RecentStats: unable to find data in memory cache]" Oct 30 02:06:29.601489 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:29.601456 6572 generic.go:356] "Generic (PLEG): container finished" podID="ca478104-e26b-4bd8-8e86-684c2807e4aa" containerID="f8fa7aa42375178d7395200f1506c25ee03f10da39cbe8e87647edefa51750d9" exitCode=0 Oct 30 02:06:29.601664 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:29.601648 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4" event={"ID":"ca478104-e26b-4bd8-8e86-684c2807e4aa","Type":"ContainerDied","Data":"f8fa7aa42375178d7395200f1506c25ee03f10da39cbe8e87647edefa51750d9"} Oct 30 02:06:29.722849 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:06:29.722819 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19de5ac4_66b5_4440_bb74_b945358bbc21.slice/crio-4437a928f7bb7f481d93784d07a669b4b2b3e18ca60f352ab7a5a612b8942ef9 WatchSource:0}: Error finding container 4437a928f7bb7f481d93784d07a669b4b2b3e18ca60f352ab7a5a612b8942ef9: Status 404 returned error can't find the container with id 4437a928f7bb7f481d93784d07a669b4b2b3e18ca60f352ab7a5a612b8942ef9 Oct 30 02:06:29.725851 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:29.725737 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-666655759c-nj68p"] Oct 30 02:06:29.739527 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:29.739473 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4" Oct 30 02:06:29.851618 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:29.851572 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca478104-e26b-4bd8-8e86-684c2807e4aa-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca478104-e26b-4bd8-8e86-684c2807e4aa-provider-credentials\") pod \"ca478104-e26b-4bd8-8e86-684c2807e4aa\" (UID: \"ca478104-e26b-4bd8-8e86-684c2807e4aa\") " Oct 30 02:06:29.851826 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:29.851740 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca478104-e26b-4bd8-8e86-684c2807e4aa-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca478104-e26b-4bd8-8e86-684c2807e4aa-routing-overlay-volume\") pod \"ca478104-e26b-4bd8-8e86-684c2807e4aa\" (UID: \"ca478104-e26b-4bd8-8e86-684c2807e4aa\") " Oct 30 02:06:29.852007 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:29.851976 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca478104-e26b-4bd8-8e86-684c2807e4aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca478104-e26b-4bd8-8e86-684c2807e4aa-plugins-config-volume\") pod \"ca478104-e26b-4bd8-8e86-684c2807e4aa\" (UID: \"ca478104-e26b-4bd8-8e86-684c2807e4aa\") " Oct 30 02:06:29.852133 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:29.852033 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca478104-e26b-4bd8-8e86-684c2807e4aa-kube-api-access-cbl5g\" (UniqueName: \"kubernetes.io/projected/ca478104-e26b-4bd8-8e86-684c2807e4aa-kube-api-access-cbl5g\") pod \"ca478104-e26b-4bd8-8e86-684c2807e4aa\" (UID: \"ca478104-e26b-4bd8-8e86-684c2807e4aa\") " Oct 30 02:06:29.853500 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:29.853470 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca478104-e26b-4bd8-8e86-684c2807e4aa-routing-overlay-volume" pod "ca478104-e26b-4bd8-8e86-684c2807e4aa" (UID: "ca478104-e26b-4bd8-8e86-684c2807e4aa"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:29.854104 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:29.854071 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca478104-e26b-4bd8-8e86-684c2807e4aa-plugins-config-volume" pod "ca478104-e26b-4bd8-8e86-684c2807e4aa" (UID: "ca478104-e26b-4bd8-8e86-684c2807e4aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:29.854883 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:29.854854 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca478104-e26b-4bd8-8e86-684c2807e4aa-kube-api-access-cbl5g" pod "ca478104-e26b-4bd8-8e86-684c2807e4aa" (UID: "ca478104-e26b-4bd8-8e86-684c2807e4aa"). InnerVolumeSpecName "kube-api-access-cbl5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:29.855692 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:29.855670 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca478104-e26b-4bd8-8e86-684c2807e4aa-provider-credentials" pod "ca478104-e26b-4bd8-8e86-684c2807e4aa" (UID: "ca478104-e26b-4bd8-8e86-684c2807e4aa"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:29.923902 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:29.918703 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:29.955959 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:29.954891 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca478104-e26b-4bd8-8e86-684c2807e4aa-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:29.955959 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:29.954925 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbl5g\" (UniqueName: \"kubernetes.io/projected/ca478104-e26b-4bd8-8e86-684c2807e4aa-kube-api-access-cbl5g\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:29.955959 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:29.954940 6572 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca478104-e26b-4bd8-8e86-684c2807e4aa-provider-credentials\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:29.955959 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:29.954956 6572 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca478104-e26b-4bd8-8e86-684c2807e4aa-routing-overlay-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:30.545916 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:30.545837 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="285a2ade-2eaa-4b72-8a06-26e9a9395442" path="/var/lib/kubelet/pods/285a2ade-2eaa-4b72-8a06-26e9a9395442/volumes" Oct 30 02:06:30.612805 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:30.612767 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4" Oct 30 02:06:30.612805 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:30.612792 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4" event={"ID":"ca478104-e26b-4bd8-8e86-684c2807e4aa","Type":"ContainerDied","Data":"06674034d1417ebf0b0a8cc62fd504e18a6ebef5f7128f47e488aefd6b60423e"} Oct 30 02:06:30.613010 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:30.612857 6572 scope.go:122] "RemoveContainer" containerID="f8fa7aa42375178d7395200f1506c25ee03f10da39cbe8e87647edefa51750d9" Oct 30 02:06:30.615116 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:30.615089 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-666655759c-nj68p" event={"ID":"19de5ac4-66b5-4440-bb74-b945358bbc21","Type":"ContainerStarted","Data":"b6daf4b40ebe0775df822f76ab748fe2c56013b0a26f2ce3b2b3d1571488c2b1"} Oct 30 02:06:30.615259 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:30.615141 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-666655759c-nj68p" event={"ID":"19de5ac4-66b5-4440-bb74-b945358bbc21","Type":"ContainerStarted","Data":"4437a928f7bb7f481d93784d07a669b4b2b3e18ca60f352ab7a5a612b8942ef9"} Oct 30 02:06:30.615463 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:30.615444 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-666655759c-nj68p" Oct 30 02:06:30.664135 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:30.664108 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4"] Oct 30 02:06:30.675384 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:30.675355 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-679dp8vq4"] Oct 30 02:06:30.713063 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:30.713014 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-666655759c-nj68p" podStartSLOduration=2.7130010049999997 podStartE2EDuration="2.713001005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:06:30.70827839 +0000 UTC m=+2916.284703793" watchObservedRunningTime="2026-10-30 02:06:30.713001005 +0000 UTC m=+2916.289426406" Oct 30 02:06:30.912895 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:30.912860 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:31.108368 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:31.108332 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:31.809998 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:31.809016 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:32.172407 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:32.172379 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-666655759c-nj68p"] Oct 30 02:06:32.200181 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:32.200156 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-79cb95d98d-6vf9k"] Oct 30 02:06:32.201002 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:32.200985 6572 cpu_manager.go:414] "containerMap: removing container" podUID="ca478104-e26b-4bd8-8e86-684c2807e4aa" containerName="payload-processing" Oct 30 02:06:32.201051 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:32.201003 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca478104-e26b-4bd8-8e86-684c2807e4aa" containerName="payload-processing" Oct 30 02:06:32.201148 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:32.201139 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca478104-e26b-4bd8-8e86-684c2807e4aa" containerName="payload-processing" Oct 30 02:06:32.206524 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:32.206505 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79cb95d98d-6vf9k" Oct 30 02:06:32.218031 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:32.218006 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc02da2-4164-4e44-9f7b-6b6a4ec63d96-plugins-config-volume\") pod \"payload-processing-external-model-79cb95d98d-6vf9k\" (UID: \"5dc02da2-4164-4e44-9f7b-6b6a4ec63d96\") " pod="models-as-a-service/payload-processing-external-model-79cb95d98d-6vf9k" Oct 30 02:06:32.218140 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:32.218047 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rstg8\" (UniqueName: \"kubernetes.io/projected/5dc02da2-4164-4e44-9f7b-6b6a4ec63d96-kube-api-access-rstg8\") pod \"payload-processing-external-model-79cb95d98d-6vf9k\" (UID: \"5dc02da2-4164-4e44-9f7b-6b6a4ec63d96\") " pod="models-as-a-service/payload-processing-external-model-79cb95d98d-6vf9k" Oct 30 02:06:32.218140 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:32.218129 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5dc02da2-4164-4e44-9f7b-6b6a4ec63d96-provider-credentials\") pod \"payload-processing-external-model-79cb95d98d-6vf9k\" (UID: \"5dc02da2-4164-4e44-9f7b-6b6a4ec63d96\") " pod="models-as-a-service/payload-processing-external-model-79cb95d98d-6vf9k" Oct 30 02:06:32.218345 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:32.218323 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5dc02da2-4164-4e44-9f7b-6b6a4ec63d96-routing-overlay-volume\") pod \"payload-processing-external-model-79cb95d98d-6vf9k\" (UID: \"5dc02da2-4164-4e44-9f7b-6b6a4ec63d96\") " pod="models-as-a-service/payload-processing-external-model-79cb95d98d-6vf9k" Oct 30 02:06:32.225010 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:32.224988 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-79cb95d98d-6vf9k"] Oct 30 02:06:32.231454 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:32.231433 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ed9d1b8f9283\"" Oct 30 02:06:32.319685 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:32.319649 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc02da2-4164-4e44-9f7b-6b6a4ec63d96-plugins-config-volume\") pod \"payload-processing-external-model-79cb95d98d-6vf9k\" (UID: \"5dc02da2-4164-4e44-9f7b-6b6a4ec63d96\") " pod="models-as-a-service/payload-processing-external-model-79cb95d98d-6vf9k" Oct 30 02:06:32.319888 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:32.319696 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rstg8\" (UniqueName: \"kubernetes.io/projected/5dc02da2-4164-4e44-9f7b-6b6a4ec63d96-kube-api-access-rstg8\") pod \"payload-processing-external-model-79cb95d98d-6vf9k\" (UID: \"5dc02da2-4164-4e44-9f7b-6b6a4ec63d96\") " pod="models-as-a-service/payload-processing-external-model-79cb95d98d-6vf9k" Oct 30 02:06:32.319888 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:32.319733 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5dc02da2-4164-4e44-9f7b-6b6a4ec63d96-provider-credentials\") pod \"payload-processing-external-model-79cb95d98d-6vf9k\" (UID: \"5dc02da2-4164-4e44-9f7b-6b6a4ec63d96\") " pod="models-as-a-service/payload-processing-external-model-79cb95d98d-6vf9k" Oct 30 02:06:32.319888 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:32.319874 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5dc02da2-4164-4e44-9f7b-6b6a4ec63d96-routing-overlay-volume\") pod \"payload-processing-external-model-79cb95d98d-6vf9k\" (UID: \"5dc02da2-4164-4e44-9f7b-6b6a4ec63d96\") " pod="models-as-a-service/payload-processing-external-model-79cb95d98d-6vf9k" Oct 30 02:06:32.320479 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:32.320454 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5dc02da2-4164-4e44-9f7b-6b6a4ec63d96-routing-overlay-volume\") pod \"payload-processing-external-model-79cb95d98d-6vf9k\" (UID: \"5dc02da2-4164-4e44-9f7b-6b6a4ec63d96\") " pod="models-as-a-service/payload-processing-external-model-79cb95d98d-6vf9k" Oct 30 02:06:32.320593 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:32.320525 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc02da2-4164-4e44-9f7b-6b6a4ec63d96-plugins-config-volume\") pod \"payload-processing-external-model-79cb95d98d-6vf9k\" (UID: \"5dc02da2-4164-4e44-9f7b-6b6a4ec63d96\") " pod="models-as-a-service/payload-processing-external-model-79cb95d98d-6vf9k" Oct 30 02:06:32.322514 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:32.322492 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5dc02da2-4164-4e44-9f7b-6b6a4ec63d96-provider-credentials\") pod \"payload-processing-external-model-79cb95d98d-6vf9k\" (UID: \"5dc02da2-4164-4e44-9f7b-6b6a4ec63d96\") " pod="models-as-a-service/payload-processing-external-model-79cb95d98d-6vf9k" Oct 30 02:06:32.339381 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:32.339350 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rstg8\" (UniqueName: \"kubernetes.io/projected/5dc02da2-4164-4e44-9f7b-6b6a4ec63d96-kube-api-access-rstg8\") pod \"payload-processing-external-model-79cb95d98d-6vf9k\" (UID: \"5dc02da2-4164-4e44-9f7b-6b6a4ec63d96\") " pod="models-as-a-service/payload-processing-external-model-79cb95d98d-6vf9k" Oct 30 02:06:32.525474 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:32.525388 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79cb95d98d-6vf9k" Oct 30 02:06:32.544406 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:32.544376 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca478104-e26b-4bd8-8e86-684c2807e4aa" path="/var/lib/kubelet/pods/ca478104-e26b-4bd8-8e86-684c2807e4aa/volumes" Oct 30 02:06:32.835865 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:06:32.835837 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dc02da2_4164_4e44_9f7b_6b6a4ec63d96.slice/crio-1929b9280dbadc739d0e3c25fa6eeea44c4dba7d90340f18907c71dd1f054a58 WatchSource:0}: Error finding container 1929b9280dbadc739d0e3c25fa6eeea44c4dba7d90340f18907c71dd1f054a58: Status 404 returned error can't find the container with id 1929b9280dbadc739d0e3c25fa6eeea44c4dba7d90340f18907c71dd1f054a58 Oct 30 02:06:32.836572 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:32.836453 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-79cb95d98d-6vf9k"] Oct 30 02:06:33.474526 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:33.474486 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:33.648698 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:33.648661 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79cb95d98d-6vf9k" event={"ID":"5dc02da2-4164-4e44-9f7b-6b6a4ec63d96","Type":"ContainerStarted","Data":"d2eea272e418c66998e2209db659afa6f8898c11f0d070f5befff131ca8888db"} Oct 30 02:06:33.648891 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:33.648713 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79cb95d98d-6vf9k" event={"ID":"5dc02da2-4164-4e44-9f7b-6b6a4ec63d96","Type":"ContainerStarted","Data":"1929b9280dbadc739d0e3c25fa6eeea44c4dba7d90340f18907c71dd1f054a58"} Oct 30 02:06:33.648978 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:33.648955 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-79cb95d98d-6vf9k" Oct 30 02:06:33.689413 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:33.689368 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-79cb95d98d-6vf9k" podStartSLOduration=1.689354673 podStartE2EDuration="1.689354673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:06:33.687585904 +0000 UTC m=+2919.264011306" watchObservedRunningTime="2026-10-30 02:06:33.689354673 +0000 UTC m=+2919.265780075" Oct 30 02:06:33.954771 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:33.954275 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:34.147355 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:34.147305 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:34.336549 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:34.336460 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:34.572815 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:34.564796 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:34.753301 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:34.753266 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:35.001634 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:35.001598 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv"] Oct 30 02:06:35.159710 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:35.159618 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:35.373768 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:35.370357 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:36.984849 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:36.984709 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:37.194349 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:37.194313 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:37.404087 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:37.403989 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:37.528677 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:37.528636 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-79cb95d98d-6vf9k"] Oct 30 02:06:37.531532 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:37.531506 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-79cb95d98d-6vf9k" Oct 30 02:06:37.559189 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:37.559161 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-666655759c-s9mw5"] Oct 30 02:06:37.564096 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:37.564077 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-666655759c-s9mw5" Oct 30 02:06:37.582951 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:37.582924 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-666655759c-s9mw5"] Oct 30 02:06:37.639975 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:37.639332 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-666655759c-nj68p" podUID="19de5ac4-66b5-4440-bb74-b945358bbc21" containerName="payload-processing" containerID="cri-o://b6daf4b40ebe0775df822f76ab748fe2c56013b0a26f2ce3b2b3d1571488c2b1" gracePeriod=25 Oct 30 02:06:37.700659 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:37.697783 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wsts\" (UniqueName: \"kubernetes.io/projected/a989accf-0147-4cd4-8d86-a5f769044d7a-kube-api-access-8wsts\") pod \"payload-processing-external-model-666655759c-s9mw5\" (UID: \"a989accf-0147-4cd4-8d86-a5f769044d7a\") " pod="models-as-a-service/payload-processing-external-model-666655759c-s9mw5" Oct 30 02:06:37.700659 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:37.697957 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a989accf-0147-4cd4-8d86-a5f769044d7a-plugins-config-volume\") pod \"payload-processing-external-model-666655759c-s9mw5\" (UID: \"a989accf-0147-4cd4-8d86-a5f769044d7a\") " pod="models-as-a-service/payload-processing-external-model-666655759c-s9mw5" Oct 30 02:06:37.700659 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:37.698049 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a989accf-0147-4cd4-8d86-a5f769044d7a-routing-overlay-volume\") pod \"payload-processing-external-model-666655759c-s9mw5\" (UID: \"a989accf-0147-4cd4-8d86-a5f769044d7a\") " pod="models-as-a-service/payload-processing-external-model-666655759c-s9mw5" Oct 30 02:06:37.700659 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:37.698202 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a989accf-0147-4cd4-8d86-a5f769044d7a-provider-credentials\") pod \"payload-processing-external-model-666655759c-s9mw5\" (UID: \"a989accf-0147-4cd4-8d86-a5f769044d7a\") " pod="models-as-a-service/payload-processing-external-model-666655759c-s9mw5" Oct 30 02:06:37.802945 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:37.802890 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a989accf-0147-4cd4-8d86-a5f769044d7a-routing-overlay-volume\") pod \"payload-processing-external-model-666655759c-s9mw5\" (UID: \"a989accf-0147-4cd4-8d86-a5f769044d7a\") " pod="models-as-a-service/payload-processing-external-model-666655759c-s9mw5" Oct 30 02:06:37.803338 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:37.803319 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a989accf-0147-4cd4-8d86-a5f769044d7a-provider-credentials\") pod \"payload-processing-external-model-666655759c-s9mw5\" (UID: \"a989accf-0147-4cd4-8d86-a5f769044d7a\") " pod="models-as-a-service/payload-processing-external-model-666655759c-s9mw5" Oct 30 02:06:37.803600 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:37.803571 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wsts\" (UniqueName: \"kubernetes.io/projected/a989accf-0147-4cd4-8d86-a5f769044d7a-kube-api-access-8wsts\") pod \"payload-processing-external-model-666655759c-s9mw5\" (UID: \"a989accf-0147-4cd4-8d86-a5f769044d7a\") " pod="models-as-a-service/payload-processing-external-model-666655759c-s9mw5" Oct 30 02:06:37.803894 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:37.803863 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a989accf-0147-4cd4-8d86-a5f769044d7a-plugins-config-volume\") pod \"payload-processing-external-model-666655759c-s9mw5\" (UID: \"a989accf-0147-4cd4-8d86-a5f769044d7a\") " pod="models-as-a-service/payload-processing-external-model-666655759c-s9mw5" Oct 30 02:06:37.805401 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:37.805363 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a989accf-0147-4cd4-8d86-a5f769044d7a-plugins-config-volume\") pod \"payload-processing-external-model-666655759c-s9mw5\" (UID: \"a989accf-0147-4cd4-8d86-a5f769044d7a\") " pod="models-as-a-service/payload-processing-external-model-666655759c-s9mw5" Oct 30 02:06:37.806405 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:37.806380 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a989accf-0147-4cd4-8d86-a5f769044d7a-routing-overlay-volume\") pod \"payload-processing-external-model-666655759c-s9mw5\" (UID: \"a989accf-0147-4cd4-8d86-a5f769044d7a\") " pod="models-as-a-service/payload-processing-external-model-666655759c-s9mw5" Oct 30 02:06:37.835792 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:37.826523 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a989accf-0147-4cd4-8d86-a5f769044d7a-provider-credentials\") pod \"payload-processing-external-model-666655759c-s9mw5\" (UID: \"a989accf-0147-4cd4-8d86-a5f769044d7a\") " pod="models-as-a-service/payload-processing-external-model-666655759c-s9mw5" Oct 30 02:06:37.877088 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:37.877029 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wsts\" (UniqueName: \"kubernetes.io/projected/a989accf-0147-4cd4-8d86-a5f769044d7a-kube-api-access-8wsts\") pod \"payload-processing-external-model-666655759c-s9mw5\" (UID: \"a989accf-0147-4cd4-8d86-a5f769044d7a\") " pod="models-as-a-service/payload-processing-external-model-666655759c-s9mw5" Oct 30 02:06:37.879539 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:37.879434 6572 generic.go:356] "Generic (PLEG): container finished" podID="19de5ac4-66b5-4440-bb74-b945358bbc21" containerID="b6daf4b40ebe0775df822f76ab748fe2c56013b0a26f2ce3b2b3d1571488c2b1" exitCode=0 Oct 30 02:06:37.879539 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:37.879482 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-666655759c-nj68p" event={"ID":"19de5ac4-66b5-4440-bb74-b945358bbc21","Type":"ContainerDied","Data":"b6daf4b40ebe0775df822f76ab748fe2c56013b0a26f2ce3b2b3d1571488c2b1"} Oct 30 02:06:37.882522 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:37.882455 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-666655759c-s9mw5" Oct 30 02:06:38.085813 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:38.085783 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-666655759c-nj68p" Oct 30 02:06:38.210714 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:38.210622 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19de5ac4-66b5-4440-bb74-b945358bbc21-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19de5ac4-66b5-4440-bb74-b945358bbc21-plugins-config-volume\") pod \"19de5ac4-66b5-4440-bb74-b945358bbc21\" (UID: \"19de5ac4-66b5-4440-bb74-b945358bbc21\") " Oct 30 02:06:38.210714 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:38.210669 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19de5ac4-66b5-4440-bb74-b945358bbc21-kube-api-access-l5rts\" (UniqueName: \"kubernetes.io/projected/19de5ac4-66b5-4440-bb74-b945358bbc21-kube-api-access-l5rts\") pod \"19de5ac4-66b5-4440-bb74-b945358bbc21\" (UID: \"19de5ac4-66b5-4440-bb74-b945358bbc21\") " Oct 30 02:06:38.211055 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:38.210829 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19de5ac4-66b5-4440-bb74-b945358bbc21-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19de5ac4-66b5-4440-bb74-b945358bbc21-routing-overlay-volume\") pod \"19de5ac4-66b5-4440-bb74-b945358bbc21\" (UID: \"19de5ac4-66b5-4440-bb74-b945358bbc21\") " Oct 30 02:06:38.211124 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:38.211093 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19de5ac4-66b5-4440-bb74-b945358bbc21-provider-credentials\" (UniqueName: \"kubernetes.io/projected/19de5ac4-66b5-4440-bb74-b945358bbc21-provider-credentials\") pod \"19de5ac4-66b5-4440-bb74-b945358bbc21\" (UID: \"19de5ac4-66b5-4440-bb74-b945358bbc21\") " Oct 30 02:06:38.211182 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:38.211155 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19de5ac4-66b5-4440-bb74-b945358bbc21-plugins-config-volume" pod "19de5ac4-66b5-4440-bb74-b945358bbc21" (UID: "19de5ac4-66b5-4440-bb74-b945358bbc21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:38.211484 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:38.211462 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19de5ac4-66b5-4440-bb74-b945358bbc21-routing-overlay-volume" pod "19de5ac4-66b5-4440-bb74-b945358bbc21" (UID: "19de5ac4-66b5-4440-bb74-b945358bbc21"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:38.211936 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:38.211876 6572 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19de5ac4-66b5-4440-bb74-b945358bbc21-routing-overlay-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:38.211936 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:38.211942 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19de5ac4-66b5-4440-bb74-b945358bbc21-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:38.213512 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:38.213488 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19de5ac4-66b5-4440-bb74-b945358bbc21-kube-api-access-l5rts" pod "19de5ac4-66b5-4440-bb74-b945358bbc21" (UID: "19de5ac4-66b5-4440-bb74-b945358bbc21"). InnerVolumeSpecName "kube-api-access-l5rts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:38.213938 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:38.213902 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19de5ac4-66b5-4440-bb74-b945358bbc21-provider-credentials" pod "19de5ac4-66b5-4440-bb74-b945358bbc21" (UID: "19de5ac4-66b5-4440-bb74-b945358bbc21"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:38.234338 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:38.233913 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-666655759c-s9mw5"] Oct 30 02:06:38.300079 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:38.300040 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:38.313132 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:38.313105 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5rts\" (UniqueName: \"kubernetes.io/projected/19de5ac4-66b5-4440-bb74-b945358bbc21-kube-api-access-l5rts\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:38.313132 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:38.313131 6572 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19de5ac4-66b5-4440-bb74-b945358bbc21-provider-credentials\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:38.493821 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:38.491317 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:38.894517 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:38.894433 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-666655759c-nj68p" Oct 30 02:06:38.894517 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:38.894452 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-666655759c-nj68p" event={"ID":"19de5ac4-66b5-4440-bb74-b945358bbc21","Type":"ContainerDied","Data":"4437a928f7bb7f481d93784d07a669b4b2b3e18ca60f352ab7a5a612b8942ef9"} Oct 30 02:06:38.894741 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:38.894527 6572 scope.go:122] "RemoveContainer" containerID="b6daf4b40ebe0775df822f76ab748fe2c56013b0a26f2ce3b2b3d1571488c2b1" Oct 30 02:06:38.896916 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:38.896812 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-666655759c-s9mw5" event={"ID":"a989accf-0147-4cd4-8d86-a5f769044d7a","Type":"ContainerStarted","Data":"6cc1e5b4ba85d9aea0b1753f220753da1ceeb2d0b4f3405f53ed5db5000cd08f"} Oct 30 02:06:38.896916 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:38.896873 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-666655759c-s9mw5" event={"ID":"a989accf-0147-4cd4-8d86-a5f769044d7a","Type":"ContainerStarted","Data":"5c7c7a53564fe07df7300b08cadfb3d1e327cef609283c9e94eebe9f2b697ca9"} Oct 30 02:06:38.897152 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:38.896930 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-666655759c-s9mw5" Oct 30 02:06:38.945502 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:38.945475 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-666655759c-nj68p"] Oct 30 02:06:38.961813 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:38.961783 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-666655759c-nj68p"] Oct 30 02:06:38.992829 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:38.992781 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-666655759c-s9mw5" podStartSLOduration=1.992762163 podStartE2EDuration="1.992762163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:06:38.988085526 +0000 UTC m=+2924.564510939" watchObservedRunningTime="2026-10-30 02:06:38.992762163 +0000 UTC m=+2924.569187559" Oct 30 02:06:39.187392 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:39.187344 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:39.393276 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:39.393240 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-666655759c-s9mw5"] Oct 30 02:06:39.430778 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:39.430729 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-79cb95d98d-sps9h"] Oct 30 02:06:39.432056 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:39.432031 6572 cpu_manager.go:414] "containerMap: removing container" podUID="19de5ac4-66b5-4440-bb74-b945358bbc21" containerName="payload-processing" Oct 30 02:06:39.432176 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:39.432057 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19de5ac4-66b5-4440-bb74-b945358bbc21" containerName="payload-processing" Oct 30 02:06:39.432300 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:39.432283 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="19de5ac4-66b5-4440-bb74-b945358bbc21" containerName="payload-processing" Oct 30 02:06:39.438353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:39.438289 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79cb95d98d-sps9h" Oct 30 02:06:39.456639 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:39.456612 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-79cb95d98d-sps9h"] Oct 30 02:06:39.493077 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:39.493031 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9073118-db7d-4711-a22a-39f8de904c10-plugins-config-volume\") pod \"payload-processing-external-model-79cb95d98d-sps9h\" (UID: \"d9073118-db7d-4711-a22a-39f8de904c10\") " pod="models-as-a-service/payload-processing-external-model-79cb95d98d-sps9h" Oct 30 02:06:39.493260 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:39.493091 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj86c\" (UniqueName: \"kubernetes.io/projected/d9073118-db7d-4711-a22a-39f8de904c10-kube-api-access-wj86c\") pod \"payload-processing-external-model-79cb95d98d-sps9h\" (UID: \"d9073118-db7d-4711-a22a-39f8de904c10\") " pod="models-as-a-service/payload-processing-external-model-79cb95d98d-sps9h" Oct 30 02:06:39.493365 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:39.493314 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d9073118-db7d-4711-a22a-39f8de904c10-provider-credentials\") pod \"payload-processing-external-model-79cb95d98d-sps9h\" (UID: \"d9073118-db7d-4711-a22a-39f8de904c10\") " pod="models-as-a-service/payload-processing-external-model-79cb95d98d-sps9h" Oct 30 02:06:39.493365 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:39.493354 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d9073118-db7d-4711-a22a-39f8de904c10-routing-overlay-volume\") pod \"payload-processing-external-model-79cb95d98d-sps9h\" (UID: \"d9073118-db7d-4711-a22a-39f8de904c10\") " pod="models-as-a-service/payload-processing-external-model-79cb95d98d-sps9h" Oct 30 02:06:39.598725 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:39.598683 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d9073118-db7d-4711-a22a-39f8de904c10-provider-credentials\") pod \"payload-processing-external-model-79cb95d98d-sps9h\" (UID: \"d9073118-db7d-4711-a22a-39f8de904c10\") " pod="models-as-a-service/payload-processing-external-model-79cb95d98d-sps9h" Oct 30 02:06:39.599017 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:39.598811 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d9073118-db7d-4711-a22a-39f8de904c10-routing-overlay-volume\") pod \"payload-processing-external-model-79cb95d98d-sps9h\" (UID: \"d9073118-db7d-4711-a22a-39f8de904c10\") " pod="models-as-a-service/payload-processing-external-model-79cb95d98d-sps9h" Oct 30 02:06:39.599148 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:39.599130 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9073118-db7d-4711-a22a-39f8de904c10-plugins-config-volume\") pod \"payload-processing-external-model-79cb95d98d-sps9h\" (UID: \"d9073118-db7d-4711-a22a-39f8de904c10\") " pod="models-as-a-service/payload-processing-external-model-79cb95d98d-sps9h" Oct 30 02:06:39.599198 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:39.599161 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj86c\" (UniqueName: \"kubernetes.io/projected/d9073118-db7d-4711-a22a-39f8de904c10-kube-api-access-wj86c\") pod \"payload-processing-external-model-79cb95d98d-sps9h\" (UID: \"d9073118-db7d-4711-a22a-39f8de904c10\") " pod="models-as-a-service/payload-processing-external-model-79cb95d98d-sps9h" Oct 30 02:06:39.599634 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:39.599611 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d9073118-db7d-4711-a22a-39f8de904c10-routing-overlay-volume\") pod \"payload-processing-external-model-79cb95d98d-sps9h\" (UID: \"d9073118-db7d-4711-a22a-39f8de904c10\") " pod="models-as-a-service/payload-processing-external-model-79cb95d98d-sps9h" Oct 30 02:06:39.599925 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:39.599908 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9073118-db7d-4711-a22a-39f8de904c10-plugins-config-volume\") pod \"payload-processing-external-model-79cb95d98d-sps9h\" (UID: \"d9073118-db7d-4711-a22a-39f8de904c10\") " pod="models-as-a-service/payload-processing-external-model-79cb95d98d-sps9h" Oct 30 02:06:39.601425 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:39.601406 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d9073118-db7d-4711-a22a-39f8de904c10-provider-credentials\") pod \"payload-processing-external-model-79cb95d98d-sps9h\" (UID: \"d9073118-db7d-4711-a22a-39f8de904c10\") " pod="models-as-a-service/payload-processing-external-model-79cb95d98d-sps9h" Oct 30 02:06:39.617860 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:39.617836 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj86c\" (UniqueName: \"kubernetes.io/projected/d9073118-db7d-4711-a22a-39f8de904c10-kube-api-access-wj86c\") pod \"payload-processing-external-model-79cb95d98d-sps9h\" (UID: \"d9073118-db7d-4711-a22a-39f8de904c10\") " pod="models-as-a-service/payload-processing-external-model-79cb95d98d-sps9h" Oct 30 02:06:39.754998 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:39.754927 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79cb95d98d-sps9h" Oct 30 02:06:40.002044 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:40.002006 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv" podUID="048aae92-605f-4b4b-bfc3-b9f8e87a3a10" containerName="payload-processing" containerID="cri-o://bb9ce9c7ce6658bf2bdc6fc8967ef625eb057b17d60351a3db8d0664d682a545" gracePeriod=25 Oct 30 02:06:40.065736 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:40.065707 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-79cb95d98d-sps9h"] Oct 30 02:06:40.103557 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:06:40.103527 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9073118_db7d_4711_a22a_39f8de904c10.slice/crio-73ef6401be715f3c498ed9c7aa362cec01dafa0872432231c456511a3f17c500 WatchSource:0}: Error finding container 73ef6401be715f3c498ed9c7aa362cec01dafa0872432231c456511a3f17c500: Status 404 returned error can't find the container with id 73ef6401be715f3c498ed9c7aa362cec01dafa0872432231c456511a3f17c500 Oct 30 02:06:40.271343 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:40.271285 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv" Oct 30 02:06:40.308576 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:40.308546 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/048aae92-605f-4b4b-bfc3-b9f8e87a3a10-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048aae92-605f-4b4b-bfc3-b9f8e87a3a10-plugins-config-volume\") pod \"048aae92-605f-4b4b-bfc3-b9f8e87a3a10\" (UID: \"048aae92-605f-4b4b-bfc3-b9f8e87a3a10\") " Oct 30 02:06:40.308736 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:40.308625 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/048aae92-605f-4b4b-bfc3-b9f8e87a3a10-kube-api-access-9dpx4\" (UniqueName: \"kubernetes.io/projected/048aae92-605f-4b4b-bfc3-b9f8e87a3a10-kube-api-access-9dpx4\") pod \"048aae92-605f-4b4b-bfc3-b9f8e87a3a10\" (UID: \"048aae92-605f-4b4b-bfc3-b9f8e87a3a10\") " Oct 30 02:06:40.308850 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:40.308831 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/048aae92-605f-4b4b-bfc3-b9f8e87a3a10-provider-credentials\" (UniqueName: \"kubernetes.io/projected/048aae92-605f-4b4b-bfc3-b9f8e87a3a10-provider-credentials\") pod \"048aae92-605f-4b4b-bfc3-b9f8e87a3a10\" (UID: \"048aae92-605f-4b4b-bfc3-b9f8e87a3a10\") " Oct 30 02:06:40.308984 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:40.308963 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/048aae92-605f-4b4b-bfc3-b9f8e87a3a10-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/048aae92-605f-4b4b-bfc3-b9f8e87a3a10-routing-overlay-volume\") pod \"048aae92-605f-4b4b-bfc3-b9f8e87a3a10\" (UID: \"048aae92-605f-4b4b-bfc3-b9f8e87a3a10\") " Oct 30 02:06:40.309083 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:40.309004 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/048aae92-605f-4b4b-bfc3-b9f8e87a3a10-plugins-config-volume" pod "048aae92-605f-4b4b-bfc3-b9f8e87a3a10" (UID: "048aae92-605f-4b4b-bfc3-b9f8e87a3a10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:40.309442 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:40.309416 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/048aae92-605f-4b4b-bfc3-b9f8e87a3a10-routing-overlay-volume" pod "048aae92-605f-4b4b-bfc3-b9f8e87a3a10" (UID: "048aae92-605f-4b4b-bfc3-b9f8e87a3a10"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:40.309585 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:40.309568 6572 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/048aae92-605f-4b4b-bfc3-b9f8e87a3a10-routing-overlay-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:40.309668 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:40.309585 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048aae92-605f-4b4b-bfc3-b9f8e87a3a10-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:40.310971 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:40.310951 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/048aae92-605f-4b4b-bfc3-b9f8e87a3a10-provider-credentials" pod "048aae92-605f-4b4b-bfc3-b9f8e87a3a10" (UID: "048aae92-605f-4b4b-bfc3-b9f8e87a3a10"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:40.311184 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:40.311170 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/048aae92-605f-4b4b-bfc3-b9f8e87a3a10-kube-api-access-9dpx4" pod "048aae92-605f-4b4b-bfc3-b9f8e87a3a10" (UID: "048aae92-605f-4b4b-bfc3-b9f8e87a3a10"). InnerVolumeSpecName "kube-api-access-9dpx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:40.411518 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:40.411487 6572 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/048aae92-605f-4b4b-bfc3-b9f8e87a3a10-provider-credentials\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:40.411518 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:40.411513 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dpx4\" (UniqueName: \"kubernetes.io/projected/048aae92-605f-4b4b-bfc3-b9f8e87a3a10-kube-api-access-9dpx4\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:40.544200 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:40.544125 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19de5ac4-66b5-4440-bb74-b945358bbc21" path="/var/lib/kubelet/pods/19de5ac4-66b5-4440-bb74-b945358bbc21/volumes" Oct 30 02:06:40.941209 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:40.941175 6572 generic.go:356] "Generic (PLEG): container finished" podID="048aae92-605f-4b4b-bfc3-b9f8e87a3a10" containerID="bb9ce9c7ce6658bf2bdc6fc8967ef625eb057b17d60351a3db8d0664d682a545" exitCode=0 Oct 30 02:06:40.941645 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:40.941626 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv" event={"ID":"048aae92-605f-4b4b-bfc3-b9f8e87a3a10","Type":"ContainerDied","Data":"bb9ce9c7ce6658bf2bdc6fc8967ef625eb057b17d60351a3db8d0664d682a545"} Oct 30 02:06:40.941817 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:40.941783 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv" event={"ID":"048aae92-605f-4b4b-bfc3-b9f8e87a3a10","Type":"ContainerDied","Data":"c2548e7cddb7c28985a96cc6cad4ea591bab99ff976c9dfc780183a7d5997f96"} Oct 30 02:06:40.941966 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:40.941954 6572 scope.go:122] "RemoveContainer" containerID="bb9ce9c7ce6658bf2bdc6fc8967ef625eb057b17d60351a3db8d0664d682a545" Oct 30 02:06:40.942181 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:40.942169 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv" Oct 30 02:06:40.947843 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:40.947804 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79cb95d98d-sps9h" event={"ID":"d9073118-db7d-4711-a22a-39f8de904c10","Type":"ContainerStarted","Data":"52021ce52ced2a8109a965e62a8a15481d9a5442215795ddd4d76be633fb806c"} Oct 30 02:06:40.947969 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:40.947876 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79cb95d98d-sps9h" event={"ID":"d9073118-db7d-4711-a22a-39f8de904c10","Type":"ContainerStarted","Data":"73ef6401be715f3c498ed9c7aa362cec01dafa0872432231c456511a3f17c500"} Oct 30 02:06:40.948420 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:40.948388 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-79cb95d98d-sps9h" Oct 30 02:06:40.958791 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:40.958767 6572 scope.go:122] "RemoveContainer" containerID="bb9ce9c7ce6658bf2bdc6fc8967ef625eb057b17d60351a3db8d0664d682a545" Oct 30 02:06:40.959108 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:06:40.959078 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb9ce9c7ce6658bf2bdc6fc8967ef625eb057b17d60351a3db8d0664d682a545\": container with ID starting with bb9ce9c7ce6658bf2bdc6fc8967ef625eb057b17d60351a3db8d0664d682a545 not found: ID does not exist" containerID="bb9ce9c7ce6658bf2bdc6fc8967ef625eb057b17d60351a3db8d0664d682a545" Oct 30 02:06:40.959206 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:40.959114 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb9ce9c7ce6658bf2bdc6fc8967ef625eb057b17d60351a3db8d0664d682a545"} err="failed to get container status \"bb9ce9c7ce6658bf2bdc6fc8967ef625eb057b17d60351a3db8d0664d682a545\": rpc error: code = NotFound desc = could not find container \"bb9ce9c7ce6658bf2bdc6fc8967ef625eb057b17d60351a3db8d0664d682a545\": container with ID starting with bb9ce9c7ce6658bf2bdc6fc8967ef625eb057b17d60351a3db8d0664d682a545 not found: ID does not exist" Oct 30 02:06:41.047434 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:41.047404 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:41.139044 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:41.139013 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv"] Oct 30 02:06:41.157827 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:41.157801 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2e618f/payload-processing-external-model-e2e-extmod-b-2e618f-b7f5b82pv"] Oct 30 02:06:41.217959 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:41.217871 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-79cb95d98d-sps9h" podStartSLOduration=2.217854686 podStartE2EDuration="2.217854686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:06:41.21696678 +0000 UTC m=+2926.793392183" watchObservedRunningTime="2026-10-30 02:06:41.217854686 +0000 UTC m=+2926.794280091" Oct 30 02:06:42.079073 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:42.079042 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:42.317562 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:42.317529 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:42.529777 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:42.529681 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-79cb95d98d-6vf9k" podUID="5dc02da2-4164-4e44-9f7b-6b6a4ec63d96" containerName="payload-processing" containerID="cri-o://d2eea272e418c66998e2209db659afa6f8898c11f0d070f5befff131ca8888db" gracePeriod=25 Oct 30 02:06:42.544163 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:42.544136 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="048aae92-605f-4b4b-bfc3-b9f8e87a3a10" path="/var/lib/kubelet/pods/048aae92-605f-4b4b-bfc3-b9f8e87a3a10/volumes" Oct 30 02:06:42.559507 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:42.559479 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:42.608003 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:06:42.607972 6572 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dc02da2_4164_4e44_9f7b_6b6a4ec63d96.slice/crio-d2eea272e418c66998e2209db659afa6f8898c11f0d070f5befff131ca8888db.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dc02da2_4164_4e44_9f7b_6b6a4ec63d96.slice/crio-conmon-d2eea272e418c66998e2209db659afa6f8898c11f0d070f5befff131ca8888db.scope\": RecentStats: unable to find data in memory cache]" Oct 30 02:06:42.772804 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:42.771847 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:42.861377 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:42.861350 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79cb95d98d-6vf9k" Oct 30 02:06:42.952762 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:42.952717 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dc02da2-4164-4e44-9f7b-6b6a4ec63d96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc02da2-4164-4e44-9f7b-6b6a4ec63d96-plugins-config-volume\") pod \"5dc02da2-4164-4e44-9f7b-6b6a4ec63d96\" (UID: \"5dc02da2-4164-4e44-9f7b-6b6a4ec63d96\") " Oct 30 02:06:42.952960 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:42.952792 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dc02da2-4164-4e44-9f7b-6b6a4ec63d96-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5dc02da2-4164-4e44-9f7b-6b6a4ec63d96-routing-overlay-volume\") pod \"5dc02da2-4164-4e44-9f7b-6b6a4ec63d96\" (UID: \"5dc02da2-4164-4e44-9f7b-6b6a4ec63d96\") " Oct 30 02:06:42.952960 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:42.952880 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dc02da2-4164-4e44-9f7b-6b6a4ec63d96-kube-api-access-rstg8\" (UniqueName: \"kubernetes.io/projected/5dc02da2-4164-4e44-9f7b-6b6a4ec63d96-kube-api-access-rstg8\") pod \"5dc02da2-4164-4e44-9f7b-6b6a4ec63d96\" (UID: \"5dc02da2-4164-4e44-9f7b-6b6a4ec63d96\") " Oct 30 02:06:42.952960 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:42.952918 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dc02da2-4164-4e44-9f7b-6b6a4ec63d96-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5dc02da2-4164-4e44-9f7b-6b6a4ec63d96-provider-credentials\") pod \"5dc02da2-4164-4e44-9f7b-6b6a4ec63d96\" (UID: \"5dc02da2-4164-4e44-9f7b-6b6a4ec63d96\") " Oct 30 02:06:42.953158 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:42.953133 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dc02da2-4164-4e44-9f7b-6b6a4ec63d96-routing-overlay-volume" pod "5dc02da2-4164-4e44-9f7b-6b6a4ec63d96" (UID: "5dc02da2-4164-4e44-9f7b-6b6a4ec63d96"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:42.953219 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:42.953162 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dc02da2-4164-4e44-9f7b-6b6a4ec63d96-plugins-config-volume" pod "5dc02da2-4164-4e44-9f7b-6b6a4ec63d96" (UID: "5dc02da2-4164-4e44-9f7b-6b6a4ec63d96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:42.953559 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:42.953532 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc02da2-4164-4e44-9f7b-6b6a4ec63d96-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:42.953559 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:42.953557 6572 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5dc02da2-4164-4e44-9f7b-6b6a4ec63d96-routing-overlay-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:42.960944 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:42.955634 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc02da2-4164-4e44-9f7b-6b6a4ec63d96-kube-api-access-rstg8" pod "5dc02da2-4164-4e44-9f7b-6b6a4ec63d96" (UID: "5dc02da2-4164-4e44-9f7b-6b6a4ec63d96"). InnerVolumeSpecName "kube-api-access-rstg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:42.960944 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:42.956172 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc02da2-4164-4e44-9f7b-6b6a4ec63d96-provider-credentials" pod "5dc02da2-4164-4e44-9f7b-6b6a4ec63d96" (UID: "5dc02da2-4164-4e44-9f7b-6b6a4ec63d96"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:42.965362 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:42.965333 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5"] Oct 30 02:06:42.966542 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:42.966519 6572 cpu_manager.go:414] "containerMap: removing container" podUID="048aae92-605f-4b4b-bfc3-b9f8e87a3a10" containerName="payload-processing" Oct 30 02:06:42.966643 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:42.966543 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="048aae92-605f-4b4b-bfc3-b9f8e87a3a10" containerName="payload-processing" Oct 30 02:06:42.966643 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:42.966572 6572 cpu_manager.go:414] "containerMap: removing container" podUID="5dc02da2-4164-4e44-9f7b-6b6a4ec63d96" containerName="payload-processing" Oct 30 02:06:42.966643 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:42.966580 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dc02da2-4164-4e44-9f7b-6b6a4ec63d96" containerName="payload-processing" Oct 30 02:06:42.966874 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:42.966859 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="048aae92-605f-4b4b-bfc3-b9f8e87a3a10" containerName="payload-processing" Oct 30 02:06:42.966922 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:42.966880 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dc02da2-4164-4e44-9f7b-6b6a4ec63d96" containerName="payload-processing" Oct 30 02:06:42.971599 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:42.971582 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:06:42.973908 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:42.973877 6572 generic.go:356] "Generic (PLEG): container finished" podID="5dc02da2-4164-4e44-9f7b-6b6a4ec63d96" containerID="d2eea272e418c66998e2209db659afa6f8898c11f0d070f5befff131ca8888db" exitCode=0 Oct 30 02:06:42.974703 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:42.974686 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79cb95d98d-6vf9k" Oct 30 02:06:42.975856 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:42.975827 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79cb95d98d-6vf9k" event={"ID":"5dc02da2-4164-4e44-9f7b-6b6a4ec63d96","Type":"ContainerDied","Data":"d2eea272e418c66998e2209db659afa6f8898c11f0d070f5befff131ca8888db"} Oct 30 02:06:42.975953 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:42.975892 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79cb95d98d-6vf9k" event={"ID":"5dc02da2-4164-4e44-9f7b-6b6a4ec63d96","Type":"ContainerDied","Data":"1929b9280dbadc739d0e3c25fa6eeea44c4dba7d90340f18907c71dd1f054a58"} Oct 30 02:06:42.975953 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:42.975924 6572 scope.go:122] "RemoveContainer" containerID="d2eea272e418c66998e2209db659afa6f8898c11f0d070f5befff131ca8888db" Oct 30 02:06:42.994380 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:42.994353 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5"] Oct 30 02:06:43.001650 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:42.998132 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f89697-openshift-default-dockercfg-6j4rg\"" Oct 30 02:06:43.001650 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:42.999387 6572 scope.go:122] "RemoveContainer" containerID="d2eea272e418c66998e2209db659afa6f8898c11f0d070f5befff131ca8888db" Oct 30 02:06:43.001650 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:06:42.999865 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2eea272e418c66998e2209db659afa6f8898c11f0d070f5befff131ca8888db\": container with ID starting with d2eea272e418c66998e2209db659afa6f8898c11f0d070f5befff131ca8888db not found: ID does not exist" containerID="d2eea272e418c66998e2209db659afa6f8898c11f0d070f5befff131ca8888db" Oct 30 02:06:43.001650 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:42.999923 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2eea272e418c66998e2209db659afa6f8898c11f0d070f5befff131ca8888db"} err="failed to get container status \"d2eea272e418c66998e2209db659afa6f8898c11f0d070f5befff131ca8888db\": rpc error: code = NotFound desc = could not find container \"d2eea272e418c66998e2209db659afa6f8898c11f0d070f5befff131ca8888db\": container with ID starting with d2eea272e418c66998e2209db659afa6f8898c11f0d070f5befff131ca8888db not found: ID does not exist" Oct 30 02:06:43.063773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.058976 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-istio-envoy\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5\" (UID: \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:06:43.078694 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.075978 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-workload-certs\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5\" (UID: \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:06:43.078694 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.076038 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-workload-socket\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5\" (UID: \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:06:43.078694 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.076133 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-istio-token\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5\" (UID: \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:06:43.078694 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.076170 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-istio-data\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5\" (UID: \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:06:43.078694 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.076233 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-istio-podinfo\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5\" (UID: \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:06:43.078694 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.076307 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-istiod-ca-cert\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5\" (UID: \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:06:43.078694 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.076347 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn9pf\" (UniqueName: \"kubernetes.io/projected/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-kube-api-access-fn9pf\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5\" (UID: \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:06:43.078694 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.076470 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-credential-socket\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5\" (UID: \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:06:43.078694 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.076784 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rstg8\" (UniqueName: \"kubernetes.io/projected/5dc02da2-4164-4e44-9f7b-6b6a4ec63d96-kube-api-access-rstg8\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:43.078694 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.076806 6572 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5dc02da2-4164-4e44-9f7b-6b6a4ec63d96-provider-credentials\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:43.173775 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.171721 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:43.180783 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.180365 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-79cb95d98d-6vf9k"] Oct 30 02:06:43.184789 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.181428 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-istio-envoy\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5\" (UID: \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:06:43.184789 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.181525 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-workload-certs\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5\" (UID: \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:06:43.184789 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.181565 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-workload-socket\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5\" (UID: \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:06:43.184789 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.181641 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-istio-token\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5\" (UID: \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:06:43.184789 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.181679 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-istio-data\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5\" (UID: \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:06:43.184789 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.181740 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-istio-podinfo\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5\" (UID: \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:06:43.184789 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.181828 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-istiod-ca-cert\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5\" (UID: \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:06:43.184789 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.181865 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn9pf\" (UniqueName: \"kubernetes.io/projected/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-kube-api-access-fn9pf\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5\" (UID: \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:06:43.184789 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.181947 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-credential-socket\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5\" (UID: \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:06:43.184789 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.182470 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-credential-socket\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5\" (UID: \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:06:43.200778 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.189199 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-istio-envoy\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5\" (UID: \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:06:43.200778 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.189464 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-workload-certs\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5\" (UID: \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:06:43.200778 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.189698 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-workload-socket\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5\" (UID: \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:06:43.208776 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.206306 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-istio-data\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5\" (UID: \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:06:43.208776 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.207631 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-istiod-ca-cert\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5\" (UID: \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:06:43.225767 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.214307 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-istio-podinfo\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5\" (UID: \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:06:43.244793 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.244484 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn9pf\" (UniqueName: \"kubernetes.io/projected/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-kube-api-access-fn9pf\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5\" (UID: \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:06:43.247082 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.245435 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-istio-token\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5\" (UID: \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:06:43.247082 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.246803 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-79cb95d98d-6vf9k"] Oct 30 02:06:43.313108 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.313070 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:06:43.582809 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.582683 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:43.701563 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:06:43.701528 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod166123b7_cdef_49d7_8e6c_2a2a993b9bb8.slice/crio-1aa9835bd0feaf2a602684ab6b4cee6a47897e6a79212c74abf012018bbb1722 WatchSource:0}: Error finding container 1aa9835bd0feaf2a602684ab6b4cee6a47897e6a79212c74abf012018bbb1722: Status 404 returned error can't find the container with id 1aa9835bd0feaf2a602684ab6b4cee6a47897e6a79212c74abf012018bbb1722 Oct 30 02:06:43.702012 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.701989 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5"] Oct 30 02:06:43.704888 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.704851 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:06:43.704983 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.704934 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:06:43.705028 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.704982 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:06:43.988500 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.988462 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" event={"ID":"166123b7-cdef-49d7-8e6c-2a2a993b9bb8","Type":"ContainerStarted","Data":"1b1062c93a39d9f873c2611743f1ba3423beb5cc68ddf14d4b57baa66761cd63"} Oct 30 02:06:43.988722 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:43.988528 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" event={"ID":"166123b7-cdef-49d7-8e6c-2a2a993b9bb8","Type":"ContainerStarted","Data":"1aa9835bd0feaf2a602684ab6b4cee6a47897e6a79212c74abf012018bbb1722"} Oct 30 02:06:44.037284 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:44.037230 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" podStartSLOduration=2.037215641 podStartE2EDuration="2.037215641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:06:44.036004647 +0000 UTC m=+2929.612430050" watchObservedRunningTime="2026-10-30 02:06:44.037215641 +0000 UTC m=+2929.613641042" Oct 30 02:06:44.313740 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:44.313668 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:06:44.320081 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:44.320059 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:06:44.545096 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:44.545063 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dc02da2-4164-4e44-9f7b-6b6a4ec63d96" path="/var/lib/kubelet/pods/5dc02da2-4164-4e44-9f7b-6b6a4ec63d96/volumes" Oct 30 02:06:44.646978 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:44.646882 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:44.689690 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:06:44.689655 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:06:45.054816 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:45.053214 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:45.163872 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:45.163085 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:06:45.170233 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:45.170199 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:06:45.184894 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:45.184853 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-79cb95d98d-sps9h"] Oct 30 02:06:45.187849 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:45.187825 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-79cb95d98d-sps9h" Oct 30 02:06:45.370373 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:45.370297 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w"] Oct 30 02:06:45.370815 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:45.370493 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" podUID="99a74259-b0af-4c30-b41a-75835331b838" containerName="istio-proxy" containerID="cri-o://9b1720bdfd7678e36efbccbf9cea31fb49cd331ea9d68d469e23965e80dcd39e" gracePeriod=30 Oct 30 02:06:45.402047 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:45.402016 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:45.948145 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:45.948111 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-666655759c-s9mw5" podUID="a989accf-0147-4cd4-8d86-a5f769044d7a" containerName="payload-processing" containerID="cri-o://6cc1e5b4ba85d9aea0b1753f220753da1ceeb2d0b4f3405f53ed5db5000cd08f" gracePeriod=25 Oct 30 02:06:46.183471 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:46.178281 6572 generic.go:356] "Generic (PLEG): container finished" podID="a989accf-0147-4cd4-8d86-a5f769044d7a" containerID="6cc1e5b4ba85d9aea0b1753f220753da1ceeb2d0b4f3405f53ed5db5000cd08f" exitCode=0 Oct 30 02:06:46.183471 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:46.179603 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-666655759c-s9mw5" event={"ID":"a989accf-0147-4cd4-8d86-a5f769044d7a","Type":"ContainerDied","Data":"6cc1e5b4ba85d9aea0b1753f220753da1ceeb2d0b4f3405f53ed5db5000cd08f"} Oct 30 02:06:46.249364 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:46.249336 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-666655759c-s9mw5" Oct 30 02:06:46.388616 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:46.388579 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a989accf-0147-4cd4-8d86-a5f769044d7a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a989accf-0147-4cd4-8d86-a5f769044d7a-plugins-config-volume\") pod \"a989accf-0147-4cd4-8d86-a5f769044d7a\" (UID: \"a989accf-0147-4cd4-8d86-a5f769044d7a\") " Oct 30 02:06:46.389089 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:46.388647 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a989accf-0147-4cd4-8d86-a5f769044d7a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a989accf-0147-4cd4-8d86-a5f769044d7a-provider-credentials\") pod \"a989accf-0147-4cd4-8d86-a5f769044d7a\" (UID: \"a989accf-0147-4cd4-8d86-a5f769044d7a\") " Oct 30 02:06:46.389089 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:46.388715 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a989accf-0147-4cd4-8d86-a5f769044d7a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a989accf-0147-4cd4-8d86-a5f769044d7a-routing-overlay-volume\") pod \"a989accf-0147-4cd4-8d86-a5f769044d7a\" (UID: \"a989accf-0147-4cd4-8d86-a5f769044d7a\") " Oct 30 02:06:46.389089 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:46.388881 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a989accf-0147-4cd4-8d86-a5f769044d7a-kube-api-access-8wsts\" (UniqueName: \"kubernetes.io/projected/a989accf-0147-4cd4-8d86-a5f769044d7a-kube-api-access-8wsts\") pod \"a989accf-0147-4cd4-8d86-a5f769044d7a\" (UID: \"a989accf-0147-4cd4-8d86-a5f769044d7a\") " Oct 30 02:06:46.389089 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:46.389071 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a989accf-0147-4cd4-8d86-a5f769044d7a-plugins-config-volume" pod "a989accf-0147-4cd4-8d86-a5f769044d7a" (UID: "a989accf-0147-4cd4-8d86-a5f769044d7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:46.389089 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:46.389082 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a989accf-0147-4cd4-8d86-a5f769044d7a-routing-overlay-volume" pod "a989accf-0147-4cd4-8d86-a5f769044d7a" (UID: "a989accf-0147-4cd4-8d86-a5f769044d7a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:46.389667 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:46.389645 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a989accf-0147-4cd4-8d86-a5f769044d7a-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:46.389879 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:46.389671 6572 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a989accf-0147-4cd4-8d86-a5f769044d7a-routing-overlay-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:46.391105 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:46.391079 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a989accf-0147-4cd4-8d86-a5f769044d7a-provider-credentials" pod "a989accf-0147-4cd4-8d86-a5f769044d7a" (UID: "a989accf-0147-4cd4-8d86-a5f769044d7a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:46.391266 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:46.391249 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a989accf-0147-4cd4-8d86-a5f769044d7a-kube-api-access-8wsts" pod "a989accf-0147-4cd4-8d86-a5f769044d7a" (UID: "a989accf-0147-4cd4-8d86-a5f769044d7a"). InnerVolumeSpecName "kube-api-access-8wsts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:46.491266 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:46.491188 6572 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a989accf-0147-4cd4-8d86-a5f769044d7a-provider-credentials\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:46.491266 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:46.491219 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wsts\" (UniqueName: \"kubernetes.io/projected/a989accf-0147-4cd4-8d86-a5f769044d7a-kube-api-access-8wsts\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:46.685210 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:46.685169 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:46.935855 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:46.935823 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:47.254728 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:47.254640 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:47.318809 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:47.317892 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-666655759c-s9mw5" Oct 30 02:06:47.318809 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:47.318657 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-666655759c-s9mw5" event={"ID":"a989accf-0147-4cd4-8d86-a5f769044d7a","Type":"ContainerDied","Data":"5c7c7a53564fe07df7300b08cadfb3d1e327cef609283c9e94eebe9f2b697ca9"} Oct 30 02:06:47.318809 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:47.318714 6572 scope.go:122] "RemoveContainer" containerID="6cc1e5b4ba85d9aea0b1753f220753da1ceeb2d0b4f3405f53ed5db5000cd08f" Oct 30 02:06:47.483340 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:47.483304 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-666655759c-s9mw5"] Oct 30 02:06:47.490810 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:47.489969 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-666655759c-s9mw5"] Oct 30 02:06:47.579849 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:47.579125 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:47.811769 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:47.811030 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:48.354178 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:48.354144 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:48.623820 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:48.621326 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:48.665772 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:48.665049 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a989accf-0147-4cd4-8d86-a5f769044d7a" path="/var/lib/kubelet/pods/a989accf-0147-4cd4-8d86-a5f769044d7a/volumes" Oct 30 02:06:49.033768 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:49.033092 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:50.185424 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.185347 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-79cb95d98d-sps9h" podUID="d9073118-db7d-4711-a22a-39f8de904c10" containerName="payload-processing" containerID="cri-o://52021ce52ced2a8109a965e62a8a15481d9a5442215795ddd4d76be633fb806c" gracePeriod=25 Oct 30 02:06:50.325234 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.325204 6572 generic.go:356] "Generic (PLEG): container finished" podID="d9073118-db7d-4711-a22a-39f8de904c10" containerID="52021ce52ced2a8109a965e62a8a15481d9a5442215795ddd4d76be633fb806c" exitCode=0 Oct 30 02:06:50.325406 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.325242 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79cb95d98d-sps9h" event={"ID":"d9073118-db7d-4711-a22a-39f8de904c10","Type":"ContainerDied","Data":"52021ce52ced2a8109a965e62a8a15481d9a5442215795ddd4d76be633fb806c"} Oct 30 02:06:50.517407 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.517382 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79cb95d98d-sps9h" Oct 30 02:06:50.640210 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.640185 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" Oct 30 02:06:50.689627 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.689599 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9073118-db7d-4711-a22a-39f8de904c10-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d9073118-db7d-4711-a22a-39f8de904c10-provider-credentials\") pod \"d9073118-db7d-4711-a22a-39f8de904c10\" (UID: \"d9073118-db7d-4711-a22a-39f8de904c10\") " Oct 30 02:06:50.689785 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.689675 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9073118-db7d-4711-a22a-39f8de904c10-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9073118-db7d-4711-a22a-39f8de904c10-plugins-config-volume\") pod \"d9073118-db7d-4711-a22a-39f8de904c10\" (UID: \"d9073118-db7d-4711-a22a-39f8de904c10\") " Oct 30 02:06:50.689841 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.689799 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9073118-db7d-4711-a22a-39f8de904c10-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d9073118-db7d-4711-a22a-39f8de904c10-routing-overlay-volume\") pod \"d9073118-db7d-4711-a22a-39f8de904c10\" (UID: \"d9073118-db7d-4711-a22a-39f8de904c10\") " Oct 30 02:06:50.690017 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.689999 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9073118-db7d-4711-a22a-39f8de904c10-kube-api-access-wj86c\" (UniqueName: \"kubernetes.io/projected/d9073118-db7d-4711-a22a-39f8de904c10-kube-api-access-wj86c\") pod \"d9073118-db7d-4711-a22a-39f8de904c10\" (UID: \"d9073118-db7d-4711-a22a-39f8de904c10\") " Oct 30 02:06:50.690112 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.690091 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9073118-db7d-4711-a22a-39f8de904c10-routing-overlay-volume" pod "d9073118-db7d-4711-a22a-39f8de904c10" (UID: "d9073118-db7d-4711-a22a-39f8de904c10"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:50.690159 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.690130 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9073118-db7d-4711-a22a-39f8de904c10-plugins-config-volume" pod "d9073118-db7d-4711-a22a-39f8de904c10" (UID: "d9073118-db7d-4711-a22a-39f8de904c10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:50.691089 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.691009 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9073118-db7d-4711-a22a-39f8de904c10-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:50.691089 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.691039 6572 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d9073118-db7d-4711-a22a-39f8de904c10-routing-overlay-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:50.692056 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.692036 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9073118-db7d-4711-a22a-39f8de904c10-provider-credentials" pod "d9073118-db7d-4711-a22a-39f8de904c10" (UID: "d9073118-db7d-4711-a22a-39f8de904c10"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:50.692150 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.692132 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9073118-db7d-4711-a22a-39f8de904c10-kube-api-access-wj86c" pod "d9073118-db7d-4711-a22a-39f8de904c10" (UID: "d9073118-db7d-4711-a22a-39f8de904c10"). InnerVolumeSpecName "kube-api-access-wj86c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:50.792463 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.792379 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99a74259-b0af-4c30-b41a-75835331b838-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99a74259-b0af-4c30-b41a-75835331b838-istiod-ca-cert\") pod \"99a74259-b0af-4c30-b41a-75835331b838\" (UID: \"99a74259-b0af-4c30-b41a-75835331b838\") " Oct 30 02:06:50.792683 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.792590 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99a74259-b0af-4c30-b41a-75835331b838-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99a74259-b0af-4c30-b41a-75835331b838-istio-envoy\") pod \"99a74259-b0af-4c30-b41a-75835331b838\" (UID: \"99a74259-b0af-4c30-b41a-75835331b838\") " Oct 30 02:06:50.792879 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.792843 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99a74259-b0af-4c30-b41a-75835331b838-istiod-ca-cert" pod "99a74259-b0af-4c30-b41a-75835331b838" (UID: "99a74259-b0af-4c30-b41a-75835331b838"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:50.793207 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.793062 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99a74259-b0af-4c30-b41a-75835331b838-kube-api-access-lgqms\" (UniqueName: \"kubernetes.io/projected/99a74259-b0af-4c30-b41a-75835331b838-kube-api-access-lgqms\") pod \"99a74259-b0af-4c30-b41a-75835331b838\" (UID: \"99a74259-b0af-4c30-b41a-75835331b838\") " Oct 30 02:06:50.793207 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.793118 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99a74259-b0af-4c30-b41a-75835331b838-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99a74259-b0af-4c30-b41a-75835331b838-workload-certs\") pod \"99a74259-b0af-4c30-b41a-75835331b838\" (UID: \"99a74259-b0af-4c30-b41a-75835331b838\") " Oct 30 02:06:50.793524 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.793176 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99a74259-b0af-4c30-b41a-75835331b838-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99a74259-b0af-4c30-b41a-75835331b838-istio-podinfo\") pod \"99a74259-b0af-4c30-b41a-75835331b838\" (UID: \"99a74259-b0af-4c30-b41a-75835331b838\") " Oct 30 02:06:50.793524 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.793471 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99a74259-b0af-4c30-b41a-75835331b838-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a74259-b0af-4c30-b41a-75835331b838-workload-socket\") pod \"99a74259-b0af-4c30-b41a-75835331b838\" (UID: \"99a74259-b0af-4c30-b41a-75835331b838\") " Oct 30 02:06:50.793716 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.793689 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a74259-b0af-4c30-b41a-75835331b838-workload-certs" pod "99a74259-b0af-4c30-b41a-75835331b838" (UID: "99a74259-b0af-4c30-b41a-75835331b838"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:50.793716 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.793707 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99a74259-b0af-4c30-b41a-75835331b838-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a74259-b0af-4c30-b41a-75835331b838-credential-socket\") pod \"99a74259-b0af-4c30-b41a-75835331b838\" (UID: \"99a74259-b0af-4c30-b41a-75835331b838\") " Oct 30 02:06:50.793878 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.793791 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99a74259-b0af-4c30-b41a-75835331b838-istio-token\" (UniqueName: \"kubernetes.io/projected/99a74259-b0af-4c30-b41a-75835331b838-istio-token\") pod \"99a74259-b0af-4c30-b41a-75835331b838\" (UID: \"99a74259-b0af-4c30-b41a-75835331b838\") " Oct 30 02:06:50.793919 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.793878 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99a74259-b0af-4c30-b41a-75835331b838-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99a74259-b0af-4c30-b41a-75835331b838-istio-data\") pod \"99a74259-b0af-4c30-b41a-75835331b838\" (UID: \"99a74259-b0af-4c30-b41a-75835331b838\") " Oct 30 02:06:50.794502 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.794260 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a74259-b0af-4c30-b41a-75835331b838-workload-socket" pod "99a74259-b0af-4c30-b41a-75835331b838" (UID: "99a74259-b0af-4c30-b41a-75835331b838"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:50.794502 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.794279 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a74259-b0af-4c30-b41a-75835331b838-credential-socket" pod "99a74259-b0af-4c30-b41a-75835331b838" (UID: "99a74259-b0af-4c30-b41a-75835331b838"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:50.795121 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.794682 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a74259-b0af-4c30-b41a-75835331b838-istio-data" pod "99a74259-b0af-4c30-b41a-75835331b838" (UID: "99a74259-b0af-4c30-b41a-75835331b838"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:50.795341 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.795315 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a74259-b0af-4c30-b41a-75835331b838-istio-envoy" pod "99a74259-b0af-4c30-b41a-75835331b838" (UID: "99a74259-b0af-4c30-b41a-75835331b838"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:50.795501 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.795476 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99a74259-b0af-4c30-b41a-75835331b838-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:50.795555 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.795524 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a74259-b0af-4c30-b41a-75835331b838-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:50.795555 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.795540 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj86c\" (UniqueName: \"kubernetes.io/projected/d9073118-db7d-4711-a22a-39f8de904c10-kube-api-access-wj86c\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:50.795633 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.795556 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a74259-b0af-4c30-b41a-75835331b838-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:50.795633 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.795571 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99a74259-b0af-4c30-b41a-75835331b838-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:50.795633 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.795585 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99a74259-b0af-4c30-b41a-75835331b838-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:50.795633 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.795600 6572 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d9073118-db7d-4711-a22a-39f8de904c10-provider-credentials\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:50.795633 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.795614 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99a74259-b0af-4c30-b41a-75835331b838-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:50.796467 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.796443 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99a74259-b0af-4c30-b41a-75835331b838-istio-podinfo" pod "99a74259-b0af-4c30-b41a-75835331b838" (UID: "99a74259-b0af-4c30-b41a-75835331b838"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:06:50.796636 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.796618 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99a74259-b0af-4c30-b41a-75835331b838-istio-token" pod "99a74259-b0af-4c30-b41a-75835331b838" (UID: "99a74259-b0af-4c30-b41a-75835331b838"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:50.798381 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.798361 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99a74259-b0af-4c30-b41a-75835331b838-kube-api-access-lgqms" pod "99a74259-b0af-4c30-b41a-75835331b838" (UID: "99a74259-b0af-4c30-b41a-75835331b838"). InnerVolumeSpecName "kube-api-access-lgqms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:50.938775 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.926028 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:50.938775 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.933604 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99a74259-b0af-4c30-b41a-75835331b838-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:50.938775 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.938160 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgqms\" (UniqueName: \"kubernetes.io/projected/99a74259-b0af-4c30-b41a-75835331b838-kube-api-access-lgqms\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:50.938775 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:50.938189 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99a74259-b0af-4c30-b41a-75835331b838-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:06:51.336392 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:51.336361 6572 generic.go:356] "Generic (PLEG): container finished" podID="99a74259-b0af-4c30-b41a-75835331b838" containerID="9b1720bdfd7678e36efbccbf9cea31fb49cd331ea9d68d469e23965e80dcd39e" exitCode=0 Oct 30 02:06:51.336920 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:51.336409 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" Oct 30 02:06:51.336920 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:51.336453 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" event={"ID":"99a74259-b0af-4c30-b41a-75835331b838","Type":"ContainerDied","Data":"9b1720bdfd7678e36efbccbf9cea31fb49cd331ea9d68d469e23965e80dcd39e"} Oct 30 02:06:51.336920 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:51.336521 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w" event={"ID":"99a74259-b0af-4c30-b41a-75835331b838","Type":"ContainerDied","Data":"323df7d26b4372961b78499a49ae8c47e9364cbedc44028d7b6d9aa9e82b43f4"} Oct 30 02:06:51.336920 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:51.336560 6572 scope.go:122] "RemoveContainer" containerID="9b1720bdfd7678e36efbccbf9cea31fb49cd331ea9d68d469e23965e80dcd39e" Oct 30 02:06:51.342565 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:51.342537 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79cb95d98d-sps9h" event={"ID":"d9073118-db7d-4711-a22a-39f8de904c10","Type":"ContainerDied","Data":"73ef6401be715f3c498ed9c7aa362cec01dafa0872432231c456511a3f17c500"} Oct 30 02:06:51.342713 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:51.342641 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79cb95d98d-sps9h" Oct 30 02:06:51.351945 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:51.351926 6572 scope.go:122] "RemoveContainer" containerID="9b1720bdfd7678e36efbccbf9cea31fb49cd331ea9d68d469e23965e80dcd39e" Oct 30 02:06:51.352405 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:06:51.352378 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b1720bdfd7678e36efbccbf9cea31fb49cd331ea9d68d469e23965e80dcd39e\": container with ID starting with 9b1720bdfd7678e36efbccbf9cea31fb49cd331ea9d68d469e23965e80dcd39e not found: ID does not exist" containerID="9b1720bdfd7678e36efbccbf9cea31fb49cd331ea9d68d469e23965e80dcd39e" Oct 30 02:06:51.352550 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:51.352412 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b1720bdfd7678e36efbccbf9cea31fb49cd331ea9d68d469e23965e80dcd39e"} err="failed to get container status \"9b1720bdfd7678e36efbccbf9cea31fb49cd331ea9d68d469e23965e80dcd39e\": rpc error: code = NotFound desc = could not find container \"9b1720bdfd7678e36efbccbf9cea31fb49cd331ea9d68d469e23965e80dcd39e\": container with ID starting with 9b1720bdfd7678e36efbccbf9cea31fb49cd331ea9d68d469e23965e80dcd39e not found: ID does not exist" Oct 30 02:06:51.352550 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:51.352434 6572 scope.go:122] "RemoveContainer" containerID="52021ce52ced2a8109a965e62a8a15481d9a5442215795ddd4d76be633fb806c" Oct 30 02:06:51.406232 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:51.406204 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-79cb95d98d-sps9h"] Oct 30 02:06:51.410647 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:51.410621 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-79cb95d98d-sps9h"] Oct 30 02:06:51.453922 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:51.453895 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w"] Oct 30 02:06:51.460150 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:51.460124 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-mp96w"] Oct 30 02:06:51.464493 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:51.464437 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:51.754033 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:51.753948 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:52.545710 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:52.545678 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99a74259-b0af-4c30-b41a-75835331b838" path="/var/lib/kubelet/pods/99a74259-b0af-4c30-b41a-75835331b838/volumes" Oct 30 02:06:52.546144 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:52.546127 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9073118-db7d-4711-a22a-39f8de904c10" path="/var/lib/kubelet/pods/d9073118-db7d-4711-a22a-39f8de904c10/volumes" Oct 30 02:06:52.815191 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:52.814104 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:53.054200 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:53.054162 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70cbc47d-7v8z4"] Oct 30 02:06:53.066914 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:53.066833 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70cbc47d-7v8z4"] Oct 30 02:06:53.164837 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:53.164804 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:53.535781 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:53.532933 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:54.545193 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:54.545161 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfe3389b-960e-45cc-a22a-634e4a64bbcc" path="/var/lib/kubelet/pods/dfe3389b-960e-45cc-a22a-634e4a64bbcc/volumes" Oct 30 02:06:54.637710 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:54.637678 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:56.194769 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:56.194548 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:57.707096 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:57.707057 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:57.990422 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:57.990343 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-56bd9db94-6l572"] Oct 30 02:06:58.574358 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:58.574327 6572 scope.go:122] "RemoveContainer" containerID="e03bd87c6bfb1bf25658f4fee7a82b071f670363bcbb2445ef42f0a8bfa731a9" Oct 30 02:06:58.586476 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:58.586451 6572 scope.go:122] "RemoveContainer" containerID="c624cb2e4813b6439ba0ce52d8bf23d0ceeb9dc743e7a1fd6d02958bc71d7a65" Oct 30 02:06:58.596720 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:58.596702 6572 scope.go:122] "RemoveContainer" containerID="fb52c385c51c09c4f3502cabd8790fc7a2bea5186ba64ed5dd071a2607ae0fca" Oct 30 02:06:58.606874 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:58.606856 6572 scope.go:122] "RemoveContainer" containerID="a16cf0994fb06470a6c97f1c889e39c2d94b628d040081252a1c2ab55449209f" Oct 30 02:06:59.455029 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:59.454997 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:59.661192 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:06:59.661159 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:01.213003 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:01.212963 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:02.803704 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:02.803644 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:02.991489 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:02.991450 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-56bd9db94-6l572" podUID="584d49a8-b8f8-4012-9740-763c8e930126" containerName="payload-processing" containerID="cri-o://01922007307b03523cae1737abdfbb242a524c757e38783c635f9869cc37f763" gracePeriod=25 Oct 30 02:07:03.397664 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:03.397633 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56bd9db94-6l572" Oct 30 02:07:03.417103 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:03.417074 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/584d49a8-b8f8-4012-9740-763c8e930126-provider-credentials\" (UniqueName: \"kubernetes.io/projected/584d49a8-b8f8-4012-9740-763c8e930126-provider-credentials\") pod \"584d49a8-b8f8-4012-9740-763c8e930126\" (UID: \"584d49a8-b8f8-4012-9740-763c8e930126\") " Oct 30 02:07:03.417254 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:03.417219 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/584d49a8-b8f8-4012-9740-763c8e930126-kube-api-access-8jfwg\" (UniqueName: \"kubernetes.io/projected/584d49a8-b8f8-4012-9740-763c8e930126-kube-api-access-8jfwg\") pod \"584d49a8-b8f8-4012-9740-763c8e930126\" (UID: \"584d49a8-b8f8-4012-9740-763c8e930126\") " Oct 30 02:07:03.417310 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:03.417279 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/584d49a8-b8f8-4012-9740-763c8e930126-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/584d49a8-b8f8-4012-9740-763c8e930126-routing-overlay-volume\") pod \"584d49a8-b8f8-4012-9740-763c8e930126\" (UID: \"584d49a8-b8f8-4012-9740-763c8e930126\") " Oct 30 02:07:03.417377 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:03.417308 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/584d49a8-b8f8-4012-9740-763c8e930126-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/584d49a8-b8f8-4012-9740-763c8e930126-plugins-config-volume\") pod \"584d49a8-b8f8-4012-9740-763c8e930126\" (UID: \"584d49a8-b8f8-4012-9740-763c8e930126\") " Oct 30 02:07:03.417679 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:03.417653 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/584d49a8-b8f8-4012-9740-763c8e930126-routing-overlay-volume" pod "584d49a8-b8f8-4012-9740-763c8e930126" (UID: "584d49a8-b8f8-4012-9740-763c8e930126"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:07:03.417798 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:03.417777 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/584d49a8-b8f8-4012-9740-763c8e930126-plugins-config-volume" pod "584d49a8-b8f8-4012-9740-763c8e930126" (UID: "584d49a8-b8f8-4012-9740-763c8e930126"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:07:03.419386 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:03.419360 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/584d49a8-b8f8-4012-9740-763c8e930126-provider-credentials" pod "584d49a8-b8f8-4012-9740-763c8e930126" (UID: "584d49a8-b8f8-4012-9740-763c8e930126"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:07:03.419622 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:03.419596 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/584d49a8-b8f8-4012-9740-763c8e930126-kube-api-access-8jfwg" pod "584d49a8-b8f8-4012-9740-763c8e930126" (UID: "584d49a8-b8f8-4012-9740-763c8e930126"). InnerVolumeSpecName "kube-api-access-8jfwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:07:03.518631 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:03.518595 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jfwg\" (UniqueName: \"kubernetes.io/projected/584d49a8-b8f8-4012-9740-763c8e930126-kube-api-access-8jfwg\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:07:03.518631 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:03.518631 6572 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/584d49a8-b8f8-4012-9740-763c8e930126-routing-overlay-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:07:03.518869 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:03.518645 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/584d49a8-b8f8-4012-9740-763c8e930126-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:07:03.518869 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:03.518659 6572 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/584d49a8-b8f8-4012-9740-763c8e930126-provider-credentials\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:07:03.692878 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:03.688961 6572 generic.go:356] "Generic (PLEG): container finished" podID="584d49a8-b8f8-4012-9740-763c8e930126" containerID="01922007307b03523cae1737abdfbb242a524c757e38783c635f9869cc37f763" exitCode=0 Oct 30 02:07:03.692878 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:03.689016 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56bd9db94-6l572" event={"ID":"584d49a8-b8f8-4012-9740-763c8e930126","Type":"ContainerDied","Data":"01922007307b03523cae1737abdfbb242a524c757e38783c635f9869cc37f763"} Oct 30 02:07:03.692878 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:03.689068 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56bd9db94-6l572" event={"ID":"584d49a8-b8f8-4012-9740-763c8e930126","Type":"ContainerDied","Data":"c78894e8c63e810f028ad8ff54530275f95909ca191420640089bcb817106784"} Oct 30 02:07:03.692878 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:03.689099 6572 scope.go:122] "RemoveContainer" containerID="01922007307b03523cae1737abdfbb242a524c757e38783c635f9869cc37f763" Oct 30 02:07:03.692878 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:03.689264 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56bd9db94-6l572" Oct 30 02:07:03.830447 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:03.829067 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-56bd9db94-6l572"] Oct 30 02:07:03.841988 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:03.840582 6572 scope.go:122] "RemoveContainer" containerID="01922007307b03523cae1737abdfbb242a524c757e38783c635f9869cc37f763" Oct 30 02:07:03.846786 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:03.844559 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-56bd9db94-6l572"] Oct 30 02:07:03.846786 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:07:03.846184 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01922007307b03523cae1737abdfbb242a524c757e38783c635f9869cc37f763\": container with ID starting with 01922007307b03523cae1737abdfbb242a524c757e38783c635f9869cc37f763 not found: ID does not exist" containerID="01922007307b03523cae1737abdfbb242a524c757e38783c635f9869cc37f763" Oct 30 02:07:03.846786 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:03.846222 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01922007307b03523cae1737abdfbb242a524c757e38783c635f9869cc37f763"} err="failed to get container status \"01922007307b03523cae1737abdfbb242a524c757e38783c635f9869cc37f763\": rpc error: code = NotFound desc = could not find container \"01922007307b03523cae1737abdfbb242a524c757e38783c635f9869cc37f763\": container with ID starting with 01922007307b03523cae1737abdfbb242a524c757e38783c635f9869cc37f763 not found: ID does not exist" Oct 30 02:07:04.551451 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:04.551407 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="584d49a8-b8f8-4012-9740-763c8e930126" path="/var/lib/kubelet/pods/584d49a8-b8f8-4012-9740-763c8e930126/volumes" Oct 30 02:07:04.626773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:04.625777 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:06.674016 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:06.672599 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:08.338529 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:08.338493 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:09.818701 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:09.818642 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:11.418325 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:11.418294 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:13.076184 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:13.076112 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:14.695872 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:14.695703 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:14.717191 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:07:14.717159 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:07:16.293451 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:16.293408 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:17.916836 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:17.913157 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:19.504677 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:19.504636 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:21.101805 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:21.101697 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:22.638877 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:22.638830 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:22.826927 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:22.826892 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcefc294-xxdgz"] Oct 30 02:07:22.827639 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:22.827624 6572 cpu_manager.go:414] "containerMap: removing container" podUID="a989accf-0147-4cd4-8d86-a5f769044d7a" containerName="payload-processing" Oct 30 02:07:22.827688 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:22.827641 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a989accf-0147-4cd4-8d86-a5f769044d7a" containerName="payload-processing" Oct 30 02:07:22.827735 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:22.827689 6572 cpu_manager.go:414] "containerMap: removing container" podUID="d9073118-db7d-4711-a22a-39f8de904c10" containerName="payload-processing" Oct 30 02:07:22.827735 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:22.827694 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9073118-db7d-4711-a22a-39f8de904c10" containerName="payload-processing" Oct 30 02:07:22.827735 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:22.827709 6572 cpu_manager.go:414] "containerMap: removing container" podUID="99a74259-b0af-4c30-b41a-75835331b838" containerName="istio-proxy" Oct 30 02:07:22.827735 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:22.827714 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99a74259-b0af-4c30-b41a-75835331b838" containerName="istio-proxy" Oct 30 02:07:22.827735 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:22.827726 6572 cpu_manager.go:414] "containerMap: removing container" podUID="584d49a8-b8f8-4012-9740-763c8e930126" containerName="payload-processing" Oct 30 02:07:22.827735 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:22.827731 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="584d49a8-b8f8-4012-9740-763c8e930126" containerName="payload-processing" Oct 30 02:07:22.827947 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:22.827859 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9073118-db7d-4711-a22a-39f8de904c10" containerName="payload-processing" Oct 30 02:07:22.827947 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:22.827870 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="99a74259-b0af-4c30-b41a-75835331b838" containerName="istio-proxy" Oct 30 02:07:22.827947 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:22.827878 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="584d49a8-b8f8-4012-9740-763c8e930126" containerName="payload-processing" Oct 30 02:07:22.827947 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:22.827888 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="a989accf-0147-4cd4-8d86-a5f769044d7a" containerName="payload-processing" Oct 30 02:07:22.831566 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:22.831549 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcefc294-xxdgz" Oct 30 02:07:22.850056 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:22.850022 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcefc294-xxdgz"] Oct 30 02:07:22.856263 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:22.856235 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:07:22.856407 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:22.856263 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:07:22.856407 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:22.856280 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:07:22.857007 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:22.856987 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:07:22.900232 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:22.900147 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml8hd\" (UniqueName: \"kubernetes.io/projected/db0796b5-2d24-4831-9d44-d1b1f5bfd792-kube-api-access-ml8hd\") pod \"maas-api-revoke-sub-bcefc294-xxdgz\" (UID: \"db0796b5-2d24-4831-9d44-d1b1f5bfd792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcefc294-xxdgz" Oct 30 02:07:22.900408 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:22.900300 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db0796b5-2d24-4831-9d44-d1b1f5bfd792-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcefc294-xxdgz\" (UID: \"db0796b5-2d24-4831-9d44-d1b1f5bfd792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcefc294-xxdgz" Oct 30 02:07:22.900579 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:22.900551 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db0796b5-2d24-4831-9d44-d1b1f5bfd792-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcefc294-xxdgz\" (UID: \"db0796b5-2d24-4831-9d44-d1b1f5bfd792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcefc294-xxdgz" Oct 30 02:07:23.001908 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:23.001876 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml8hd\" (UniqueName: \"kubernetes.io/projected/db0796b5-2d24-4831-9d44-d1b1f5bfd792-kube-api-access-ml8hd\") pod \"maas-api-revoke-sub-bcefc294-xxdgz\" (UID: \"db0796b5-2d24-4831-9d44-d1b1f5bfd792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcefc294-xxdgz" Oct 30 02:07:23.002095 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:23.001945 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db0796b5-2d24-4831-9d44-d1b1f5bfd792-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcefc294-xxdgz\" (UID: \"db0796b5-2d24-4831-9d44-d1b1f5bfd792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcefc294-xxdgz" Oct 30 02:07:23.002095 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:23.002085 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db0796b5-2d24-4831-9d44-d1b1f5bfd792-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcefc294-xxdgz\" (UID: \"db0796b5-2d24-4831-9d44-d1b1f5bfd792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcefc294-xxdgz" Oct 30 02:07:23.002815 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:23.002795 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db0796b5-2d24-4831-9d44-d1b1f5bfd792-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcefc294-xxdgz\" (UID: \"db0796b5-2d24-4831-9d44-d1b1f5bfd792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcefc294-xxdgz" Oct 30 02:07:23.004537 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:23.004513 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db0796b5-2d24-4831-9d44-d1b1f5bfd792-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcefc294-xxdgz\" (UID: \"db0796b5-2d24-4831-9d44-d1b1f5bfd792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcefc294-xxdgz" Oct 30 02:07:23.021368 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:23.021342 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml8hd\" (UniqueName: \"kubernetes.io/projected/db0796b5-2d24-4831-9d44-d1b1f5bfd792-kube-api-access-ml8hd\") pod \"maas-api-revoke-sub-bcefc294-xxdgz\" (UID: \"db0796b5-2d24-4831-9d44-d1b1f5bfd792\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcefc294-xxdgz" Oct 30 02:07:23.146948 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:23.146917 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcefc294-xxdgz" Oct 30 02:07:23.447938 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:23.447909 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcefc294-xxdgz"] Oct 30 02:07:23.448585 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:07:23.448565 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb0796b5_2d24_4831_9d44_d1b1f5bfd792.slice/crio-f05570e7fe34afe66fbaa2861a199a6ff47a48ac45ee7de4cdec873b400a3b6a WatchSource:0}: Error finding container f05570e7fe34afe66fbaa2861a199a6ff47a48ac45ee7de4cdec873b400a3b6a: Status 404 returned error can't find the container with id f05570e7fe34afe66fbaa2861a199a6ff47a48ac45ee7de4cdec873b400a3b6a Oct 30 02:07:23.976709 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:23.976600 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcefc294-xxdgz" event={"ID":"db0796b5-2d24-4831-9d44-d1b1f5bfd792","Type":"ContainerStarted","Data":"b622469261665873552005e23154592dfe9c2b6c6e25a0178286c78aba235f58"} Oct 30 02:07:23.977290 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:23.976718 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcefc294-xxdgz" event={"ID":"db0796b5-2d24-4831-9d44-d1b1f5bfd792","Type":"ContainerStarted","Data":"f05570e7fe34afe66fbaa2861a199a6ff47a48ac45ee7de4cdec873b400a3b6a"} Oct 30 02:07:24.020396 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:24.020341 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcefc294-xxdgz" podStartSLOduration=2.020326376 podStartE2EDuration="2.020326376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:07:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:07:24.014863514 +0000 UTC m=+2969.591288916" watchObservedRunningTime="2026-10-30 02:07:24.020326376 +0000 UTC m=+2969.596751774" Oct 30 02:07:24.159148 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:24.158403 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:24.992133 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:24.992084 6572 generic.go:356] "Generic (PLEG): container finished" podID="db0796b5-2d24-4831-9d44-d1b1f5bfd792" containerID="b622469261665873552005e23154592dfe9c2b6c6e25a0178286c78aba235f58" exitCode=0 Oct 30 02:07:24.992722 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:24.992170 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcefc294-xxdgz" event={"ID":"db0796b5-2d24-4831-9d44-d1b1f5bfd792","Type":"ContainerDied","Data":"b622469261665873552005e23154592dfe9c2b6c6e25a0178286c78aba235f58"} Oct 30 02:07:25.797125 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:25.797084 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:26.179534 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:26.179505 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcefc294-xxdgz" Oct 30 02:07:26.257694 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:26.257641 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db0796b5-2d24-4831-9d44-d1b1f5bfd792-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db0796b5-2d24-4831-9d44-d1b1f5bfd792-maas-api-openshift-ca\") pod \"db0796b5-2d24-4831-9d44-d1b1f5bfd792\" (UID: \"db0796b5-2d24-4831-9d44-d1b1f5bfd792\") " Oct 30 02:07:26.257906 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:26.257719 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db0796b5-2d24-4831-9d44-d1b1f5bfd792-kube-api-access-ml8hd\" (UniqueName: \"kubernetes.io/projected/db0796b5-2d24-4831-9d44-d1b1f5bfd792-kube-api-access-ml8hd\") pod \"db0796b5-2d24-4831-9d44-d1b1f5bfd792\" (UID: \"db0796b5-2d24-4831-9d44-d1b1f5bfd792\") " Oct 30 02:07:26.258077 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:26.258047 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db0796b5-2d24-4831-9d44-d1b1f5bfd792-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db0796b5-2d24-4831-9d44-d1b1f5bfd792-maas-api-xks-ca\") pod \"db0796b5-2d24-4831-9d44-d1b1f5bfd792\" (UID: \"db0796b5-2d24-4831-9d44-d1b1f5bfd792\") " Oct 30 02:07:26.258149 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:26.258118 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db0796b5-2d24-4831-9d44-d1b1f5bfd792-maas-api-openshift-ca" pod "db0796b5-2d24-4831-9d44-d1b1f5bfd792" (UID: "db0796b5-2d24-4831-9d44-d1b1f5bfd792"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:07:26.258812 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:26.258786 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db0796b5-2d24-4831-9d44-d1b1f5bfd792-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:07:26.260837 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:26.260807 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db0796b5-2d24-4831-9d44-d1b1f5bfd792-kube-api-access-ml8hd" pod "db0796b5-2d24-4831-9d44-d1b1f5bfd792" (UID: "db0796b5-2d24-4831-9d44-d1b1f5bfd792"). InnerVolumeSpecName "kube-api-access-ml8hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:07:26.260837 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:26.260819 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db0796b5-2d24-4831-9d44-d1b1f5bfd792-maas-api-xks-ca" pod "db0796b5-2d24-4831-9d44-d1b1f5bfd792" (UID: "db0796b5-2d24-4831-9d44-d1b1f5bfd792"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:07:26.360458 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:26.360365 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db0796b5-2d24-4831-9d44-d1b1f5bfd792-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:07:26.360458 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:26.360399 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml8hd\" (UniqueName: \"kubernetes.io/projected/db0796b5-2d24-4831-9d44-d1b1f5bfd792-kube-api-access-ml8hd\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:07:27.010999 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:27.010967 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcefc294-xxdgz" event={"ID":"db0796b5-2d24-4831-9d44-d1b1f5bfd792","Type":"ContainerDied","Data":"f05570e7fe34afe66fbaa2861a199a6ff47a48ac45ee7de4cdec873b400a3b6a"} Oct 30 02:07:27.011197 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:27.011004 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcefc294-xxdgz" Oct 30 02:07:27.011197 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:27.011021 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f05570e7fe34afe66fbaa2861a199a6ff47a48ac45ee7de4cdec873b400a3b6a" Oct 30 02:07:27.444329 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:27.444293 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:29.046980 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:29.046949 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:30.619529 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:30.619492 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:32.203392 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:32.203354 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:33.823085 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:33.822987 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:34.109113 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:34.109016 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:34.386822 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:34.386699 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:34.823124 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:34.823076 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-64d945b46f-8fq9n"] Oct 30 02:07:34.832787 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:34.829134 6572 cpu_manager.go:414] "containerMap: removing container" podUID="db0796b5-2d24-4831-9d44-d1b1f5bfd792" containerName="revoke-keys" Oct 30 02:07:34.833091 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:34.833059 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db0796b5-2d24-4831-9d44-d1b1f5bfd792" containerName="revoke-keys" Oct 30 02:07:34.833552 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:34.833529 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="db0796b5-2d24-4831-9d44-d1b1f5bfd792" containerName="revoke-keys" Oct 30 02:07:34.839218 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:34.839189 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-64d945b46f-8fq9n" Oct 30 02:07:34.867349 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:34.864500 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-84d872aa7a07\"" Oct 30 02:07:34.872789 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:34.867848 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-64d945b46f-8fq9n"] Oct 30 02:07:34.876438 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:34.876082 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-nfgtq\"" Oct 30 02:07:34.876438 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:34.876214 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 02:07:34.876438 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:34.876271 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 02:07:34.876438 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:34.876339 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 02:07:34.876438 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:34.876390 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 02:07:34.969899 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:34.969857 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db-provider-credentials\") pod \"payload-processing-external-model-64d945b46f-8fq9n\" (UID: \"54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db\") " pod="models-as-a-service/payload-processing-external-model-64d945b46f-8fq9n" Oct 30 02:07:34.970119 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:34.969992 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsgpm\" (UniqueName: \"kubernetes.io/projected/54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db-kube-api-access-gsgpm\") pod \"payload-processing-external-model-64d945b46f-8fq9n\" (UID: \"54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db\") " pod="models-as-a-service/payload-processing-external-model-64d945b46f-8fq9n" Oct 30 02:07:34.970119 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:34.970043 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db-plugins-config-volume\") pod \"payload-processing-external-model-64d945b46f-8fq9n\" (UID: \"54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db\") " pod="models-as-a-service/payload-processing-external-model-64d945b46f-8fq9n" Oct 30 02:07:34.970698 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:34.970252 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db-routing-overlay-volume\") pod \"payload-processing-external-model-64d945b46f-8fq9n\" (UID: \"54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db\") " pod="models-as-a-service/payload-processing-external-model-64d945b46f-8fq9n" Oct 30 02:07:35.072256 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:35.072195 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db-routing-overlay-volume\") pod \"payload-processing-external-model-64d945b46f-8fq9n\" (UID: \"54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db\") " pod="models-as-a-service/payload-processing-external-model-64d945b46f-8fq9n" Oct 30 02:07:35.072473 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:35.072374 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db-provider-credentials\") pod \"payload-processing-external-model-64d945b46f-8fq9n\" (UID: \"54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db\") " pod="models-as-a-service/payload-processing-external-model-64d945b46f-8fq9n" Oct 30 02:07:35.072473 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:35.072448 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsgpm\" (UniqueName: \"kubernetes.io/projected/54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db-kube-api-access-gsgpm\") pod \"payload-processing-external-model-64d945b46f-8fq9n\" (UID: \"54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db\") " pod="models-as-a-service/payload-processing-external-model-64d945b46f-8fq9n" Oct 30 02:07:35.072587 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:35.072487 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db-plugins-config-volume\") pod \"payload-processing-external-model-64d945b46f-8fq9n\" (UID: \"54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db\") " pod="models-as-a-service/payload-processing-external-model-64d945b46f-8fq9n" Oct 30 02:07:35.073172 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:35.073103 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db-routing-overlay-volume\") pod \"payload-processing-external-model-64d945b46f-8fq9n\" (UID: \"54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db\") " pod="models-as-a-service/payload-processing-external-model-64d945b46f-8fq9n" Oct 30 02:07:35.073432 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:35.073405 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db-plugins-config-volume\") pod \"payload-processing-external-model-64d945b46f-8fq9n\" (UID: \"54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db\") " pod="models-as-a-service/payload-processing-external-model-64d945b46f-8fq9n" Oct 30 02:07:35.075314 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:35.075290 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db-provider-credentials\") pod \"payload-processing-external-model-64d945b46f-8fq9n\" (UID: \"54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db\") " pod="models-as-a-service/payload-processing-external-model-64d945b46f-8fq9n" Oct 30 02:07:35.109710 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:35.109673 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsgpm\" (UniqueName: \"kubernetes.io/projected/54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db-kube-api-access-gsgpm\") pod \"payload-processing-external-model-64d945b46f-8fq9n\" (UID: \"54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db\") " pod="models-as-a-service/payload-processing-external-model-64d945b46f-8fq9n" Oct 30 02:07:35.181069 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:35.181027 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-64d945b46f-8fq9n" Oct 30 02:07:35.533544 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:07:35.533504 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54b0c2a9_5229_42d1_83a2_8e6ccc7aa2db.slice/crio-ae6d283d0ee820dc073890b557709e3b818c6386120d4520e604deed586449cd WatchSource:0}: Error finding container ae6d283d0ee820dc073890b557709e3b818c6386120d4520e604deed586449cd: Status 404 returned error can't find the container with id ae6d283d0ee820dc073890b557709e3b818c6386120d4520e604deed586449cd Oct 30 02:07:35.534181 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:35.534150 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-64d945b46f-8fq9n"] Oct 30 02:07:36.021882 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:36.021845 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:36.109949 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:36.109896 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-64d945b46f-8fq9n" event={"ID":"54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db","Type":"ContainerStarted","Data":"126c10a442941063352accc0a118390659b679d6da39fba5ccf24759a9cc19eb"} Oct 30 02:07:36.110147 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:36.109967 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-64d945b46f-8fq9n" event={"ID":"54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db","Type":"ContainerStarted","Data":"ae6d283d0ee820dc073890b557709e3b818c6386120d4520e604deed586449cd"} Oct 30 02:07:36.110147 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:36.110138 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-64d945b46f-8fq9n" Oct 30 02:07:36.229774 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:36.217118 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-64d945b46f-8fq9n" podStartSLOduration=2.217093367 podStartE2EDuration="2.217093367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:07:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:07:36.177195136 +0000 UTC m=+2981.753620539" watchObservedRunningTime="2026-10-30 02:07:36.217093367 +0000 UTC m=+2981.793518771" Oct 30 02:07:36.282830 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:36.274318 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:36.492144 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:36.492097 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:36.779769 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:36.779166 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:37.353770 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:37.353537 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:37.579774 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:37.578469 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:38.096451 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:38.096408 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:38.747779 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:38.747312 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:38.842885 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:38.842493 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt"] Oct 30 02:07:38.851601 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:38.851573 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt" Oct 30 02:07:38.856345 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:38.856315 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt"] Oct 30 02:07:38.874855 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:38.871731 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-92f0cc\"/\"agc-cred-73f3254bcb73\"" Oct 30 02:07:38.874855 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:38.871757 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-92f0cc\"/\"routing-overlay\"" Oct 30 02:07:38.878812 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:38.878785 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-92f0cc\"/\"payload-processing-external-model-plugins-e2e-extmod-a-92f0cc\"" Oct 30 02:07:38.878934 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:38.878839 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-92f0cc\"/\"openshift-service-ca.crt\"" Oct 30 02:07:38.878934 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:38.878883 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-92f0cc\"/\"payload-processing-external-model-e2e--c7e40432-dockercfg-vk62m\"" Oct 30 02:07:38.879067 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:38.878937 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-92f0cc\"/\"kube-root-ca.crt\"" Oct 30 02:07:39.011836 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:39.011457 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/87500d13-55e9-4243-b3f9-617d55ce4202-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt\" (UID: \"87500d13-55e9-4243-b3f9-617d55ce4202\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt" Oct 30 02:07:39.011836 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:39.011785 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87500d13-55e9-4243-b3f9-617d55ce4202-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt\" (UID: \"87500d13-55e9-4243-b3f9-617d55ce4202\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt" Oct 30 02:07:39.012115 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:39.011901 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87500d13-55e9-4243-b3f9-617d55ce4202-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt\" (UID: \"87500d13-55e9-4243-b3f9-617d55ce4202\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt" Oct 30 02:07:39.012115 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:39.012005 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvnds\" (UniqueName: \"kubernetes.io/projected/87500d13-55e9-4243-b3f9-617d55ce4202-kube-api-access-lvnds\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt\" (UID: \"87500d13-55e9-4243-b3f9-617d55ce4202\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt" Oct 30 02:07:39.118773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:39.118238 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87500d13-55e9-4243-b3f9-617d55ce4202-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt\" (UID: \"87500d13-55e9-4243-b3f9-617d55ce4202\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt" Oct 30 02:07:39.143774 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:39.119626 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87500d13-55e9-4243-b3f9-617d55ce4202-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt\" (UID: \"87500d13-55e9-4243-b3f9-617d55ce4202\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt" Oct 30 02:07:39.143774 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:39.120457 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87500d13-55e9-4243-b3f9-617d55ce4202-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt\" (UID: \"87500d13-55e9-4243-b3f9-617d55ce4202\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt" Oct 30 02:07:39.143774 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:39.119741 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87500d13-55e9-4243-b3f9-617d55ce4202-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt\" (UID: \"87500d13-55e9-4243-b3f9-617d55ce4202\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt" Oct 30 02:07:39.143774 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:39.120607 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvnds\" (UniqueName: \"kubernetes.io/projected/87500d13-55e9-4243-b3f9-617d55ce4202-kube-api-access-lvnds\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt\" (UID: \"87500d13-55e9-4243-b3f9-617d55ce4202\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt" Oct 30 02:07:39.143774 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:39.120806 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/87500d13-55e9-4243-b3f9-617d55ce4202-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt\" (UID: \"87500d13-55e9-4243-b3f9-617d55ce4202\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt" Oct 30 02:07:39.160131 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:39.158722 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvnds\" (UniqueName: \"kubernetes.io/projected/87500d13-55e9-4243-b3f9-617d55ce4202-kube-api-access-lvnds\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt\" (UID: \"87500d13-55e9-4243-b3f9-617d55ce4202\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt" Oct 30 02:07:39.183773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:39.173306 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/87500d13-55e9-4243-b3f9-617d55ce4202-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt\" (UID: \"87500d13-55e9-4243-b3f9-617d55ce4202\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt" Oct 30 02:07:39.183773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:39.173576 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt" Oct 30 02:07:39.537204 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:39.537124 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt"] Oct 30 02:07:39.537560 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:07:39.537535 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87500d13_55e9_4243_b3f9_617d55ce4202.slice/crio-b825b19ec567504d9d080da2c8e9f6de945ca5a02d68ed32ad98fa16dcd5fdd2 WatchSource:0}: Error finding container b825b19ec567504d9d080da2c8e9f6de945ca5a02d68ed32ad98fa16dcd5fdd2: Status 404 returned error can't find the container with id b825b19ec567504d9d080da2c8e9f6de945ca5a02d68ed32ad98fa16dcd5fdd2 Oct 30 02:07:39.681861 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:39.681826 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:40.188829 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:40.187367 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:40.343385 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:40.343211 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt" event={"ID":"87500d13-55e9-4243-b3f9-617d55ce4202","Type":"ContainerStarted","Data":"7d3cf9e67d2d884b0a506acc8748b743a25c426531cf173aa983fc003105bf81"} Oct 30 02:07:40.343385 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:40.343284 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt" event={"ID":"87500d13-55e9-4243-b3f9-617d55ce4202","Type":"ContainerStarted","Data":"b825b19ec567504d9d080da2c8e9f6de945ca5a02d68ed32ad98fa16dcd5fdd2"} Oct 30 02:07:40.357778 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:40.354679 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt" Oct 30 02:07:40.404198 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:40.402682 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt" podStartSLOduration=2.402664381 podStartE2EDuration="2.402664381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:07:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:07:40.401236715 +0000 UTC m=+2985.977662118" watchObservedRunningTime="2026-10-30 02:07:40.402664381 +0000 UTC m=+2985.979089785" Oct 30 02:07:40.428781 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:40.425634 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:40.632262 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:40.632166 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:42.053452 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:42.053405 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:43.604819 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:43.604784 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:44.688086 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:07:44.688048 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:07:45.191461 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:45.191365 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:46.787506 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:46.787472 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:47.122114 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:47.122024 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-64d945b46f-8fq9n" Oct 30 02:07:48.419205 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:48.419166 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:49.457141 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:49.457106 6572 generic.go:356] "Generic (PLEG): container finished" podID="ee995a27-1255-4c9c-88c4-c6e4eb58a77d" containerID="e12add2d220062db183af8b624107420e07785b347a77a42019f4da8c03eff4c" exitCode=0 Oct 30 02:07:49.457543 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:49.457179 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" event={"ID":"ee995a27-1255-4c9c-88c4-c6e4eb58a77d","Type":"ContainerDied","Data":"e12add2d220062db183af8b624107420e07785b347a77a42019f4da8c03eff4c"} Oct 30 02:07:49.457671 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:49.457655 6572 scope.go:122] "RemoveContainer" containerID="e12add2d220062db183af8b624107420e07785b347a77a42019f4da8c03eff4c" Oct 30 02:07:49.459783 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:49.459729 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:07:49.459870 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:49.459829 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:07:49.459870 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:49.459863 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:07:50.107701 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:50.107646 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:50.469806 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:50.469771 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" event={"ID":"ee995a27-1255-4c9c-88c4-c6e4eb58a77d","Type":"ContainerStarted","Data":"62afb4d5e249e315edbcb3f4fff9df302ef2ea37a401bf2d1fea6e47542ec553"} Oct 30 02:07:50.736185 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:50.736101 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:07:50.738570 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:50.738533 6572 patch_prober.go:29] interesting pod/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused" start-of-body= Oct 30 02:07:50.738803 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:50.738590 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" podUID="ee995a27-1255-4c9c-88c4-c6e4eb58a77d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused" Oct 30 02:07:51.745008 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:51.744970 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:51.750580 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:51.747201 6572 patch_prober.go:29] interesting pod/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused" start-of-body= Oct 30 02:07:51.751118 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:51.751086 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" podUID="ee995a27-1255-4c9c-88c4-c6e4eb58a77d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused" Oct 30 02:07:52.394274 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:52.394242 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt" Oct 30 02:07:52.739959 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:52.739918 6572 patch_prober.go:29] interesting pod/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused" start-of-body= Oct 30 02:07:52.740316 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:52.740278 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" podUID="ee995a27-1255-4c9c-88c4-c6e4eb58a77d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused" Oct 30 02:07:53.546494 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:53.546453 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:53.736336 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:53.736295 6572 patch_prober.go:29] interesting pod/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused" start-of-body= Oct 30 02:07:53.736518 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:53.736354 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" podUID="ee995a27-1255-4c9c-88c4-c6e4eb58a77d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused" Oct 30 02:07:54.082728 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:54.082692 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:54.736028 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:54.735997 6572 patch_prober.go:29] interesting pod/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused" start-of-body= Oct 30 02:07:54.736443 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:54.736069 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" podUID="ee995a27-1255-4c9c-88c4-c6e4eb58a77d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused" Oct 30 02:07:55.032115 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:55.027033 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:07:55.202937 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:55.202891 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:55.431208 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:55.431167 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:55.736467 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:55.736373 6572 patch_prober.go:29] interesting pod/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused" start-of-body= Oct 30 02:07:55.736467 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:55.736434 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" podUID="ee995a27-1255-4c9c-88c4-c6e4eb58a77d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused" Oct 30 02:07:56.781129 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:56.781090 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:07:56.784285 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:56.782956 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:07:56.904769 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:56.903912 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:58.504692 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:58.504650 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:58.701611 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:58.701575 6572 scope.go:122] "RemoveContainer" containerID="7296d4b88560babfcab03c1eb67e8fba9c9f8ff3ec6e84ddb528dcfee89f161e" Oct 30 02:07:58.716947 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:58.716920 6572 scope.go:122] "RemoveContainer" containerID="155596eb43aee35ea78b451d49cec556135657f20aa015270f254909dceee0d2" Oct 30 02:07:58.728426 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:07:58.728405 6572 scope.go:122] "RemoveContainer" containerID="a33c844c7622991d252f438cbb9b069895ef8c33ef29c79086e68696f777de02" Oct 30 02:08:00.088054 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:00.088019 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:01.919575 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:01.919538 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:03.797769 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:03.793371 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:05.813917 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:05.813867 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" podUID="9a56da7a-9b92-40e2-b08b-43cf591b5221" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:08:05.907982 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:05.907936 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" podUID="317505ef-238c-458e-a74e-1198806b94d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 02:08:05.955904 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:05.955852 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" podUID="e76339df-9659-4bea-80a4-5f18c4a73fd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:08:05.975119 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:05.975072 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" podUID="cb20484a-51e2-4581-a629-e92d4234784e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:08:06.117674 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:06.117499 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:06.313125 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:06.311855 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" podUID="fac3ca55-64af-42d9-90b1-0e821b893282" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:08:06.832253 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:06.832206 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" podUID="9a56da7a-9b92-40e2-b08b-43cf591b5221" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 02:08:06.923774 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:06.902803 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" Oct 30 02:08:06.923774 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:06.923118 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" podUID="317505ef-238c-458e-a74e-1198806b94d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 02:08:06.923774 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:06.923629 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" Oct 30 02:08:07.176848 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:07.151090 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" podUID="e76339df-9659-4bea-80a4-5f18c4a73fd6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 30 02:08:07.526632 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:07.526543 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" Oct 30 02:08:07.614151 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:07.532495 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" podUID="0ee3f8e3-9b55-4ce5-afaf-86780a26391e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 30 02:08:07.633774 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:07.621019 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" Oct 30 02:08:07.633774 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:07.621082 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" Oct 30 02:08:07.666279 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:07.666249 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" Oct 30 02:08:07.843309 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:07.840454 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:08.053638 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:08.053601 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:09.483841 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:09.483710 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:11.206193 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:11.206158 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:12.714219 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:12.711637 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:14.493848 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:14.493688 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:14.604089 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:14.604053 6572 generic.go:356] "Generic (PLEG): container finished" podID="166123b7-cdef-49d7-8e6c-2a2a993b9bb8" containerID="1b1062c93a39d9f873c2611743f1ba3423beb5cc68ddf14d4b57baa66761cd63" exitCode=0 Oct 30 02:08:14.604286 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:14.604119 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" event={"ID":"166123b7-cdef-49d7-8e6c-2a2a993b9bb8","Type":"ContainerDied","Data":"1b1062c93a39d9f873c2611743f1ba3423beb5cc68ddf14d4b57baa66761cd63"} Oct 30 02:08:14.604603 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:14.604589 6572 scope.go:122] "RemoveContainer" containerID="1b1062c93a39d9f873c2611743f1ba3423beb5cc68ddf14d4b57baa66761cd63" Oct 30 02:08:14.606701 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:14.606664 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:08:14.606804 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:14.606762 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:08:14.606852 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:14.606802 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:08:14.706945 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:08:14.706897 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:08:15.163527 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:15.163487 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:08:15.638070 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:15.637971 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" event={"ID":"166123b7-cdef-49d7-8e6c-2a2a993b9bb8","Type":"ContainerStarted","Data":"1edaf94c752d86ca6cb8f395b33fcd7f8ba41d4f91cb5bc654b4cbc9c6d1fd4e"} Oct 30 02:08:16.268571 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:16.268462 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:16.317774 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:16.317710 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:08:16.317997 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:16.317943 6572 patch_prober.go:29] interesting pod/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.88:15021/healthz/ready\": dial tcp 10.133.1.88:15021: connect: connection refused" start-of-body= Oct 30 02:08:16.318078 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:16.317996 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" podUID="166123b7-cdef-49d7-8e6c-2a2a993b9bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.88:15021/healthz/ready\": dial tcp 10.133.1.88:15021: connect: connection refused" Oct 30 02:08:17.319772 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:17.319280 6572 patch_prober.go:29] interesting pod/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.88:15021/healthz/ready\": dial tcp 10.133.1.88:15021: connect: connection refused" start-of-body= Oct 30 02:08:17.319772 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:17.319363 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" podUID="166123b7-cdef-49d7-8e6c-2a2a993b9bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.88:15021/healthz/ready\": dial tcp 10.133.1.88:15021: connect: connection refused" Oct 30 02:08:17.821719 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:17.821575 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:18.321771 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:18.321232 6572 patch_prober.go:29] interesting pod/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.88:15021/healthz/ready\": dial tcp 10.133.1.88:15021: connect: connection refused" start-of-body= Oct 30 02:08:18.321771 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:18.321296 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" podUID="166123b7-cdef-49d7-8e6c-2a2a993b9bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.88:15021/healthz/ready\": dial tcp 10.133.1.88:15021: connect: connection refused" Oct 30 02:08:18.603829 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:18.603707 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:19.325775 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:19.321249 6572 patch_prober.go:29] interesting pod/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.88:15021/healthz/ready\": dial tcp 10.133.1.88:15021: connect: connection refused" start-of-body= Oct 30 02:08:19.325775 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:19.321334 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" podUID="166123b7-cdef-49d7-8e6c-2a2a993b9bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.88:15021/healthz/ready\": dial tcp 10.133.1.88:15021: connect: connection refused" Oct 30 02:08:19.574149 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:19.574113 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:20.317783 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:20.316121 6572 patch_prober.go:29] interesting pod/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.88:15021/healthz/ready\": dial tcp 10.133.1.88:15021: connect: connection refused" start-of-body= Oct 30 02:08:20.317783 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:20.316183 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" podUID="166123b7-cdef-49d7-8e6c-2a2a993b9bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.88:15021/healthz/ready\": dial tcp 10.133.1.88:15021: connect: connection refused" Oct 30 02:08:21.206095 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:21.206056 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:21.319085 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:21.319046 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:08:21.319607 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:21.319590 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:08:21.320429 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:21.320407 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:08:22.820773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:22.814630 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:23.643204 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:23.643170 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8167a986-mx528"] Oct 30 02:08:23.647278 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:23.647255 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8167a986-mx528" Oct 30 02:08:23.669228 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:23.669199 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8167a986-mx528"] Oct 30 02:08:23.673306 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:23.673279 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:08:23.673645 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:23.673377 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:08:23.673645 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:23.673498 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:08:23.673645 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:23.673539 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:08:23.685775 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:23.685725 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42xst\" (UniqueName: \"kubernetes.io/projected/0dd2d590-698e-4769-800f-55d530d7ca2a-kube-api-access-42xst\") pod \"maas-api-revoke-sub-8167a986-mx528\" (UID: \"0dd2d590-698e-4769-800f-55d530d7ca2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8167a986-mx528" Oct 30 02:08:23.686191 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:23.686163 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd2d590-698e-4769-800f-55d530d7ca2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8167a986-mx528\" (UID: \"0dd2d590-698e-4769-800f-55d530d7ca2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8167a986-mx528" Oct 30 02:08:23.686435 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:23.686414 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd2d590-698e-4769-800f-55d530d7ca2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8167a986-mx528\" (UID: \"0dd2d590-698e-4769-800f-55d530d7ca2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8167a986-mx528" Oct 30 02:08:23.788332 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:23.788292 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42xst\" (UniqueName: \"kubernetes.io/projected/0dd2d590-698e-4769-800f-55d530d7ca2a-kube-api-access-42xst\") pod \"maas-api-revoke-sub-8167a986-mx528\" (UID: \"0dd2d590-698e-4769-800f-55d530d7ca2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8167a986-mx528" Oct 30 02:08:23.788543 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:23.788497 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd2d590-698e-4769-800f-55d530d7ca2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8167a986-mx528\" (UID: \"0dd2d590-698e-4769-800f-55d530d7ca2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8167a986-mx528" Oct 30 02:08:23.788623 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:23.788607 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd2d590-698e-4769-800f-55d530d7ca2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8167a986-mx528\" (UID: \"0dd2d590-698e-4769-800f-55d530d7ca2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8167a986-mx528" Oct 30 02:08:23.789485 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:23.789455 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd2d590-698e-4769-800f-55d530d7ca2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8167a986-mx528\" (UID: \"0dd2d590-698e-4769-800f-55d530d7ca2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8167a986-mx528" Oct 30 02:08:23.791261 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:23.791235 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd2d590-698e-4769-800f-55d530d7ca2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8167a986-mx528\" (UID: \"0dd2d590-698e-4769-800f-55d530d7ca2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8167a986-mx528" Oct 30 02:08:23.807899 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:23.807871 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42xst\" (UniqueName: \"kubernetes.io/projected/0dd2d590-698e-4769-800f-55d530d7ca2a-kube-api-access-42xst\") pod \"maas-api-revoke-sub-8167a986-mx528\" (UID: \"0dd2d590-698e-4769-800f-55d530d7ca2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8167a986-mx528" Oct 30 02:08:23.961791 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:23.961760 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8167a986-mx528" Oct 30 02:08:24.276484 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:08:24.276441 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd2d590_698e_4769_800f_55d530d7ca2a.slice/crio-4f730e2da73ebffa6d1456e9799762cafb752ff868f185226ecbfbefee3559e1 WatchSource:0}: Error finding container 4f730e2da73ebffa6d1456e9799762cafb752ff868f185226ecbfbefee3559e1: Status 404 returned error can't find the container with id 4f730e2da73ebffa6d1456e9799762cafb752ff868f185226ecbfbefee3559e1 Oct 30 02:08:24.277144 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:24.277121 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8167a986-mx528"] Oct 30 02:08:24.775094 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:24.774993 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8167a986-mx528" event={"ID":"0dd2d590-698e-4769-800f-55d530d7ca2a","Type":"ContainerStarted","Data":"ad8b6ba999d4e6c37f44f4539621a51987e497a3e485bda1ad10c4dc627b218d"} Oct 30 02:08:24.775094 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:24.775055 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8167a986-mx528" event={"ID":"0dd2d590-698e-4769-800f-55d530d7ca2a","Type":"ContainerStarted","Data":"4f730e2da73ebffa6d1456e9799762cafb752ff868f185226ecbfbefee3559e1"} Oct 30 02:08:24.815281 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:24.815226 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8167a986-mx528" podStartSLOduration=1.815209523 podStartE2EDuration="1.815209523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:08:24.813430822 +0000 UTC m=+3030.389856223" watchObservedRunningTime="2026-10-30 02:08:24.815209523 +0000 UTC m=+3030.391634929" Oct 30 02:08:25.396005 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:25.395963 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:25.786839 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:25.786800 6572 generic.go:356] "Generic (PLEG): container finished" podID="0dd2d590-698e-4769-800f-55d530d7ca2a" containerID="ad8b6ba999d4e6c37f44f4539621a51987e497a3e485bda1ad10c4dc627b218d" exitCode=0 Oct 30 02:08:25.787038 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:25.786879 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8167a986-mx528" event={"ID":"0dd2d590-698e-4769-800f-55d530d7ca2a","Type":"ContainerDied","Data":"ad8b6ba999d4e6c37f44f4539621a51987e497a3e485bda1ad10c4dc627b218d"} Oct 30 02:08:26.006150 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:26.006114 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:26.989089 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:26.989049 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8167a986-mx528" Oct 30 02:08:27.041597 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:27.041565 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dd2d590-698e-4769-800f-55d530d7ca2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd2d590-698e-4769-800f-55d530d7ca2a-maas-api-openshift-ca\") pod \"0dd2d590-698e-4769-800f-55d530d7ca2a\" (UID: \"0dd2d590-698e-4769-800f-55d530d7ca2a\") " Oct 30 02:08:27.041860 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:27.041663 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dd2d590-698e-4769-800f-55d530d7ca2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd2d590-698e-4769-800f-55d530d7ca2a-maas-api-xks-ca\") pod \"0dd2d590-698e-4769-800f-55d530d7ca2a\" (UID: \"0dd2d590-698e-4769-800f-55d530d7ca2a\") " Oct 30 02:08:27.042021 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:27.041987 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dd2d590-698e-4769-800f-55d530d7ca2a-kube-api-access-42xst\" (UniqueName: \"kubernetes.io/projected/0dd2d590-698e-4769-800f-55d530d7ca2a-kube-api-access-42xst\") pod \"0dd2d590-698e-4769-800f-55d530d7ca2a\" (UID: \"0dd2d590-698e-4769-800f-55d530d7ca2a\") " Oct 30 02:08:27.042499 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:27.042464 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dd2d590-698e-4769-800f-55d530d7ca2a-maas-api-openshift-ca" pod "0dd2d590-698e-4769-800f-55d530d7ca2a" (UID: "0dd2d590-698e-4769-800f-55d530d7ca2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:08:27.042898 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:27.042871 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd2d590-698e-4769-800f-55d530d7ca2a-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:08:27.045283 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:27.045260 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd2d590-698e-4769-800f-55d530d7ca2a-kube-api-access-42xst" pod "0dd2d590-698e-4769-800f-55d530d7ca2a" (UID: "0dd2d590-698e-4769-800f-55d530d7ca2a"). InnerVolumeSpecName "kube-api-access-42xst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:08:27.055963 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:27.055902 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dd2d590-698e-4769-800f-55d530d7ca2a-maas-api-xks-ca" pod "0dd2d590-698e-4769-800f-55d530d7ca2a" (UID: "0dd2d590-698e-4769-800f-55d530d7ca2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:08:27.144529 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:27.144501 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd2d590-698e-4769-800f-55d530d7ca2a-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:08:27.144529 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:27.144526 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42xst\" (UniqueName: \"kubernetes.io/projected/0dd2d590-698e-4769-800f-55d530d7ca2a-kube-api-access-42xst\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:08:27.753809 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:27.753772 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:27.808052 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:27.808022 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8167a986-mx528" Oct 30 02:08:27.808270 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:27.808054 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8167a986-mx528" event={"ID":"0dd2d590-698e-4769-800f-55d530d7ca2a","Type":"ContainerDied","Data":"4f730e2da73ebffa6d1456e9799762cafb752ff868f185226ecbfbefee3559e1"} Oct 30 02:08:27.808270 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:27.808104 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f730e2da73ebffa6d1456e9799762cafb752ff868f185226ecbfbefee3559e1" Oct 30 02:08:28.354724 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:28.354691 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:29.287787 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:29.287704 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:29.499292 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:29.499248 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:29.700898 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:29.700861 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:30.000597 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.000556 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf"] Oct 30 02:08:30.001951 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.001922 6572 cpu_manager.go:414] "containerMap: removing container" podUID="0dd2d590-698e-4769-800f-55d530d7ca2a" containerName="revoke-keys" Oct 30 02:08:30.001951 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.001950 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dd2d590-698e-4769-800f-55d530d7ca2a" containerName="revoke-keys" Oct 30 02:08:30.002181 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.002150 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dd2d590-698e-4769-800f-55d530d7ca2a" containerName="revoke-keys" Oct 30 02:08:30.006152 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.006128 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:08:30.030646 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.030615 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf"] Oct 30 02:08:30.032446 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.032423 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-601d4b-openshift-default-dockercfg-7nw2n\"" Oct 30 02:08:30.122375 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.122332 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1147e056-c134-4bd0-8c0f-19de2e33eead-workload-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf\" (UID: \"1147e056-c134-4bd0-8c0f-19de2e33eead\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:08:30.122615 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.122388 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1147e056-c134-4bd0-8c0f-19de2e33eead-istio-podinfo\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf\" (UID: \"1147e056-c134-4bd0-8c0f-19de2e33eead\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:08:30.122615 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.122506 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1147e056-c134-4bd0-8c0f-19de2e33eead-istio-data\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf\" (UID: \"1147e056-c134-4bd0-8c0f-19de2e33eead\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:08:30.122615 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.122594 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1147e056-c134-4bd0-8c0f-19de2e33eead-credential-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf\" (UID: \"1147e056-c134-4bd0-8c0f-19de2e33eead\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:08:30.122877 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.122625 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1147e056-c134-4bd0-8c0f-19de2e33eead-istio-token\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf\" (UID: \"1147e056-c134-4bd0-8c0f-19de2e33eead\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:08:30.122877 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.122730 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1147e056-c134-4bd0-8c0f-19de2e33eead-istiod-ca-cert\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf\" (UID: \"1147e056-c134-4bd0-8c0f-19de2e33eead\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:08:30.122877 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.122851 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1147e056-c134-4bd0-8c0f-19de2e33eead-istio-envoy\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf\" (UID: \"1147e056-c134-4bd0-8c0f-19de2e33eead\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:08:30.123026 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.122898 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1147e056-c134-4bd0-8c0f-19de2e33eead-workload-certs\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf\" (UID: \"1147e056-c134-4bd0-8c0f-19de2e33eead\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:08:30.123026 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.122959 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvk9g\" (UniqueName: \"kubernetes.io/projected/1147e056-c134-4bd0-8c0f-19de2e33eead-kube-api-access-dvk9g\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf\" (UID: \"1147e056-c134-4bd0-8c0f-19de2e33eead\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:08:30.132198 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.132150 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:30.236770 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.227993 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvk9g\" (UniqueName: \"kubernetes.io/projected/1147e056-c134-4bd0-8c0f-19de2e33eead-kube-api-access-dvk9g\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf\" (UID: \"1147e056-c134-4bd0-8c0f-19de2e33eead\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:08:30.236770 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.228172 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1147e056-c134-4bd0-8c0f-19de2e33eead-workload-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf\" (UID: \"1147e056-c134-4bd0-8c0f-19de2e33eead\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:08:30.236770 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.228223 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1147e056-c134-4bd0-8c0f-19de2e33eead-istio-podinfo\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf\" (UID: \"1147e056-c134-4bd0-8c0f-19de2e33eead\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:08:30.236770 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.228275 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1147e056-c134-4bd0-8c0f-19de2e33eead-istio-data\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf\" (UID: \"1147e056-c134-4bd0-8c0f-19de2e33eead\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:08:30.236770 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.228321 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1147e056-c134-4bd0-8c0f-19de2e33eead-credential-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf\" (UID: \"1147e056-c134-4bd0-8c0f-19de2e33eead\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:08:30.236770 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.228346 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1147e056-c134-4bd0-8c0f-19de2e33eead-istio-token\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf\" (UID: \"1147e056-c134-4bd0-8c0f-19de2e33eead\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:08:30.236770 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.228418 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1147e056-c134-4bd0-8c0f-19de2e33eead-istiod-ca-cert\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf\" (UID: \"1147e056-c134-4bd0-8c0f-19de2e33eead\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:08:30.236770 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.228498 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1147e056-c134-4bd0-8c0f-19de2e33eead-istio-envoy\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf\" (UID: \"1147e056-c134-4bd0-8c0f-19de2e33eead\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:08:30.236770 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.228534 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1147e056-c134-4bd0-8c0f-19de2e33eead-workload-certs\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf\" (UID: \"1147e056-c134-4bd0-8c0f-19de2e33eead\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:08:30.236770 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.229083 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1147e056-c134-4bd0-8c0f-19de2e33eead-workload-certs\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf\" (UID: \"1147e056-c134-4bd0-8c0f-19de2e33eead\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:08:30.236770 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.229696 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1147e056-c134-4bd0-8c0f-19de2e33eead-workload-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf\" (UID: \"1147e056-c134-4bd0-8c0f-19de2e33eead\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:08:30.267821 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.251741 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1147e056-c134-4bd0-8c0f-19de2e33eead-istio-data\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf\" (UID: \"1147e056-c134-4bd0-8c0f-19de2e33eead\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:08:30.267821 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.252577 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1147e056-c134-4bd0-8c0f-19de2e33eead-istiod-ca-cert\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf\" (UID: \"1147e056-c134-4bd0-8c0f-19de2e33eead\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:08:30.267821 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.253589 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1147e056-c134-4bd0-8c0f-19de2e33eead-credential-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf\" (UID: \"1147e056-c134-4bd0-8c0f-19de2e33eead\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:08:30.268709 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.268641 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1147e056-c134-4bd0-8c0f-19de2e33eead-istio-envoy\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf\" (UID: \"1147e056-c134-4bd0-8c0f-19de2e33eead\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:08:30.320237 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.320200 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1147e056-c134-4bd0-8c0f-19de2e33eead-istio-podinfo\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf\" (UID: \"1147e056-c134-4bd0-8c0f-19de2e33eead\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:08:30.321205 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.321180 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvk9g\" (UniqueName: \"kubernetes.io/projected/1147e056-c134-4bd0-8c0f-19de2e33eead-kube-api-access-dvk9g\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf\" (UID: \"1147e056-c134-4bd0-8c0f-19de2e33eead\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:08:30.321680 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.321656 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1147e056-c134-4bd0-8c0f-19de2e33eead-istio-token\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf\" (UID: \"1147e056-c134-4bd0-8c0f-19de2e33eead\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:08:30.368772 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.351558 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:30.619607 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.619511 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:08:30.994123 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:30.991986 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf"] Oct 30 02:08:31.003436 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:08:31.003402 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1147e056_c134_4bd0_8c0f_19de2e33eead.slice/crio-53f5437e0c0bd5ab34a4b6dec802fb47130d1d363eb1a94aca70eebec279ab63 WatchSource:0}: Error finding container 53f5437e0c0bd5ab34a4b6dec802fb47130d1d363eb1a94aca70eebec279ab63: Status 404 returned error can't find the container with id 53f5437e0c0bd5ab34a4b6dec802fb47130d1d363eb1a94aca70eebec279ab63 Oct 30 02:08:31.011330 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:31.009350 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:08:31.011330 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:31.009451 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:08:31.011330 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:31.009515 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:08:31.092600 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:31.092083 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:31.843486 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:31.843452 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" event={"ID":"1147e056-c134-4bd0-8c0f-19de2e33eead","Type":"ContainerStarted","Data":"c2eb3bb8f543d76951b0ba49b3b61bfbbd9ddb686d3e939d0a1a72b452844742"} Oct 30 02:08:31.843962 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:31.843515 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" event={"ID":"1147e056-c134-4bd0-8c0f-19de2e33eead","Type":"ContainerStarted","Data":"53f5437e0c0bd5ab34a4b6dec802fb47130d1d363eb1a94aca70eebec279ab63"} Oct 30 02:08:31.915574 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:31.915515 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" podStartSLOduration=2.9155017279999997 podStartE2EDuration="2.915501728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:08:31.899955064 +0000 UTC m=+3037.476380478" watchObservedRunningTime="2026-10-30 02:08:31.915501728 +0000 UTC m=+3037.491927129" Oct 30 02:08:32.641270 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:32.640147 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:08:32.669590 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:32.669266 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:08:32.671285 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:32.671244 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:32.852057 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:32.852023 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:08:32.853032 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:32.853010 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:08:33.809121 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:33.808725 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:34.261257 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:34.261226 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:34.507008 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:34.506956 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:34.955469 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:34.954555 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:35.184605 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:35.184571 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-64d945b46f-8fq9n"] Oct 30 02:08:35.224460 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:35.224373 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:35.472292 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:35.472253 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:35.725766 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:35.723354 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:36.759322 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:36.759278 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:37.203113 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:37.203075 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:37.428374 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:37.428343 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:37.719959 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:37.719922 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:37.935997 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:37.935954 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:38.141477 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:38.141391 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:38.178855 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:38.178815 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-601d4b-b48cfb7b-zk29s"] Oct 30 02:08:38.182578 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:38.182557 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-601d4b-b48cfb7b-zk29s" Oct 30 02:08:38.196865 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:38.196837 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c580bbf-cee6-4937-8ea7-fd44837e076d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-601d4b-b48cfb7b-zk29s\" (UID: \"7c580bbf-cee6-4937-8ea7-fd44837e076d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-601d4b-b48cfb7b-zk29s" Oct 30 02:08:38.197067 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:38.197046 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwmkg\" (UniqueName: \"kubernetes.io/projected/7c580bbf-cee6-4937-8ea7-fd44837e076d-kube-api-access-pwmkg\") pod \"payload-processing-e2e-inf-b-601d4b-b48cfb7b-zk29s\" (UID: \"7c580bbf-cee6-4937-8ea7-fd44837e076d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-601d4b-b48cfb7b-zk29s" Oct 30 02:08:38.215081 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:38.215046 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-601d4b\"" Oct 30 02:08:38.215607 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:38.215497 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-601d4b-dockercfg-9d42c\"" Oct 30 02:08:38.244183 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:38.244145 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-601d4b-b48cfb7b-zk29s"] Oct 30 02:08:38.250012 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:38.249974 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-601d4b-565cc7b475-mhc5s"] Oct 30 02:08:38.255163 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:38.255137 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-601d4b-565cc7b475-mhc5s" Oct 30 02:08:38.264416 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:38.264377 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-601d4b-565cc7b475-mhc5s"] Oct 30 02:08:38.418831 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:38.363740 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnzcr\" (UniqueName: \"kubernetes.io/projected/f7f30150-5a1b-4c5b-8044-cfe1774eb76b-kube-api-access-cnzcr\") pod \"payload-pre-processing-e2e-inf-b-601d4b-565cc7b475-mhc5s\" (UID: \"f7f30150-5a1b-4c5b-8044-cfe1774eb76b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-601d4b-565cc7b475-mhc5s" Oct 30 02:08:38.443005 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:38.442959 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c580bbf-cee6-4937-8ea7-fd44837e076d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-601d4b-b48cfb7b-zk29s\" (UID: \"7c580bbf-cee6-4937-8ea7-fd44837e076d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-601d4b-b48cfb7b-zk29s" Oct 30 02:08:38.443350 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:38.443328 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7f30150-5a1b-4c5b-8044-cfe1774eb76b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-601d4b-565cc7b475-mhc5s\" (UID: \"f7f30150-5a1b-4c5b-8044-cfe1774eb76b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-601d4b-565cc7b475-mhc5s" Oct 30 02:08:38.469776 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:38.451143 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwmkg\" (UniqueName: \"kubernetes.io/projected/7c580bbf-cee6-4937-8ea7-fd44837e076d-kube-api-access-pwmkg\") pod \"payload-processing-e2e-inf-b-601d4b-b48cfb7b-zk29s\" (UID: \"7c580bbf-cee6-4937-8ea7-fd44837e076d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-601d4b-b48cfb7b-zk29s" Oct 30 02:08:38.510770 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:38.488189 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c580bbf-cee6-4937-8ea7-fd44837e076d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-601d4b-b48cfb7b-zk29s\" (UID: \"7c580bbf-cee6-4937-8ea7-fd44837e076d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-601d4b-b48cfb7b-zk29s" Oct 30 02:08:38.555453 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:38.555420 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwmkg\" (UniqueName: \"kubernetes.io/projected/7c580bbf-cee6-4937-8ea7-fd44837e076d-kube-api-access-pwmkg\") pod \"payload-processing-e2e-inf-b-601d4b-b48cfb7b-zk29s\" (UID: \"7c580bbf-cee6-4937-8ea7-fd44837e076d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-601d4b-b48cfb7b-zk29s" Oct 30 02:08:38.556533 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:38.556506 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:38.598908 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:38.598874 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7f30150-5a1b-4c5b-8044-cfe1774eb76b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-601d4b-565cc7b475-mhc5s\" (UID: \"f7f30150-5a1b-4c5b-8044-cfe1774eb76b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-601d4b-565cc7b475-mhc5s" Oct 30 02:08:38.599117 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:38.599071 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnzcr\" (UniqueName: \"kubernetes.io/projected/f7f30150-5a1b-4c5b-8044-cfe1774eb76b-kube-api-access-cnzcr\") pod \"payload-pre-processing-e2e-inf-b-601d4b-565cc7b475-mhc5s\" (UID: \"f7f30150-5a1b-4c5b-8044-cfe1774eb76b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-601d4b-565cc7b475-mhc5s" Oct 30 02:08:38.599667 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:38.599637 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7f30150-5a1b-4c5b-8044-cfe1774eb76b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-601d4b-565cc7b475-mhc5s\" (UID: \"f7f30150-5a1b-4c5b-8044-cfe1774eb76b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-601d4b-565cc7b475-mhc5s" Oct 30 02:08:38.626562 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:38.626523 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnzcr\" (UniqueName: \"kubernetes.io/projected/f7f30150-5a1b-4c5b-8044-cfe1774eb76b-kube-api-access-cnzcr\") pod \"payload-pre-processing-e2e-inf-b-601d4b-565cc7b475-mhc5s\" (UID: \"f7f30150-5a1b-4c5b-8044-cfe1774eb76b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-601d4b-565cc7b475-mhc5s" Oct 30 02:08:38.777452 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:38.777363 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:38.798007 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:38.797977 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-601d4b-b48cfb7b-zk29s" Oct 30 02:08:38.873671 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:38.873435 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-601d4b-565cc7b475-mhc5s" Oct 30 02:08:39.017778 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:39.008114 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:39.116696 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:08:39.116657 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c580bbf_cee6_4937_8ea7_fd44837e076d.slice/crio-a186c539ebe4b1ac879ae3add7b014acf5b55b133c221e4210c16a83ab6b08c1 WatchSource:0}: Error finding container a186c539ebe4b1ac879ae3add7b014acf5b55b133c221e4210c16a83ab6b08c1: Status 404 returned error can't find the container with id a186c539ebe4b1ac879ae3add7b014acf5b55b133c221e4210c16a83ab6b08c1 Oct 30 02:08:39.118466 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:39.118409 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-601d4b-b48cfb7b-zk29s"] Oct 30 02:08:39.201242 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:39.201217 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-601d4b-565cc7b475-mhc5s"] Oct 30 02:08:39.205148 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:08:39.205122 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f30150_5a1b_4c5b_8044_cfe1774eb76b.slice/crio-be991266a78995a43ff2c607186b9594db6bb4ed2fd0b6d19842a371c869fb72 WatchSource:0}: Error finding container be991266a78995a43ff2c607186b9594db6bb4ed2fd0b6d19842a371c869fb72: Status 404 returned error can't find the container with id be991266a78995a43ff2c607186b9594db6bb4ed2fd0b6d19842a371c869fb72 Oct 30 02:08:39.240621 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:39.240598 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:39.977741 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:39.977705 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-601d4b-565cc7b475-mhc5s" event={"ID":"f7f30150-5a1b-4c5b-8044-cfe1774eb76b","Type":"ContainerStarted","Data":"6433cfff0722adfd087c6fa1004d86ed4ebb409fe889485bccce6d7c835f9a05"} Oct 30 02:08:39.977936 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:39.977784 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-601d4b-565cc7b475-mhc5s" event={"ID":"f7f30150-5a1b-4c5b-8044-cfe1774eb76b","Type":"ContainerStarted","Data":"be991266a78995a43ff2c607186b9594db6bb4ed2fd0b6d19842a371c869fb72"} Oct 30 02:08:39.977936 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:39.977832 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-601d4b-565cc7b475-mhc5s" Oct 30 02:08:39.980311 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:39.980272 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-601d4b-b48cfb7b-zk29s" event={"ID":"7c580bbf-cee6-4937-8ea7-fd44837e076d","Type":"ContainerStarted","Data":"7bcf40de3cb91740af4e29865456475c612e1a3ef6a87dc9fa9aa259f69d165e"} Oct 30 02:08:39.980455 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:39.980322 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-601d4b-b48cfb7b-zk29s" event={"ID":"7c580bbf-cee6-4937-8ea7-fd44837e076d","Type":"ContainerStarted","Data":"a186c539ebe4b1ac879ae3add7b014acf5b55b133c221e4210c16a83ab6b08c1"} Oct 30 02:08:39.980455 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:39.980414 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-601d4b-b48cfb7b-zk29s" Oct 30 02:08:40.021439 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:40.021383 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-601d4b-565cc7b475-mhc5s" podStartSLOduration=2.021365049 podStartE2EDuration="2.021365049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:08:40.017381708 +0000 UTC m=+3045.593807111" watchObservedRunningTime="2026-10-30 02:08:40.021365049 +0000 UTC m=+3045.597790453" Oct 30 02:08:40.064172 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:40.064118 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-601d4b-b48cfb7b-zk29s" podStartSLOduration=2.064104824 podStartE2EDuration="2.064104824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:08:40.060079796 +0000 UTC m=+3045.636505200" watchObservedRunningTime="2026-10-30 02:08:40.064104824 +0000 UTC m=+3045.640530226" Oct 30 02:08:40.190785 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:40.186881 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-64d945b46f-8fq9n" podUID="54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db" containerName="payload-processing" containerID="cri-o://126c10a442941063352accc0a118390659b679d6da39fba5ccf24759a9cc19eb" gracePeriod=25 Oct 30 02:08:40.421435 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:40.421015 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:40.511945 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:40.511863 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-22e99d-6c8f7f8c7f-pbfcx"] Oct 30 02:08:40.516517 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:40.516495 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22e99d-6c8f7f8c7f-pbfcx" Oct 30 02:08:40.532698 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:40.532674 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-22e99d\"" Oct 30 02:08:40.532698 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:40.532688 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-22e99d-dockercfg-rdmdm\"" Oct 30 02:08:40.549626 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:40.549598 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-22e99d-6c8f7f8c7f-pbfcx"] Oct 30 02:08:40.563233 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:40.561681 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-64d945b46f-8fq9n" Oct 30 02:08:40.647773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:40.644239 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db-plugins-config-volume\") pod \"54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db\" (UID: \"54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db\") " Oct 30 02:08:40.647773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:40.644312 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db-provider-credentials\" (UniqueName: \"kubernetes.io/projected/54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db-provider-credentials\") pod \"54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db\" (UID: \"54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db\") " Oct 30 02:08:40.647773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:40.644581 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db-routing-overlay-volume\") pod \"54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db\" (UID: \"54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db\") " Oct 30 02:08:40.647773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:40.644768 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db-kube-api-access-gsgpm\" (UniqueName: \"kubernetes.io/projected/54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db-kube-api-access-gsgpm\") pod \"54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db\" (UID: \"54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db\") " Oct 30 02:08:40.647773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:40.645099 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hnh6\" (UniqueName: \"kubernetes.io/projected/1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a-kube-api-access-7hnh6\") pod \"payload-pre-processing-e2e-inf-b-22e99d-6c8f7f8c7f-pbfcx\" (UID: \"1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22e99d-6c8f7f8c7f-pbfcx" Oct 30 02:08:40.647773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:40.645476 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-22e99d-6c8f7f8c7f-pbfcx\" (UID: \"1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22e99d-6c8f7f8c7f-pbfcx" Oct 30 02:08:40.774837 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:40.773028 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db-plugins-config-volume" pod "54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db" (UID: "54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:08:40.783818 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:40.782009 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db-routing-overlay-volume" pod "54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db" (UID: "54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:08:40.838789 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:40.835402 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-22e99d-6c8f7f8c7f-pbfcx\" (UID: \"1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22e99d-6c8f7f8c7f-pbfcx" Oct 30 02:08:40.838789 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:40.835546 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hnh6\" (UniqueName: \"kubernetes.io/projected/1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a-kube-api-access-7hnh6\") pod \"payload-pre-processing-e2e-inf-b-22e99d-6c8f7f8c7f-pbfcx\" (UID: \"1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22e99d-6c8f7f8c7f-pbfcx" Oct 30 02:08:40.838789 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:40.835818 6572 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db-routing-overlay-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:08:40.838789 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:40.835836 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:08:40.840719 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:40.840496 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-22e99d-6c8f7f8c7f-pbfcx\" (UID: \"1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22e99d-6c8f7f8c7f-pbfcx" Oct 30 02:08:40.840964 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:40.840932 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db-kube-api-access-gsgpm" pod "54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db" (UID: "54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db"). InnerVolumeSpecName "kube-api-access-gsgpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:08:40.841696 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:40.841554 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:40.843058 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:40.843030 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db-provider-credentials" pod "54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db" (UID: "54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:08:40.864777 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:40.864711 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hnh6\" (UniqueName: \"kubernetes.io/projected/1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a-kube-api-access-7hnh6\") pod \"payload-pre-processing-e2e-inf-b-22e99d-6c8f7f8c7f-pbfcx\" (UID: \"1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22e99d-6c8f7f8c7f-pbfcx" Oct 30 02:08:40.899567 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:40.899541 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22e99d-6c8f7f8c7f-pbfcx" Oct 30 02:08:40.938232 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:40.938201 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsgpm\" (UniqueName: \"kubernetes.io/projected/54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db-kube-api-access-gsgpm\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:08:40.938387 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:40.938235 6572 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db-provider-credentials\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:08:40.995055 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:40.994855 6572 generic.go:356] "Generic (PLEG): container finished" podID="54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db" containerID="126c10a442941063352accc0a118390659b679d6da39fba5ccf24759a9cc19eb" exitCode=0 Oct 30 02:08:40.995055 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:40.994908 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-64d945b46f-8fq9n" Oct 30 02:08:40.995055 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:40.994946 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-64d945b46f-8fq9n" event={"ID":"54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db","Type":"ContainerDied","Data":"126c10a442941063352accc0a118390659b679d6da39fba5ccf24759a9cc19eb"} Oct 30 02:08:40.995055 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:40.995047 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-64d945b46f-8fq9n" event={"ID":"54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db","Type":"ContainerDied","Data":"ae6d283d0ee820dc073890b557709e3b818c6386120d4520e604deed586449cd"} Oct 30 02:08:40.995413 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:40.995095 6572 scope.go:122] "RemoveContainer" containerID="126c10a442941063352accc0a118390659b679d6da39fba5ccf24759a9cc19eb" Oct 30 02:08:41.012878 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:41.012856 6572 scope.go:122] "RemoveContainer" containerID="126c10a442941063352accc0a118390659b679d6da39fba5ccf24759a9cc19eb" Oct 30 02:08:41.013186 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:08:41.013158 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"126c10a442941063352accc0a118390659b679d6da39fba5ccf24759a9cc19eb\": container with ID starting with 126c10a442941063352accc0a118390659b679d6da39fba5ccf24759a9cc19eb not found: ID does not exist" containerID="126c10a442941063352accc0a118390659b679d6da39fba5ccf24759a9cc19eb" Oct 30 02:08:41.013306 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:41.013196 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"126c10a442941063352accc0a118390659b679d6da39fba5ccf24759a9cc19eb"} err="failed to get container status \"126c10a442941063352accc0a118390659b679d6da39fba5ccf24759a9cc19eb\": rpc error: code = NotFound desc = could not find container \"126c10a442941063352accc0a118390659b679d6da39fba5ccf24759a9cc19eb\": container with ID starting with 126c10a442941063352accc0a118390659b679d6da39fba5ccf24759a9cc19eb not found: ID does not exist" Oct 30 02:08:41.171830 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:41.154138 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-64d945b46f-8fq9n"] Oct 30 02:08:41.176929 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:41.176810 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:41.270778 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:41.252192 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-64d945b46f-8fq9n"] Oct 30 02:08:41.350046 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:41.330986 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-22e99d-6c8f7f8c7f-pbfcx"] Oct 30 02:08:42.009425 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:42.009325 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22e99d-6c8f7f8c7f-pbfcx" event={"ID":"1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a","Type":"ContainerStarted","Data":"89958f4d886480638e260c469a27e7f3101c6984c5b79052808339a90db17663"} Oct 30 02:08:42.009425 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:42.009393 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22e99d-6c8f7f8c7f-pbfcx" event={"ID":"1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a","Type":"ContainerStarted","Data":"f053131fdfb324c855bbac137e18dee00e2f7886aa6bccb14ff2d957d0a0ccdd"} Oct 30 02:08:42.009743 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:42.009723 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22e99d-6c8f7f8c7f-pbfcx" Oct 30 02:08:42.068321 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:42.068263 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22e99d-6c8f7f8c7f-pbfcx" podStartSLOduration=2.068246459 podStartE2EDuration="2.068246459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:08:42.060730381 +0000 UTC m=+3047.637155783" watchObservedRunningTime="2026-10-30 02:08:42.068246459 +0000 UTC m=+3047.644672007" Oct 30 02:08:42.549624 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:42.549584 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db" path="/var/lib/kubelet/pods/54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db/volumes" Oct 30 02:08:42.809490 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:42.809374 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:44.537801 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:44.537705 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:44.690970 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:08:44.690934 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:08:46.193682 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:46.191395 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:47.890590 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:47.890552 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:49.284013 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:49.281692 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:49.502239 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:49.502206 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:49.711552 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:49.711514 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:50.742973 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:50.742934 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:51.000214 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.000138 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-601d4b-b48cfb7b-zk29s" Oct 30 02:08:51.000214 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.000207 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-601d4b-565cc7b475-mhc5s" Oct 30 02:08:51.153092 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.153062 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg"] Oct 30 02:08:51.154258 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.154240 6572 cpu_manager.go:414] "containerMap: removing container" podUID="54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db" containerName="payload-processing" Oct 30 02:08:51.154349 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.154260 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db" containerName="payload-processing" Oct 30 02:08:51.154495 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.154483 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="54b0c2a9-5229-42d1-83a2-8e6ccc7aa2db" containerName="payload-processing" Oct 30 02:08:51.160139 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.160116 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg" Oct 30 02:08:51.174958 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.174912 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg"] Oct 30 02:08:51.177406 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.177384 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-817fd7\"/\"agc-cred-a7fb0b09dfb8\"" Oct 30 02:08:51.177505 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.177409 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-817fd7\"/\"payload-processing-external-model-e2e--6dbff1fe-dockercfg-vpv7w\"" Oct 30 02:08:51.178026 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.178006 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-817fd7\"/\"openshift-service-ca.crt\"" Oct 30 02:08:51.178026 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.178011 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-817fd7\"/\"payload-processing-external-model-plugins-e2e-extmod-a-817fd7\"" Oct 30 02:08:51.178157 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.178051 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-817fd7\"/\"routing-overlay\"" Oct 30 02:08:51.178157 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.178071 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-817fd7\"/\"kube-root-ca.crt\"" Oct 30 02:08:51.363717 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.363626 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/614bbbec-a79d-48cd-97c1-294455b786e0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg\" (UID: \"614bbbec-a79d-48cd-97c1-294455b786e0\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg" Oct 30 02:08:51.365682 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.365657 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/614bbbec-a79d-48cd-97c1-294455b786e0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg\" (UID: \"614bbbec-a79d-48cd-97c1-294455b786e0\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg" Oct 30 02:08:51.366087 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.366046 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhf2x\" (UniqueName: \"kubernetes.io/projected/614bbbec-a79d-48cd-97c1-294455b786e0-kube-api-access-bhf2x\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg\" (UID: \"614bbbec-a79d-48cd-97c1-294455b786e0\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg" Oct 30 02:08:51.366354 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.366332 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/614bbbec-a79d-48cd-97c1-294455b786e0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg\" (UID: \"614bbbec-a79d-48cd-97c1-294455b786e0\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg" Oct 30 02:08:51.468353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.468315 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhf2x\" (UniqueName: \"kubernetes.io/projected/614bbbec-a79d-48cd-97c1-294455b786e0-kube-api-access-bhf2x\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg\" (UID: \"614bbbec-a79d-48cd-97c1-294455b786e0\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg" Oct 30 02:08:51.468565 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.468407 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/614bbbec-a79d-48cd-97c1-294455b786e0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg\" (UID: \"614bbbec-a79d-48cd-97c1-294455b786e0\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg" Oct 30 02:08:51.468655 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.468632 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/614bbbec-a79d-48cd-97c1-294455b786e0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg\" (UID: \"614bbbec-a79d-48cd-97c1-294455b786e0\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg" Oct 30 02:08:51.468721 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.468687 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/614bbbec-a79d-48cd-97c1-294455b786e0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg\" (UID: \"614bbbec-a79d-48cd-97c1-294455b786e0\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg" Oct 30 02:08:51.469217 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.469187 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/614bbbec-a79d-48cd-97c1-294455b786e0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg\" (UID: \"614bbbec-a79d-48cd-97c1-294455b786e0\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg" Oct 30 02:08:51.469441 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.469419 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/614bbbec-a79d-48cd-97c1-294455b786e0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg\" (UID: \"614bbbec-a79d-48cd-97c1-294455b786e0\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg" Oct 30 02:08:51.471338 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.471319 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/614bbbec-a79d-48cd-97c1-294455b786e0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg\" (UID: \"614bbbec-a79d-48cd-97c1-294455b786e0\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg" Oct 30 02:08:51.491487 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.491460 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhf2x\" (UniqueName: \"kubernetes.io/projected/614bbbec-a79d-48cd-97c1-294455b786e0-kube-api-access-bhf2x\") pod \"payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg\" (UID: \"614bbbec-a79d-48cd-97c1-294455b786e0\") " pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg" Oct 30 02:08:51.661149 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.661119 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:51.779045 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.779011 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg" Oct 30 02:08:51.858191 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.858150 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-f6e11e/payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x"] Oct 30 02:08:51.864990 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.864329 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f6e11e/payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x" Oct 30 02:08:51.883880 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.882879 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f6e11e\"/\"routing-overlay\"" Oct 30 02:08:51.883880 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.882995 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f6e11e\"/\"payload-processing-external-model-plugins-e2e-extmod-b-f6e11e\"" Oct 30 02:08:51.883880 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.883055 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f6e11e\"/\"openshift-service-ca.crt\"" Oct 30 02:08:51.883880 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.883114 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f6e11e\"/\"agc-cred-87b232792c72\"" Oct 30 02:08:51.883880 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.883203 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f6e11e\"/\"kube-root-ca.crt\"" Oct 30 02:08:51.886097 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.884991 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:51.886460 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.886385 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eccc8133-8a38-46fc-99af-285ecdd04bb0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x\" (UID: \"eccc8133-8a38-46fc-99af-285ecdd04bb0\") " pod="ai-tenant-e2e-extmod-b-f6e11e/payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x" Oct 30 02:08:51.886460 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.886447 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eccc8133-8a38-46fc-99af-285ecdd04bb0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x\" (UID: \"eccc8133-8a38-46fc-99af-285ecdd04bb0\") " pod="ai-tenant-e2e-extmod-b-f6e11e/payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x" Oct 30 02:08:51.886666 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.886640 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84p5q\" (UniqueName: \"kubernetes.io/projected/eccc8133-8a38-46fc-99af-285ecdd04bb0-kube-api-access-84p5q\") pod \"payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x\" (UID: \"eccc8133-8a38-46fc-99af-285ecdd04bb0\") " pod="ai-tenant-e2e-extmod-b-f6e11e/payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x" Oct 30 02:08:51.886945 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.886926 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eccc8133-8a38-46fc-99af-285ecdd04bb0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x\" (UID: \"eccc8133-8a38-46fc-99af-285ecdd04bb0\") " pod="ai-tenant-e2e-extmod-b-f6e11e/payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x" Oct 30 02:08:51.892515 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.892491 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f6e11e/payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x"] Oct 30 02:08:51.892710 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:51.892690 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f6e11e\"/\"payload-processing-external-model-e2e--329ff17e-dockercfg-c542x\"" Oct 30 02:08:52.060830 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:52.008151 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eccc8133-8a38-46fc-99af-285ecdd04bb0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x\" (UID: \"eccc8133-8a38-46fc-99af-285ecdd04bb0\") " pod="ai-tenant-e2e-extmod-b-f6e11e/payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x" Oct 30 02:08:52.061578 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:52.030590 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eccc8133-8a38-46fc-99af-285ecdd04bb0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x\" (UID: \"eccc8133-8a38-46fc-99af-285ecdd04bb0\") " pod="ai-tenant-e2e-extmod-b-f6e11e/payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x" Oct 30 02:08:52.062043 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:52.062018 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84p5q\" (UniqueName: \"kubernetes.io/projected/eccc8133-8a38-46fc-99af-285ecdd04bb0-kube-api-access-84p5q\") pod \"payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x\" (UID: \"eccc8133-8a38-46fc-99af-285ecdd04bb0\") " pod="ai-tenant-e2e-extmod-b-f6e11e/payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x" Oct 30 02:08:52.195869 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:52.167843 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eccc8133-8a38-46fc-99af-285ecdd04bb0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x\" (UID: \"eccc8133-8a38-46fc-99af-285ecdd04bb0\") " pod="ai-tenant-e2e-extmod-b-f6e11e/payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x" Oct 30 02:08:52.195869 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:52.168037 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eccc8133-8a38-46fc-99af-285ecdd04bb0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x\" (UID: \"eccc8133-8a38-46fc-99af-285ecdd04bb0\") " pod="ai-tenant-e2e-extmod-b-f6e11e/payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x" Oct 30 02:08:52.195869 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:52.169205 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eccc8133-8a38-46fc-99af-285ecdd04bb0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x\" (UID: \"eccc8133-8a38-46fc-99af-285ecdd04bb0\") " pod="ai-tenant-e2e-extmod-b-f6e11e/payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x" Oct 30 02:08:52.195869 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:52.193365 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eccc8133-8a38-46fc-99af-285ecdd04bb0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x\" (UID: \"eccc8133-8a38-46fc-99af-285ecdd04bb0\") " pod="ai-tenant-e2e-extmod-b-f6e11e/payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x" Oct 30 02:08:52.211628 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:52.211573 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg"] Oct 30 02:08:52.215854 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:52.215824 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84p5q\" (UniqueName: \"kubernetes.io/projected/eccc8133-8a38-46fc-99af-285ecdd04bb0-kube-api-access-84p5q\") pod \"payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x\" (UID: \"eccc8133-8a38-46fc-99af-285ecdd04bb0\") " pod="ai-tenant-e2e-extmod-b-f6e11e/payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x" Oct 30 02:08:52.226825 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:08:52.226788 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod614bbbec_a79d_48cd_97c1_294455b786e0.slice/crio-4a99771e7a6b4e4ea486639328a8eb3cd499067ae64c82488ce73f8bd6db1fd3 WatchSource:0}: Error finding container 4a99771e7a6b4e4ea486639328a8eb3cd499067ae64c82488ce73f8bd6db1fd3: Status 404 returned error can't find the container with id 4a99771e7a6b4e4ea486639328a8eb3cd499067ae64c82488ce73f8bd6db1fd3 Oct 30 02:08:52.496591 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:52.496564 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f6e11e/payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x" Oct 30 02:08:52.803068 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:08:52.803033 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeccc8133_8a38_46fc_99af_285ecdd04bb0.slice/crio-46c0703e82a2ca520fb527e63e12234dfa9756fe884c364fef83f4441a3a8772 WatchSource:0}: Error finding container 46c0703e82a2ca520fb527e63e12234dfa9756fe884c364fef83f4441a3a8772: Status 404 returned error can't find the container with id 46c0703e82a2ca520fb527e63e12234dfa9756fe884c364fef83f4441a3a8772 Oct 30 02:08:52.803560 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:52.803536 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f6e11e/payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x"] Oct 30 02:08:53.021140 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:53.021105 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22e99d-6c8f7f8c7f-pbfcx" Oct 30 02:08:53.185636 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:53.185600 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg" event={"ID":"614bbbec-a79d-48cd-97c1-294455b786e0","Type":"ContainerStarted","Data":"a1dbb45919cc80451a369cb0c1d7c81341586ad7366881cff44b371955c0ce39"} Oct 30 02:08:53.185858 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:53.185661 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg" event={"ID":"614bbbec-a79d-48cd-97c1-294455b786e0","Type":"ContainerStarted","Data":"4a99771e7a6b4e4ea486639328a8eb3cd499067ae64c82488ce73f8bd6db1fd3"} Oct 30 02:08:53.185858 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:53.185692 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg" Oct 30 02:08:53.187381 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:53.187353 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f6e11e/payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x" event={"ID":"eccc8133-8a38-46fc-99af-285ecdd04bb0","Type":"ContainerStarted","Data":"a18dac67dd224c926a3a13e29cc6c9000a720e5a2c34a47dcb1ecbaed651d19b"} Oct 30 02:08:53.187515 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:53.187404 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f6e11e/payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x" event={"ID":"eccc8133-8a38-46fc-99af-285ecdd04bb0","Type":"ContainerStarted","Data":"46c0703e82a2ca520fb527e63e12234dfa9756fe884c364fef83f4441a3a8772"} Oct 30 02:08:53.187515 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:53.187439 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-f6e11e/payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x" Oct 30 02:08:53.228741 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:53.228697 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg" podStartSLOduration=2.228683787 podStartE2EDuration="2.228683787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:08:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:08:53.224677817 +0000 UTC m=+3058.801103219" watchObservedRunningTime="2026-10-30 02:08:53.228683787 +0000 UTC m=+3058.805109188" Oct 30 02:08:53.268346 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:53.268305 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-f6e11e/payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x" podStartSLOduration=2.268292155 podStartE2EDuration="2.268292155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:08:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:08:53.265583856 +0000 UTC m=+3058.842009257" watchObservedRunningTime="2026-10-30 02:08:53.268292155 +0000 UTC m=+3058.844717597" Oct 30 02:08:53.308055 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:53.308011 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:54.934931 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:54.934879 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:56.716139 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:56.716101 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:58.404554 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:08:58.404505 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:00.039811 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:00.039771 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:01.694903 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:01.694864 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:03.405428 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:03.405392 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:04.200027 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:04.199999 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg" Oct 30 02:09:04.200216 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:04.200066 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-f6e11e/payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x" Oct 30 02:09:04.998076 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:04.998040 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:06.632331 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:06.632295 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:08.219472 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:08.219441 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:08.420283 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:08.420238 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:09.792274 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:09.792238 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:10.320286 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:10.320249 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:11.505472 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:11.505437 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:13.066871 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:13.066838 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:14.669765 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:14.669720 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:14.689885 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:09:14.689843 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:09:16.504155 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:16.504120 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:18.362782 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:18.362724 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:20.233312 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:20.233278 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:22.195859 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:22.195817 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:23.940197 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:23.940156 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:25.035214 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:25.035176 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a75e7c1-6pfh4"] Oct 30 02:09:25.045740 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:25.045709 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a75e7c1-6pfh4"] Oct 30 02:09:25.680367 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:25.680323 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:26.545804 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:26.545728 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5a194d9-ae2d-43ba-8241-91b03c367658" path="/var/lib/kubelet/pods/c5a194d9-ae2d-43ba-8241-91b03c367658/volumes" Oct 30 02:09:27.620824 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:27.620787 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:27.875522 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:27.873282 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:29.064956 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:29.064917 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:30.728900 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:30.728845 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:32.399385 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:32.399348 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:32.635919 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:32.635884 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:33.556909 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:33.556875 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:33.772050 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:33.772016 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:34.421533 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:34.421379 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:36.135606 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:36.135568 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:37.769707 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:37.769671 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:39.422345 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:39.422299 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:41.096906 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:41.096873 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:42.823216 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:42.823065 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:44.621929 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:44.615708 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:44.697915 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:09:44.697875 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:09:46.241818 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:46.241731 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:47.958577 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:47.958529 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:49.728438 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:49.728400 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:51.414281 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:51.414235 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:53.016602 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:53.016547 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:54.640397 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:54.640362 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:56.266910 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:56.266878 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:57.945809 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:57.944722 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:59.612482 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:09:59.612391 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:00.379613 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:00.379572 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f"] Oct 30 02:10:00.435532 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:00.435478 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f"] Oct 30 02:10:00.436008 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:00.435979 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" Oct 30 02:10:00.513867 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:00.513826 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzd9b\" (UniqueName: \"kubernetes.io/projected/364b9686-9847-4ef4-a5b1-7f611a248b64-kube-api-access-wzd9b\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f\" (UID: \"364b9686-9847-4ef4-a5b1-7f611a248b64\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" Oct 30 02:10:00.514050 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:00.513908 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/364b9686-9847-4ef4-a5b1-7f611a248b64-istio-envoy\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f\" (UID: \"364b9686-9847-4ef4-a5b1-7f611a248b64\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" Oct 30 02:10:00.514107 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:00.514094 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/364b9686-9847-4ef4-a5b1-7f611a248b64-workload-certs\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f\" (UID: \"364b9686-9847-4ef4-a5b1-7f611a248b64\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" Oct 30 02:10:00.514155 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:00.514128 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/364b9686-9847-4ef4-a5b1-7f611a248b64-credential-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f\" (UID: \"364b9686-9847-4ef4-a5b1-7f611a248b64\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" Oct 30 02:10:00.514155 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:00.514146 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/364b9686-9847-4ef4-a5b1-7f611a248b64-istio-data\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f\" (UID: \"364b9686-9847-4ef4-a5b1-7f611a248b64\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" Oct 30 02:10:00.514290 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:00.514270 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/364b9686-9847-4ef4-a5b1-7f611a248b64-istiod-ca-cert\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f\" (UID: \"364b9686-9847-4ef4-a5b1-7f611a248b64\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" Oct 30 02:10:00.514361 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:00.514315 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/364b9686-9847-4ef4-a5b1-7f611a248b64-istio-token\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f\" (UID: \"364b9686-9847-4ef4-a5b1-7f611a248b64\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" Oct 30 02:10:00.514484 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:00.514460 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/364b9686-9847-4ef4-a5b1-7f611a248b64-workload-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f\" (UID: \"364b9686-9847-4ef4-a5b1-7f611a248b64\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" Oct 30 02:10:00.514552 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:00.514536 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/364b9686-9847-4ef4-a5b1-7f611a248b64-istio-podinfo\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f\" (UID: \"364b9686-9847-4ef4-a5b1-7f611a248b64\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" Oct 30 02:10:00.615735 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:00.615697 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/364b9686-9847-4ef4-a5b1-7f611a248b64-workload-certs\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f\" (UID: \"364b9686-9847-4ef4-a5b1-7f611a248b64\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" Oct 30 02:10:00.616243 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:00.615743 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/364b9686-9847-4ef4-a5b1-7f611a248b64-credential-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f\" (UID: \"364b9686-9847-4ef4-a5b1-7f611a248b64\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" Oct 30 02:10:00.616243 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:00.615791 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/364b9686-9847-4ef4-a5b1-7f611a248b64-istio-data\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f\" (UID: \"364b9686-9847-4ef4-a5b1-7f611a248b64\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" Oct 30 02:10:00.616243 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:00.615826 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/364b9686-9847-4ef4-a5b1-7f611a248b64-istiod-ca-cert\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f\" (UID: \"364b9686-9847-4ef4-a5b1-7f611a248b64\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" Oct 30 02:10:00.616243 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:00.615858 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/364b9686-9847-4ef4-a5b1-7f611a248b64-istio-token\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f\" (UID: \"364b9686-9847-4ef4-a5b1-7f611a248b64\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" Oct 30 02:10:00.616243 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:00.615972 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/364b9686-9847-4ef4-a5b1-7f611a248b64-workload-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f\" (UID: \"364b9686-9847-4ef4-a5b1-7f611a248b64\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" Oct 30 02:10:00.616243 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:00.616021 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/364b9686-9847-4ef4-a5b1-7f611a248b64-istio-podinfo\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f\" (UID: \"364b9686-9847-4ef4-a5b1-7f611a248b64\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" Oct 30 02:10:00.616243 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:00.616115 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzd9b\" (UniqueName: \"kubernetes.io/projected/364b9686-9847-4ef4-a5b1-7f611a248b64-kube-api-access-wzd9b\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f\" (UID: \"364b9686-9847-4ef4-a5b1-7f611a248b64\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" Oct 30 02:10:00.616243 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:00.616151 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/364b9686-9847-4ef4-a5b1-7f611a248b64-istio-envoy\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f\" (UID: \"364b9686-9847-4ef4-a5b1-7f611a248b64\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" Oct 30 02:10:00.616243 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:00.616152 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/364b9686-9847-4ef4-a5b1-7f611a248b64-workload-certs\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f\" (UID: \"364b9686-9847-4ef4-a5b1-7f611a248b64\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" Oct 30 02:10:00.616837 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:00.616406 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/364b9686-9847-4ef4-a5b1-7f611a248b64-credential-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f\" (UID: \"364b9686-9847-4ef4-a5b1-7f611a248b64\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" Oct 30 02:10:00.616837 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:00.616645 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/364b9686-9847-4ef4-a5b1-7f611a248b64-workload-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f\" (UID: \"364b9686-9847-4ef4-a5b1-7f611a248b64\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" Oct 30 02:10:00.616837 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:00.616691 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/364b9686-9847-4ef4-a5b1-7f611a248b64-istio-data\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f\" (UID: \"364b9686-9847-4ef4-a5b1-7f611a248b64\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" Oct 30 02:10:00.616837 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:00.616781 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/364b9686-9847-4ef4-a5b1-7f611a248b64-istiod-ca-cert\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f\" (UID: \"364b9686-9847-4ef4-a5b1-7f611a248b64\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" Oct 30 02:10:00.618549 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:00.618518 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/364b9686-9847-4ef4-a5b1-7f611a248b64-istio-envoy\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f\" (UID: \"364b9686-9847-4ef4-a5b1-7f611a248b64\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" Oct 30 02:10:00.618786 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:00.618733 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/364b9686-9847-4ef4-a5b1-7f611a248b64-istio-podinfo\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f\" (UID: \"364b9686-9847-4ef4-a5b1-7f611a248b64\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" Oct 30 02:10:00.636838 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:00.636783 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/364b9686-9847-4ef4-a5b1-7f611a248b64-istio-token\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f\" (UID: \"364b9686-9847-4ef4-a5b1-7f611a248b64\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" Oct 30 02:10:00.637141 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:00.637118 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzd9b\" (UniqueName: \"kubernetes.io/projected/364b9686-9847-4ef4-a5b1-7f611a248b64-kube-api-access-wzd9b\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f\" (UID: \"364b9686-9847-4ef4-a5b1-7f611a248b64\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" Oct 30 02:10:00.807736 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:00.807704 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" Oct 30 02:10:01.245842 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:01.245811 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:01.775640 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:01.775602 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f"] Oct 30 02:10:01.781952 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:10:01.781920 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod364b9686_9847_4ef4_a5b1_7f611a248b64.slice/crio-65792ac4d49d8f61f59f4fd740c9d681f0d7ba20936b5de2f3ed6d0bd125366d WatchSource:0}: Error finding container 65792ac4d49d8f61f59f4fd740c9d681f0d7ba20936b5de2f3ed6d0bd125366d: Status 404 returned error can't find the container with id 65792ac4d49d8f61f59f4fd740c9d681f0d7ba20936b5de2f3ed6d0bd125366d Oct 30 02:10:01.785328 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:01.785281 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:10:01.785430 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:01.785380 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:10:01.785430 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:01.785412 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:10:02.107070 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:02.106978 6572 generic.go:356] "Generic (PLEG): container finished" podID="1147e056-c134-4bd0-8c0f-19de2e33eead" containerID="c2eb3bb8f543d76951b0ba49b3b61bfbbd9ddb686d3e939d0a1a72b452844742" exitCode=0 Oct 30 02:10:02.107070 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:02.107051 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" event={"ID":"1147e056-c134-4bd0-8c0f-19de2e33eead","Type":"ContainerDied","Data":"c2eb3bb8f543d76951b0ba49b3b61bfbbd9ddb686d3e939d0a1a72b452844742"} Oct 30 02:10:02.107816 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:02.107633 6572 scope.go:122] "RemoveContainer" containerID="c2eb3bb8f543d76951b0ba49b3b61bfbbd9ddb686d3e939d0a1a72b452844742" Oct 30 02:10:02.110003 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:02.109971 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" event={"ID":"364b9686-9847-4ef4-a5b1-7f611a248b64","Type":"ContainerStarted","Data":"90ea182817356e05150d77f344aaa6ec82d66a25f3c17c8ed6afec9995a0d2e1"} Oct 30 02:10:02.110109 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:02.110031 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" event={"ID":"364b9686-9847-4ef4-a5b1-7f611a248b64","Type":"ContainerStarted","Data":"65792ac4d49d8f61f59f4fd740c9d681f0d7ba20936b5de2f3ed6d0bd125366d"} Oct 30 02:10:02.110230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:02.110202 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:10:02.110284 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:02.110271 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:10:02.110328 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:02.110306 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:10:02.207315 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:02.203563 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" podStartSLOduration=2.203542695 podStartE2EDuration="2.203542695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:10:02.197986923 +0000 UTC m=+3127.774412327" watchObservedRunningTime="2026-10-30 02:10:02.203542695 +0000 UTC m=+3127.779968098" Oct 30 02:10:02.832779 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:02.827637 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" Oct 30 02:10:02.833441 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:02.833407 6572 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.99:15021/healthz/ready\": dial tcp 10.133.1.99:15021: connect: connection refused" start-of-body= Oct 30 02:10:02.833650 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:02.833623 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" podUID="364b9686-9847-4ef4-a5b1-7f611a248b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.99:15021/healthz/ready\": dial tcp 10.133.1.99:15021: connect: connection refused" Oct 30 02:10:02.852602 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:02.852567 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:10:03.016780 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:03.015449 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:03.169419 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:03.169386 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" event={"ID":"1147e056-c134-4bd0-8c0f-19de2e33eead","Type":"ContainerStarted","Data":"1f0b6965d1cee50ed6490374ca948a7d8b74666b0e6d63385c139e3fd39b2261"} Oct 30 02:10:03.626952 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:03.626826 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:10:03.627785 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:03.627667 6572 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" start-of-body= Oct 30 02:10:03.627785 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:03.627741 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" podUID="1147e056-c134-4bd0-8c0f-19de2e33eead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" Oct 30 02:10:03.814786 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:03.814295 6572 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.99:15021/healthz/ready\": dial tcp 10.133.1.99:15021: connect: connection refused" start-of-body= Oct 30 02:10:03.814786 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:03.814376 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" podUID="364b9686-9847-4ef4-a5b1-7f611a248b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.99:15021/healthz/ready\": dial tcp 10.133.1.99:15021: connect: connection refused" Oct 30 02:10:04.638772 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:04.638123 6572 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" start-of-body= Oct 30 02:10:04.638772 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:04.638201 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" podUID="1147e056-c134-4bd0-8c0f-19de2e33eead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" Oct 30 02:10:04.786773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:04.786282 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:04.811767 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:04.811144 6572 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.99:15021/healthz/ready\": dial tcp 10.133.1.99:15021: connect: connection refused" start-of-body= Oct 30 02:10:04.811767 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:04.811218 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" podUID="364b9686-9847-4ef4-a5b1-7f611a248b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.99:15021/healthz/ready\": dial tcp 10.133.1.99:15021: connect: connection refused" Oct 30 02:10:05.634774 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:05.622210 6572 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" start-of-body= Oct 30 02:10:05.634774 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:05.622272 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" podUID="1147e056-c134-4bd0-8c0f-19de2e33eead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" Oct 30 02:10:05.813774 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:05.810172 6572 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.99:15021/healthz/ready\": dial tcp 10.133.1.99:15021: connect: connection refused" start-of-body= Oct 30 02:10:05.813774 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:05.810234 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" podUID="364b9686-9847-4ef4-a5b1-7f611a248b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.99:15021/healthz/ready\": dial tcp 10.133.1.99:15021: connect: connection refused" Oct 30 02:10:06.626478 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:06.626439 6572 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" start-of-body= Oct 30 02:10:06.626941 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:06.626905 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" podUID="1147e056-c134-4bd0-8c0f-19de2e33eead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" Oct 30 02:10:06.824781 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:06.810214 6572 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.99:15021/healthz/ready\": dial tcp 10.133.1.99:15021: connect: connection refused" start-of-body= Oct 30 02:10:06.824781 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:06.810279 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" podUID="364b9686-9847-4ef4-a5b1-7f611a248b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.99:15021/healthz/ready\": dial tcp 10.133.1.99:15021: connect: connection refused" Oct 30 02:10:07.629780 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:07.629198 6572 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" start-of-body= Oct 30 02:10:07.629780 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:07.629276 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" podUID="1147e056-c134-4bd0-8c0f-19de2e33eead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" Oct 30 02:10:07.813775 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:07.810176 6572 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.99:15021/healthz/ready\": dial tcp 10.133.1.99:15021: connect: connection refused" start-of-body= Oct 30 02:10:07.813775 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:07.810238 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" podUID="364b9686-9847-4ef4-a5b1-7f611a248b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.99:15021/healthz/ready\": dial tcp 10.133.1.99:15021: connect: connection refused" Oct 30 02:10:08.620743 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:08.620706 6572 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" start-of-body= Oct 30 02:10:08.621220 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:08.620811 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" podUID="1147e056-c134-4bd0-8c0f-19de2e33eead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" Oct 30 02:10:08.705214 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:08.705173 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:08.814667 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:08.814631 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" Oct 30 02:10:08.815147 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:08.815130 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" Oct 30 02:10:08.815924 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:08.815900 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" Oct 30 02:10:08.966405 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:08.966356 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:09.626092 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:09.626063 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:10:09.626598 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:09.626576 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:10:09.627288 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:09.627271 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:10:09.939506 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:09.939467 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:11.532214 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:11.532114 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:13.136085 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:13.136038 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:14.691806 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:10:14.691762 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:10:14.798998 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:14.798946 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:16.438861 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:16.438826 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:18.160335 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:18.160291 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:19.670992 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:19.670939 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:21.288546 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:21.288509 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:22.947354 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:22.947303 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:24.686985 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:24.686939 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:26.546686 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:26.546649 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:27.996575 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:27.996536 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:28.428450 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:28.428416 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:30.389391 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:30.389352 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:31.166485 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:31.166445 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:31.697196 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:31.697163 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:32.100409 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:32.100288 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:33.687822 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:33.687688 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:35.208434 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:35.208392 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:36.827463 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:36.827424 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:38.382732 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:38.382690 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:40.125967 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:40.125928 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:41.791346 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:41.791309 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:43.367495 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:43.367457 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:44.694580 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:10:44.694539 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:10:45.026449 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:45.026340 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:45.499384 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:45.499340 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7"] Oct 30 02:10:45.505216 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:45.505185 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" Oct 30 02:10:45.528340 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:45.528306 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7"] Oct 30 02:10:45.687899 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:45.687860 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9562e58-739e-44e4-a2ca-448684bddd75-istio-data\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7\" (UID: \"a9562e58-739e-44e4-a2ca-448684bddd75\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" Oct 30 02:10:45.687899 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:45.687907 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9562e58-739e-44e4-a2ca-448684bddd75-istiod-ca-cert\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7\" (UID: \"a9562e58-739e-44e4-a2ca-448684bddd75\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" Oct 30 02:10:45.688127 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:45.688014 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9562e58-739e-44e4-a2ca-448684bddd75-istio-podinfo\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7\" (UID: \"a9562e58-739e-44e4-a2ca-448684bddd75\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" Oct 30 02:10:45.688127 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:45.688054 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9562e58-739e-44e4-a2ca-448684bddd75-istio-envoy\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7\" (UID: \"a9562e58-739e-44e4-a2ca-448684bddd75\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" Oct 30 02:10:45.688289 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:45.688269 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9562e58-739e-44e4-a2ca-448684bddd75-workload-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7\" (UID: \"a9562e58-739e-44e4-a2ca-448684bddd75\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" Oct 30 02:10:45.688338 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:45.688319 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq7tc\" (UniqueName: \"kubernetes.io/projected/a9562e58-739e-44e4-a2ca-448684bddd75-kube-api-access-vq7tc\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7\" (UID: \"a9562e58-739e-44e4-a2ca-448684bddd75\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" Oct 30 02:10:45.688431 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:45.688412 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9562e58-739e-44e4-a2ca-448684bddd75-credential-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7\" (UID: \"a9562e58-739e-44e4-a2ca-448684bddd75\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" Oct 30 02:10:45.688497 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:45.688481 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9562e58-739e-44e4-a2ca-448684bddd75-workload-certs\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7\" (UID: \"a9562e58-739e-44e4-a2ca-448684bddd75\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" Oct 30 02:10:45.688591 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:45.688572 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9562e58-739e-44e4-a2ca-448684bddd75-istio-token\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7\" (UID: \"a9562e58-739e-44e4-a2ca-448684bddd75\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" Oct 30 02:10:45.789818 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:45.789697 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9562e58-739e-44e4-a2ca-448684bddd75-istio-data\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7\" (UID: \"a9562e58-739e-44e4-a2ca-448684bddd75\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" Oct 30 02:10:45.789818 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:45.789807 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9562e58-739e-44e4-a2ca-448684bddd75-istiod-ca-cert\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7\" (UID: \"a9562e58-739e-44e4-a2ca-448684bddd75\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" Oct 30 02:10:45.790367 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:45.789862 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9562e58-739e-44e4-a2ca-448684bddd75-istio-podinfo\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7\" (UID: \"a9562e58-739e-44e4-a2ca-448684bddd75\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" Oct 30 02:10:45.790367 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:45.789893 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9562e58-739e-44e4-a2ca-448684bddd75-istio-envoy\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7\" (UID: \"a9562e58-739e-44e4-a2ca-448684bddd75\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" Oct 30 02:10:45.790367 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:45.790006 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9562e58-739e-44e4-a2ca-448684bddd75-workload-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7\" (UID: \"a9562e58-739e-44e4-a2ca-448684bddd75\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" Oct 30 02:10:45.790367 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:45.790056 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq7tc\" (UniqueName: \"kubernetes.io/projected/a9562e58-739e-44e4-a2ca-448684bddd75-kube-api-access-vq7tc\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7\" (UID: \"a9562e58-739e-44e4-a2ca-448684bddd75\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" Oct 30 02:10:45.790367 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:45.790094 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9562e58-739e-44e4-a2ca-448684bddd75-credential-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7\" (UID: \"a9562e58-739e-44e4-a2ca-448684bddd75\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" Oct 30 02:10:45.790367 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:45.790151 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9562e58-739e-44e4-a2ca-448684bddd75-workload-certs\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7\" (UID: \"a9562e58-739e-44e4-a2ca-448684bddd75\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" Oct 30 02:10:45.790367 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:45.790171 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9562e58-739e-44e4-a2ca-448684bddd75-istio-data\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7\" (UID: \"a9562e58-739e-44e4-a2ca-448684bddd75\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" Oct 30 02:10:45.790367 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:45.790200 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9562e58-739e-44e4-a2ca-448684bddd75-istio-token\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7\" (UID: \"a9562e58-739e-44e4-a2ca-448684bddd75\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" Oct 30 02:10:45.790823 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:45.790446 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9562e58-739e-44e4-a2ca-448684bddd75-credential-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7\" (UID: \"a9562e58-739e-44e4-a2ca-448684bddd75\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" Oct 30 02:10:45.790823 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:45.790481 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9562e58-739e-44e4-a2ca-448684bddd75-workload-certs\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7\" (UID: \"a9562e58-739e-44e4-a2ca-448684bddd75\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" Oct 30 02:10:45.790823 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:45.790533 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9562e58-739e-44e4-a2ca-448684bddd75-istiod-ca-cert\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7\" (UID: \"a9562e58-739e-44e4-a2ca-448684bddd75\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" Oct 30 02:10:45.790823 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:45.790658 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9562e58-739e-44e4-a2ca-448684bddd75-workload-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7\" (UID: \"a9562e58-739e-44e4-a2ca-448684bddd75\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" Oct 30 02:10:45.792607 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:45.792586 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9562e58-739e-44e4-a2ca-448684bddd75-istio-envoy\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7\" (UID: \"a9562e58-739e-44e4-a2ca-448684bddd75\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" Oct 30 02:10:45.792889 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:45.792871 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9562e58-739e-44e4-a2ca-448684bddd75-istio-podinfo\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7\" (UID: \"a9562e58-739e-44e4-a2ca-448684bddd75\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" Oct 30 02:10:45.810070 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:45.810040 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9562e58-739e-44e4-a2ca-448684bddd75-istio-token\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7\" (UID: \"a9562e58-739e-44e4-a2ca-448684bddd75\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" Oct 30 02:10:45.810212 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:45.810100 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq7tc\" (UniqueName: \"kubernetes.io/projected/a9562e58-739e-44e4-a2ca-448684bddd75-kube-api-access-vq7tc\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7\" (UID: \"a9562e58-739e-44e4-a2ca-448684bddd75\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" Oct 30 02:10:45.838361 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:45.838319 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" Oct 30 02:10:46.163935 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:46.163899 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7"] Oct 30 02:10:46.173590 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:10:46.173563 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9562e58_739e_44e4_a2ca_448684bddd75.slice/crio-91ea36b2b356575e5990f1c9b9c8905048809ddf83d1f3389983b8485bfd10fa WatchSource:0}: Error finding container 91ea36b2b356575e5990f1c9b9c8905048809ddf83d1f3389983b8485bfd10fa: Status 404 returned error can't find the container with id 91ea36b2b356575e5990f1c9b9c8905048809ddf83d1f3389983b8485bfd10fa Oct 30 02:10:46.177343 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:46.177302 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:10:46.177447 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:46.177376 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:10:46.177447 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:46.177406 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:10:46.602993 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:46.602903 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" event={"ID":"a9562e58-739e-44e4-a2ca-448684bddd75","Type":"ContainerStarted","Data":"40ed7f4ae4dc6cf2f2f4eaf65012a6383221347affc60499a9bc46e91171d0c6"} Oct 30 02:10:46.602993 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:46.602969 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" event={"ID":"a9562e58-739e-44e4-a2ca-448684bddd75","Type":"ContainerStarted","Data":"91ea36b2b356575e5990f1c9b9c8905048809ddf83d1f3389983b8485bfd10fa"} Oct 30 02:10:46.669782 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:46.669116 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" podStartSLOduration=1.669092163 podStartE2EDuration="1.669092163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:10:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:10:46.654277447 +0000 UTC m=+3172.230702849" watchObservedRunningTime="2026-10-30 02:10:46.669092163 +0000 UTC m=+3172.245517565" Oct 30 02:10:46.692341 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:46.692298 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:46.839304 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:46.839257 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" Oct 30 02:10:47.841013 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:47.840973 6572 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.100:15021/healthz/ready\": dial tcp 10.133.1.100:15021: connect: connection refused" start-of-body= Oct 30 02:10:47.841440 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:47.841030 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" podUID="a9562e58-739e-44e4-a2ca-448684bddd75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.100:15021/healthz/ready\": dial tcp 10.133.1.100:15021: connect: connection refused" Oct 30 02:10:48.404984 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:48.404945 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:48.839359 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:48.839321 6572 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.100:15021/healthz/ready\": dial tcp 10.133.1.100:15021: connect: connection refused" start-of-body= Oct 30 02:10:48.839553 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:48.839374 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" podUID="a9562e58-739e-44e4-a2ca-448684bddd75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.100:15021/healthz/ready\": dial tcp 10.133.1.100:15021: connect: connection refused" Oct 30 02:10:49.838895 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:49.838859 6572 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.100:15021/healthz/ready\": dial tcp 10.133.1.100:15021: connect: connection refused" start-of-body= Oct 30 02:10:49.839324 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:49.838911 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" podUID="a9562e58-739e-44e4-a2ca-448684bddd75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.100:15021/healthz/ready\": dial tcp 10.133.1.100:15021: connect: connection refused" Oct 30 02:10:50.417881 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:50.415565 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:50.839715 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:50.839629 6572 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.100:15021/healthz/ready\": dial tcp 10.133.1.100:15021: connect: connection refused" start-of-body= Oct 30 02:10:50.839715 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:50.839694 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" podUID="a9562e58-739e-44e4-a2ca-448684bddd75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.100:15021/healthz/ready\": dial tcp 10.133.1.100:15021: connect: connection refused" Oct 30 02:10:51.839061 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:51.839019 6572 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.100:15021/healthz/ready\": dial tcp 10.133.1.100:15021: connect: connection refused" start-of-body= Oct 30 02:10:51.839268 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:51.839074 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" podUID="a9562e58-739e-44e4-a2ca-448684bddd75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.100:15021/healthz/ready\": dial tcp 10.133.1.100:15021: connect: connection refused" Oct 30 02:10:52.041882 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:52.041846 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87f0af23-82nbd"] Oct 30 02:10:52.053073 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:52.053043 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87f0af23-82nbd"] Oct 30 02:10:52.293216 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:52.293184 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:52.545881 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:52.545789 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4159ee6d-021b-4328-8481-e426234b034c" path="/var/lib/kubelet/pods/4159ee6d-021b-4328-8481-e426234b034c/volumes" Oct 30 02:10:52.839175 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:52.839085 6572 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.100:15021/healthz/ready\": dial tcp 10.133.1.100:15021: connect: connection refused" start-of-body= Oct 30 02:10:52.839175 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:52.839139 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" podUID="a9562e58-739e-44e4-a2ca-448684bddd75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.100:15021/healthz/ready\": dial tcp 10.133.1.100:15021: connect: connection refused" Oct 30 02:10:53.898201 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:53.898161 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" Oct 30 02:10:53.905269 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:53.905234 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" Oct 30 02:10:53.919311 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:53.919275 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" Oct 30 02:10:53.919868 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:53.919309 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:55.849060 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:55.849022 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:56.223135 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:56.223098 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:56.612554 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:56.612458 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:56.873672 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:56.873575 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bd81604-p5fjs"] Oct 30 02:10:56.877461 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:56.877438 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd81604-p5fjs" Oct 30 02:10:56.892582 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:56.892549 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:10:56.892797 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:56.892643 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:10:56.894251 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:56.894208 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e537af8-9a0b-4ff3-bb95-00cd183fa206-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bd81604-p5fjs\" (UID: \"5e537af8-9a0b-4ff3-bb95-00cd183fa206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd81604-p5fjs" Oct 30 02:10:56.894513 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:56.894489 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e537af8-9a0b-4ff3-bb95-00cd183fa206-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bd81604-p5fjs\" (UID: \"5e537af8-9a0b-4ff3-bb95-00cd183fa206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd81604-p5fjs" Oct 30 02:10:56.894599 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:56.894539 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flrg9\" (UniqueName: \"kubernetes.io/projected/5e537af8-9a0b-4ff3-bb95-00cd183fa206-kube-api-access-flrg9\") pod \"maas-api-revoke-sub-3bd81604-p5fjs\" (UID: \"5e537af8-9a0b-4ff3-bb95-00cd183fa206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd81604-p5fjs" Oct 30 02:10:56.905156 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:56.905122 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:10:56.905406 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:56.905297 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:10:56.911492 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:56.911429 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bd81604-p5fjs"] Oct 30 02:10:56.997266 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:56.996980 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e537af8-9a0b-4ff3-bb95-00cd183fa206-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bd81604-p5fjs\" (UID: \"5e537af8-9a0b-4ff3-bb95-00cd183fa206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd81604-p5fjs" Oct 30 02:10:56.997266 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:56.997197 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e537af8-9a0b-4ff3-bb95-00cd183fa206-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bd81604-p5fjs\" (UID: \"5e537af8-9a0b-4ff3-bb95-00cd183fa206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd81604-p5fjs" Oct 30 02:10:56.998572 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:56.997590 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flrg9\" (UniqueName: \"kubernetes.io/projected/5e537af8-9a0b-4ff3-bb95-00cd183fa206-kube-api-access-flrg9\") pod \"maas-api-revoke-sub-3bd81604-p5fjs\" (UID: \"5e537af8-9a0b-4ff3-bb95-00cd183fa206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd81604-p5fjs" Oct 30 02:10:57.000179 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:57.000120 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e537af8-9a0b-4ff3-bb95-00cd183fa206-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bd81604-p5fjs\" (UID: \"5e537af8-9a0b-4ff3-bb95-00cd183fa206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd81604-p5fjs" Oct 30 02:10:57.013198 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:57.013159 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e537af8-9a0b-4ff3-bb95-00cd183fa206-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bd81604-p5fjs\" (UID: \"5e537af8-9a0b-4ff3-bb95-00cd183fa206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd81604-p5fjs" Oct 30 02:10:57.018777 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:57.018725 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flrg9\" (UniqueName: \"kubernetes.io/projected/5e537af8-9a0b-4ff3-bb95-00cd183fa206-kube-api-access-flrg9\") pod \"maas-api-revoke-sub-3bd81604-p5fjs\" (UID: \"5e537af8-9a0b-4ff3-bb95-00cd183fa206\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd81604-p5fjs" Oct 30 02:10:57.192431 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:57.192391 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd81604-p5fjs" Oct 30 02:10:57.499011 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:57.498978 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bd81604-p5fjs"] Oct 30 02:10:57.500887 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:10:57.500856 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e537af8_9a0b_4ff3_bb95_00cd183fa206.slice/crio-175985ecbb417547c472d447899dc40a832dac13aff6e4a99a04f6c35b92bf70 WatchSource:0}: Error finding container 175985ecbb417547c472d447899dc40a832dac13aff6e4a99a04f6c35b92bf70: Status 404 returned error can't find the container with id 175985ecbb417547c472d447899dc40a832dac13aff6e4a99a04f6c35b92bf70 Oct 30 02:10:57.611771 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:57.611036 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:57.733640 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:57.733607 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd81604-p5fjs" event={"ID":"5e537af8-9a0b-4ff3-bb95-00cd183fa206","Type":"ContainerStarted","Data":"175985ecbb417547c472d447899dc40a832dac13aff6e4a99a04f6c35b92bf70"} Oct 30 02:10:58.746855 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:58.746740 6572 generic.go:356] "Generic (PLEG): container finished" podID="5e537af8-9a0b-4ff3-bb95-00cd183fa206" containerID="99ebf356dbc2d2e637ef5e8a9aca934caf6338ac02c72957dbc862f30a7e4b15" exitCode=0 Oct 30 02:10:58.747252 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:58.746844 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd81604-p5fjs" event={"ID":"5e537af8-9a0b-4ff3-bb95-00cd183fa206","Type":"ContainerDied","Data":"99ebf356dbc2d2e637ef5e8a9aca934caf6338ac02c72957dbc862f30a7e4b15"} Oct 30 02:10:58.813086 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:58.813055 6572 scope.go:122] "RemoveContainer" containerID="f1ed0d42d984a5df6191b633744d8a2870c9e5f0bd2c647c881b0d8d961e0ec1" Oct 30 02:10:59.200872 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:59.200833 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:59.410770 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:59.410656 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:59.942864 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:10:59.942833 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd81604-p5fjs" Oct 30 02:11:00.046249 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.046160 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e537af8-9a0b-4ff3-bb95-00cd183fa206-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e537af8-9a0b-4ff3-bb95-00cd183fa206-maas-api-xks-ca\") pod \"5e537af8-9a0b-4ff3-bb95-00cd183fa206\" (UID: \"5e537af8-9a0b-4ff3-bb95-00cd183fa206\") " Oct 30 02:11:00.046414 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.046276 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e537af8-9a0b-4ff3-bb95-00cd183fa206-kube-api-access-flrg9\" (UniqueName: \"kubernetes.io/projected/5e537af8-9a0b-4ff3-bb95-00cd183fa206-kube-api-access-flrg9\") pod \"5e537af8-9a0b-4ff3-bb95-00cd183fa206\" (UID: \"5e537af8-9a0b-4ff3-bb95-00cd183fa206\") " Oct 30 02:11:00.046414 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.046377 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e537af8-9a0b-4ff3-bb95-00cd183fa206-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e537af8-9a0b-4ff3-bb95-00cd183fa206-maas-api-openshift-ca\") pod \"5e537af8-9a0b-4ff3-bb95-00cd183fa206\" (UID: \"5e537af8-9a0b-4ff3-bb95-00cd183fa206\") " Oct 30 02:11:00.046715 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.046691 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e537af8-9a0b-4ff3-bb95-00cd183fa206-maas-api-openshift-ca" pod "5e537af8-9a0b-4ff3-bb95-00cd183fa206" (UID: "5e537af8-9a0b-4ff3-bb95-00cd183fa206"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:11:00.047077 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.047059 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e537af8-9a0b-4ff3-bb95-00cd183fa206-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:11:00.049455 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.049428 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e537af8-9a0b-4ff3-bb95-00cd183fa206-maas-api-xks-ca" pod "5e537af8-9a0b-4ff3-bb95-00cd183fa206" (UID: "5e537af8-9a0b-4ff3-bb95-00cd183fa206"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:11:00.049571 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.049462 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e537af8-9a0b-4ff3-bb95-00cd183fa206-kube-api-access-flrg9" pod "5e537af8-9a0b-4ff3-bb95-00cd183fa206" (UID: "5e537af8-9a0b-4ff3-bb95-00cd183fa206"). InnerVolumeSpecName "kube-api-access-flrg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:11:00.148391 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.148352 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e537af8-9a0b-4ff3-bb95-00cd183fa206-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:11:00.148391 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.148394 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flrg9\" (UniqueName: \"kubernetes.io/projected/5e537af8-9a0b-4ff3-bb95-00cd183fa206-kube-api-access-flrg9\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:11:00.768678 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.768648 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd81604-p5fjs" Oct 30 02:11:00.768939 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.768652 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd81604-p5fjs" event={"ID":"5e537af8-9a0b-4ff3-bb95-00cd183fa206","Type":"ContainerDied","Data":"175985ecbb417547c472d447899dc40a832dac13aff6e4a99a04f6c35b92bf70"} Oct 30 02:11:00.768939 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.768820 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="175985ecbb417547c472d447899dc40a832dac13aff6e4a99a04f6c35b92bf70" Oct 30 02:11:00.796587 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.796544 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb"] Oct 30 02:11:00.798850 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.798820 6572 cpu_manager.go:414] "containerMap: removing container" podUID="5e537af8-9a0b-4ff3-bb95-00cd183fa206" containerName="revoke-keys" Oct 30 02:11:00.799074 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.799057 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e537af8-9a0b-4ff3-bb95-00cd183fa206" containerName="revoke-keys" Oct 30 02:11:00.799799 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.799774 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e537af8-9a0b-4ff3-bb95-00cd183fa206" containerName="revoke-keys" Oct 30 02:11:00.815658 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.815619 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb" Oct 30 02:11:00.817890 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.817854 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb"] Oct 30 02:11:00.836091 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.836045 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-22e99d\"/\"default-dockercfg-zmjpf\"" Oct 30 02:11:00.836269 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.836120 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-22e99d\"/\"test-model-22e99d-kserve-self-signed-certs\"" Oct 30 02:11:00.843200 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.843171 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-22e99d\"/\"openshift-service-ca.crt\"" Oct 30 02:11:00.843428 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.843324 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-22e99d\"/\"kube-root-ca.crt\"" Oct 30 02:11:00.861856 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.861738 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3636031c-7bdd-4706-ab91-dfcf7bdb21be-model-cache\") pod \"test-model-22e99d-kserve-5cc5465c4f-ctcfb\" (UID: \"3636031c-7bdd-4706-ab91-dfcf7bdb21be\") " pod="ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb" Oct 30 02:11:00.861856 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.861799 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3636031c-7bdd-4706-ab91-dfcf7bdb21be-tls-certs\") pod \"test-model-22e99d-kserve-5cc5465c4f-ctcfb\" (UID: \"3636031c-7bdd-4706-ab91-dfcf7bdb21be\") " pod="ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb" Oct 30 02:11:00.862120 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.861961 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3636031c-7bdd-4706-ab91-dfcf7bdb21be-tmp-dir\") pod \"test-model-22e99d-kserve-5cc5465c4f-ctcfb\" (UID: \"3636031c-7bdd-4706-ab91-dfcf7bdb21be\") " pod="ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb" Oct 30 02:11:00.862120 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.862012 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3636031c-7bdd-4706-ab91-dfcf7bdb21be-dshm\") pod \"test-model-22e99d-kserve-5cc5465c4f-ctcfb\" (UID: \"3636031c-7bdd-4706-ab91-dfcf7bdb21be\") " pod="ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb" Oct 30 02:11:00.862120 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.862078 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w24v5\" (UniqueName: \"kubernetes.io/projected/3636031c-7bdd-4706-ab91-dfcf7bdb21be-kube-api-access-w24v5\") pod \"test-model-22e99d-kserve-5cc5465c4f-ctcfb\" (UID: \"3636031c-7bdd-4706-ab91-dfcf7bdb21be\") " pod="ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb" Oct 30 02:11:00.862304 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.862186 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3636031c-7bdd-4706-ab91-dfcf7bdb21be-home\") pod \"test-model-22e99d-kserve-5cc5465c4f-ctcfb\" (UID: \"3636031c-7bdd-4706-ab91-dfcf7bdb21be\") " pod="ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb" Oct 30 02:11:00.894067 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.894032 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:00.963384 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.963340 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3636031c-7bdd-4706-ab91-dfcf7bdb21be-model-cache\") pod \"test-model-22e99d-kserve-5cc5465c4f-ctcfb\" (UID: \"3636031c-7bdd-4706-ab91-dfcf7bdb21be\") " pod="ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb" Oct 30 02:11:00.963963 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.963387 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3636031c-7bdd-4706-ab91-dfcf7bdb21be-tls-certs\") pod \"test-model-22e99d-kserve-5cc5465c4f-ctcfb\" (UID: \"3636031c-7bdd-4706-ab91-dfcf7bdb21be\") " pod="ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb" Oct 30 02:11:00.963963 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.963504 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3636031c-7bdd-4706-ab91-dfcf7bdb21be-tmp-dir\") pod \"test-model-22e99d-kserve-5cc5465c4f-ctcfb\" (UID: \"3636031c-7bdd-4706-ab91-dfcf7bdb21be\") " pod="ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb" Oct 30 02:11:00.963963 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.963557 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3636031c-7bdd-4706-ab91-dfcf7bdb21be-dshm\") pod \"test-model-22e99d-kserve-5cc5465c4f-ctcfb\" (UID: \"3636031c-7bdd-4706-ab91-dfcf7bdb21be\") " pod="ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb" Oct 30 02:11:00.963963 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.963579 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w24v5\" (UniqueName: \"kubernetes.io/projected/3636031c-7bdd-4706-ab91-dfcf7bdb21be-kube-api-access-w24v5\") pod \"test-model-22e99d-kserve-5cc5465c4f-ctcfb\" (UID: \"3636031c-7bdd-4706-ab91-dfcf7bdb21be\") " pod="ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb" Oct 30 02:11:00.963963 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.963628 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3636031c-7bdd-4706-ab91-dfcf7bdb21be-home\") pod \"test-model-22e99d-kserve-5cc5465c4f-ctcfb\" (UID: \"3636031c-7bdd-4706-ab91-dfcf7bdb21be\") " pod="ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb" Oct 30 02:11:00.963963 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.963914 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3636031c-7bdd-4706-ab91-dfcf7bdb21be-model-cache\") pod \"test-model-22e99d-kserve-5cc5465c4f-ctcfb\" (UID: \"3636031c-7bdd-4706-ab91-dfcf7bdb21be\") " pod="ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb" Oct 30 02:11:00.963963 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.963951 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3636031c-7bdd-4706-ab91-dfcf7bdb21be-tmp-dir\") pod \"test-model-22e99d-kserve-5cc5465c4f-ctcfb\" (UID: \"3636031c-7bdd-4706-ab91-dfcf7bdb21be\") " pod="ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb" Oct 30 02:11:00.964236 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.964196 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3636031c-7bdd-4706-ab91-dfcf7bdb21be-home\") pod \"test-model-22e99d-kserve-5cc5465c4f-ctcfb\" (UID: \"3636031c-7bdd-4706-ab91-dfcf7bdb21be\") " pod="ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb" Oct 30 02:11:00.965955 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.965928 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3636031c-7bdd-4706-ab91-dfcf7bdb21be-dshm\") pod \"test-model-22e99d-kserve-5cc5465c4f-ctcfb\" (UID: \"3636031c-7bdd-4706-ab91-dfcf7bdb21be\") " pod="ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb" Oct 30 02:11:00.966237 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.966218 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3636031c-7bdd-4706-ab91-dfcf7bdb21be-tls-certs\") pod \"test-model-22e99d-kserve-5cc5465c4f-ctcfb\" (UID: \"3636031c-7bdd-4706-ab91-dfcf7bdb21be\") " pod="ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb" Oct 30 02:11:00.987576 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:00.987539 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w24v5\" (UniqueName: \"kubernetes.io/projected/3636031c-7bdd-4706-ab91-dfcf7bdb21be-kube-api-access-w24v5\") pod \"test-model-22e99d-kserve-5cc5465c4f-ctcfb\" (UID: \"3636031c-7bdd-4706-ab91-dfcf7bdb21be\") " pod="ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb" Oct 30 02:11:01.139673 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:01.137114 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:01.139673 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:01.138869 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb" Oct 30 02:11:01.526974 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:01.526926 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb"] Oct 30 02:11:01.531001 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:11:01.530961 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3636031c_7bdd_4706_ab91_dfcf7bdb21be.slice/crio-f5214bc28b2bea305d1159954934a889cbc0deb16246c4a34fb3e944ceefdd9f WatchSource:0}: Error finding container f5214bc28b2bea305d1159954934a889cbc0deb16246c4a34fb3e944ceefdd9f: Status 404 returned error can't find the container with id f5214bc28b2bea305d1159954934a889cbc0deb16246c4a34fb3e944ceefdd9f Oct 30 02:11:01.779671 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:01.779574 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb" event={"ID":"3636031c-7bdd-4706-ab91-dfcf7bdb21be","Type":"ContainerStarted","Data":"bcb2293f39beb059f491b70c11f115a6bf691a4390bbe5c5f64fef9d62fff6cd"} Oct 30 02:11:01.779671 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:01.779618 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb" event={"ID":"3636031c-7bdd-4706-ab91-dfcf7bdb21be","Type":"ContainerStarted","Data":"f5214bc28b2bea305d1159954934a889cbc0deb16246c4a34fb3e944ceefdd9f"} Oct 30 02:11:01.827359 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:01.827291 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb" podStartSLOduration=1.8272732440000001 podStartE2EDuration="1.827273244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:11:01.821914167 +0000 UTC m=+3187.398339594" watchObservedRunningTime="2026-10-30 02:11:01.827273244 +0000 UTC m=+3187.403698722" Oct 30 02:11:02.139984 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:02.139893 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb" Oct 30 02:11:02.700777 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:02.698227 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:03.034793 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:03.034670 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcc8afe2-2l9cz"] Oct 30 02:11:03.045809 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:03.045775 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcc8afe2-2l9cz"] Oct 30 02:11:04.310661 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:04.310624 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:04.547010 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:04.546976 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b1b3330-e331-4b2f-b300-13c65a309666" path="/var/lib/kubelet/pods/5b1b3330-e331-4b2f-b300-13c65a309666/volumes" Oct 30 02:11:05.865619 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:05.865584 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:07.575766 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:07.575582 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c951947-rhrl4"] Oct 30 02:11:07.592771 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:07.583995 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c951947-rhrl4" Oct 30 02:11:07.607207 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:07.607173 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:11:07.607914 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:07.607582 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:11:07.608105 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:07.607737 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:11:07.608205 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:07.607785 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:07.608348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:07.607865 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:11:07.611325 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:07.611297 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c951947-rhrl4"] Oct 30 02:11:07.700678 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:07.700639 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9a81242-e317-4648-ab26-831696c7e08e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c951947-rhrl4\" (UID: \"f9a81242-e317-4648-ab26-831696c7e08e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c951947-rhrl4" Oct 30 02:11:07.700941 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:07.700915 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgkqb\" (UniqueName: \"kubernetes.io/projected/f9a81242-e317-4648-ab26-831696c7e08e-kube-api-access-dgkqb\") pod \"maas-api-revoke-sub-6c951947-rhrl4\" (UID: \"f9a81242-e317-4648-ab26-831696c7e08e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c951947-rhrl4" Oct 30 02:11:07.700991 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:07.700960 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9a81242-e317-4648-ab26-831696c7e08e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c951947-rhrl4\" (UID: \"f9a81242-e317-4648-ab26-831696c7e08e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c951947-rhrl4" Oct 30 02:11:07.802936 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:07.802899 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgkqb\" (UniqueName: \"kubernetes.io/projected/f9a81242-e317-4648-ab26-831696c7e08e-kube-api-access-dgkqb\") pod \"maas-api-revoke-sub-6c951947-rhrl4\" (UID: \"f9a81242-e317-4648-ab26-831696c7e08e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c951947-rhrl4" Oct 30 02:11:07.803151 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:07.802944 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9a81242-e317-4648-ab26-831696c7e08e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c951947-rhrl4\" (UID: \"f9a81242-e317-4648-ab26-831696c7e08e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c951947-rhrl4" Oct 30 02:11:07.803151 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:07.803051 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9a81242-e317-4648-ab26-831696c7e08e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c951947-rhrl4\" (UID: \"f9a81242-e317-4648-ab26-831696c7e08e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c951947-rhrl4" Oct 30 02:11:07.803711 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:07.803685 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9a81242-e317-4648-ab26-831696c7e08e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c951947-rhrl4\" (UID: \"f9a81242-e317-4648-ab26-831696c7e08e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c951947-rhrl4" Oct 30 02:11:07.805601 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:07.805574 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9a81242-e317-4648-ab26-831696c7e08e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c951947-rhrl4\" (UID: \"f9a81242-e317-4648-ab26-831696c7e08e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c951947-rhrl4" Oct 30 02:11:07.823119 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:07.823087 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgkqb\" (UniqueName: \"kubernetes.io/projected/f9a81242-e317-4648-ab26-831696c7e08e-kube-api-access-dgkqb\") pod \"maas-api-revoke-sub-6c951947-rhrl4\" (UID: \"f9a81242-e317-4648-ab26-831696c7e08e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c951947-rhrl4" Oct 30 02:11:07.939087 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:07.939045 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c951947-rhrl4" Oct 30 02:11:08.242433 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:11:08.242395 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9a81242_e317_4648_ab26_831696c7e08e.slice/crio-236ac57714d0c82f957edc355dbe0627bdbc46d90ed2474b25c908f1d466b7b6 WatchSource:0}: Error finding container 236ac57714d0c82f957edc355dbe0627bdbc46d90ed2474b25c908f1d466b7b6: Status 404 returned error can't find the container with id 236ac57714d0c82f957edc355dbe0627bdbc46d90ed2474b25c908f1d466b7b6 Oct 30 02:11:08.243398 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:08.243363 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c951947-rhrl4"] Oct 30 02:11:08.413734 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:08.413695 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:08.902350 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:08.902261 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c951947-rhrl4" event={"ID":"f9a81242-e317-4648-ab26-831696c7e08e","Type":"ContainerStarted","Data":"7486ce9f73778af1673217a6d2e89ed7f8c09a043082594bd2c334b8ac034518"} Oct 30 02:11:08.902350 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:08.902320 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c951947-rhrl4" event={"ID":"f9a81242-e317-4648-ab26-831696c7e08e","Type":"ContainerStarted","Data":"236ac57714d0c82f957edc355dbe0627bdbc46d90ed2474b25c908f1d466b7b6"} Oct 30 02:11:08.948375 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:08.948314 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c951947-rhrl4" podStartSLOduration=1.948295638 podStartE2EDuration="1.948295638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:11:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:11:08.942381661 +0000 UTC m=+3194.518807063" watchObservedRunningTime="2026-10-30 02:11:08.948295638 +0000 UTC m=+3194.524721067" Oct 30 02:11:09.239871 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:09.234366 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:09.913228 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:09.913192 6572 generic.go:356] "Generic (PLEG): container finished" podID="f9a81242-e317-4648-ab26-831696c7e08e" containerID="7486ce9f73778af1673217a6d2e89ed7f8c09a043082594bd2c334b8ac034518" exitCode=0 Oct 30 02:11:09.913708 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:09.913279 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c951947-rhrl4" event={"ID":"f9a81242-e317-4648-ab26-831696c7e08e","Type":"ContainerDied","Data":"7486ce9f73778af1673217a6d2e89ed7f8c09a043082594bd2c334b8ac034518"} Oct 30 02:11:10.893777 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:10.889156 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:11.093489 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:11.093463 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c951947-rhrl4" Oct 30 02:11:11.140812 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:11.140764 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb" Oct 30 02:11:11.155461 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:11.155380 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb" Oct 30 02:11:11.161600 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:11.161570 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9a81242-e317-4648-ab26-831696c7e08e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9a81242-e317-4648-ab26-831696c7e08e-maas-api-openshift-ca\") pod \"f9a81242-e317-4648-ab26-831696c7e08e\" (UID: \"f9a81242-e317-4648-ab26-831696c7e08e\") " Oct 30 02:11:11.161912 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:11.161894 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9a81242-e317-4648-ab26-831696c7e08e-kube-api-access-dgkqb\" (UniqueName: \"kubernetes.io/projected/f9a81242-e317-4648-ab26-831696c7e08e-kube-api-access-dgkqb\") pod \"f9a81242-e317-4648-ab26-831696c7e08e\" (UID: \"f9a81242-e317-4648-ab26-831696c7e08e\") " Oct 30 02:11:11.162007 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:11.161931 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9a81242-e317-4648-ab26-831696c7e08e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9a81242-e317-4648-ab26-831696c7e08e-maas-api-xks-ca\") pod \"f9a81242-e317-4648-ab26-831696c7e08e\" (UID: \"f9a81242-e317-4648-ab26-831696c7e08e\") " Oct 30 02:11:11.162095 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:11.162058 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9a81242-e317-4648-ab26-831696c7e08e-maas-api-openshift-ca" pod "f9a81242-e317-4648-ab26-831696c7e08e" (UID: "f9a81242-e317-4648-ab26-831696c7e08e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:11:11.162638 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:11.162611 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9a81242-e317-4648-ab26-831696c7e08e-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:11:11.164454 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:11.164430 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9a81242-e317-4648-ab26-831696c7e08e-kube-api-access-dgkqb" pod "f9a81242-e317-4648-ab26-831696c7e08e" (UID: "f9a81242-e317-4648-ab26-831696c7e08e"). InnerVolumeSpecName "kube-api-access-dgkqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:11:11.166231 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:11.166204 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9a81242-e317-4648-ab26-831696c7e08e-maas-api-xks-ca" pod "f9a81242-e317-4648-ab26-831696c7e08e" (UID: "f9a81242-e317-4648-ab26-831696c7e08e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:11:11.264793 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:11.264729 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgkqb\" (UniqueName: \"kubernetes.io/projected/f9a81242-e317-4648-ab26-831696c7e08e-kube-api-access-dgkqb\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:11:11.264793 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:11.264789 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9a81242-e317-4648-ab26-831696c7e08e-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:11:11.934596 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:11.934565 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c951947-rhrl4" Oct 30 02:11:11.934815 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:11.934612 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c951947-rhrl4" event={"ID":"f9a81242-e317-4648-ab26-831696c7e08e","Type":"ContainerDied","Data":"236ac57714d0c82f957edc355dbe0627bdbc46d90ed2474b25c908f1d466b7b6"} Oct 30 02:11:11.934815 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:11.934657 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="236ac57714d0c82f957edc355dbe0627bdbc46d90ed2474b25c908f1d466b7b6" Oct 30 02:11:11.947016 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:11.946988 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb" Oct 30 02:11:12.454695 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:12.454659 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:14.104541 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:14.104506 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:14.690438 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:11:14.690400 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:11:15.752681 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:15.752640 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:16.036951 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:16.036863 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e7e449c-wg98f"] Oct 30 02:11:16.048073 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:16.048034 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e7e449c-wg98f"] Oct 30 02:11:16.546082 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:16.546047 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9058c6dd-6552-4d20-830e-14e0e2c90745" path="/var/lib/kubelet/pods/9058c6dd-6552-4d20-830e-14e0e2c90745/volumes" Oct 30 02:11:17.403567 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:17.403529 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:19.002081 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:19.002048 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:19.166126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:19.166089 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt"] Oct 30 02:11:19.349157 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:19.349047 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:19.566873 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:19.566835 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:19.985381 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:19.985339 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:20.241127 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:20.241034 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:20.947389 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:20.947353 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:21.038947 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:21.038911 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e64858ed-tljsz"] Oct 30 02:11:21.049180 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:21.049147 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e64858ed-tljsz"] Oct 30 02:11:22.527948 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:22.527913 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:22.550390 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:22.550353 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4de42b0f-0785-47fc-9b4a-7a58b82e2934" path="/var/lib/kubelet/pods/4de42b0f-0785-47fc-9b4a-7a58b82e2934/volumes" Oct 30 02:11:24.120527 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:24.120488 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:24.167347 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:24.167299 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt" podUID="87500d13-55e9-4243-b3f9-617d55ce4202" containerName="payload-processing" containerID="cri-o://7d3cf9e67d2d884b0a506acc8748b743a25c426531cf173aa983fc003105bf81" gracePeriod=25 Oct 30 02:11:24.510837 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:24.510802 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt" Oct 30 02:11:24.566631 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:24.566580 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87500d13-55e9-4243-b3f9-617d55ce4202-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87500d13-55e9-4243-b3f9-617d55ce4202-routing-overlay-volume\") pod \"87500d13-55e9-4243-b3f9-617d55ce4202\" (UID: \"87500d13-55e9-4243-b3f9-617d55ce4202\") " Oct 30 02:11:24.566882 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:24.566863 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87500d13-55e9-4243-b3f9-617d55ce4202-provider-credentials\" (UniqueName: \"kubernetes.io/projected/87500d13-55e9-4243-b3f9-617d55ce4202-provider-credentials\") pod \"87500d13-55e9-4243-b3f9-617d55ce4202\" (UID: \"87500d13-55e9-4243-b3f9-617d55ce4202\") " Oct 30 02:11:24.566983 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:24.566945 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87500d13-55e9-4243-b3f9-617d55ce4202-kube-api-access-lvnds\" (UniqueName: \"kubernetes.io/projected/87500d13-55e9-4243-b3f9-617d55ce4202-kube-api-access-lvnds\") pod \"87500d13-55e9-4243-b3f9-617d55ce4202\" (UID: \"87500d13-55e9-4243-b3f9-617d55ce4202\") " Oct 30 02:11:24.566983 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:24.566974 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87500d13-55e9-4243-b3f9-617d55ce4202-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87500d13-55e9-4243-b3f9-617d55ce4202-plugins-config-volume\") pod \"87500d13-55e9-4243-b3f9-617d55ce4202\" (UID: \"87500d13-55e9-4243-b3f9-617d55ce4202\") " Oct 30 02:11:24.568386 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:24.568354 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87500d13-55e9-4243-b3f9-617d55ce4202-plugins-config-volume" pod "87500d13-55e9-4243-b3f9-617d55ce4202" (UID: "87500d13-55e9-4243-b3f9-617d55ce4202"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:11:24.568800 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:24.568735 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87500d13-55e9-4243-b3f9-617d55ce4202-routing-overlay-volume" pod "87500d13-55e9-4243-b3f9-617d55ce4202" (UID: "87500d13-55e9-4243-b3f9-617d55ce4202"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:11:24.572079 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:24.572043 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87500d13-55e9-4243-b3f9-617d55ce4202-kube-api-access-lvnds" pod "87500d13-55e9-4243-b3f9-617d55ce4202" (UID: "87500d13-55e9-4243-b3f9-617d55ce4202"). InnerVolumeSpecName "kube-api-access-lvnds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:11:24.573242 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:24.573212 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87500d13-55e9-4243-b3f9-617d55ce4202-provider-credentials" pod "87500d13-55e9-4243-b3f9-617d55ce4202" (UID: "87500d13-55e9-4243-b3f9-617d55ce4202"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:11:24.669975 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:24.669934 6572 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87500d13-55e9-4243-b3f9-617d55ce4202-routing-overlay-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:11:24.669975 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:24.669975 6572 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/87500d13-55e9-4243-b3f9-617d55ce4202-provider-credentials\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:11:24.670191 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:24.669990 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvnds\" (UniqueName: \"kubernetes.io/projected/87500d13-55e9-4243-b3f9-617d55ce4202-kube-api-access-lvnds\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:11:24.670191 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:24.670005 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87500d13-55e9-4243-b3f9-617d55ce4202-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:11:25.171115 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:25.171076 6572 generic.go:356] "Generic (PLEG): container finished" podID="87500d13-55e9-4243-b3f9-617d55ce4202" containerID="7d3cf9e67d2d884b0a506acc8748b743a25c426531cf173aa983fc003105bf81" exitCode=0 Oct 30 02:11:25.171708 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:25.171129 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt" Oct 30 02:11:25.171708 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:25.171158 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt" event={"ID":"87500d13-55e9-4243-b3f9-617d55ce4202","Type":"ContainerDied","Data":"7d3cf9e67d2d884b0a506acc8748b743a25c426531cf173aa983fc003105bf81"} Oct 30 02:11:25.171708 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:25.171214 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt" event={"ID":"87500d13-55e9-4243-b3f9-617d55ce4202","Type":"ContainerDied","Data":"b825b19ec567504d9d080da2c8e9f6de945ca5a02d68ed32ad98fa16dcd5fdd2"} Oct 30 02:11:25.171708 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:25.171236 6572 scope.go:122] "RemoveContainer" containerID="7d3cf9e67d2d884b0a506acc8748b743a25c426531cf173aa983fc003105bf81" Oct 30 02:11:25.184627 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:25.184586 6572 scope.go:122] "RemoveContainer" containerID="7d3cf9e67d2d884b0a506acc8748b743a25c426531cf173aa983fc003105bf81" Oct 30 02:11:25.185001 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:11:25.184971 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d3cf9e67d2d884b0a506acc8748b743a25c426531cf173aa983fc003105bf81\": container with ID starting with 7d3cf9e67d2d884b0a506acc8748b743a25c426531cf173aa983fc003105bf81 not found: ID does not exist" containerID="7d3cf9e67d2d884b0a506acc8748b743a25c426531cf173aa983fc003105bf81" Oct 30 02:11:25.185101 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:25.185009 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d3cf9e67d2d884b0a506acc8748b743a25c426531cf173aa983fc003105bf81"} err="failed to get container status \"7d3cf9e67d2d884b0a506acc8748b743a25c426531cf173aa983fc003105bf81\": rpc error: code = NotFound desc = could not find container \"7d3cf9e67d2d884b0a506acc8748b743a25c426531cf173aa983fc003105bf81\": container with ID starting with 7d3cf9e67d2d884b0a506acc8748b743a25c426531cf173aa983fc003105bf81 not found: ID does not exist" Oct 30 02:11:25.223502 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:25.223460 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt"] Oct 30 02:11:25.235267 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:25.235225 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-7f65gwkbt"] Oct 30 02:11:25.859213 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:25.859081 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:26.047908 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:26.047869 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2560735-2cn2b"] Oct 30 02:11:26.058429 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:26.058399 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2560735-2cn2b"] Oct 30 02:11:26.545935 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:26.545900 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c8d84a7-9256-4836-b428-26e184dd11f7" path="/var/lib/kubelet/pods/2c8d84a7-9256-4836-b428-26e184dd11f7/volumes" Oct 30 02:11:26.546475 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:26.546458 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87500d13-55e9-4243-b3f9-617d55ce4202" path="/var/lib/kubelet/pods/87500d13-55e9-4243-b3f9-617d55ce4202/volumes" Oct 30 02:11:26.845133 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:26.844040 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:27.061835 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:27.061793 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:27.428826 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:27.428783 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:27.637207 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:27.637156 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:27.768777 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:27.768663 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw"] Oct 30 02:11:27.769661 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:27.769635 6572 cpu_manager.go:414] "containerMap: removing container" podUID="f9a81242-e317-4648-ab26-831696c7e08e" containerName="revoke-keys" Oct 30 02:11:27.769661 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:27.769656 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9a81242-e317-4648-ab26-831696c7e08e" containerName="revoke-keys" Oct 30 02:11:27.769950 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:27.769715 6572 cpu_manager.go:414] "containerMap: removing container" podUID="87500d13-55e9-4243-b3f9-617d55ce4202" containerName="payload-processing" Oct 30 02:11:27.769950 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:27.769721 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87500d13-55e9-4243-b3f9-617d55ce4202" containerName="payload-processing" Oct 30 02:11:27.769950 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:27.769876 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9a81242-e317-4648-ab26-831696c7e08e" containerName="revoke-keys" Oct 30 02:11:27.769950 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:27.769894 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="87500d13-55e9-4243-b3f9-617d55ce4202" containerName="payload-processing" Oct 30 02:11:27.774899 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:27.774878 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:11:27.803982 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:27.803944 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-340ae6-openshift-default-dockercfg-k5b4c\"" Oct 30 02:11:27.815519 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:27.815478 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw"] Oct 30 02:11:27.938773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:27.929489 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d32bbe46-7de7-4307-a62d-ee16d8b59741-workload-certs\") pod \"e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw\" (UID: \"d32bbe46-7de7-4307-a62d-ee16d8b59741\") " pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:11:27.938773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:27.929583 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d32bbe46-7de7-4307-a62d-ee16d8b59741-istio-podinfo\") pod \"e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw\" (UID: \"d32bbe46-7de7-4307-a62d-ee16d8b59741\") " pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:11:27.938773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:27.929700 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32bbe46-7de7-4307-a62d-ee16d8b59741-credential-socket\") pod \"e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw\" (UID: \"d32bbe46-7de7-4307-a62d-ee16d8b59741\") " pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:11:27.938773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:27.929772 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d32bbe46-7de7-4307-a62d-ee16d8b59741-istio-token\") pod \"e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw\" (UID: \"d32bbe46-7de7-4307-a62d-ee16d8b59741\") " pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:11:27.938773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:27.929955 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d32bbe46-7de7-4307-a62d-ee16d8b59741-istio-data\") pod \"e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw\" (UID: \"d32bbe46-7de7-4307-a62d-ee16d8b59741\") " pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:11:27.938773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:27.930057 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d32bbe46-7de7-4307-a62d-ee16d8b59741-istio-envoy\") pod \"e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw\" (UID: \"d32bbe46-7de7-4307-a62d-ee16d8b59741\") " pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:11:27.938773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:27.930093 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32bbe46-7de7-4307-a62d-ee16d8b59741-workload-socket\") pod \"e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw\" (UID: \"d32bbe46-7de7-4307-a62d-ee16d8b59741\") " pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:11:27.938773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:27.930167 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g52cc\" (UniqueName: \"kubernetes.io/projected/d32bbe46-7de7-4307-a62d-ee16d8b59741-kube-api-access-g52cc\") pod \"e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw\" (UID: \"d32bbe46-7de7-4307-a62d-ee16d8b59741\") " pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:11:27.938773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:27.930225 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d32bbe46-7de7-4307-a62d-ee16d8b59741-istiod-ca-cert\") pod \"e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw\" (UID: \"d32bbe46-7de7-4307-a62d-ee16d8b59741\") " pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:11:27.938773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:27.930420 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:28.039833 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:28.038801 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d32bbe46-7de7-4307-a62d-ee16d8b59741-workload-certs\") pod \"e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw\" (UID: \"d32bbe46-7de7-4307-a62d-ee16d8b59741\") " pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:11:28.039833 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:28.038887 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d32bbe46-7de7-4307-a62d-ee16d8b59741-istio-podinfo\") pod \"e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw\" (UID: \"d32bbe46-7de7-4307-a62d-ee16d8b59741\") " pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:11:28.039833 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:28.038990 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32bbe46-7de7-4307-a62d-ee16d8b59741-credential-socket\") pod \"e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw\" (UID: \"d32bbe46-7de7-4307-a62d-ee16d8b59741\") " pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:11:28.039833 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:28.039032 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d32bbe46-7de7-4307-a62d-ee16d8b59741-istio-token\") pod \"e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw\" (UID: \"d32bbe46-7de7-4307-a62d-ee16d8b59741\") " pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:11:28.039833 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:28.039109 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d32bbe46-7de7-4307-a62d-ee16d8b59741-istio-data\") pod \"e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw\" (UID: \"d32bbe46-7de7-4307-a62d-ee16d8b59741\") " pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:11:28.039833 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:28.039190 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d32bbe46-7de7-4307-a62d-ee16d8b59741-istio-envoy\") pod \"e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw\" (UID: \"d32bbe46-7de7-4307-a62d-ee16d8b59741\") " pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:11:28.039833 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:28.039221 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32bbe46-7de7-4307-a62d-ee16d8b59741-workload-socket\") pod \"e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw\" (UID: \"d32bbe46-7de7-4307-a62d-ee16d8b59741\") " pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:11:28.039833 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:28.039293 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g52cc\" (UniqueName: \"kubernetes.io/projected/d32bbe46-7de7-4307-a62d-ee16d8b59741-kube-api-access-g52cc\") pod \"e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw\" (UID: \"d32bbe46-7de7-4307-a62d-ee16d8b59741\") " pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:11:28.039833 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:28.039348 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d32bbe46-7de7-4307-a62d-ee16d8b59741-istiod-ca-cert\") pod \"e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw\" (UID: \"d32bbe46-7de7-4307-a62d-ee16d8b59741\") " pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:11:28.067784 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:28.067033 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d32bbe46-7de7-4307-a62d-ee16d8b59741-istiod-ca-cert\") pod \"e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw\" (UID: \"d32bbe46-7de7-4307-a62d-ee16d8b59741\") " pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:11:28.069443 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:28.069267 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32bbe46-7de7-4307-a62d-ee16d8b59741-workload-socket\") pod \"e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw\" (UID: \"d32bbe46-7de7-4307-a62d-ee16d8b59741\") " pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:11:28.069785 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:28.069764 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d32bbe46-7de7-4307-a62d-ee16d8b59741-istio-data\") pod \"e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw\" (UID: \"d32bbe46-7de7-4307-a62d-ee16d8b59741\") " pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:11:28.070078 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:28.070057 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d32bbe46-7de7-4307-a62d-ee16d8b59741-workload-certs\") pod \"e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw\" (UID: \"d32bbe46-7de7-4307-a62d-ee16d8b59741\") " pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:11:28.070340 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:28.070282 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32bbe46-7de7-4307-a62d-ee16d8b59741-credential-socket\") pod \"e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw\" (UID: \"d32bbe46-7de7-4307-a62d-ee16d8b59741\") " pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:11:28.108839 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:28.091884 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d32bbe46-7de7-4307-a62d-ee16d8b59741-istio-envoy\") pod \"e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw\" (UID: \"d32bbe46-7de7-4307-a62d-ee16d8b59741\") " pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:11:28.108839 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:28.098900 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d32bbe46-7de7-4307-a62d-ee16d8b59741-istio-token\") pod \"e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw\" (UID: \"d32bbe46-7de7-4307-a62d-ee16d8b59741\") " pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:11:28.108839 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:28.108834 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g52cc\" (UniqueName: \"kubernetes.io/projected/d32bbe46-7de7-4307-a62d-ee16d8b59741-kube-api-access-g52cc\") pod \"e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw\" (UID: \"d32bbe46-7de7-4307-a62d-ee16d8b59741\") " pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:11:28.122773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:28.117484 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d32bbe46-7de7-4307-a62d-ee16d8b59741-istio-podinfo\") pod \"e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw\" (UID: \"d32bbe46-7de7-4307-a62d-ee16d8b59741\") " pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:11:28.122773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:28.120544 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:28.354162 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:28.354072 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:28.391634 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:28.391600 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:11:28.718267 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:28.718234 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw"] Oct 30 02:11:28.718672 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:11:28.718650 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd32bbe46_7de7_4307_a62d_ee16d8b59741.slice/crio-5e5b46695b4578f68d4a914dbcf145c16ce9de34356c16da454489233da66424 WatchSource:0}: Error finding container 5e5b46695b4578f68d4a914dbcf145c16ce9de34356c16da454489233da66424: Status 404 returned error can't find the container with id 5e5b46695b4578f68d4a914dbcf145c16ce9de34356c16da454489233da66424 Oct 30 02:11:28.721886 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:28.721852 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:11:28.721968 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:28.721924 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:11:28.721968 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:28.721953 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:11:29.221116 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:29.221075 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" event={"ID":"d32bbe46-7de7-4307-a62d-ee16d8b59741","Type":"ContainerStarted","Data":"bb1dffbbc8f6b7d54e0b3092038eeadb2860e27159c7f4fcde7bb5b692499bee"} Oct 30 02:11:29.221358 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:29.221145 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" event={"ID":"d32bbe46-7de7-4307-a62d-ee16d8b59741","Type":"ContainerStarted","Data":"5e5b46695b4578f68d4a914dbcf145c16ce9de34356c16da454489233da66424"} Oct 30 02:11:29.290131 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:29.280218 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:29.307822 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:29.306420 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" podStartSLOduration=2.30640081 podStartE2EDuration="2.30640081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:11:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:11:29.280383202 +0000 UTC m=+3214.856808628" watchObservedRunningTime="2026-10-30 02:11:29.30640081 +0000 UTC m=+3214.882826212" Oct 30 02:11:29.392828 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:29.392741 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:11:29.411771 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:29.405988 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:11:30.231703 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:30.231672 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:11:30.232702 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:30.232684 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:11:30.880499 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:30.880179 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:31.905597 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:31.905562 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:32.133514 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:32.133478 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:32.765767 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:32.765028 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:33.622324 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:33.622283 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:33.644402 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:33.644368 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-340ae6-58897f9598-pv72n"] Oct 30 02:11:33.649699 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:33.649678 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-340ae6-58897f9598-pv72n" Oct 30 02:11:33.669457 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:33.669429 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-340ae6-58897f9598-pv72n"] Oct 30 02:11:33.673728 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:33.673704 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-340ae6\"" Oct 30 02:11:33.770282 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:33.770191 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxz79\" (UniqueName: \"kubernetes.io/projected/90887a7d-bb7c-4c14-9e63-0866a137918b-kube-api-access-wxz79\") pod \"payload-processing-e2e-inf-a-340ae6-58897f9598-pv72n\" (UID: \"90887a7d-bb7c-4c14-9e63-0866a137918b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-340ae6-58897f9598-pv72n" Oct 30 02:11:33.838772 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:33.811137 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90887a7d-bb7c-4c14-9e63-0866a137918b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-340ae6-58897f9598-pv72n\" (UID: \"90887a7d-bb7c-4c14-9e63-0866a137918b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-340ae6-58897f9598-pv72n" Oct 30 02:11:33.839283 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:33.770852 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-340ae6-dockercfg-d9z9d\"" Oct 30 02:11:33.864857 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:33.847301 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:33.947776 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:33.947142 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxz79\" (UniqueName: \"kubernetes.io/projected/90887a7d-bb7c-4c14-9e63-0866a137918b-kube-api-access-wxz79\") pod \"payload-processing-e2e-inf-a-340ae6-58897f9598-pv72n\" (UID: \"90887a7d-bb7c-4c14-9e63-0866a137918b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-340ae6-58897f9598-pv72n" Oct 30 02:11:33.947776 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:33.947218 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90887a7d-bb7c-4c14-9e63-0866a137918b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-340ae6-58897f9598-pv72n\" (UID: \"90887a7d-bb7c-4c14-9e63-0866a137918b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-340ae6-58897f9598-pv72n" Oct 30 02:11:33.953771 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:33.948286 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90887a7d-bb7c-4c14-9e63-0866a137918b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-340ae6-58897f9598-pv72n\" (UID: \"90887a7d-bb7c-4c14-9e63-0866a137918b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-340ae6-58897f9598-pv72n" Oct 30 02:11:34.012853 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:34.004299 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxz79\" (UniqueName: \"kubernetes.io/projected/90887a7d-bb7c-4c14-9e63-0866a137918b-kube-api-access-wxz79\") pod \"payload-processing-e2e-inf-a-340ae6-58897f9598-pv72n\" (UID: \"90887a7d-bb7c-4c14-9e63-0866a137918b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-340ae6-58897f9598-pv72n" Oct 30 02:11:34.267141 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:34.267036 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-340ae6-58897f9598-pv72n" Oct 30 02:11:34.379181 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:34.379139 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:34.651882 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:34.651843 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-340ae6-58897f9598-pv72n"] Oct 30 02:11:34.652292 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:11:34.652248 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90887a7d_bb7c_4c14_9e63_0866a137918b.slice/crio-48b02ba3e409cb71716fab213b45327a608fdbc8743f4d5e1a7e9d5698c1f5e4 WatchSource:0}: Error finding container 48b02ba3e409cb71716fab213b45327a608fdbc8743f4d5e1a7e9d5698c1f5e4: Status 404 returned error can't find the container with id 48b02ba3e409cb71716fab213b45327a608fdbc8743f4d5e1a7e9d5698c1f5e4 Oct 30 02:11:35.289769 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:35.289708 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-340ae6-58897f9598-pv72n" event={"ID":"90887a7d-bb7c-4c14-9e63-0866a137918b","Type":"ContainerStarted","Data":"7140f05e250959aca84f6f44e72dced7ca12dedc0ab92f8a0d336adb8aa18e6d"} Oct 30 02:11:35.290004 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:35.289812 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-340ae6-58897f9598-pv72n" event={"ID":"90887a7d-bb7c-4c14-9e63-0866a137918b","Type":"ContainerStarted","Data":"48b02ba3e409cb71716fab213b45327a608fdbc8743f4d5e1a7e9d5698c1f5e4"} Oct 30 02:11:35.290004 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:35.289848 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-340ae6-58897f9598-pv72n" Oct 30 02:11:35.331461 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:35.331264 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-340ae6-58897f9598-pv72n" podStartSLOduration=2.331244838 podStartE2EDuration="2.331244838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:11:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:11:35.326093739 +0000 UTC m=+3220.902519142" watchObservedRunningTime="2026-10-30 02:11:35.331244838 +0000 UTC m=+3220.907670302" Oct 30 02:11:36.038776 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:36.032851 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:37.934771 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:37.929347 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:39.559676 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:39.559638 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:41.326258 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:41.326210 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:42.994472 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:42.994421 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:44.500789 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:44.497184 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:44.690303 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:11:44.690265 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:11:46.219152 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:46.219100 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:46.302888 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:46.302853 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-340ae6-58897f9598-pv72n" Oct 30 02:11:47.692780 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:47.689953 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:49.463730 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:49.463693 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:51.139891 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:51.139849 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:52.766502 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:52.765567 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:54.427089 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:54.427053 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:56.009418 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:56.009378 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:57.516721 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:57.516682 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:58.849961 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:58.849927 6572 scope.go:122] "RemoveContainer" containerID="7cb0d34bf6ec8a80d917a26b231ce19f2ead0e374febd3931cc39d9c405f3577" Oct 30 02:11:59.201507 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:11:59.201464 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:00.770978 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:00.770924 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:02.405697 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:02.405657 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:03.650363 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:03.650275 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76881e36-25w2k"] Oct 30 02:12:03.656743 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:03.656722 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76881e36-25w2k" Oct 30 02:12:03.680827 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:03.680802 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76881e36-25w2k"] Oct 30 02:12:03.685528 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:03.685500 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:12:03.685723 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:03.685507 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:12:03.685723 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:03.685576 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:12:03.685897 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:03.685609 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:12:03.793111 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:03.793077 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59ee674a-29e5-4865-a2db-15aa20fee9d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76881e36-25w2k\" (UID: \"59ee674a-29e5-4865-a2db-15aa20fee9d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76881e36-25w2k" Oct 30 02:12:03.793316 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:03.793292 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tlh9\" (UniqueName: \"kubernetes.io/projected/59ee674a-29e5-4865-a2db-15aa20fee9d7-kube-api-access-4tlh9\") pod \"maas-api-revoke-sub-76881e36-25w2k\" (UID: \"59ee674a-29e5-4865-a2db-15aa20fee9d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76881e36-25w2k" Oct 30 02:12:03.793367 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:03.793327 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59ee674a-29e5-4865-a2db-15aa20fee9d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76881e36-25w2k\" (UID: \"59ee674a-29e5-4865-a2db-15aa20fee9d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76881e36-25w2k" Oct 30 02:12:03.894511 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:03.894467 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tlh9\" (UniqueName: \"kubernetes.io/projected/59ee674a-29e5-4865-a2db-15aa20fee9d7-kube-api-access-4tlh9\") pod \"maas-api-revoke-sub-76881e36-25w2k\" (UID: \"59ee674a-29e5-4865-a2db-15aa20fee9d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76881e36-25w2k" Oct 30 02:12:03.894710 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:03.894525 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59ee674a-29e5-4865-a2db-15aa20fee9d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76881e36-25w2k\" (UID: \"59ee674a-29e5-4865-a2db-15aa20fee9d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76881e36-25w2k" Oct 30 02:12:03.894710 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:03.894683 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59ee674a-29e5-4865-a2db-15aa20fee9d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76881e36-25w2k\" (UID: \"59ee674a-29e5-4865-a2db-15aa20fee9d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76881e36-25w2k" Oct 30 02:12:03.895256 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:03.895232 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59ee674a-29e5-4865-a2db-15aa20fee9d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76881e36-25w2k\" (UID: \"59ee674a-29e5-4865-a2db-15aa20fee9d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76881e36-25w2k" Oct 30 02:12:03.897133 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:03.897114 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59ee674a-29e5-4865-a2db-15aa20fee9d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76881e36-25w2k\" (UID: \"59ee674a-29e5-4865-a2db-15aa20fee9d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76881e36-25w2k" Oct 30 02:12:03.914926 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:03.914904 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tlh9\" (UniqueName: \"kubernetes.io/projected/59ee674a-29e5-4865-a2db-15aa20fee9d7-kube-api-access-4tlh9\") pod \"maas-api-revoke-sub-76881e36-25w2k\" (UID: \"59ee674a-29e5-4865-a2db-15aa20fee9d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76881e36-25w2k" Oct 30 02:12:03.973850 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:03.973823 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76881e36-25w2k" Oct 30 02:12:04.127358 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:04.127323 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:04.288555 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:04.288514 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76881e36-25w2k"] Oct 30 02:12:04.289387 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:12:04.289356 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ee674a_29e5_4865_a2db_15aa20fee9d7.slice/crio-11bf3e50566db2cc1817b8b125657c4ed76e4a0a90ab7670de9dd0d46c36e926 WatchSource:0}: Error finding container 11bf3e50566db2cc1817b8b125657c4ed76e4a0a90ab7670de9dd0d46c36e926: Status 404 returned error can't find the container with id 11bf3e50566db2cc1817b8b125657c4ed76e4a0a90ab7670de9dd0d46c36e926 Oct 30 02:12:04.602870 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:04.602723 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76881e36-25w2k" event={"ID":"59ee674a-29e5-4865-a2db-15aa20fee9d7","Type":"ContainerStarted","Data":"11bf3e50566db2cc1817b8b125657c4ed76e4a0a90ab7670de9dd0d46c36e926"} Oct 30 02:12:05.614738 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:05.614650 6572 generic.go:356] "Generic (PLEG): container finished" podID="59ee674a-29e5-4865-a2db-15aa20fee9d7" containerID="11a77fac2bdfb09297c3636c11ce40cf79069a4dd4bcefa8d36db5f1058655d0" exitCode=0 Oct 30 02:12:05.615203 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:05.614734 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76881e36-25w2k" event={"ID":"59ee674a-29e5-4865-a2db-15aa20fee9d7","Type":"ContainerDied","Data":"11a77fac2bdfb09297c3636c11ce40cf79069a4dd4bcefa8d36db5f1058655d0"} Oct 30 02:12:05.793789 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:05.793511 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:06.813838 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:06.813801 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76881e36-25w2k" Oct 30 02:12:06.881547 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:06.881452 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59ee674a-29e5-4865-a2db-15aa20fee9d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59ee674a-29e5-4865-a2db-15aa20fee9d7-maas-api-xks-ca\") pod \"59ee674a-29e5-4865-a2db-15aa20fee9d7\" (UID: \"59ee674a-29e5-4865-a2db-15aa20fee9d7\") " Oct 30 02:12:06.881777 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:06.881738 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59ee674a-29e5-4865-a2db-15aa20fee9d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59ee674a-29e5-4865-a2db-15aa20fee9d7-maas-api-openshift-ca\") pod \"59ee674a-29e5-4865-a2db-15aa20fee9d7\" (UID: \"59ee674a-29e5-4865-a2db-15aa20fee9d7\") " Oct 30 02:12:06.881892 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:06.881876 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59ee674a-29e5-4865-a2db-15aa20fee9d7-kube-api-access-4tlh9\" (UniqueName: \"kubernetes.io/projected/59ee674a-29e5-4865-a2db-15aa20fee9d7-kube-api-access-4tlh9\") pod \"59ee674a-29e5-4865-a2db-15aa20fee9d7\" (UID: \"59ee674a-29e5-4865-a2db-15aa20fee9d7\") " Oct 30 02:12:06.882201 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:06.882168 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59ee674a-29e5-4865-a2db-15aa20fee9d7-maas-api-openshift-ca" pod "59ee674a-29e5-4865-a2db-15aa20fee9d7" (UID: "59ee674a-29e5-4865-a2db-15aa20fee9d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:12:06.882654 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:06.882628 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59ee674a-29e5-4865-a2db-15aa20fee9d7-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:12:06.884577 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:06.884548 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59ee674a-29e5-4865-a2db-15aa20fee9d7-maas-api-xks-ca" pod "59ee674a-29e5-4865-a2db-15aa20fee9d7" (UID: "59ee674a-29e5-4865-a2db-15aa20fee9d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:12:06.884577 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:06.884564 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ee674a-29e5-4865-a2db-15aa20fee9d7-kube-api-access-4tlh9" pod "59ee674a-29e5-4865-a2db-15aa20fee9d7" (UID: "59ee674a-29e5-4865-a2db-15aa20fee9d7"). InnerVolumeSpecName "kube-api-access-4tlh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:12:06.985237 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:06.985199 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59ee674a-29e5-4865-a2db-15aa20fee9d7-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:12:06.985237 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:06.985231 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tlh9\" (UniqueName: \"kubernetes.io/projected/59ee674a-29e5-4865-a2db-15aa20fee9d7-kube-api-access-4tlh9\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:12:07.369160 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:07.369121 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:07.635886 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:07.635803 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76881e36-25w2k" Oct 30 02:12:07.635886 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:07.635829 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76881e36-25w2k" event={"ID":"59ee674a-29e5-4865-a2db-15aa20fee9d7","Type":"ContainerDied","Data":"11bf3e50566db2cc1817b8b125657c4ed76e4a0a90ab7670de9dd0d46c36e926"} Oct 30 02:12:07.636092 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:07.635891 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11bf3e50566db2cc1817b8b125657c4ed76e4a0a90ab7670de9dd0d46c36e926" Oct 30 02:12:08.514338 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:08.514301 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:09.944069 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:09.944023 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:10.433107 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:10.432792 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:12.104815 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:12.104774 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:13.605643 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:13.605531 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:14.753221 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:14.753178 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:14.823774 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:12:14.819949 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:12:15.211542 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:15.211506 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9f4cb55-hj454"] Oct 30 02:12:15.212438 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:15.212420 6572 cpu_manager.go:414] "containerMap: removing container" podUID="59ee674a-29e5-4865-a2db-15aa20fee9d7" containerName="revoke-keys" Oct 30 02:12:15.212508 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:15.212440 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59ee674a-29e5-4865-a2db-15aa20fee9d7" containerName="revoke-keys" Oct 30 02:12:15.212563 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:15.212554 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="59ee674a-29e5-4865-a2db-15aa20fee9d7" containerName="revoke-keys" Oct 30 02:12:15.217982 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:15.217153 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9f4cb55-hj454" Oct 30 02:12:15.231602 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:15.231568 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:12:15.240112 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:15.240072 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9f4cb55-hj454"] Oct 30 02:12:15.240846 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:15.240822 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:12:15.248153 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:15.248105 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ad57d82-5011-4542-9301-3f042e876dde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9f4cb55-hj454\" (UID: \"1ad57d82-5011-4542-9301-3f042e876dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9f4cb55-hj454" Oct 30 02:12:15.248366 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:15.248186 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ad57d82-5011-4542-9301-3f042e876dde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9f4cb55-hj454\" (UID: \"1ad57d82-5011-4542-9301-3f042e876dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9f4cb55-hj454" Oct 30 02:12:15.248366 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:15.248310 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mddbh\" (UniqueName: \"kubernetes.io/projected/1ad57d82-5011-4542-9301-3f042e876dde-kube-api-access-mddbh\") pod \"maas-api-revoke-sub-a9f4cb55-hj454\" (UID: \"1ad57d82-5011-4542-9301-3f042e876dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9f4cb55-hj454" Oct 30 02:12:15.311572 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:15.311538 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:12:15.311858 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:15.311588 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:12:15.402439 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:15.401175 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ad57d82-5011-4542-9301-3f042e876dde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9f4cb55-hj454\" (UID: \"1ad57d82-5011-4542-9301-3f042e876dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9f4cb55-hj454" Oct 30 02:12:15.402439 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:15.401262 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ad57d82-5011-4542-9301-3f042e876dde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9f4cb55-hj454\" (UID: \"1ad57d82-5011-4542-9301-3f042e876dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9f4cb55-hj454" Oct 30 02:12:15.402439 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:15.401417 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mddbh\" (UniqueName: \"kubernetes.io/projected/1ad57d82-5011-4542-9301-3f042e876dde-kube-api-access-mddbh\") pod \"maas-api-revoke-sub-a9f4cb55-hj454\" (UID: \"1ad57d82-5011-4542-9301-3f042e876dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9f4cb55-hj454" Oct 30 02:12:15.403890 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:15.403668 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:15.404293 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:15.404258 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ad57d82-5011-4542-9301-3f042e876dde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9f4cb55-hj454\" (UID: \"1ad57d82-5011-4542-9301-3f042e876dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9f4cb55-hj454" Oct 30 02:12:15.420575 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:15.420532 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ad57d82-5011-4542-9301-3f042e876dde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9f4cb55-hj454\" (UID: \"1ad57d82-5011-4542-9301-3f042e876dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9f4cb55-hj454" Oct 30 02:12:15.425902 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:15.425859 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mddbh\" (UniqueName: \"kubernetes.io/projected/1ad57d82-5011-4542-9301-3f042e876dde-kube-api-access-mddbh\") pod \"maas-api-revoke-sub-a9f4cb55-hj454\" (UID: \"1ad57d82-5011-4542-9301-3f042e876dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9f4cb55-hj454" Oct 30 02:12:15.535952 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:15.535862 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9f4cb55-hj454" Oct 30 02:12:15.848640 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:15.848604 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9f4cb55-hj454"] Oct 30 02:12:15.849388 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:12:15.849358 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad57d82_5011_4542_9301_3f042e876dde.slice/crio-990eccf376568c00750a0fd8dd46a375d04a7362a78d4a412e69486119d9a76b WatchSource:0}: Error finding container 990eccf376568c00750a0fd8dd46a375d04a7362a78d4a412e69486119d9a76b: Status 404 returned error can't find the container with id 990eccf376568c00750a0fd8dd46a375d04a7362a78d4a412e69486119d9a76b Oct 30 02:12:16.766842 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:16.766805 6572 generic.go:356] "Generic (PLEG): container finished" podID="1ad57d82-5011-4542-9301-3f042e876dde" containerID="351149746ac04315c86c2e99e4fad956d61874677b1683b62bb8259e030906f6" exitCode=0 Oct 30 02:12:16.767049 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:16.766886 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9f4cb55-hj454" event={"ID":"1ad57d82-5011-4542-9301-3f042e876dde","Type":"ContainerDied","Data":"351149746ac04315c86c2e99e4fad956d61874677b1683b62bb8259e030906f6"} Oct 30 02:12:16.767049 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:16.766948 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9f4cb55-hj454" event={"ID":"1ad57d82-5011-4542-9301-3f042e876dde","Type":"ContainerStarted","Data":"990eccf376568c00750a0fd8dd46a375d04a7362a78d4a412e69486119d9a76b"} Oct 30 02:12:16.968784 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:16.968661 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:17.977379 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:17.977338 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9f4cb55-hj454" Oct 30 02:12:18.048994 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:18.048900 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ad57d82-5011-4542-9301-3f042e876dde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ad57d82-5011-4542-9301-3f042e876dde-maas-api-xks-ca\") pod \"1ad57d82-5011-4542-9301-3f042e876dde\" (UID: \"1ad57d82-5011-4542-9301-3f042e876dde\") " Oct 30 02:12:18.049172 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:18.049022 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ad57d82-5011-4542-9301-3f042e876dde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ad57d82-5011-4542-9301-3f042e876dde-maas-api-openshift-ca\") pod \"1ad57d82-5011-4542-9301-3f042e876dde\" (UID: \"1ad57d82-5011-4542-9301-3f042e876dde\") " Oct 30 02:12:18.049172 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:18.049091 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ad57d82-5011-4542-9301-3f042e876dde-kube-api-access-mddbh\" (UniqueName: \"kubernetes.io/projected/1ad57d82-5011-4542-9301-3f042e876dde-kube-api-access-mddbh\") pod \"1ad57d82-5011-4542-9301-3f042e876dde\" (UID: \"1ad57d82-5011-4542-9301-3f042e876dde\") " Oct 30 02:12:18.049405 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:18.049378 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ad57d82-5011-4542-9301-3f042e876dde-maas-api-openshift-ca" pod "1ad57d82-5011-4542-9301-3f042e876dde" (UID: "1ad57d82-5011-4542-9301-3f042e876dde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:12:18.050115 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:18.050091 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ad57d82-5011-4542-9301-3f042e876dde-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:12:18.051169 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:18.051145 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ad57d82-5011-4542-9301-3f042e876dde-maas-api-xks-ca" pod "1ad57d82-5011-4542-9301-3f042e876dde" (UID: "1ad57d82-5011-4542-9301-3f042e876dde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:12:18.051277 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:18.051258 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ad57d82-5011-4542-9301-3f042e876dde-kube-api-access-mddbh" pod "1ad57d82-5011-4542-9301-3f042e876dde" (UID: "1ad57d82-5011-4542-9301-3f042e876dde"). InnerVolumeSpecName "kube-api-access-mddbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:12:18.151667 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:18.151629 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ad57d82-5011-4542-9301-3f042e876dde-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:12:18.151667 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:18.151660 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mddbh\" (UniqueName: \"kubernetes.io/projected/1ad57d82-5011-4542-9301-3f042e876dde-kube-api-access-mddbh\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:12:18.496781 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:18.496218 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:18.788133 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:18.788048 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9f4cb55-hj454" Oct 30 02:12:18.788133 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:18.788082 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9f4cb55-hj454" event={"ID":"1ad57d82-5011-4542-9301-3f042e876dde","Type":"ContainerDied","Data":"990eccf376568c00750a0fd8dd46a375d04a7362a78d4a412e69486119d9a76b"} Oct 30 02:12:18.788133 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:18.788130 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="990eccf376568c00750a0fd8dd46a375d04a7362a78d4a412e69486119d9a76b" Oct 30 02:12:20.012782 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:20.008887 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:21.606843 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:21.603807 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:23.191574 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:23.191534 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:23.397787 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:23.395341 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:23.971373 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:23.971339 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:24.191663 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:24.191623 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:24.926970 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:24.926928 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:26.602009 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:26.601814 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:26.614549 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:26.614515 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df3d431b-hflx7"] Oct 30 02:12:26.625332 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:26.625291 6572 cpu_manager.go:414] "containerMap: removing container" podUID="1ad57d82-5011-4542-9301-3f042e876dde" containerName="revoke-keys" Oct 30 02:12:26.634055 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:26.634012 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ad57d82-5011-4542-9301-3f042e876dde" containerName="revoke-keys" Oct 30 02:12:26.634951 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:26.634919 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ad57d82-5011-4542-9301-3f042e876dde" containerName="revoke-keys" Oct 30 02:12:26.650095 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:26.650058 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df3d431b-hflx7"] Oct 30 02:12:26.650401 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:26.650197 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df3d431b-hflx7" Oct 30 02:12:26.666368 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:26.666330 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztmf5\" (UniqueName: \"kubernetes.io/projected/0143bfe1-c549-4cac-baa1-e207c659e9f2-kube-api-access-ztmf5\") pod \"maas-api-revoke-sub-df3d431b-hflx7\" (UID: \"0143bfe1-c549-4cac-baa1-e207c659e9f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df3d431b-hflx7" Oct 30 02:12:26.666629 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:26.666607 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0143bfe1-c549-4cac-baa1-e207c659e9f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df3d431b-hflx7\" (UID: \"0143bfe1-c549-4cac-baa1-e207c659e9f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df3d431b-hflx7" Oct 30 02:12:26.666839 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:26.666820 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0143bfe1-c549-4cac-baa1-e207c659e9f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df3d431b-hflx7\" (UID: \"0143bfe1-c549-4cac-baa1-e207c659e9f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df3d431b-hflx7" Oct 30 02:12:26.676962 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:26.676930 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:12:26.677235 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:26.677016 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:12:26.677235 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:26.677061 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:12:26.677375 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:26.677107 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:12:26.768799 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:26.768706 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0143bfe1-c549-4cac-baa1-e207c659e9f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df3d431b-hflx7\" (UID: \"0143bfe1-c549-4cac-baa1-e207c659e9f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df3d431b-hflx7" Oct 30 02:12:26.769002 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:26.768834 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0143bfe1-c549-4cac-baa1-e207c659e9f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df3d431b-hflx7\" (UID: \"0143bfe1-c549-4cac-baa1-e207c659e9f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df3d431b-hflx7" Oct 30 02:12:26.769002 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:26.768935 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztmf5\" (UniqueName: \"kubernetes.io/projected/0143bfe1-c549-4cac-baa1-e207c659e9f2-kube-api-access-ztmf5\") pod \"maas-api-revoke-sub-df3d431b-hflx7\" (UID: \"0143bfe1-c549-4cac-baa1-e207c659e9f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df3d431b-hflx7" Oct 30 02:12:26.769606 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:26.769583 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0143bfe1-c549-4cac-baa1-e207c659e9f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df3d431b-hflx7\" (UID: \"0143bfe1-c549-4cac-baa1-e207c659e9f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df3d431b-hflx7" Oct 30 02:12:26.771266 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:26.771243 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0143bfe1-c549-4cac-baa1-e207c659e9f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df3d431b-hflx7\" (UID: \"0143bfe1-c549-4cac-baa1-e207c659e9f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df3d431b-hflx7" Oct 30 02:12:26.787847 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:26.787814 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztmf5\" (UniqueName: \"kubernetes.io/projected/0143bfe1-c549-4cac-baa1-e207c659e9f2-kube-api-access-ztmf5\") pod \"maas-api-revoke-sub-df3d431b-hflx7\" (UID: \"0143bfe1-c549-4cac-baa1-e207c659e9f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df3d431b-hflx7" Oct 30 02:12:26.975257 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:26.975224 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df3d431b-hflx7" Oct 30 02:12:27.032558 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:27.032509 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcefc294-xxdgz"] Oct 30 02:12:27.038498 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:27.038454 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcefc294-xxdgz"] Oct 30 02:12:27.296864 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:12:27.296828 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0143bfe1_c549_4cac_baa1_e207c659e9f2.slice/crio-70d7dfe8d678adc3c182dc655a379ba338026e992a140733917f6844dec353b9 WatchSource:0}: Error finding container 70d7dfe8d678adc3c182dc655a379ba338026e992a140733917f6844dec353b9: Status 404 returned error can't find the container with id 70d7dfe8d678adc3c182dc655a379ba338026e992a140733917f6844dec353b9 Oct 30 02:12:27.297041 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:27.296927 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df3d431b-hflx7"] Oct 30 02:12:27.951437 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:27.951396 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df3d431b-hflx7" event={"ID":"0143bfe1-c549-4cac-baa1-e207c659e9f2","Type":"ContainerStarted","Data":"108ca68d130758ca46835ec61951cd311b5aed54c53c1f406a1e9c1ce85c220a"} Oct 30 02:12:27.951931 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:27.951467 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df3d431b-hflx7" event={"ID":"0143bfe1-c549-4cac-baa1-e207c659e9f2","Type":"ContainerStarted","Data":"70d7dfe8d678adc3c182dc655a379ba338026e992a140733917f6844dec353b9"} Oct 30 02:12:27.992355 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:27.992229 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df3d431b-hflx7" podStartSLOduration=1.992213357 podStartE2EDuration="1.992213357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:12:27.988446239 +0000 UTC m=+3273.564871645" watchObservedRunningTime="2026-10-30 02:12:27.992213357 +0000 UTC m=+3273.568638759" Oct 30 02:12:28.200822 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:28.189136 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:28.546068 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:28.546025 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db0796b5-2d24-4831-9d44-d1b1f5bfd792" path="/var/lib/kubelet/pods/db0796b5-2d24-4831-9d44-d1b1f5bfd792/volumes" Oct 30 02:12:28.963085 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:28.963048 6572 generic.go:356] "Generic (PLEG): container finished" podID="0143bfe1-c549-4cac-baa1-e207c659e9f2" containerID="108ca68d130758ca46835ec61951cd311b5aed54c53c1f406a1e9c1ce85c220a" exitCode=0 Oct 30 02:12:28.963566 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:28.963199 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df3d431b-hflx7" event={"ID":"0143bfe1-c549-4cac-baa1-e207c659e9f2","Type":"ContainerDied","Data":"108ca68d130758ca46835ec61951cd311b5aed54c53c1f406a1e9c1ce85c220a"} Oct 30 02:12:29.783774 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:29.783713 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:30.179298 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:30.179268 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df3d431b-hflx7" Oct 30 02:12:30.234429 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:30.234255 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0143bfe1-c549-4cac-baa1-e207c659e9f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0143bfe1-c549-4cac-baa1-e207c659e9f2-maas-api-xks-ca\") pod \"0143bfe1-c549-4cac-baa1-e207c659e9f2\" (UID: \"0143bfe1-c549-4cac-baa1-e207c659e9f2\") " Oct 30 02:12:30.235556 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:30.235082 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0143bfe1-c549-4cac-baa1-e207c659e9f2-kube-api-access-ztmf5\" (UniqueName: \"kubernetes.io/projected/0143bfe1-c549-4cac-baa1-e207c659e9f2-kube-api-access-ztmf5\") pod \"0143bfe1-c549-4cac-baa1-e207c659e9f2\" (UID: \"0143bfe1-c549-4cac-baa1-e207c659e9f2\") " Oct 30 02:12:30.235556 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:30.235160 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0143bfe1-c549-4cac-baa1-e207c659e9f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0143bfe1-c549-4cac-baa1-e207c659e9f2-maas-api-openshift-ca\") pod \"0143bfe1-c549-4cac-baa1-e207c659e9f2\" (UID: \"0143bfe1-c549-4cac-baa1-e207c659e9f2\") " Oct 30 02:12:30.236608 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:30.236583 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0143bfe1-c549-4cac-baa1-e207c659e9f2-maas-api-openshift-ca" pod "0143bfe1-c549-4cac-baa1-e207c659e9f2" (UID: "0143bfe1-c549-4cac-baa1-e207c659e9f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:12:30.238533 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:30.238500 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0143bfe1-c549-4cac-baa1-e207c659e9f2-maas-api-xks-ca" pod "0143bfe1-c549-4cac-baa1-e207c659e9f2" (UID: "0143bfe1-c549-4cac-baa1-e207c659e9f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:12:30.239331 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:30.239304 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0143bfe1-c549-4cac-baa1-e207c659e9f2-kube-api-access-ztmf5" pod "0143bfe1-c549-4cac-baa1-e207c659e9f2" (UID: "0143bfe1-c549-4cac-baa1-e207c659e9f2"). InnerVolumeSpecName "kube-api-access-ztmf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:12:30.337558 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:30.337517 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0143bfe1-c549-4cac-baa1-e207c659e9f2-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:12:30.337558 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:30.337552 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztmf5\" (UniqueName: \"kubernetes.io/projected/0143bfe1-c549-4cac-baa1-e207c659e9f2-kube-api-access-ztmf5\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:12:30.337558 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:30.337562 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0143bfe1-c549-4cac-baa1-e207c659e9f2-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:12:30.985135 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:30.985106 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df3d431b-hflx7" Oct 30 02:12:30.985343 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:30.985142 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df3d431b-hflx7" event={"ID":"0143bfe1-c549-4cac-baa1-e207c659e9f2","Type":"ContainerDied","Data":"70d7dfe8d678adc3c182dc655a379ba338026e992a140733917f6844dec353b9"} Oct 30 02:12:30.985343 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:30.985195 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70d7dfe8d678adc3c182dc655a379ba338026e992a140733917f6844dec353b9" Oct 30 02:12:31.479775 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:31.479037 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:33.192648 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:33.192615 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:34.673015 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:34.670704 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:35.414775 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:35.407194 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:35.573351 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:35.573310 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-717065-cv844"] Oct 30 02:12:35.574296 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:35.574275 6572 cpu_manager.go:414] "containerMap: removing container" podUID="0143bfe1-c549-4cac-baa1-e207c659e9f2" containerName="revoke-keys" Oct 30 02:12:35.574381 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:35.574296 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0143bfe1-c549-4cac-baa1-e207c659e9f2" containerName="revoke-keys" Oct 30 02:12:35.574446 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:35.574436 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="0143bfe1-c549-4cac-baa1-e207c659e9f2" containerName="revoke-keys" Oct 30 02:12:35.578884 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:35.578860 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-717065-cv844" Oct 30 02:12:35.600063 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:35.600029 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-717065-cv844"] Oct 30 02:12:35.602828 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:35.602795 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:12:35.603029 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:35.602876 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:12:35.608012 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:35.607602 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:12:35.608012 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:35.607727 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:12:35.745328 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:35.745234 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/319e8315-1f3e-4ffa-a417-38b4625f1ae4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-717065-cv844\" (UID: \"319e8315-1f3e-4ffa-a417-38b4625f1ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-717065-cv844" Oct 30 02:12:35.745812 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:35.745480 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/319e8315-1f3e-4ffa-a417-38b4625f1ae4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-717065-cv844\" (UID: \"319e8315-1f3e-4ffa-a417-38b4625f1ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-717065-cv844" Oct 30 02:12:35.745812 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:35.745548 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w99n6\" (UniqueName: \"kubernetes.io/projected/319e8315-1f3e-4ffa-a417-38b4625f1ae4-kube-api-access-w99n6\") pod \"maas-api-revoke-keys-e2e-inf-a-717065-cv844\" (UID: \"319e8315-1f3e-4ffa-a417-38b4625f1ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-717065-cv844" Oct 30 02:12:35.847805 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:35.847740 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w99n6\" (UniqueName: \"kubernetes.io/projected/319e8315-1f3e-4ffa-a417-38b4625f1ae4-kube-api-access-w99n6\") pod \"maas-api-revoke-keys-e2e-inf-a-717065-cv844\" (UID: \"319e8315-1f3e-4ffa-a417-38b4625f1ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-717065-cv844" Oct 30 02:12:35.848607 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:35.848578 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/319e8315-1f3e-4ffa-a417-38b4625f1ae4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-717065-cv844\" (UID: \"319e8315-1f3e-4ffa-a417-38b4625f1ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-717065-cv844" Oct 30 02:12:35.849058 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:35.849030 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/319e8315-1f3e-4ffa-a417-38b4625f1ae4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-717065-cv844\" (UID: \"319e8315-1f3e-4ffa-a417-38b4625f1ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-717065-cv844" Oct 30 02:12:35.849996 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:35.849974 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/319e8315-1f3e-4ffa-a417-38b4625f1ae4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-717065-cv844\" (UID: \"319e8315-1f3e-4ffa-a417-38b4625f1ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-717065-cv844" Oct 30 02:12:35.852579 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:35.852549 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/319e8315-1f3e-4ffa-a417-38b4625f1ae4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-717065-cv844\" (UID: \"319e8315-1f3e-4ffa-a417-38b4625f1ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-717065-cv844" Oct 30 02:12:35.868847 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:35.868811 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w99n6\" (UniqueName: \"kubernetes.io/projected/319e8315-1f3e-4ffa-a417-38b4625f1ae4-kube-api-access-w99n6\") pod \"maas-api-revoke-keys-e2e-inf-a-717065-cv844\" (UID: \"319e8315-1f3e-4ffa-a417-38b4625f1ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-717065-cv844" Oct 30 02:12:35.896301 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:35.896256 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-717065-cv844" Oct 30 02:12:36.208633 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:36.208604 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-717065-cv844"] Oct 30 02:12:36.208808 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:12:36.208696 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod319e8315_1f3e_4ffa_a417_38b4625f1ae4.slice/crio-6d7e5567c96387baea07718916f9a76e3180a32fc1360da6e0e46ef9084a3df4 WatchSource:0}: Error finding container 6d7e5567c96387baea07718916f9a76e3180a32fc1360da6e0e46ef9084a3df4: Status 404 returned error can't find the container with id 6d7e5567c96387baea07718916f9a76e3180a32fc1360da6e0e46ef9084a3df4 Oct 30 02:12:36.403205 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:36.403142 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:37.052675 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:37.052635 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-717065-cv844" event={"ID":"319e8315-1f3e-4ffa-a417-38b4625f1ae4","Type":"ContainerStarted","Data":"47f1efe4a4f4a15f4e0fc6666deacf9383ce3f3063116397289e9acdc009be1f"} Oct 30 02:12:37.053198 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:37.052692 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-717065-cv844" event={"ID":"319e8315-1f3e-4ffa-a417-38b4625f1ae4","Type":"ContainerStarted","Data":"6d7e5567c96387baea07718916f9a76e3180a32fc1360da6e0e46ef9084a3df4"} Oct 30 02:12:37.096768 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:37.096696 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-717065-cv844" podStartSLOduration=2.096679036 podStartE2EDuration="2.096679036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:12:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:12:37.091438793 +0000 UTC m=+3282.667864194" watchObservedRunningTime="2026-10-30 02:12:37.096679036 +0000 UTC m=+3282.673104437" Oct 30 02:12:37.505149 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:37.505107 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07155f44-bp5t5"] Oct 30 02:12:37.510368 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:37.510340 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07155f44-bp5t5" Oct 30 02:12:37.527768 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:37.527713 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07155f44-bp5t5"] Oct 30 02:12:37.612267 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:37.612159 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8a485b3-17b6-4680-ba7b-02a2f7a795c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07155f44-bp5t5\" (UID: \"d8a485b3-17b6-4680-ba7b-02a2f7a795c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07155f44-bp5t5" Oct 30 02:12:37.612465 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:37.612404 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2w2j\" (UniqueName: \"kubernetes.io/projected/d8a485b3-17b6-4680-ba7b-02a2f7a795c4-kube-api-access-b2w2j\") pod \"maas-api-revoke-sub-07155f44-bp5t5\" (UID: \"d8a485b3-17b6-4680-ba7b-02a2f7a795c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07155f44-bp5t5" Oct 30 02:12:37.612585 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:37.612560 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8a485b3-17b6-4680-ba7b-02a2f7a795c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07155f44-bp5t5\" (UID: \"d8a485b3-17b6-4680-ba7b-02a2f7a795c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07155f44-bp5t5" Oct 30 02:12:37.714736 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:37.714690 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2w2j\" (UniqueName: \"kubernetes.io/projected/d8a485b3-17b6-4680-ba7b-02a2f7a795c4-kube-api-access-b2w2j\") pod \"maas-api-revoke-sub-07155f44-bp5t5\" (UID: \"d8a485b3-17b6-4680-ba7b-02a2f7a795c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07155f44-bp5t5" Oct 30 02:12:37.714976 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:37.714846 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8a485b3-17b6-4680-ba7b-02a2f7a795c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07155f44-bp5t5\" (UID: \"d8a485b3-17b6-4680-ba7b-02a2f7a795c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07155f44-bp5t5" Oct 30 02:12:37.715028 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:37.715016 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8a485b3-17b6-4680-ba7b-02a2f7a795c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07155f44-bp5t5\" (UID: \"d8a485b3-17b6-4680-ba7b-02a2f7a795c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07155f44-bp5t5" Oct 30 02:12:37.715569 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:37.715547 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8a485b3-17b6-4680-ba7b-02a2f7a795c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07155f44-bp5t5\" (UID: \"d8a485b3-17b6-4680-ba7b-02a2f7a795c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07155f44-bp5t5" Oct 30 02:12:37.717603 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:37.717574 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8a485b3-17b6-4680-ba7b-02a2f7a795c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07155f44-bp5t5\" (UID: \"d8a485b3-17b6-4680-ba7b-02a2f7a795c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07155f44-bp5t5" Oct 30 02:12:37.734067 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:37.734036 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2w2j\" (UniqueName: \"kubernetes.io/projected/d8a485b3-17b6-4680-ba7b-02a2f7a795c4-kube-api-access-b2w2j\") pod \"maas-api-revoke-sub-07155f44-bp5t5\" (UID: \"d8a485b3-17b6-4680-ba7b-02a2f7a795c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07155f44-bp5t5" Oct 30 02:12:37.828290 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:37.828242 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07155f44-bp5t5" Oct 30 02:12:38.003783 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:38.002349 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:38.067674 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:38.067634 6572 generic.go:356] "Generic (PLEG): container finished" podID="319e8315-1f3e-4ffa-a417-38b4625f1ae4" containerID="47f1efe4a4f4a15f4e0fc6666deacf9383ce3f3063116397289e9acdc009be1f" exitCode=0 Oct 30 02:12:38.068232 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:38.067732 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-717065-cv844" event={"ID":"319e8315-1f3e-4ffa-a417-38b4625f1ae4","Type":"ContainerDied","Data":"47f1efe4a4f4a15f4e0fc6666deacf9383ce3f3063116397289e9acdc009be1f"} Oct 30 02:12:38.142808 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:12:38.142774 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8a485b3_17b6_4680_ba7b_02a2f7a795c4.slice/crio-31713e7640d048b1130d8cb80ac36e74d844d83308564c2d8b327a8ecb578076 WatchSource:0}: Error finding container 31713e7640d048b1130d8cb80ac36e74d844d83308564c2d8b327a8ecb578076: Status 404 returned error can't find the container with id 31713e7640d048b1130d8cb80ac36e74d844d83308564c2d8b327a8ecb578076 Oct 30 02:12:38.143717 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:38.143682 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07155f44-bp5t5"] Oct 30 02:12:39.079132 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:39.079097 6572 generic.go:356] "Generic (PLEG): container finished" podID="d8a485b3-17b6-4680-ba7b-02a2f7a795c4" containerID="e1b71fb9c00d0cf0e10eab9c758e5da21009f41fa0035989497f827999f92e83" exitCode=0 Oct 30 02:12:39.079582 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:39.079182 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07155f44-bp5t5" event={"ID":"d8a485b3-17b6-4680-ba7b-02a2f7a795c4","Type":"ContainerDied","Data":"e1b71fb9c00d0cf0e10eab9c758e5da21009f41fa0035989497f827999f92e83"} Oct 30 02:12:39.079582 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:39.079242 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07155f44-bp5t5" event={"ID":"d8a485b3-17b6-4680-ba7b-02a2f7a795c4","Type":"ContainerStarted","Data":"31713e7640d048b1130d8cb80ac36e74d844d83308564c2d8b327a8ecb578076"} Oct 30 02:12:39.295465 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:39.295392 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-717065-cv844" Oct 30 02:12:39.346119 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:39.346050 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/319e8315-1f3e-4ffa-a417-38b4625f1ae4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/319e8315-1f3e-4ffa-a417-38b4625f1ae4-maas-api-xks-ca\") pod \"319e8315-1f3e-4ffa-a417-38b4625f1ae4\" (UID: \"319e8315-1f3e-4ffa-a417-38b4625f1ae4\") " Oct 30 02:12:39.346416 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:39.346391 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/319e8315-1f3e-4ffa-a417-38b4625f1ae4-kube-api-access-w99n6\" (UniqueName: \"kubernetes.io/projected/319e8315-1f3e-4ffa-a417-38b4625f1ae4-kube-api-access-w99n6\") pod \"319e8315-1f3e-4ffa-a417-38b4625f1ae4\" (UID: \"319e8315-1f3e-4ffa-a417-38b4625f1ae4\") " Oct 30 02:12:39.346564 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:39.346539 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/319e8315-1f3e-4ffa-a417-38b4625f1ae4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/319e8315-1f3e-4ffa-a417-38b4625f1ae4-maas-api-openshift-ca\") pod \"319e8315-1f3e-4ffa-a417-38b4625f1ae4\" (UID: \"319e8315-1f3e-4ffa-a417-38b4625f1ae4\") " Oct 30 02:12:39.348068 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:39.347984 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/319e8315-1f3e-4ffa-a417-38b4625f1ae4-maas-api-openshift-ca" pod "319e8315-1f3e-4ffa-a417-38b4625f1ae4" (UID: "319e8315-1f3e-4ffa-a417-38b4625f1ae4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:12:39.353811 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:39.350370 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/319e8315-1f3e-4ffa-a417-38b4625f1ae4-maas-api-xks-ca" pod "319e8315-1f3e-4ffa-a417-38b4625f1ae4" (UID: "319e8315-1f3e-4ffa-a417-38b4625f1ae4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:12:39.353811 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:39.350414 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/319e8315-1f3e-4ffa-a417-38b4625f1ae4-kube-api-access-w99n6" pod "319e8315-1f3e-4ffa-a417-38b4625f1ae4" (UID: "319e8315-1f3e-4ffa-a417-38b4625f1ae4"). InnerVolumeSpecName "kube-api-access-w99n6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:12:39.449341 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:39.449299 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w99n6\" (UniqueName: \"kubernetes.io/projected/319e8315-1f3e-4ffa-a417-38b4625f1ae4-kube-api-access-w99n6\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:12:39.449341 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:39.449338 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/319e8315-1f3e-4ffa-a417-38b4625f1ae4-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:12:39.449341 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:39.449351 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/319e8315-1f3e-4ffa-a417-38b4625f1ae4-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:12:39.528225 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:39.522971 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:40.100549 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:40.100506 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-717065-cv844" Oct 30 02:12:40.101190 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:40.100545 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-717065-cv844" event={"ID":"319e8315-1f3e-4ffa-a417-38b4625f1ae4","Type":"ContainerDied","Data":"6d7e5567c96387baea07718916f9a76e3180a32fc1360da6e0e46ef9084a3df4"} Oct 30 02:12:40.101190 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:40.100639 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d7e5567c96387baea07718916f9a76e3180a32fc1360da6e0e46ef9084a3df4" Oct 30 02:12:40.275208 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:40.275177 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07155f44-bp5t5" Oct 30 02:12:40.364655 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:40.364548 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8a485b3-17b6-4680-ba7b-02a2f7a795c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8a485b3-17b6-4680-ba7b-02a2f7a795c4-maas-api-openshift-ca\") pod \"d8a485b3-17b6-4680-ba7b-02a2f7a795c4\" (UID: \"d8a485b3-17b6-4680-ba7b-02a2f7a795c4\") " Oct 30 02:12:40.364929 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:40.364906 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8a485b3-17b6-4680-ba7b-02a2f7a795c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8a485b3-17b6-4680-ba7b-02a2f7a795c4-maas-api-xks-ca\") pod \"d8a485b3-17b6-4680-ba7b-02a2f7a795c4\" (UID: \"d8a485b3-17b6-4680-ba7b-02a2f7a795c4\") " Oct 30 02:12:40.365132 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:40.365090 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8a485b3-17b6-4680-ba7b-02a2f7a795c4-maas-api-openshift-ca" pod "d8a485b3-17b6-4680-ba7b-02a2f7a795c4" (UID: "d8a485b3-17b6-4680-ba7b-02a2f7a795c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:12:40.365222 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:40.365134 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8a485b3-17b6-4680-ba7b-02a2f7a795c4-kube-api-access-b2w2j\" (UniqueName: \"kubernetes.io/projected/d8a485b3-17b6-4680-ba7b-02a2f7a795c4-kube-api-access-b2w2j\") pod \"d8a485b3-17b6-4680-ba7b-02a2f7a795c4\" (UID: \"d8a485b3-17b6-4680-ba7b-02a2f7a795c4\") " Oct 30 02:12:40.365943 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:40.365913 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8a485b3-17b6-4680-ba7b-02a2f7a795c4-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:12:40.367999 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:40.367967 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8a485b3-17b6-4680-ba7b-02a2f7a795c4-kube-api-access-b2w2j" pod "d8a485b3-17b6-4680-ba7b-02a2f7a795c4" (UID: "d8a485b3-17b6-4680-ba7b-02a2f7a795c4"). InnerVolumeSpecName "kube-api-access-b2w2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:12:40.368319 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:40.368289 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8a485b3-17b6-4680-ba7b-02a2f7a795c4-maas-api-xks-ca" pod "d8a485b3-17b6-4680-ba7b-02a2f7a795c4" (UID: "d8a485b3-17b6-4680-ba7b-02a2f7a795c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:12:40.468007 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:40.467961 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8a485b3-17b6-4680-ba7b-02a2f7a795c4-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:12:40.468007 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:40.467997 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2w2j\" (UniqueName: \"kubernetes.io/projected/d8a485b3-17b6-4680-ba7b-02a2f7a795c4-kube-api-access-b2w2j\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:12:40.971774 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:40.971513 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:41.111552 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:41.111519 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07155f44-bp5t5" Oct 30 02:12:41.112024 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:41.111550 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07155f44-bp5t5" event={"ID":"d8a485b3-17b6-4680-ba7b-02a2f7a795c4","Type":"ContainerDied","Data":"31713e7640d048b1130d8cb80ac36e74d844d83308564c2d8b327a8ecb578076"} Oct 30 02:12:41.112024 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:41.111603 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31713e7640d048b1130d8cb80ac36e74d844d83308564c2d8b327a8ecb578076" Oct 30 02:12:42.511827 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:42.511785 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:44.198054 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:44.195400 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:44.690579 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:12:44.690541 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:12:45.693767 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:45.693096 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:46.080504 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:46.080400 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-717065-656f9c7c8d-mv46m"] Oct 30 02:12:46.323960 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:46.323923 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:46.507587 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:46.507547 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:47.313498 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:47.313462 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:48.863983 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:48.863945 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:49.168488 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:49.168451 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg"] Oct 30 02:12:49.332187 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:49.332148 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:49.520119 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:49.520013 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:50.041025 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:50.040980 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f6e11e/payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x"] Oct 30 02:12:50.156189 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:50.156149 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:50.373121 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:50.373020 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:50.845688 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:50.845653 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:51.081674 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:51.081621 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-717065-656f9c7c8d-mv46m" podUID="5e718bf0-89f5-487a-a823-cdd6081e7431" containerName="payload-processing" containerID="cri-o://e80013f7a054d07f891b138509c9d9b49e1f9e53ffd696640443f3a8bb1e3164" gracePeriod=25 Oct 30 02:12:52.412867 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:52.412817 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:53.771774 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:53.771722 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-717065-656f9c7c8d-mv46m" podUID="5e718bf0-89f5-487a-a823-cdd6081e7431" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:12:54.005175 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:54.005137 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:54.169126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:54.169076 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg" podUID="614bbbec-a79d-48cd-97c1-294455b786e0" containerName="payload-processing" containerID="cri-o://a1dbb45919cc80451a369cb0c1d7c81341586ad7366881cff44b371955c0ce39" gracePeriod=25 Oct 30 02:12:54.198323 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:12:54.198298 6572 logging.go:55] [core] [Channel #14351 SubChannel #14353]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.97:50052", ServerName: "10.133.1.97:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.97:50052: connect: connection refused" Oct 30 02:12:54.319339 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:54.319305 6572 generic.go:356] "Generic (PLEG): container finished" podID="614bbbec-a79d-48cd-97c1-294455b786e0" containerID="a1dbb45919cc80451a369cb0c1d7c81341586ad7366881cff44b371955c0ce39" exitCode=0 Oct 30 02:12:54.319555 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:54.319354 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg" event={"ID":"614bbbec-a79d-48cd-97c1-294455b786e0","Type":"ContainerDied","Data":"a1dbb45919cc80451a369cb0c1d7c81341586ad7366881cff44b371955c0ce39"} Oct 30 02:12:54.482948 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:54.482916 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg" Oct 30 02:12:54.661407 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:54.661358 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/614bbbec-a79d-48cd-97c1-294455b786e0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/614bbbec-a79d-48cd-97c1-294455b786e0-provider-credentials\") pod \"614bbbec-a79d-48cd-97c1-294455b786e0\" (UID: \"614bbbec-a79d-48cd-97c1-294455b786e0\") " Oct 30 02:12:54.661623 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:54.661472 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/614bbbec-a79d-48cd-97c1-294455b786e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/614bbbec-a79d-48cd-97c1-294455b786e0-plugins-config-volume\") pod \"614bbbec-a79d-48cd-97c1-294455b786e0\" (UID: \"614bbbec-a79d-48cd-97c1-294455b786e0\") " Oct 30 02:12:54.661731 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:54.661704 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/614bbbec-a79d-48cd-97c1-294455b786e0-kube-api-access-bhf2x\" (UniqueName: \"kubernetes.io/projected/614bbbec-a79d-48cd-97c1-294455b786e0-kube-api-access-bhf2x\") pod \"614bbbec-a79d-48cd-97c1-294455b786e0\" (UID: \"614bbbec-a79d-48cd-97c1-294455b786e0\") " Oct 30 02:12:54.661867 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:54.661847 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/614bbbec-a79d-48cd-97c1-294455b786e0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/614bbbec-a79d-48cd-97c1-294455b786e0-routing-overlay-volume\") pod \"614bbbec-a79d-48cd-97c1-294455b786e0\" (UID: \"614bbbec-a79d-48cd-97c1-294455b786e0\") " Oct 30 02:12:54.661991 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:54.661965 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/614bbbec-a79d-48cd-97c1-294455b786e0-plugins-config-volume" pod "614bbbec-a79d-48cd-97c1-294455b786e0" (UID: "614bbbec-a79d-48cd-97c1-294455b786e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:12:54.662230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:54.662203 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/614bbbec-a79d-48cd-97c1-294455b786e0-routing-overlay-volume" pod "614bbbec-a79d-48cd-97c1-294455b786e0" (UID: "614bbbec-a79d-48cd-97c1-294455b786e0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:12:54.662650 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:54.662629 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/614bbbec-a79d-48cd-97c1-294455b786e0-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:12:54.662650 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:54.662651 6572 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/614bbbec-a79d-48cd-97c1-294455b786e0-routing-overlay-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:12:54.663883 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:54.663766 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/614bbbec-a79d-48cd-97c1-294455b786e0-provider-credentials" pod "614bbbec-a79d-48cd-97c1-294455b786e0" (UID: "614bbbec-a79d-48cd-97c1-294455b786e0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:12:54.664025 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:54.664000 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/614bbbec-a79d-48cd-97c1-294455b786e0-kube-api-access-bhf2x" pod "614bbbec-a79d-48cd-97c1-294455b786e0" (UID: "614bbbec-a79d-48cd-97c1-294455b786e0"). InnerVolumeSpecName "kube-api-access-bhf2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:12:54.764701 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:54.764612 6572 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/614bbbec-a79d-48cd-97c1-294455b786e0-provider-credentials\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:12:54.764701 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:54.764644 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhf2x\" (UniqueName: \"kubernetes.io/projected/614bbbec-a79d-48cd-97c1-294455b786e0-kube-api-access-bhf2x\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:12:55.042229 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:55.042129 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-f6e11e/payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x" podUID="eccc8133-8a38-46fc-99af-285ecdd04bb0" containerName="payload-processing" containerID="cri-o://a18dac67dd224c926a3a13e29cc6c9000a720e5a2c34a47dcb1ecbaed651d19b" gracePeriod=25 Oct 30 02:12:55.198871 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:55.197963 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg" podUID="614bbbec-a79d-48cd-97c1-294455b786e0" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.1.97:50052\" within 1s: context deadline exceeded" Oct 30 02:12:55.338101 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:55.338053 6572 generic.go:356] "Generic (PLEG): container finished" podID="eccc8133-8a38-46fc-99af-285ecdd04bb0" containerID="a18dac67dd224c926a3a13e29cc6c9000a720e5a2c34a47dcb1ecbaed651d19b" exitCode=0 Oct 30 02:12:55.338569 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:55.338154 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f6e11e/payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x" event={"ID":"eccc8133-8a38-46fc-99af-285ecdd04bb0","Type":"ContainerDied","Data":"a18dac67dd224c926a3a13e29cc6c9000a720e5a2c34a47dcb1ecbaed651d19b"} Oct 30 02:12:55.340042 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:55.339994 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f6e11e/payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x" event={"ID":"eccc8133-8a38-46fc-99af-285ecdd04bb0","Type":"ContainerDied","Data":"46c0703e82a2ca520fb527e63e12234dfa9756fe884c364fef83f4441a3a8772"} Oct 30 02:12:55.340195 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:55.340066 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46c0703e82a2ca520fb527e63e12234dfa9756fe884c364fef83f4441a3a8772" Oct 30 02:12:55.341819 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:55.341791 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg" event={"ID":"614bbbec-a79d-48cd-97c1-294455b786e0","Type":"ContainerDied","Data":"4a99771e7a6b4e4ea486639328a8eb3cd499067ae64c82488ce73f8bd6db1fd3"} Oct 30 02:12:55.341983 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:55.341855 6572 scope.go:122] "RemoveContainer" containerID="a1dbb45919cc80451a369cb0c1d7c81341586ad7366881cff44b371955c0ce39" Oct 30 02:12:55.342062 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:55.342047 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg" Oct 30 02:12:55.363815 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:55.363741 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f6e11e/payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x" Oct 30 02:12:55.396654 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:55.396606 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg"] Oct 30 02:12:55.401688 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:55.401650 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-817fd7/payload-processing-external-model-e2e-extmod-a-817fd7-5f59llpzg"] Oct 30 02:12:55.509772 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:55.492499 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eccc8133-8a38-46fc-99af-285ecdd04bb0-kube-api-access-84p5q\" (UniqueName: \"kubernetes.io/projected/eccc8133-8a38-46fc-99af-285ecdd04bb0-kube-api-access-84p5q\") pod \"eccc8133-8a38-46fc-99af-285ecdd04bb0\" (UID: \"eccc8133-8a38-46fc-99af-285ecdd04bb0\") " Oct 30 02:12:55.509772 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:55.492577 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eccc8133-8a38-46fc-99af-285ecdd04bb0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/eccc8133-8a38-46fc-99af-285ecdd04bb0-provider-credentials\") pod \"eccc8133-8a38-46fc-99af-285ecdd04bb0\" (UID: \"eccc8133-8a38-46fc-99af-285ecdd04bb0\") " Oct 30 02:12:55.509772 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:55.492711 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eccc8133-8a38-46fc-99af-285ecdd04bb0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eccc8133-8a38-46fc-99af-285ecdd04bb0-plugins-config-volume\") pod \"eccc8133-8a38-46fc-99af-285ecdd04bb0\" (UID: \"eccc8133-8a38-46fc-99af-285ecdd04bb0\") " Oct 30 02:12:55.509772 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:55.492877 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eccc8133-8a38-46fc-99af-285ecdd04bb0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eccc8133-8a38-46fc-99af-285ecdd04bb0-routing-overlay-volume\") pod \"eccc8133-8a38-46fc-99af-285ecdd04bb0\" (UID: \"eccc8133-8a38-46fc-99af-285ecdd04bb0\") " Oct 30 02:12:55.535772 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:55.528006 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eccc8133-8a38-46fc-99af-285ecdd04bb0-routing-overlay-volume" pod "eccc8133-8a38-46fc-99af-285ecdd04bb0" (UID: "eccc8133-8a38-46fc-99af-285ecdd04bb0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:12:55.535772 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:55.535339 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:55.661800 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:55.633677 6572 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eccc8133-8a38-46fc-99af-285ecdd04bb0-routing-overlay-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:12:55.661800 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:55.633934 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eccc8133-8a38-46fc-99af-285ecdd04bb0-provider-credentials" pod "eccc8133-8a38-46fc-99af-285ecdd04bb0" (UID: "eccc8133-8a38-46fc-99af-285ecdd04bb0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:12:55.780439 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:55.746626 6572 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eccc8133-8a38-46fc-99af-285ecdd04bb0-provider-credentials\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:12:55.785787 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:55.785721 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eccc8133-8a38-46fc-99af-285ecdd04bb0-plugins-config-volume" pod "eccc8133-8a38-46fc-99af-285ecdd04bb0" (UID: "eccc8133-8a38-46fc-99af-285ecdd04bb0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:12:55.801796 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:55.801734 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eccc8133-8a38-46fc-99af-285ecdd04bb0-kube-api-access-84p5q" pod "eccc8133-8a38-46fc-99af-285ecdd04bb0" (UID: "eccc8133-8a38-46fc-99af-285ecdd04bb0"). InnerVolumeSpecName "kube-api-access-84p5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:12:55.848027 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:55.847944 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84p5q\" (UniqueName: \"kubernetes.io/projected/eccc8133-8a38-46fc-99af-285ecdd04bb0-kube-api-access-84p5q\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:12:55.848027 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:55.847972 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eccc8133-8a38-46fc-99af-285ecdd04bb0-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.352347 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:56.352312 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f6e11e/payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x" Oct 30 02:12:56.408279 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:56.408239 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f6e11e/payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x"] Oct 30 02:12:56.418306 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:56.418268 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-f6e11e/payload-processing-external-model-e2e-extmod-b-f6e11e-5c7cbts4x"] Oct 30 02:12:56.566700 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:56.566662 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:56.575109 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:56.575075 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="614bbbec-a79d-48cd-97c1-294455b786e0" path="/var/lib/kubelet/pods/614bbbec-a79d-48cd-97c1-294455b786e0/volumes" Oct 30 02:12:56.575651 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:56.575633 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eccc8133-8a38-46fc-99af-285ecdd04bb0" path="/var/lib/kubelet/pods/eccc8133-8a38-46fc-99af-285ecdd04bb0/volumes" Oct 30 02:12:56.685439 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:56.685403 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-817fd7-h2s4x"] Oct 30 02:12:56.686320 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:56.686303 6572 cpu_manager.go:414] "containerMap: removing container" podUID="319e8315-1f3e-4ffa-a417-38b4625f1ae4" containerName="revoke-keys" Oct 30 02:12:56.686383 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:56.686322 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="319e8315-1f3e-4ffa-a417-38b4625f1ae4" containerName="revoke-keys" Oct 30 02:12:56.686383 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:56.686354 6572 cpu_manager.go:414] "containerMap: removing container" podUID="d8a485b3-17b6-4680-ba7b-02a2f7a795c4" containerName="revoke-keys" Oct 30 02:12:56.686383 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:56.686359 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8a485b3-17b6-4680-ba7b-02a2f7a795c4" containerName="revoke-keys" Oct 30 02:12:56.686383 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:56.686372 6572 cpu_manager.go:414] "containerMap: removing container" podUID="614bbbec-a79d-48cd-97c1-294455b786e0" containerName="payload-processing" Oct 30 02:12:56.686383 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:56.686378 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="614bbbec-a79d-48cd-97c1-294455b786e0" containerName="payload-processing" Oct 30 02:12:56.686383 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:56.686384 6572 cpu_manager.go:414] "containerMap: removing container" podUID="eccc8133-8a38-46fc-99af-285ecdd04bb0" containerName="payload-processing" Oct 30 02:12:56.686576 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:56.686390 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eccc8133-8a38-46fc-99af-285ecdd04bb0" containerName="payload-processing" Oct 30 02:12:56.686576 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:56.686505 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="319e8315-1f3e-4ffa-a417-38b4625f1ae4" containerName="revoke-keys" Oct 30 02:12:56.686576 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:56.686520 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8a485b3-17b6-4680-ba7b-02a2f7a795c4" containerName="revoke-keys" Oct 30 02:12:56.686576 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:56.686528 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="614bbbec-a79d-48cd-97c1-294455b786e0" containerName="payload-processing" Oct 30 02:12:56.686576 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:56.686534 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="eccc8133-8a38-46fc-99af-285ecdd04bb0" containerName="payload-processing" Oct 30 02:12:56.692115 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:56.692092 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-817fd7-h2s4x" Oct 30 02:12:56.707034 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:56.707002 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:12:56.707188 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:56.707070 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:12:56.709404 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:56.709374 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-817fd7-h2s4x"] Oct 30 02:12:56.713686 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:56.713652 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:12:56.713686 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:56.713659 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:12:56.760271 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:56.760233 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:56.769275 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:56.769224 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f30d5e-bb68-452b-ac26-489a345d5416-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-817fd7-h2s4x\" (UID: \"54f30d5e-bb68-452b-ac26-489a345d5416\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-817fd7-h2s4x" Oct 30 02:12:56.769526 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:56.769505 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mb9m\" (UniqueName: \"kubernetes.io/projected/54f30d5e-bb68-452b-ac26-489a345d5416-kube-api-access-2mb9m\") pod \"maas-api-revoke-keys-e2e-extmod-a-817fd7-h2s4x\" (UID: \"54f30d5e-bb68-452b-ac26-489a345d5416\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-817fd7-h2s4x" Oct 30 02:12:56.769589 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:56.769558 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f30d5e-bb68-452b-ac26-489a345d5416-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-817fd7-h2s4x\" (UID: \"54f30d5e-bb68-452b-ac26-489a345d5416\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-817fd7-h2s4x" Oct 30 02:12:56.870976 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:56.870940 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mb9m\" (UniqueName: \"kubernetes.io/projected/54f30d5e-bb68-452b-ac26-489a345d5416-kube-api-access-2mb9m\") pod \"maas-api-revoke-keys-e2e-extmod-a-817fd7-h2s4x\" (UID: \"54f30d5e-bb68-452b-ac26-489a345d5416\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-817fd7-h2s4x" Oct 30 02:12:56.871213 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:56.871001 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f30d5e-bb68-452b-ac26-489a345d5416-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-817fd7-h2s4x\" (UID: \"54f30d5e-bb68-452b-ac26-489a345d5416\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-817fd7-h2s4x" Oct 30 02:12:56.871281 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:56.871231 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f30d5e-bb68-452b-ac26-489a345d5416-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-817fd7-h2s4x\" (UID: \"54f30d5e-bb68-452b-ac26-489a345d5416\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-817fd7-h2s4x" Oct 30 02:12:56.871885 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:56.871856 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f30d5e-bb68-452b-ac26-489a345d5416-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-817fd7-h2s4x\" (UID: \"54f30d5e-bb68-452b-ac26-489a345d5416\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-817fd7-h2s4x" Oct 30 02:12:56.873762 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:56.873724 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f30d5e-bb68-452b-ac26-489a345d5416-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-817fd7-h2s4x\" (UID: \"54f30d5e-bb68-452b-ac26-489a345d5416\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-817fd7-h2s4x" Oct 30 02:12:56.891543 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:56.891509 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mb9m\" (UniqueName: \"kubernetes.io/projected/54f30d5e-bb68-452b-ac26-489a345d5416-kube-api-access-2mb9m\") pod \"maas-api-revoke-keys-e2e-extmod-a-817fd7-h2s4x\" (UID: \"54f30d5e-bb68-452b-ac26-489a345d5416\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-817fd7-h2s4x" Oct 30 02:12:57.008717 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:57.008626 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-817fd7-h2s4x" Oct 30 02:12:57.230384 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:57.230337 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:57.318943 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:57.318910 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-817fd7-h2s4x"] Oct 30 02:12:57.319117 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:12:57.318977 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54f30d5e_bb68_452b_ac26_489a345d5416.slice/crio-8e2f2e7f1e7bb8d575b25711d552e506b595576393e21ed4d9204ea4cdadda0d WatchSource:0}: Error finding container 8e2f2e7f1e7bb8d575b25711d552e506b595576393e21ed4d9204ea4cdadda0d: Status 404 returned error can't find the container with id 8e2f2e7f1e7bb8d575b25711d552e506b595576393e21ed4d9204ea4cdadda0d Oct 30 02:12:57.365279 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:57.365241 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-817fd7-h2s4x" event={"ID":"54f30d5e-bb68-452b-ac26-489a345d5416","Type":"ContainerStarted","Data":"8e2f2e7f1e7bb8d575b25711d552e506b595576393e21ed4d9204ea4cdadda0d"} Oct 30 02:12:58.380156 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:58.380118 6572 generic.go:356] "Generic (PLEG): container finished" podID="54f30d5e-bb68-452b-ac26-489a345d5416" containerID="73fd74bfb4ab98c1cb424ec0e390c283f20a15383f35cf6eb3fa5549ca3aa7a4" exitCode=0 Oct 30 02:12:58.380601 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:58.380200 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-817fd7-h2s4x" event={"ID":"54f30d5e-bb68-452b-ac26-489a345d5416","Type":"ContainerDied","Data":"73fd74bfb4ab98c1cb424ec0e390c283f20a15383f35cf6eb3fa5549ca3aa7a4"} Oct 30 02:12:58.799480 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:58.798302 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:58.900570 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:58.900535 6572 scope.go:122] "RemoveContainer" containerID="770cd5282723848a36b727228ceb39cd00b3d9794ecb693c87fb22f138bfa1c3" Oct 30 02:12:58.913727 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:58.913699 6572 scope.go:122] "RemoveContainer" containerID="38ac7d5ebe3d022137f5d26045b815f12733c60475b3c4d600cbad9ef6dd1614" Oct 30 02:12:58.926225 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:58.926198 6572 scope.go:122] "RemoveContainer" containerID="afe86eac8cad4c67362c7cfad93dc58cacab72be0862bce63c888c3fd711dffb" Oct 30 02:12:58.939307 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:58.939279 6572 scope.go:122] "RemoveContainer" containerID="7d2e49c06f2dfeec9f872f00dda6fdd7abf3f8119f537c6a0f7d739ae44284c2" Oct 30 02:12:59.570414 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:59.570386 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-817fd7-h2s4x" Oct 30 02:12:59.583979 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:59.583946 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54f30d5e-bb68-452b-ac26-489a345d5416-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f30d5e-bb68-452b-ac26-489a345d5416-maas-api-xks-ca\") pod \"54f30d5e-bb68-452b-ac26-489a345d5416\" (UID: \"54f30d5e-bb68-452b-ac26-489a345d5416\") " Oct 30 02:12:59.584117 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:59.584039 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54f30d5e-bb68-452b-ac26-489a345d5416-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f30d5e-bb68-452b-ac26-489a345d5416-maas-api-openshift-ca\") pod \"54f30d5e-bb68-452b-ac26-489a345d5416\" (UID: \"54f30d5e-bb68-452b-ac26-489a345d5416\") " Oct 30 02:12:59.584279 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:59.584262 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54f30d5e-bb68-452b-ac26-489a345d5416-kube-api-access-2mb9m\" (UniqueName: \"kubernetes.io/projected/54f30d5e-bb68-452b-ac26-489a345d5416-kube-api-access-2mb9m\") pod \"54f30d5e-bb68-452b-ac26-489a345d5416\" (UID: \"54f30d5e-bb68-452b-ac26-489a345d5416\") " Oct 30 02:12:59.584461 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:59.584431 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54f30d5e-bb68-452b-ac26-489a345d5416-maas-api-openshift-ca" pod "54f30d5e-bb68-452b-ac26-489a345d5416" (UID: "54f30d5e-bb68-452b-ac26-489a345d5416"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:12:59.585117 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:59.585088 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f30d5e-bb68-452b-ac26-489a345d5416-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:12:59.586457 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:59.586430 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54f30d5e-bb68-452b-ac26-489a345d5416-maas-api-xks-ca" pod "54f30d5e-bb68-452b-ac26-489a345d5416" (UID: "54f30d5e-bb68-452b-ac26-489a345d5416"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:12:59.586616 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:59.586599 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54f30d5e-bb68-452b-ac26-489a345d5416-kube-api-access-2mb9m" pod "54f30d5e-bb68-452b-ac26-489a345d5416" (UID: "54f30d5e-bb68-452b-ac26-489a345d5416"). InnerVolumeSpecName "kube-api-access-2mb9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:12:59.687049 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:59.687007 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f30d5e-bb68-452b-ac26-489a345d5416-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:12:59.687049 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:12:59.687046 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mb9m\" (UniqueName: \"kubernetes.io/projected/54f30d5e-bb68-452b-ac26-489a345d5416-kube-api-access-2mb9m\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:13:00.262775 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:00.261721 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:00.403137 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:00.403107 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-817fd7-h2s4x" Oct 30 02:13:00.403137 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:00.403132 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-817fd7-h2s4x" event={"ID":"54f30d5e-bb68-452b-ac26-489a345d5416","Type":"ContainerDied","Data":"8e2f2e7f1e7bb8d575b25711d552e506b595576393e21ed4d9204ea4cdadda0d"} Oct 30 02:13:00.403364 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:00.403180 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e2f2e7f1e7bb8d575b25711d552e506b595576393e21ed4d9204ea4cdadda0d" Oct 30 02:13:00.799528 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:00.799469 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:01.915789 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:01.915729 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:03.253879 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:03.253846 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:03.780301 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:03.780263 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-717065-656f9c7c8d-mv46m" podUID="5e718bf0-89f5-487a-a823-cdd6081e7431" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:13:03.809213 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:03.809180 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:03.887957 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:03.887922 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-717065-cv844"] Oct 30 02:13:03.899576 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:03.899541 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-717065-cv844"] Oct 30 02:13:04.017143 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:04.017101 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:04.451026 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:04.450987 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:04.561970 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:04.561938 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="319e8315-1f3e-4ffa-a417-38b4625f1ae4" path="/var/lib/kubelet/pods/319e8315-1f3e-4ffa-a417-38b4625f1ae4/volumes" Oct 30 02:13:05.833256 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:05.831540 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:07.230054 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:07.230023 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-817fd7-7dfc855d4b-4b72d"] Oct 30 02:13:07.258815 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:07.258782 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-817fd7-6f9f8d5c7f-x9gmq"] Oct 30 02:13:07.510829 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:07.489862 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:07.698660 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:07.698629 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:08.656568 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:08.656463 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:09.118845 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:09.118417 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:10.660958 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:10.660914 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:10.847537 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:10.847497 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:10.902983 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:10.902944 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq"] Oct 30 02:13:10.903225 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:10.903192 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" podUID="03f91ca9-d4d4-4e24-b535-05f521c0b3b1" containerName="istio-proxy" containerID="cri-o://d493a0f67d59ae56e1fb01dacef256ea7d0b93f23a858a8ede485cf49c1a682d" gracePeriod=30 Oct 30 02:13:11.096849 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.094041 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:11.559468 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.559440 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-717065-656f9c7c8d-mv46m" Oct 30 02:13:11.571497 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.571470 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f81908-z8dt5"] Oct 30 02:13:11.572406 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.572386 6572 cpu_manager.go:414] "containerMap: removing container" podUID="54f30d5e-bb68-452b-ac26-489a345d5416" containerName="revoke-keys" Oct 30 02:13:11.572481 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.572408 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54f30d5e-bb68-452b-ac26-489a345d5416" containerName="revoke-keys" Oct 30 02:13:11.572481 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.572431 6572 cpu_manager.go:414] "containerMap: removing container" podUID="5e718bf0-89f5-487a-a823-cdd6081e7431" containerName="payload-processing" Oct 30 02:13:11.572481 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.572436 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e718bf0-89f5-487a-a823-cdd6081e7431" containerName="payload-processing" Oct 30 02:13:11.572593 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.572584 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="54f30d5e-bb68-452b-ac26-489a345d5416" containerName="revoke-keys" Oct 30 02:13:11.572634 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.572598 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e718bf0-89f5-487a-a823-cdd6081e7431" containerName="payload-processing" Oct 30 02:13:11.576870 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.576852 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f81908-z8dt5" Oct 30 02:13:11.637711 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.635728 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:13:11.637711 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.635943 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:13:11.637711 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.635952 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:13:11.637711 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.636555 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:13:11.638252 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.638202 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f81908-z8dt5"] Oct 30 02:13:11.663016 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.662982 6572 generic.go:356] "Generic (PLEG): container finished" podID="5e718bf0-89f5-487a-a823-cdd6081e7431" containerID="e80013f7a054d07f891b138509c9d9b49e1f9e53ffd696640443f3a8bb1e3164" exitCode=0 Oct 30 02:13:11.663525 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.663041 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-717065-656f9c7c8d-mv46m" Oct 30 02:13:11.663525 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.663150 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-717065-656f9c7c8d-mv46m" event={"ID":"5e718bf0-89f5-487a-a823-cdd6081e7431","Type":"ContainerDied","Data":"e80013f7a054d07f891b138509c9d9b49e1f9e53ffd696640443f3a8bb1e3164"} Oct 30 02:13:11.663525 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.663207 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-717065-656f9c7c8d-mv46m" event={"ID":"5e718bf0-89f5-487a-a823-cdd6081e7431","Type":"ContainerDied","Data":"0d02b579652bb4b105b65af9327e16c0195f0db3c7638edf3f1287de4fd3758c"} Oct 30 02:13:11.663525 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.663239 6572 scope.go:122] "RemoveContainer" containerID="e80013f7a054d07f891b138509c9d9b49e1f9e53ffd696640443f3a8bb1e3164" Oct 30 02:13:11.672898 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.672877 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e718bf0-89f5-487a-a823-cdd6081e7431-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e718bf0-89f5-487a-a823-cdd6081e7431-plugins-config-volume\") pod \"5e718bf0-89f5-487a-a823-cdd6081e7431\" (UID: \"5e718bf0-89f5-487a-a823-cdd6081e7431\") " Oct 30 02:13:11.673288 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.673250 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e718bf0-89f5-487a-a823-cdd6081e7431-kube-api-access-kdbb5\" (UniqueName: \"kubernetes.io/projected/5e718bf0-89f5-487a-a823-cdd6081e7431-kube-api-access-kdbb5\") pod \"5e718bf0-89f5-487a-a823-cdd6081e7431\" (UID: \"5e718bf0-89f5-487a-a823-cdd6081e7431\") " Oct 30 02:13:11.674759 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.674718 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72ff6c4a-7156-401a-a334-8b11b5d3c460-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f81908-z8dt5\" (UID: \"72ff6c4a-7156-401a-a334-8b11b5d3c460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f81908-z8dt5" Oct 30 02:13:11.675080 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.675060 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72ff6c4a-7156-401a-a334-8b11b5d3c460-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f81908-z8dt5\" (UID: \"72ff6c4a-7156-401a-a334-8b11b5d3c460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f81908-z8dt5" Oct 30 02:13:11.675580 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.675561 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brkv6\" (UniqueName: \"kubernetes.io/projected/72ff6c4a-7156-401a-a334-8b11b5d3c460-kube-api-access-brkv6\") pod \"maas-api-revoke-keys-e2e-inf-b-f81908-z8dt5\" (UID: \"72ff6c4a-7156-401a-a334-8b11b5d3c460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f81908-z8dt5" Oct 30 02:13:11.683020 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.682989 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e718bf0-89f5-487a-a823-cdd6081e7431-plugins-config-volume" pod "5e718bf0-89f5-487a-a823-cdd6081e7431" (UID: "5e718bf0-89f5-487a-a823-cdd6081e7431"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:13:11.687974 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.687948 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e718bf0-89f5-487a-a823-cdd6081e7431-kube-api-access-kdbb5" pod "5e718bf0-89f5-487a-a823-cdd6081e7431" (UID: "5e718bf0-89f5-487a-a823-cdd6081e7431"). InnerVolumeSpecName "kube-api-access-kdbb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:11.689864 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.689844 6572 scope.go:122] "RemoveContainer" containerID="e80013f7a054d07f891b138509c9d9b49e1f9e53ffd696640443f3a8bb1e3164" Oct 30 02:13:11.690267 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:13:11.690239 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e80013f7a054d07f891b138509c9d9b49e1f9e53ffd696640443f3a8bb1e3164\": container with ID starting with e80013f7a054d07f891b138509c9d9b49e1f9e53ffd696640443f3a8bb1e3164 not found: ID does not exist" containerID="e80013f7a054d07f891b138509c9d9b49e1f9e53ffd696640443f3a8bb1e3164" Oct 30 02:13:11.690329 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.690279 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e80013f7a054d07f891b138509c9d9b49e1f9e53ffd696640443f3a8bb1e3164"} err="failed to get container status \"e80013f7a054d07f891b138509c9d9b49e1f9e53ffd696640443f3a8bb1e3164\": rpc error: code = NotFound desc = could not find container \"e80013f7a054d07f891b138509c9d9b49e1f9e53ffd696640443f3a8bb1e3164\": container with ID starting with e80013f7a054d07f891b138509c9d9b49e1f9e53ffd696640443f3a8bb1e3164 not found: ID does not exist" Oct 30 02:13:11.777255 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.777215 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brkv6\" (UniqueName: \"kubernetes.io/projected/72ff6c4a-7156-401a-a334-8b11b5d3c460-kube-api-access-brkv6\") pod \"maas-api-revoke-keys-e2e-inf-b-f81908-z8dt5\" (UID: \"72ff6c4a-7156-401a-a334-8b11b5d3c460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f81908-z8dt5" Oct 30 02:13:11.777515 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.777302 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72ff6c4a-7156-401a-a334-8b11b5d3c460-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f81908-z8dt5\" (UID: \"72ff6c4a-7156-401a-a334-8b11b5d3c460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f81908-z8dt5" Oct 30 02:13:11.777515 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.777373 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72ff6c4a-7156-401a-a334-8b11b5d3c460-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f81908-z8dt5\" (UID: \"72ff6c4a-7156-401a-a334-8b11b5d3c460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f81908-z8dt5" Oct 30 02:13:11.777646 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.777587 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdbb5\" (UniqueName: \"kubernetes.io/projected/5e718bf0-89f5-487a-a823-cdd6081e7431-kube-api-access-kdbb5\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:13:11.777646 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.777607 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e718bf0-89f5-487a-a823-cdd6081e7431-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:13:11.778169 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.778146 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72ff6c4a-7156-401a-a334-8b11b5d3c460-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f81908-z8dt5\" (UID: \"72ff6c4a-7156-401a-a334-8b11b5d3c460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f81908-z8dt5" Oct 30 02:13:11.779915 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.779893 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72ff6c4a-7156-401a-a334-8b11b5d3c460-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f81908-z8dt5\" (UID: \"72ff6c4a-7156-401a-a334-8b11b5d3c460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f81908-z8dt5" Oct 30 02:13:11.796010 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.795979 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brkv6\" (UniqueName: \"kubernetes.io/projected/72ff6c4a-7156-401a-a334-8b11b5d3c460-kube-api-access-brkv6\") pod \"maas-api-revoke-keys-e2e-inf-b-f81908-z8dt5\" (UID: \"72ff6c4a-7156-401a-a334-8b11b5d3c460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f81908-z8dt5" Oct 30 02:13:11.892059 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:11.892026 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f81908-z8dt5" Oct 30 02:13:12.043413 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:12.043029 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-717065-656f9c7c8d-mv46m"] Oct 30 02:13:12.049206 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:12.048926 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-717065-656f9c7c8d-mv46m"] Oct 30 02:13:12.232963 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:12.232861 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-817fd7-7dfc855d4b-4b72d" podUID="874b4b98-1b9b-4c67-8204-fa621edf5b55" containerName="payload-processing" containerID="cri-o://b6795ed4b258e4f130b254cc4e28dc22e957daa8aa367e8177ddad4bfc4a739c" gracePeriod=25 Oct 30 02:13:12.239286 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:12.239162 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f81908-z8dt5"] Oct 30 02:13:12.242653 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:13:12.242624 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72ff6c4a_7156_401a_a334_8b11b5d3c460.slice/crio-99f569310c80bbf173a0679774058856955c31d68d9c347d97ddfaa07148a2b9 WatchSource:0}: Error finding container 99f569310c80bbf173a0679774058856955c31d68d9c347d97ddfaa07148a2b9: Status 404 returned error can't find the container with id 99f569310c80bbf173a0679774058856955c31d68d9c347d97ddfaa07148a2b9 Oct 30 02:13:12.259319 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:12.259261 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-817fd7-6f9f8d5c7f-x9gmq" podUID="2db1738c-aff2-4494-a0a7-adb1d8a25ee2" containerName="payload-pre-processing" containerID="cri-o://f45d2efe2579a18fe031ce4c82b36b76e39a3c08092f0bd92a6ec3da8fcf6ac7" gracePeriod=25 Oct 30 02:13:12.547333 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:12.547254 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e718bf0-89f5-487a-a823-cdd6081e7431" path="/var/lib/kubelet/pods/5e718bf0-89f5-487a-a823-cdd6081e7431/volumes" Oct 30 02:13:12.681439 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:12.681384 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f81908-z8dt5" event={"ID":"72ff6c4a-7156-401a-a334-8b11b5d3c460","Type":"ContainerStarted","Data":"99f569310c80bbf173a0679774058856955c31d68d9c347d97ddfaa07148a2b9"} Oct 30 02:13:12.720829 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:12.720794 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:13.692958 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:13.692866 6572 generic.go:356] "Generic (PLEG): container finished" podID="72ff6c4a-7156-401a-a334-8b11b5d3c460" containerID="cf54ded7227dff6130c92f852b027353242aa3c2b699afaba1627e1ec2dd1c6a" exitCode=0 Oct 30 02:13:13.693416 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:13.692952 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f81908-z8dt5" event={"ID":"72ff6c4a-7156-401a-a334-8b11b5d3c460","Type":"ContainerDied","Data":"cf54ded7227dff6130c92f852b027353242aa3c2b699afaba1627e1ec2dd1c6a"} Oct 30 02:13:14.664952 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:14.637833 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:14.700343 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:13:14.700309 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:13:14.934779 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:14.934680 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f81908-z8dt5" Oct 30 02:13:14.974883 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:14.974844 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72ff6c4a-7156-401a-a334-8b11b5d3c460-kube-api-access-brkv6\" (UniqueName: \"kubernetes.io/projected/72ff6c4a-7156-401a-a334-8b11b5d3c460-kube-api-access-brkv6\") pod \"72ff6c4a-7156-401a-a334-8b11b5d3c460\" (UID: \"72ff6c4a-7156-401a-a334-8b11b5d3c460\") " Oct 30 02:13:14.975109 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:14.974993 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72ff6c4a-7156-401a-a334-8b11b5d3c460-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72ff6c4a-7156-401a-a334-8b11b5d3c460-maas-api-xks-ca\") pod \"72ff6c4a-7156-401a-a334-8b11b5d3c460\" (UID: \"72ff6c4a-7156-401a-a334-8b11b5d3c460\") " Oct 30 02:13:14.975109 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:14.975047 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72ff6c4a-7156-401a-a334-8b11b5d3c460-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72ff6c4a-7156-401a-a334-8b11b5d3c460-maas-api-openshift-ca\") pod \"72ff6c4a-7156-401a-a334-8b11b5d3c460\" (UID: \"72ff6c4a-7156-401a-a334-8b11b5d3c460\") " Oct 30 02:13:14.976366 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:14.976334 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72ff6c4a-7156-401a-a334-8b11b5d3c460-maas-api-openshift-ca" pod "72ff6c4a-7156-401a-a334-8b11b5d3c460" (UID: "72ff6c4a-7156-401a-a334-8b11b5d3c460"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:13:14.979017 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:14.978975 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72ff6c4a-7156-401a-a334-8b11b5d3c460-kube-api-access-brkv6" pod "72ff6c4a-7156-401a-a334-8b11b5d3c460" (UID: "72ff6c4a-7156-401a-a334-8b11b5d3c460"). InnerVolumeSpecName "kube-api-access-brkv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:14.981381 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:14.981354 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72ff6c4a-7156-401a-a334-8b11b5d3c460-maas-api-xks-ca" pod "72ff6c4a-7156-401a-a334-8b11b5d3c460" (UID: "72ff6c4a-7156-401a-a334-8b11b5d3c460"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:13:15.077472 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:15.077436 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brkv6\" (UniqueName: \"kubernetes.io/projected/72ff6c4a-7156-401a-a334-8b11b5d3c460-kube-api-access-brkv6\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:13:15.077472 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:15.077467 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72ff6c4a-7156-401a-a334-8b11b5d3c460-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:13:15.077472 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:15.077478 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72ff6c4a-7156-401a-a334-8b11b5d3c460-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:13:15.716470 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:15.716445 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f81908-z8dt5" Oct 30 02:13:15.716917 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:15.716443 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f81908-z8dt5" event={"ID":"72ff6c4a-7156-401a-a334-8b11b5d3c460","Type":"ContainerDied","Data":"99f569310c80bbf173a0679774058856955c31d68d9c347d97ddfaa07148a2b9"} Oct 30 02:13:15.716917 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:15.716574 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99f569310c80bbf173a0679774058856955c31d68d9c347d97ddfaa07148a2b9" Oct 30 02:13:16.232187 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.188733 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:16.274326 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.273894 6572 patch_prober.go:29] interesting pod/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:13:16.274326 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.273963 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" podUID="d32bbe46-7de7-4307-a62d-ee16d8b59741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:13:16.359233 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.359200 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:13:16.436828 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.436740 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-workload-certs\") pod \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\" (UID: \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\") " Oct 30 02:13:16.437191 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.437168 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-istio-data\") pod \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\" (UID: \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\") " Oct 30 02:13:16.437335 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.437318 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-istio-token\" (UniqueName: \"kubernetes.io/projected/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-istio-token\") pod \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\" (UID: \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\") " Oct 30 02:13:16.437512 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.437495 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-kube-api-access-gp929\" (UniqueName: \"kubernetes.io/projected/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-kube-api-access-gp929\") pod \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\" (UID: \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\") " Oct 30 02:13:16.437638 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.437622 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-workload-socket\") pod \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\" (UID: \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\") " Oct 30 02:13:16.437795 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.437777 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-istio-podinfo\") pod \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\" (UID: \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\") " Oct 30 02:13:16.437930 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.437915 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-credential-socket\") pod \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\" (UID: \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\") " Oct 30 02:13:16.438130 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.438112 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-istio-envoy\") pod \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\" (UID: \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\") " Oct 30 02:13:16.438442 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.438422 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-istiod-ca-cert\") pod \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\" (UID: \"03f91ca9-d4d4-4e24-b535-05f521c0b3b1\") " Oct 30 02:13:16.440354 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.439741 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-credential-socket" pod "03f91ca9-d4d4-4e24-b535-05f521c0b3b1" (UID: "03f91ca9-d4d4-4e24-b535-05f521c0b3b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:16.440354 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.440099 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-istio-data" pod "03f91ca9-d4d4-4e24-b535-05f521c0b3b1" (UID: "03f91ca9-d4d4-4e24-b535-05f521c0b3b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:16.440354 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.440286 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-workload-certs" pod "03f91ca9-d4d4-4e24-b535-05f521c0b3b1" (UID: "03f91ca9-d4d4-4e24-b535-05f521c0b3b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:16.440354 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.440331 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-workload-socket" pod "03f91ca9-d4d4-4e24-b535-05f521c0b3b1" (UID: "03f91ca9-d4d4-4e24-b535-05f521c0b3b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:16.441236 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.441212 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-istiod-ca-cert" pod "03f91ca9-d4d4-4e24-b535-05f521c0b3b1" (UID: "03f91ca9-d4d4-4e24-b535-05f521c0b3b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:13:16.447187 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.447158 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-istio-envoy" pod "03f91ca9-d4d4-4e24-b535-05f521c0b3b1" (UID: "03f91ca9-d4d4-4e24-b535-05f521c0b3b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:16.457247 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.457205 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-istio-podinfo" pod "03f91ca9-d4d4-4e24-b535-05f521c0b3b1" (UID: "03f91ca9-d4d4-4e24-b535-05f521c0b3b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:13:16.457352 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.457258 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-istio-token" pod "03f91ca9-d4d4-4e24-b535-05f521c0b3b1" (UID: "03f91ca9-d4d4-4e24-b535-05f521c0b3b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:16.459796 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.459723 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-kube-api-access-gp929" pod "03f91ca9-d4d4-4e24-b535-05f521c0b3b1" (UID: "03f91ca9-d4d4-4e24-b535-05f521c0b3b1"). InnerVolumeSpecName "kube-api-access-gp929". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:16.540574 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.540494 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:13:16.540574 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.540523 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:13:16.540574 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.540532 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:13:16.540574 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.540540 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:13:16.540574 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.540549 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp929\" (UniqueName: \"kubernetes.io/projected/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-kube-api-access-gp929\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:13:16.540574 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.540559 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:13:16.540574 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.540567 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:13:16.540574 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.540578 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:13:16.541028 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.540588 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03f91ca9-d4d4-4e24-b535-05f521c0b3b1-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:13:16.730706 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.730671 6572 generic.go:356] "Generic (PLEG): container finished" podID="03f91ca9-d4d4-4e24-b535-05f521c0b3b1" containerID="d493a0f67d59ae56e1fb01dacef256ea7d0b93f23a858a8ede485cf49c1a682d" exitCode=0 Oct 30 02:13:16.731212 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.730734 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" event={"ID":"03f91ca9-d4d4-4e24-b535-05f521c0b3b1","Type":"ContainerDied","Data":"d493a0f67d59ae56e1fb01dacef256ea7d0b93f23a858a8ede485cf49c1a682d"} Oct 30 02:13:16.731353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.731251 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" event={"ID":"03f91ca9-d4d4-4e24-b535-05f521c0b3b1","Type":"ContainerDied","Data":"cc5ca1ae192c2901e7170a38fdaf3499dae5ab9f8ebbaae35fe2886f0f229cf4"} Oct 30 02:13:16.731353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.730787 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq" Oct 30 02:13:16.731495 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.731280 6572 scope.go:122] "RemoveContainer" containerID="d493a0f67d59ae56e1fb01dacef256ea7d0b93f23a858a8ede485cf49c1a682d" Oct 30 02:13:16.745823 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.745800 6572 scope.go:122] "RemoveContainer" containerID="5ab944474ca5be3bb0733170ecb446ebd8ab1ccbfac2272c3a8d2cbf84656ca5" Oct 30 02:13:16.758987 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.758958 6572 scope.go:122] "RemoveContainer" containerID="d493a0f67d59ae56e1fb01dacef256ea7d0b93f23a858a8ede485cf49c1a682d" Oct 30 02:13:16.759344 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:13:16.759321 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d493a0f67d59ae56e1fb01dacef256ea7d0b93f23a858a8ede485cf49c1a682d\": container with ID starting with d493a0f67d59ae56e1fb01dacef256ea7d0b93f23a858a8ede485cf49c1a682d not found: ID does not exist" containerID="d493a0f67d59ae56e1fb01dacef256ea7d0b93f23a858a8ede485cf49c1a682d" Oct 30 02:13:16.759448 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.759358 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d493a0f67d59ae56e1fb01dacef256ea7d0b93f23a858a8ede485cf49c1a682d"} err="failed to get container status \"d493a0f67d59ae56e1fb01dacef256ea7d0b93f23a858a8ede485cf49c1a682d\": rpc error: code = NotFound desc = could not find container \"d493a0f67d59ae56e1fb01dacef256ea7d0b93f23a858a8ede485cf49c1a682d\": container with ID starting with d493a0f67d59ae56e1fb01dacef256ea7d0b93f23a858a8ede485cf49c1a682d not found: ID does not exist" Oct 30 02:13:16.759448 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.759386 6572 scope.go:122] "RemoveContainer" containerID="5ab944474ca5be3bb0733170ecb446ebd8ab1ccbfac2272c3a8d2cbf84656ca5" Oct 30 02:13:16.759679 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:13:16.759657 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ab944474ca5be3bb0733170ecb446ebd8ab1ccbfac2272c3a8d2cbf84656ca5\": container with ID starting with 5ab944474ca5be3bb0733170ecb446ebd8ab1ccbfac2272c3a8d2cbf84656ca5 not found: ID does not exist" containerID="5ab944474ca5be3bb0733170ecb446ebd8ab1ccbfac2272c3a8d2cbf84656ca5" Oct 30 02:13:16.759736 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.759689 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ab944474ca5be3bb0733170ecb446ebd8ab1ccbfac2272c3a8d2cbf84656ca5"} err="failed to get container status \"5ab944474ca5be3bb0733170ecb446ebd8ab1ccbfac2272c3a8d2cbf84656ca5\": rpc error: code = NotFound desc = could not find container \"5ab944474ca5be3bb0733170ecb446ebd8ab1ccbfac2272c3a8d2cbf84656ca5\": container with ID starting with 5ab944474ca5be3bb0733170ecb446ebd8ab1ccbfac2272c3a8d2cbf84656ca5 not found: ID does not exist" Oct 30 02:13:16.786466 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.786419 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq"] Oct 30 02:13:16.791826 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:16.791701 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-cxfhq"] Oct 30 02:13:17.703424 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:17.703325 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:17.745433 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:17.741985 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-817fd7-6f9f8d5c7f-x9gmq" podUID="2db1738c-aff2-4494-a0a7-adb1d8a25ee2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:13:17.748957 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:17.748865 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-817fd7-7dfc855d4b-4b72d" podUID="874b4b98-1b9b-4c67-8204-fa621edf5b55" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:13:18.546075 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:18.546027 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03f91ca9-d4d4-4e24-b535-05f521c0b3b1" path="/var/lib/kubelet/pods/03f91ca9-d4d4-4e24-b535-05f521c0b3b1/volumes" Oct 30 02:13:19.328192 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:19.328147 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:19.511640 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:19.511608 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:19.771868 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:19.771834 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:20.061894 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.061785 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg"] Oct 30 02:13:20.063168 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.063136 6572 cpu_manager.go:414] "containerMap: removing container" podUID="72ff6c4a-7156-401a-a334-8b11b5d3c460" containerName="revoke-keys" Oct 30 02:13:20.063168 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.063167 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72ff6c4a-7156-401a-a334-8b11b5d3c460" containerName="revoke-keys" Oct 30 02:13:20.063419 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.063214 6572 cpu_manager.go:414] "containerMap: removing container" podUID="03f91ca9-d4d4-4e24-b535-05f521c0b3b1" containerName="istio-proxy" Oct 30 02:13:20.063419 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.063222 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03f91ca9-d4d4-4e24-b535-05f521c0b3b1" containerName="istio-proxy" Oct 30 02:13:20.063419 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.063248 6572 cpu_manager.go:414] "containerMap: removing container" podUID="03f91ca9-d4d4-4e24-b535-05f521c0b3b1" containerName="istio-proxy" Oct 30 02:13:20.063419 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.063256 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03f91ca9-d4d4-4e24-b535-05f521c0b3b1" containerName="istio-proxy" Oct 30 02:13:20.063656 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.063434 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="03f91ca9-d4d4-4e24-b535-05f521c0b3b1" containerName="istio-proxy" Oct 30 02:13:20.063656 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.063454 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="03f91ca9-d4d4-4e24-b535-05f521c0b3b1" containerName="istio-proxy" Oct 30 02:13:20.063656 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.063478 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="72ff6c4a-7156-401a-a334-8b11b5d3c460" containerName="revoke-keys" Oct 30 02:13:20.070962 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.070934 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:13:20.088697 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.088661 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg"] Oct 30 02:13:20.092132 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.092096 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2d5aa9-openshift-default-dockercfg-zngnf\"" Oct 30 02:13:20.105224 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.105189 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e662617-4bbb-4021-8bae-ba979960bb94-credential-socket\") pod \"e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg\" (UID: \"0e662617-4bbb-4021-8bae-ba979960bb94\") " pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:13:20.105373 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.105311 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e662617-4bbb-4021-8bae-ba979960bb94-workload-certs\") pod \"e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg\" (UID: \"0e662617-4bbb-4021-8bae-ba979960bb94\") " pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:13:20.105373 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.105353 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e662617-4bbb-4021-8bae-ba979960bb94-istio-data\") pod \"e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg\" (UID: \"0e662617-4bbb-4021-8bae-ba979960bb94\") " pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:13:20.105512 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.105452 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e662617-4bbb-4021-8bae-ba979960bb94-istio-token\") pod \"e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg\" (UID: \"0e662617-4bbb-4021-8bae-ba979960bb94\") " pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:13:20.105560 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.105522 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e662617-4bbb-4021-8bae-ba979960bb94-istio-envoy\") pod \"e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg\" (UID: \"0e662617-4bbb-4021-8bae-ba979960bb94\") " pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:13:20.105605 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.105556 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e662617-4bbb-4021-8bae-ba979960bb94-istiod-ca-cert\") pod \"e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg\" (UID: \"0e662617-4bbb-4021-8bae-ba979960bb94\") " pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:13:20.105605 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.105591 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e662617-4bbb-4021-8bae-ba979960bb94-istio-podinfo\") pod \"e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg\" (UID: \"0e662617-4bbb-4021-8bae-ba979960bb94\") " pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:13:20.105859 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.105835 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psld8\" (UniqueName: \"kubernetes.io/projected/0e662617-4bbb-4021-8bae-ba979960bb94-kube-api-access-psld8\") pod \"e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg\" (UID: \"0e662617-4bbb-4021-8bae-ba979960bb94\") " pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:13:20.105949 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.105933 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e662617-4bbb-4021-8bae-ba979960bb94-workload-socket\") pod \"e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg\" (UID: \"0e662617-4bbb-4021-8bae-ba979960bb94\") " pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:13:20.207577 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.207530 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e662617-4bbb-4021-8bae-ba979960bb94-istio-envoy\") pod \"e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg\" (UID: \"0e662617-4bbb-4021-8bae-ba979960bb94\") " pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:13:20.207577 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.207576 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e662617-4bbb-4021-8bae-ba979960bb94-istiod-ca-cert\") pod \"e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg\" (UID: \"0e662617-4bbb-4021-8bae-ba979960bb94\") " pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:13:20.207888 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.207594 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e662617-4bbb-4021-8bae-ba979960bb94-istio-podinfo\") pod \"e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg\" (UID: \"0e662617-4bbb-4021-8bae-ba979960bb94\") " pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:13:20.207888 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.207731 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psld8\" (UniqueName: \"kubernetes.io/projected/0e662617-4bbb-4021-8bae-ba979960bb94-kube-api-access-psld8\") pod \"e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg\" (UID: \"0e662617-4bbb-4021-8bae-ba979960bb94\") " pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:13:20.207888 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.207816 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e662617-4bbb-4021-8bae-ba979960bb94-workload-socket\") pod \"e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg\" (UID: \"0e662617-4bbb-4021-8bae-ba979960bb94\") " pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:13:20.208075 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.207899 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e662617-4bbb-4021-8bae-ba979960bb94-credential-socket\") pod \"e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg\" (UID: \"0e662617-4bbb-4021-8bae-ba979960bb94\") " pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:13:20.208075 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.207962 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e662617-4bbb-4021-8bae-ba979960bb94-workload-certs\") pod \"e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg\" (UID: \"0e662617-4bbb-4021-8bae-ba979960bb94\") " pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:13:20.208075 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.208002 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e662617-4bbb-4021-8bae-ba979960bb94-istio-data\") pod \"e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg\" (UID: \"0e662617-4bbb-4021-8bae-ba979960bb94\") " pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:13:20.208075 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.208052 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e662617-4bbb-4021-8bae-ba979960bb94-istio-token\") pod \"e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg\" (UID: \"0e662617-4bbb-4021-8bae-ba979960bb94\") " pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:13:20.208317 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.208174 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e662617-4bbb-4021-8bae-ba979960bb94-workload-socket\") pod \"e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg\" (UID: \"0e662617-4bbb-4021-8bae-ba979960bb94\") " pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:13:20.208317 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.208274 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e662617-4bbb-4021-8bae-ba979960bb94-credential-socket\") pod \"e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg\" (UID: \"0e662617-4bbb-4021-8bae-ba979960bb94\") " pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:13:20.208482 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.208423 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e662617-4bbb-4021-8bae-ba979960bb94-workload-certs\") pod \"e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg\" (UID: \"0e662617-4bbb-4021-8bae-ba979960bb94\") " pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:13:20.208610 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.208446 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e662617-4bbb-4021-8bae-ba979960bb94-istio-data\") pod \"e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg\" (UID: \"0e662617-4bbb-4021-8bae-ba979960bb94\") " pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:13:20.208610 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.208451 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e662617-4bbb-4021-8bae-ba979960bb94-istiod-ca-cert\") pod \"e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg\" (UID: \"0e662617-4bbb-4021-8bae-ba979960bb94\") " pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:13:20.214090 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.214062 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:20.227480 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.227440 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e662617-4bbb-4021-8bae-ba979960bb94-istio-envoy\") pod \"e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg\" (UID: \"0e662617-4bbb-4021-8bae-ba979960bb94\") " pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:13:20.227826 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.227802 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psld8\" (UniqueName: \"kubernetes.io/projected/0e662617-4bbb-4021-8bae-ba979960bb94-kube-api-access-psld8\") pod \"e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg\" (UID: \"0e662617-4bbb-4021-8bae-ba979960bb94\") " pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:13:20.227918 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.227892 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e662617-4bbb-4021-8bae-ba979960bb94-istio-token\") pod \"e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg\" (UID: \"0e662617-4bbb-4021-8bae-ba979960bb94\") " pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:13:20.228023 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.228004 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e662617-4bbb-4021-8bae-ba979960bb94-istio-podinfo\") pod \"e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg\" (UID: \"0e662617-4bbb-4021-8bae-ba979960bb94\") " pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:13:20.400062 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.399968 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:20.419090 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.419051 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:13:20.995721 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:20.995568 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:21.990437 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:21.990395 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f81908-5b744797f9-5bkkh"] Oct 30 02:13:22.154495 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:22.154453 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:22.293442 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:22.293413 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg"] Oct 30 02:13:22.296851 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:13:22.296815 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e662617_4bbb_4021_8bae_ba979960bb94.slice/crio-f6bfa4b20fb2793bbdc2a4f0c1e7e1b8ec1157dab5757a042aa79d39c7c15c04 WatchSource:0}: Error finding container f6bfa4b20fb2793bbdc2a4f0c1e7e1b8ec1157dab5757a042aa79d39c7c15c04: Status 404 returned error can't find the container with id f6bfa4b20fb2793bbdc2a4f0c1e7e1b8ec1157dab5757a042aa79d39c7c15c04 Oct 30 02:13:22.300112 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:22.300071 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:13:22.300263 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:22.300156 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:13:22.300263 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:22.300188 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:13:22.336206 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:22.336162 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:22.591556 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:22.591411 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:22.854163 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:22.853153 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" event={"ID":"0e662617-4bbb-4021-8bae-ba979960bb94","Type":"ContainerStarted","Data":"275743f8bf29b2b7143c534cfc570df9a9e1ca985baed8835a56ac070c838cb9"} Oct 30 02:13:22.854163 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:22.853218 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" event={"ID":"0e662617-4bbb-4021-8bae-ba979960bb94","Type":"ContainerStarted","Data":"f6bfa4b20fb2793bbdc2a4f0c1e7e1b8ec1157dab5757a042aa79d39c7c15c04"} Oct 30 02:13:22.902505 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:22.901519 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" podStartSLOduration=2.901500538 podStartE2EDuration="2.901500538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:13:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:13:22.896211743 +0000 UTC m=+3328.472637148" watchObservedRunningTime="2026-10-30 02:13:22.901500538 +0000 UTC m=+3328.477925944" Oct 30 02:13:23.420202 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:23.420160 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:13:23.428290 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:23.428263 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:13:23.864812 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:23.864720 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:13:23.865862 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:23.865836 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:13:24.329242 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:24.329202 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:25.392684 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:25.392640 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:25.621479 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:25.621449 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:25.886464 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:25.886428 6572 generic.go:356] "Generic (PLEG): container finished" podID="d32bbe46-7de7-4307-a62d-ee16d8b59741" containerID="bb1dffbbc8f6b7d54e0b3092038eeadb2860e27159c7f4fcde7bb5b692499bee" exitCode=0 Oct 30 02:13:25.886652 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:25.886505 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" event={"ID":"d32bbe46-7de7-4307-a62d-ee16d8b59741","Type":"ContainerDied","Data":"bb1dffbbc8f6b7d54e0b3092038eeadb2860e27159c7f4fcde7bb5b692499bee"} Oct 30 02:13:25.887155 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:25.887136 6572 scope.go:122] "RemoveContainer" containerID="bb1dffbbc8f6b7d54e0b3092038eeadb2860e27159c7f4fcde7bb5b692499bee" Oct 30 02:13:25.889672 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:25.889638 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:13:25.889782 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:25.889733 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:13:25.889841 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:25.889799 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:13:26.000684 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:26.000640 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:26.904954 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:26.904913 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" event={"ID":"d32bbe46-7de7-4307-a62d-ee16d8b59741","Type":"ContainerStarted","Data":"f2f7c87a7516025f1fe86fe1725973ee44ea731d37473fe7c122fea190ce1cff"} Oct 30 02:13:26.991849 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:26.991810 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f81908-5b744797f9-5bkkh" podUID="e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497" containerName="payload-pre-processing" containerID="cri-o://3890e0823fde6ad3557cf92e1e3904f3e32b522c5138a700ea5c432c60d9bf56" gracePeriod=25 Oct 30 02:13:27.037236 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:27.037176 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:27.042067 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:27.042042 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8167a986-mx528"] Oct 30 02:13:27.048317 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:27.048290 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8167a986-mx528"] Oct 30 02:13:27.092476 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:27.092443 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2d5aa9-66754f6cd7-5sdmc"] Oct 30 02:13:27.097115 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:27.096557 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d5aa9-66754f6cd7-5sdmc" Oct 30 02:13:27.138883 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:27.138220 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-2d5aa9-dockercfg-stmkq\"" Oct 30 02:13:27.144961 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:27.144231 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2d5aa9-66754f6cd7-5sdmc"] Oct 30 02:13:27.225573 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:27.212176 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-2d5aa9\"" Oct 30 02:13:27.274773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:27.259739 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d425439-27d7-485c-b992-199e1b587c60-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2d5aa9-66754f6cd7-5sdmc\" (UID: \"4d425439-27d7-485c-b992-199e1b587c60\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d5aa9-66754f6cd7-5sdmc" Oct 30 02:13:27.274773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:27.259889 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf8fg\" (UniqueName: \"kubernetes.io/projected/4d425439-27d7-485c-b992-199e1b587c60-kube-api-access-bf8fg\") pod \"payload-pre-processing-e2e-inf-b-2d5aa9-66754f6cd7-5sdmc\" (UID: \"4d425439-27d7-485c-b992-199e1b587c60\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d5aa9-66754f6cd7-5sdmc" Oct 30 02:13:27.460863 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:27.450592 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf8fg\" (UniqueName: \"kubernetes.io/projected/4d425439-27d7-485c-b992-199e1b587c60-kube-api-access-bf8fg\") pod \"payload-pre-processing-e2e-inf-b-2d5aa9-66754f6cd7-5sdmc\" (UID: \"4d425439-27d7-485c-b992-199e1b587c60\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d5aa9-66754f6cd7-5sdmc" Oct 30 02:13:27.464461 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:27.464431 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:13:27.492447 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:27.492342 6572 patch_prober.go:29] interesting pod/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.104:15021/healthz/ready\": dial tcp 10.133.1.104:15021: connect: connection refused" start-of-body= Oct 30 02:13:27.492447 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:27.492402 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" podUID="d32bbe46-7de7-4307-a62d-ee16d8b59741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.104:15021/healthz/ready\": dial tcp 10.133.1.104:15021: connect: connection refused" Oct 30 02:13:27.510823 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:27.505114 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d425439-27d7-485c-b992-199e1b587c60-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2d5aa9-66754f6cd7-5sdmc\" (UID: \"4d425439-27d7-485c-b992-199e1b587c60\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d5aa9-66754f6cd7-5sdmc" Oct 30 02:13:27.510823 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:27.506323 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d425439-27d7-485c-b992-199e1b587c60-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2d5aa9-66754f6cd7-5sdmc\" (UID: \"4d425439-27d7-485c-b992-199e1b587c60\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d5aa9-66754f6cd7-5sdmc" Oct 30 02:13:27.510823 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:27.508234 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:27.569872 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:27.567951 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf8fg\" (UniqueName: \"kubernetes.io/projected/4d425439-27d7-485c-b992-199e1b587c60-kube-api-access-bf8fg\") pod \"payload-pre-processing-e2e-inf-b-2d5aa9-66754f6cd7-5sdmc\" (UID: \"4d425439-27d7-485c-b992-199e1b587c60\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d5aa9-66754f6cd7-5sdmc" Oct 30 02:13:27.704945 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:27.703266 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:27.742248 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:27.741323 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-817fd7-6f9f8d5c7f-x9gmq" podUID="2db1738c-aff2-4494-a0a7-adb1d8a25ee2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:13:27.742248 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:27.741952 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-817fd7-7dfc855d4b-4b72d" podUID="874b4b98-1b9b-4c67-8204-fa621edf5b55" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:13:27.747347 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:27.747277 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d5aa9-66754f6cd7-5sdmc" Oct 30 02:13:28.078659 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:28.078627 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2d5aa9-66754f6cd7-5sdmc"] Oct 30 02:13:28.079177 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:13:28.079153 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d425439_27d7_485c_b992_199e1b587c60.slice/crio-c85dd24bd97c568ac054d9abc546ffeec1b6c917308ce254f99b2ceaa8e7a762 WatchSource:0}: Error finding container c85dd24bd97c568ac054d9abc546ffeec1b6c917308ce254f99b2ceaa8e7a762: Status 404 returned error can't find the container with id c85dd24bd97c568ac054d9abc546ffeec1b6c917308ce254f99b2ceaa8e7a762 Oct 30 02:13:28.392241 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:28.392151 6572 patch_prober.go:29] interesting pod/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.104:15021/healthz/ready\": dial tcp 10.133.1.104:15021: connect: connection refused" start-of-body= Oct 30 02:13:28.392241 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:28.392213 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" podUID="d32bbe46-7de7-4307-a62d-ee16d8b59741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.104:15021/healthz/ready\": dial tcp 10.133.1.104:15021: connect: connection refused" Oct 30 02:13:28.584788 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:28.584125 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dd2d590-698e-4769-800f-55d530d7ca2a" path="/var/lib/kubelet/pods/0dd2d590-698e-4769-800f-55d530d7ca2a/volumes" Oct 30 02:13:28.957667 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:28.957629 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d5aa9-66754f6cd7-5sdmc" event={"ID":"4d425439-27d7-485c-b992-199e1b587c60","Type":"ContainerStarted","Data":"21f6cf1389d0e4f46ab99ce6b4c2ee8aa44dbbe2ff83ed9716c1ed15648682ad"} Oct 30 02:13:28.957932 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:28.957728 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d5aa9-66754f6cd7-5sdmc" event={"ID":"4d425439-27d7-485c-b992-199e1b587c60","Type":"ContainerStarted","Data":"c85dd24bd97c568ac054d9abc546ffeec1b6c917308ce254f99b2ceaa8e7a762"} Oct 30 02:13:28.957932 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:28.957917 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d5aa9-66754f6cd7-5sdmc" Oct 30 02:13:28.998822 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:28.998771 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d5aa9-66754f6cd7-5sdmc" podStartSLOduration=1.998757801 podStartE2EDuration="1.998757801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:13:28.993987293 +0000 UTC m=+3334.570412696" watchObservedRunningTime="2026-10-30 02:13:28.998757801 +0000 UTC m=+3334.575183193" Oct 30 02:13:29.271824 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:29.266061 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:29.392177 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:29.392146 6572 patch_prober.go:29] interesting pod/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.104:15021/healthz/ready\": dial tcp 10.133.1.104:15021: connect: connection refused" start-of-body= Oct 30 02:13:29.392355 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:29.392205 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" podUID="d32bbe46-7de7-4307-a62d-ee16d8b59741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.104:15021/healthz/ready\": dial tcp 10.133.1.104:15021: connect: connection refused" Oct 30 02:13:30.232054 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:30.232010 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:13:30.392229 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:30.392194 6572 patch_prober.go:29] interesting pod/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.104:15021/healthz/ready\": dial tcp 10.133.1.104:15021: connect: connection refused" start-of-body= Oct 30 02:13:30.392616 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:30.392260 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" podUID="d32bbe46-7de7-4307-a62d-ee16d8b59741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.104:15021/healthz/ready\": dial tcp 10.133.1.104:15021: connect: connection refused" Oct 30 02:13:30.585593 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:30.585503 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:31.368191 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:31.365947 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:31.373872 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:31.373850 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-817fd7-h2s4x"] Oct 30 02:13:31.377788 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:31.377722 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-817fd7-h2s4x"] Oct 30 02:13:31.392904 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:31.392879 6572 patch_prober.go:29] interesting pod/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.104:15021/healthz/ready\": dial tcp 10.133.1.104:15021: connect: connection refused" start-of-body= Oct 30 02:13:31.393226 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:31.392921 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" podUID="d32bbe46-7de7-4307-a62d-ee16d8b59741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.104:15021/healthz/ready\": dial tcp 10.133.1.104:15021: connect: connection refused" Oct 30 02:13:31.552810 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:31.552742 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:31.780499 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:31.780464 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:32.392823 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:32.392789 6572 patch_prober.go:29] interesting pod/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.104:15021/healthz/ready\": dial tcp 10.133.1.104:15021: connect: connection refused" start-of-body= Oct 30 02:13:32.393014 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:32.392859 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" podUID="d32bbe46-7de7-4307-a62d-ee16d8b59741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.104:15021/healthz/ready\": dial tcp 10.133.1.104:15021: connect: connection refused" Oct 30 02:13:32.552249 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:32.552155 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54f30d5e-bb68-452b-ac26-489a345d5416" path="/var/lib/kubelet/pods/54f30d5e-bb68-452b-ac26-489a345d5416/volumes" Oct 30 02:13:32.649841 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:32.649813 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-817fd7-7dfc855d4b-4b72d" Oct 30 02:13:32.686944 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:32.686918 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/874b4b98-1b9b-4c67-8204-fa621edf5b55-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/874b4b98-1b9b-4c67-8204-fa621edf5b55-plugins-config-volume\") pod \"874b4b98-1b9b-4c67-8204-fa621edf5b55\" (UID: \"874b4b98-1b9b-4c67-8204-fa621edf5b55\") " Oct 30 02:13:32.687339 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:32.687322 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/874b4b98-1b9b-4c67-8204-fa621edf5b55-kube-api-access-jplk2\" (UniqueName: \"kubernetes.io/projected/874b4b98-1b9b-4c67-8204-fa621edf5b55-kube-api-access-jplk2\") pod \"874b4b98-1b9b-4c67-8204-fa621edf5b55\" (UID: \"874b4b98-1b9b-4c67-8204-fa621edf5b55\") " Oct 30 02:13:32.687442 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:32.687387 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/874b4b98-1b9b-4c67-8204-fa621edf5b55-plugins-config-volume" pod "874b4b98-1b9b-4c67-8204-fa621edf5b55" (UID: "874b4b98-1b9b-4c67-8204-fa621edf5b55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:13:32.687871 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:32.687847 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/874b4b98-1b9b-4c67-8204-fa621edf5b55-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:13:32.689533 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:32.689491 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-817fd7-6f9f8d5c7f-x9gmq" Oct 30 02:13:32.690280 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:32.690256 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/874b4b98-1b9b-4c67-8204-fa621edf5b55-kube-api-access-jplk2" pod "874b4b98-1b9b-4c67-8204-fa621edf5b55" (UID: "874b4b98-1b9b-4c67-8204-fa621edf5b55"). InnerVolumeSpecName "kube-api-access-jplk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:32.789407 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:32.789353 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2db1738c-aff2-4494-a0a7-adb1d8a25ee2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2db1738c-aff2-4494-a0a7-adb1d8a25ee2-plugins-config-volume\") pod \"2db1738c-aff2-4494-a0a7-adb1d8a25ee2\" (UID: \"2db1738c-aff2-4494-a0a7-adb1d8a25ee2\") " Oct 30 02:13:32.789632 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:32.789503 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2db1738c-aff2-4494-a0a7-adb1d8a25ee2-kube-api-access-6fbdq\" (UniqueName: \"kubernetes.io/projected/2db1738c-aff2-4494-a0a7-adb1d8a25ee2-kube-api-access-6fbdq\") pod \"2db1738c-aff2-4494-a0a7-adb1d8a25ee2\" (UID: \"2db1738c-aff2-4494-a0a7-adb1d8a25ee2\") " Oct 30 02:13:32.790236 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:32.790206 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2db1738c-aff2-4494-a0a7-adb1d8a25ee2-plugins-config-volume" pod "2db1738c-aff2-4494-a0a7-adb1d8a25ee2" (UID: "2db1738c-aff2-4494-a0a7-adb1d8a25ee2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:13:32.790398 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:32.790291 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jplk2\" (UniqueName: \"kubernetes.io/projected/874b4b98-1b9b-4c67-8204-fa621edf5b55-kube-api-access-jplk2\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:13:32.792523 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:32.792493 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2db1738c-aff2-4494-a0a7-adb1d8a25ee2-kube-api-access-6fbdq" pod "2db1738c-aff2-4494-a0a7-adb1d8a25ee2" (UID: "2db1738c-aff2-4494-a0a7-adb1d8a25ee2"). InnerVolumeSpecName "kube-api-access-6fbdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:32.892111 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:32.892029 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2db1738c-aff2-4494-a0a7-adb1d8a25ee2-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:13:32.892111 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:32.892059 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fbdq\" (UniqueName: \"kubernetes.io/projected/2db1738c-aff2-4494-a0a7-adb1d8a25ee2-kube-api-access-6fbdq\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:13:32.991793 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:32.991735 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:33.066437 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:33.066400 6572 generic.go:356] "Generic (PLEG): container finished" podID="874b4b98-1b9b-4c67-8204-fa621edf5b55" containerID="b6795ed4b258e4f130b254cc4e28dc22e957daa8aa367e8177ddad4bfc4a739c" exitCode=0 Oct 30 02:13:33.066636 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:33.066453 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-817fd7-7dfc855d4b-4b72d" Oct 30 02:13:33.066636 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:33.066502 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-817fd7-7dfc855d4b-4b72d" event={"ID":"874b4b98-1b9b-4c67-8204-fa621edf5b55","Type":"ContainerDied","Data":"b6795ed4b258e4f130b254cc4e28dc22e957daa8aa367e8177ddad4bfc4a739c"} Oct 30 02:13:33.066636 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:33.066560 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-817fd7-7dfc855d4b-4b72d" event={"ID":"874b4b98-1b9b-4c67-8204-fa621edf5b55","Type":"ContainerDied","Data":"814e8bd98ee1006d90ce75c5fa106fddeb1807278168d43d7662ecf9e4170070"} Oct 30 02:13:33.066636 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:33.066588 6572 scope.go:122] "RemoveContainer" containerID="b6795ed4b258e4f130b254cc4e28dc22e957daa8aa367e8177ddad4bfc4a739c" Oct 30 02:13:33.068255 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:33.068233 6572 generic.go:356] "Generic (PLEG): container finished" podID="2db1738c-aff2-4494-a0a7-adb1d8a25ee2" containerID="f45d2efe2579a18fe031ce4c82b36b76e39a3c08092f0bd92a6ec3da8fcf6ac7" exitCode=0 Oct 30 02:13:33.068398 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:33.068285 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-817fd7-6f9f8d5c7f-x9gmq" Oct 30 02:13:33.068398 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:33.068360 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-817fd7-6f9f8d5c7f-x9gmq" event={"ID":"2db1738c-aff2-4494-a0a7-adb1d8a25ee2","Type":"ContainerDied","Data":"f45d2efe2579a18fe031ce4c82b36b76e39a3c08092f0bd92a6ec3da8fcf6ac7"} Oct 30 02:13:33.068508 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:33.068414 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-817fd7-6f9f8d5c7f-x9gmq" event={"ID":"2db1738c-aff2-4494-a0a7-adb1d8a25ee2","Type":"ContainerDied","Data":"e5f6e2e08a0adfc836ee08d1e87c3398348042777a2e376fc9d089726b44428d"} Oct 30 02:13:33.080092 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:33.080069 6572 scope.go:122] "RemoveContainer" containerID="b6795ed4b258e4f130b254cc4e28dc22e957daa8aa367e8177ddad4bfc4a739c" Oct 30 02:13:33.080354 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:13:33.080328 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6795ed4b258e4f130b254cc4e28dc22e957daa8aa367e8177ddad4bfc4a739c\": container with ID starting with b6795ed4b258e4f130b254cc4e28dc22e957daa8aa367e8177ddad4bfc4a739c not found: ID does not exist" containerID="b6795ed4b258e4f130b254cc4e28dc22e957daa8aa367e8177ddad4bfc4a739c" Oct 30 02:13:33.080454 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:33.080354 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6795ed4b258e4f130b254cc4e28dc22e957daa8aa367e8177ddad4bfc4a739c"} err="failed to get container status \"b6795ed4b258e4f130b254cc4e28dc22e957daa8aa367e8177ddad4bfc4a739c\": rpc error: code = NotFound desc = could not find container \"b6795ed4b258e4f130b254cc4e28dc22e957daa8aa367e8177ddad4bfc4a739c\": container with ID starting with b6795ed4b258e4f130b254cc4e28dc22e957daa8aa367e8177ddad4bfc4a739c not found: ID does not exist" Oct 30 02:13:33.080454 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:33.080372 6572 scope.go:122] "RemoveContainer" containerID="f45d2efe2579a18fe031ce4c82b36b76e39a3c08092f0bd92a6ec3da8fcf6ac7" Oct 30 02:13:33.094730 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:33.094710 6572 scope.go:122] "RemoveContainer" containerID="f45d2efe2579a18fe031ce4c82b36b76e39a3c08092f0bd92a6ec3da8fcf6ac7" Oct 30 02:13:33.095027 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:13:33.094991 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f45d2efe2579a18fe031ce4c82b36b76e39a3c08092f0bd92a6ec3da8fcf6ac7\": container with ID starting with f45d2efe2579a18fe031ce4c82b36b76e39a3c08092f0bd92a6ec3da8fcf6ac7 not found: ID does not exist" containerID="f45d2efe2579a18fe031ce4c82b36b76e39a3c08092f0bd92a6ec3da8fcf6ac7" Oct 30 02:13:33.095120 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:33.095029 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f45d2efe2579a18fe031ce4c82b36b76e39a3c08092f0bd92a6ec3da8fcf6ac7"} err="failed to get container status \"f45d2efe2579a18fe031ce4c82b36b76e39a3c08092f0bd92a6ec3da8fcf6ac7\": rpc error: code = NotFound desc = could not find container \"f45d2efe2579a18fe031ce4c82b36b76e39a3c08092f0bd92a6ec3da8fcf6ac7\": container with ID starting with f45d2efe2579a18fe031ce4c82b36b76e39a3c08092f0bd92a6ec3da8fcf6ac7 not found: ID does not exist" Oct 30 02:13:33.132673 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:33.132637 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-817fd7-7dfc855d4b-4b72d"] Oct 30 02:13:33.137368 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:33.137341 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-817fd7-7dfc855d4b-4b72d"] Oct 30 02:13:33.171034 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:33.171001 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-817fd7-6f9f8d5c7f-x9gmq"] Oct 30 02:13:33.176450 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:33.176422 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-817fd7-6f9f8d5c7f-x9gmq"] Oct 30 02:13:33.397740 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:33.397709 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:13:33.398972 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:33.398950 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:13:34.484828 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:34.484796 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:34.548917 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:34.548878 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2db1738c-aff2-4494-a0a7-adb1d8a25ee2" path="/var/lib/kubelet/pods/2db1738c-aff2-4494-a0a7-adb1d8a25ee2/volumes" Oct 30 02:13:34.549416 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:34.549400 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="874b4b98-1b9b-4c67-8204-fa621edf5b55" path="/var/lib/kubelet/pods/874b4b98-1b9b-4c67-8204-fa621edf5b55/volumes" Oct 30 02:13:35.995877 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:35.994862 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:36.412680 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:36.412647 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f81908-5b744797f9-5bkkh" podUID="e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:13:37.518769 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:37.511657 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:37.933673 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:37.933640 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:38.139837 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:38.139795 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:38.508786 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:38.507979 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:38.726886 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:38.726850 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f6e11e-fg5tj"] Oct 30 02:13:38.727683 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:38.727667 6572 cpu_manager.go:414] "containerMap: removing container" podUID="2db1738c-aff2-4494-a0a7-adb1d8a25ee2" containerName="payload-pre-processing" Oct 30 02:13:38.727739 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:38.727684 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2db1738c-aff2-4494-a0a7-adb1d8a25ee2" containerName="payload-pre-processing" Oct 30 02:13:38.727739 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:38.727703 6572 cpu_manager.go:414] "containerMap: removing container" podUID="874b4b98-1b9b-4c67-8204-fa621edf5b55" containerName="payload-processing" Oct 30 02:13:38.727739 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:38.727710 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="874b4b98-1b9b-4c67-8204-fa621edf5b55" containerName="payload-processing" Oct 30 02:13:38.727897 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:38.727879 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="2db1738c-aff2-4494-a0a7-adb1d8a25ee2" containerName="payload-pre-processing" Oct 30 02:13:38.727897 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:38.727890 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="874b4b98-1b9b-4c67-8204-fa621edf5b55" containerName="payload-processing" Oct 30 02:13:38.732693 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:38.732673 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f6e11e-fg5tj" Oct 30 02:13:38.750811 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:38.750776 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f6e11e-fg5tj"] Oct 30 02:13:38.755853 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:38.755827 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:13:38.755853 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:38.755835 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:13:38.775347 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:38.775253 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:13:38.776113 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:38.775623 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:13:38.798537 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:38.798489 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58622924-77ea-4ecb-b876-6d43d54997d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f6e11e-fg5tj\" (UID: \"58622924-77ea-4ecb-b876-6d43d54997d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f6e11e-fg5tj" Oct 30 02:13:38.798727 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:38.798670 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58622924-77ea-4ecb-b876-6d43d54997d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f6e11e-fg5tj\" (UID: \"58622924-77ea-4ecb-b876-6d43d54997d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f6e11e-fg5tj" Oct 30 02:13:38.798727 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:38.798707 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv7g7\" (UniqueName: \"kubernetes.io/projected/58622924-77ea-4ecb-b876-6d43d54997d1-kube-api-access-dv7g7\") pod \"maas-api-revoke-keys-e2e-extmod-b-f6e11e-fg5tj\" (UID: \"58622924-77ea-4ecb-b876-6d43d54997d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f6e11e-fg5tj" Oct 30 02:13:38.900576 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:38.900545 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58622924-77ea-4ecb-b876-6d43d54997d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f6e11e-fg5tj\" (UID: \"58622924-77ea-4ecb-b876-6d43d54997d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f6e11e-fg5tj" Oct 30 02:13:38.900792 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:38.900646 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58622924-77ea-4ecb-b876-6d43d54997d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f6e11e-fg5tj\" (UID: \"58622924-77ea-4ecb-b876-6d43d54997d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f6e11e-fg5tj" Oct 30 02:13:38.900792 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:38.900676 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv7g7\" (UniqueName: \"kubernetes.io/projected/58622924-77ea-4ecb-b876-6d43d54997d1-kube-api-access-dv7g7\") pod \"maas-api-revoke-keys-e2e-extmod-b-f6e11e-fg5tj\" (UID: \"58622924-77ea-4ecb-b876-6d43d54997d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f6e11e-fg5tj" Oct 30 02:13:38.901435 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:38.901408 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58622924-77ea-4ecb-b876-6d43d54997d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f6e11e-fg5tj\" (UID: \"58622924-77ea-4ecb-b876-6d43d54997d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f6e11e-fg5tj" Oct 30 02:13:38.903100 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:38.903078 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58622924-77ea-4ecb-b876-6d43d54997d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f6e11e-fg5tj\" (UID: \"58622924-77ea-4ecb-b876-6d43d54997d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f6e11e-fg5tj" Oct 30 02:13:38.924093 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:38.924066 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv7g7\" (UniqueName: \"kubernetes.io/projected/58622924-77ea-4ecb-b876-6d43d54997d1-kube-api-access-dv7g7\") pod \"maas-api-revoke-keys-e2e-extmod-b-f6e11e-fg5tj\" (UID: \"58622924-77ea-4ecb-b876-6d43d54997d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f6e11e-fg5tj" Oct 30 02:13:39.048688 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:39.048603 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f6e11e-fg5tj" Oct 30 02:13:39.280864 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:39.280698 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:39.357216 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:39.356243 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f6e11e-fg5tj"] Oct 30 02:13:39.360108 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:13:39.359964 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58622924_77ea_4ecb_b876_6d43d54997d1.slice/crio-5bd132b780fa3ad9e3b068ce7a0588304bf69790ec3024fe14565694517e5caf WatchSource:0}: Error finding container 5bd132b780fa3ad9e3b068ce7a0588304bf69790ec3024fe14565694517e5caf: Status 404 returned error can't find the container with id 5bd132b780fa3ad9e3b068ce7a0588304bf69790ec3024fe14565694517e5caf Oct 30 02:13:39.970683 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:39.970650 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d5aa9-66754f6cd7-5sdmc" Oct 30 02:13:40.152242 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:40.152197 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f6e11e-fg5tj" event={"ID":"58622924-77ea-4ecb-b876-6d43d54997d1","Type":"ContainerStarted","Data":"b53127a074daf4425c03a11a189306505512b1a062565ff01956a68ac0d67a27"} Oct 30 02:13:40.152437 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:40.152288 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f6e11e-fg5tj" event={"ID":"58622924-77ea-4ecb-b876-6d43d54997d1","Type":"ContainerStarted","Data":"5bd132b780fa3ad9e3b068ce7a0588304bf69790ec3024fe14565694517e5caf"} Oct 30 02:13:40.193621 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:40.193569 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f6e11e-fg5tj" podStartSLOduration=2.193554342 podStartE2EDuration="2.193554342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:13:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:13:40.189233873 +0000 UTC m=+3345.765659275" watchObservedRunningTime="2026-10-30 02:13:40.193554342 +0000 UTC m=+3345.769979743" Oct 30 02:13:40.993470 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:40.984737 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:41.164646 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:41.164605 6572 generic.go:356] "Generic (PLEG): container finished" podID="58622924-77ea-4ecb-b876-6d43d54997d1" containerID="b53127a074daf4425c03a11a189306505512b1a062565ff01956a68ac0d67a27" exitCode=0 Oct 30 02:13:41.164875 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:41.164683 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f6e11e-fg5tj" event={"ID":"58622924-77ea-4ecb-b876-6d43d54997d1","Type":"ContainerDied","Data":"b53127a074daf4425c03a11a189306505512b1a062565ff01956a68ac0d67a27"} Oct 30 02:13:42.343076 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:42.343045 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f6e11e-fg5tj" Oct 30 02:13:42.476146 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:42.476053 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58622924-77ea-4ecb-b876-6d43d54997d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58622924-77ea-4ecb-b876-6d43d54997d1-maas-api-xks-ca\") pod \"58622924-77ea-4ecb-b876-6d43d54997d1\" (UID: \"58622924-77ea-4ecb-b876-6d43d54997d1\") " Oct 30 02:13:42.476146 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:42.476108 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58622924-77ea-4ecb-b876-6d43d54997d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58622924-77ea-4ecb-b876-6d43d54997d1-maas-api-openshift-ca\") pod \"58622924-77ea-4ecb-b876-6d43d54997d1\" (UID: \"58622924-77ea-4ecb-b876-6d43d54997d1\") " Oct 30 02:13:42.476412 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:42.476391 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58622924-77ea-4ecb-b876-6d43d54997d1-kube-api-access-dv7g7\" (UniqueName: \"kubernetes.io/projected/58622924-77ea-4ecb-b876-6d43d54997d1-kube-api-access-dv7g7\") pod \"58622924-77ea-4ecb-b876-6d43d54997d1\" (UID: \"58622924-77ea-4ecb-b876-6d43d54997d1\") " Oct 30 02:13:42.476587 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:42.476566 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58622924-77ea-4ecb-b876-6d43d54997d1-maas-api-openshift-ca" pod "58622924-77ea-4ecb-b876-6d43d54997d1" (UID: "58622924-77ea-4ecb-b876-6d43d54997d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:13:42.477048 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:42.477017 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58622924-77ea-4ecb-b876-6d43d54997d1-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:13:42.478431 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:42.478404 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58622924-77ea-4ecb-b876-6d43d54997d1-maas-api-xks-ca" pod "58622924-77ea-4ecb-b876-6d43d54997d1" (UID: "58622924-77ea-4ecb-b876-6d43d54997d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:13:42.478515 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:42.478450 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58622924-77ea-4ecb-b876-6d43d54997d1-kube-api-access-dv7g7" pod "58622924-77ea-4ecb-b876-6d43d54997d1" (UID: "58622924-77ea-4ecb-b876-6d43d54997d1"). InnerVolumeSpecName "kube-api-access-dv7g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:42.578535 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:42.578500 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv7g7\" (UniqueName: \"kubernetes.io/projected/58622924-77ea-4ecb-b876-6d43d54997d1-kube-api-access-dv7g7\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:13:42.578535 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:42.578531 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58622924-77ea-4ecb-b876-6d43d54997d1-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:13:42.672858 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:42.659509 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:43.185026 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:43.185000 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f6e11e-fg5tj" Oct 30 02:13:43.185232 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:43.185034 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f6e11e-fg5tj" event={"ID":"58622924-77ea-4ecb-b876-6d43d54997d1","Type":"ContainerDied","Data":"5bd132b780fa3ad9e3b068ce7a0588304bf69790ec3024fe14565694517e5caf"} Oct 30 02:13:43.185232 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:43.185087 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bd132b780fa3ad9e3b068ce7a0588304bf69790ec3024fe14565694517e5caf" Oct 30 02:13:44.330262 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:44.328576 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:44.703384 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:13:44.703342 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:13:45.770979 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:45.770935 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:46.412535 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:46.412493 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f81908-5b744797f9-5bkkh" podUID="e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:13:47.292924 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:47.291787 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:47.303514 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:47.303299 6572 generic.go:356] "Generic (PLEG): container finished" podID="e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497" containerID="3890e0823fde6ad3557cf92e1e3904f3e32b522c5138a700ea5c432c60d9bf56" exitCode=0 Oct 30 02:13:47.303514 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:47.303450 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f81908-5b744797f9-5bkkh" event={"ID":"e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497","Type":"ContainerDied","Data":"3890e0823fde6ad3557cf92e1e3904f3e32b522c5138a700ea5c432c60d9bf56"} Oct 30 02:13:47.342959 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:47.342930 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f81908-5b744797f9-5bkkh" Oct 30 02:13:47.395060 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:47.395023 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497-kube-api-access-rshgw\" (UniqueName: \"kubernetes.io/projected/e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497-kube-api-access-rshgw\") pod \"e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497\" (UID: \"e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497\") " Oct 30 02:13:47.395286 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:47.395136 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497-plugins-config-volume\") pod \"e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497\" (UID: \"e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497\") " Oct 30 02:13:47.396407 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:47.396368 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497-plugins-config-volume" pod "e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497" (UID: "e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:13:47.398126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:47.398083 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497-kube-api-access-rshgw" pod "e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497" (UID: "e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497"). InnerVolumeSpecName "kube-api-access-rshgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:47.498101 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:47.498059 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:13:47.498101 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:47.498098 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rshgw\" (UniqueName: \"kubernetes.io/projected/e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497-kube-api-access-rshgw\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:13:48.314606 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:48.314577 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f81908-5b744797f9-5bkkh" Oct 30 02:13:48.314606 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:48.314596 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f81908-5b744797f9-5bkkh" event={"ID":"e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497","Type":"ContainerDied","Data":"bef634cde777e1c2126892ab251fcb41bdb74ba4685b24f58609e830344916c8"} Oct 30 02:13:48.315308 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:48.314653 6572 scope.go:122] "RemoveContainer" containerID="3890e0823fde6ad3557cf92e1e3904f3e32b522c5138a700ea5c432c60d9bf56" Oct 30 02:13:48.375234 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:48.375198 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f81908-5b744797f9-5bkkh"] Oct 30 02:13:48.385890 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:48.385863 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f81908-5b744797f9-5bkkh"] Oct 30 02:13:48.546405 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:48.546364 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497" path="/var/lib/kubelet/pods/e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497/volumes" Oct 30 02:13:48.821614 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:48.821580 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:49.187194 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:49.187160 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f6e11e-7d74b77d6-w25pd"] Oct 30 02:13:49.218465 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:49.218434 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f6e11e-794f9f4fc5-z7q9b"] Oct 30 02:13:49.407866 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:49.407834 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:49.598915 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:49.598834 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:50.407774 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:50.402367 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:51.885832 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:51.885794 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:53.483717 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:53.483681 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:54.188361 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:54.188280 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-f6e11e-7d74b77d6-w25pd" podUID="ddf6fa1c-dc69-45e9-b865-46d8b020731a" containerName="payload-processing" containerID="cri-o://dcbb5ce83e85e3917446d92e2c7d0a21f216d489c345b08deb3b91e33a4684c8" gracePeriod=25 Oct 30 02:13:54.219157 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:54.219101 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f6e11e-794f9f4fc5-z7q9b" podUID="9c2e3d3a-c68b-4dd6-94ea-e17f286b5738" containerName="payload-pre-processing" containerID="cri-o://d15215779b4c61392092e632c398d4317d118a00e16d2944c3dde15f9dac98be" gracePeriod=25 Oct 30 02:13:54.382804 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:54.382488 6572 generic.go:356] "Generic (PLEG): container finished" podID="ddf6fa1c-dc69-45e9-b865-46d8b020731a" containerID="dcbb5ce83e85e3917446d92e2c7d0a21f216d489c345b08deb3b91e33a4684c8" exitCode=0 Oct 30 02:13:54.382804 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:54.382530 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-f6e11e-7d74b77d6-w25pd" event={"ID":"ddf6fa1c-dc69-45e9-b865-46d8b020731a","Type":"ContainerDied","Data":"dcbb5ce83e85e3917446d92e2c7d0a21f216d489c345b08deb3b91e33a4684c8"} Oct 30 02:13:54.385161 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:54.385128 6572 generic.go:356] "Generic (PLEG): container finished" podID="9c2e3d3a-c68b-4dd6-94ea-e17f286b5738" containerID="d15215779b4c61392092e632c398d4317d118a00e16d2944c3dde15f9dac98be" exitCode=0 Oct 30 02:13:54.385345 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:54.385160 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f6e11e-794f9f4fc5-z7q9b" event={"ID":"9c2e3d3a-c68b-4dd6-94ea-e17f286b5738","Type":"ContainerDied","Data":"d15215779b4c61392092e632c398d4317d118a00e16d2944c3dde15f9dac98be"} Oct 30 02:13:54.529164 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:54.529136 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-f6e11e-7d74b77d6-w25pd" Oct 30 02:13:54.576640 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:54.576604 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f6e11e-794f9f4fc5-z7q9b" Oct 30 02:13:54.661394 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:54.661356 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c2e3d3a-c68b-4dd6-94ea-e17f286b5738-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c2e3d3a-c68b-4dd6-94ea-e17f286b5738-plugins-config-volume\") pod \"9c2e3d3a-c68b-4dd6-94ea-e17f286b5738\" (UID: \"9c2e3d3a-c68b-4dd6-94ea-e17f286b5738\") " Oct 30 02:13:54.661598 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:54.661544 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c2e3d3a-c68b-4dd6-94ea-e17f286b5738-kube-api-access-gd46n\" (UniqueName: \"kubernetes.io/projected/9c2e3d3a-c68b-4dd6-94ea-e17f286b5738-kube-api-access-gd46n\") pod \"9c2e3d3a-c68b-4dd6-94ea-e17f286b5738\" (UID: \"9c2e3d3a-c68b-4dd6-94ea-e17f286b5738\") " Oct 30 02:13:54.661716 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:54.661678 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddf6fa1c-dc69-45e9-b865-46d8b020731a-kube-api-access-87xhh\" (UniqueName: \"kubernetes.io/projected/ddf6fa1c-dc69-45e9-b865-46d8b020731a-kube-api-access-87xhh\") pod \"ddf6fa1c-dc69-45e9-b865-46d8b020731a\" (UID: \"ddf6fa1c-dc69-45e9-b865-46d8b020731a\") " Oct 30 02:13:54.661897 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:54.661879 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddf6fa1c-dc69-45e9-b865-46d8b020731a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddf6fa1c-dc69-45e9-b865-46d8b020731a-plugins-config-volume\") pod \"ddf6fa1c-dc69-45e9-b865-46d8b020731a\" (UID: \"ddf6fa1c-dc69-45e9-b865-46d8b020731a\") " Oct 30 02:13:54.662800 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:54.662198 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c2e3d3a-c68b-4dd6-94ea-e17f286b5738-plugins-config-volume" pod "9c2e3d3a-c68b-4dd6-94ea-e17f286b5738" (UID: "9c2e3d3a-c68b-4dd6-94ea-e17f286b5738"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:13:54.662800 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:54.662355 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddf6fa1c-dc69-45e9-b865-46d8b020731a-plugins-config-volume" pod "ddf6fa1c-dc69-45e9-b865-46d8b020731a" (UID: "ddf6fa1c-dc69-45e9-b865-46d8b020731a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:13:54.663410 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:54.663388 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddf6fa1c-dc69-45e9-b865-46d8b020731a-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:13:54.663530 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:54.663416 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c2e3d3a-c68b-4dd6-94ea-e17f286b5738-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:13:54.665718 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:54.665689 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c2e3d3a-c68b-4dd6-94ea-e17f286b5738-kube-api-access-gd46n" pod "9c2e3d3a-c68b-4dd6-94ea-e17f286b5738" (UID: "9c2e3d3a-c68b-4dd6-94ea-e17f286b5738"). InnerVolumeSpecName "kube-api-access-gd46n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:54.665825 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:54.665732 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddf6fa1c-dc69-45e9-b865-46d8b020731a-kube-api-access-87xhh" pod "ddf6fa1c-dc69-45e9-b865-46d8b020731a" (UID: "ddf6fa1c-dc69-45e9-b865-46d8b020731a"). InnerVolumeSpecName "kube-api-access-87xhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:54.764829 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:54.764728 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd46n\" (UniqueName: \"kubernetes.io/projected/9c2e3d3a-c68b-4dd6-94ea-e17f286b5738-kube-api-access-gd46n\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:13:54.764829 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:54.764801 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87xhh\" (UniqueName: \"kubernetes.io/projected/ddf6fa1c-dc69-45e9-b865-46d8b020731a-kube-api-access-87xhh\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:13:55.140379 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:55.140292 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:55.396613 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:55.396530 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-f6e11e-7d74b77d6-w25pd" Oct 30 02:13:55.396613 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:55.396555 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-f6e11e-7d74b77d6-w25pd" event={"ID":"ddf6fa1c-dc69-45e9-b865-46d8b020731a","Type":"ContainerDied","Data":"59b3e8f8e8f14f4907b80646898f94afe7230be6583018d54df80bd908dd7592"} Oct 30 02:13:55.396613 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:55.396613 6572 scope.go:122] "RemoveContainer" containerID="dcbb5ce83e85e3917446d92e2c7d0a21f216d489c345b08deb3b91e33a4684c8" Oct 30 02:13:55.398415 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:55.398393 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f6e11e-794f9f4fc5-z7q9b" Oct 30 02:13:55.398534 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:55.398430 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f6e11e-794f9f4fc5-z7q9b" event={"ID":"9c2e3d3a-c68b-4dd6-94ea-e17f286b5738","Type":"ContainerDied","Data":"64a01825610acb87e1ec7e0668c9dcc407991f907fdfe7e4f48dad8f9cf79150"} Oct 30 02:13:55.409962 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:55.409840 6572 scope.go:122] "RemoveContainer" containerID="d15215779b4c61392092e632c398d4317d118a00e16d2944c3dde15f9dac98be" Oct 30 02:13:55.447625 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:55.447595 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f6e11e-794f9f4fc5-z7q9b"] Oct 30 02:13:55.460000 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:55.459957 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f6e11e-794f9f4fc5-z7q9b"] Oct 30 02:13:55.487356 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:55.487318 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f6e11e-7d74b77d6-w25pd"] Oct 30 02:13:55.492287 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:55.492257 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f6e11e-7d74b77d6-w25pd"] Oct 30 02:13:56.546555 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:56.546522 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c2e3d3a-c68b-4dd6-94ea-e17f286b5738" path="/var/lib/kubelet/pods/9c2e3d3a-c68b-4dd6-94ea-e17f286b5738/volumes" Oct 30 02:13:56.547080 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:56.547064 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddf6fa1c-dc69-45e9-b865-46d8b020731a" path="/var/lib/kubelet/pods/ddf6fa1c-dc69-45e9-b865-46d8b020731a/volumes" Oct 30 02:13:56.841456 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:56.841371 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:58.395451 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:58.395415 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:59.006868 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:59.006837 6572 scope.go:122] "RemoveContainer" containerID="b622469261665873552005e23154592dfe9c2b6c6e25a0178286c78aba235f58" Oct 30 02:13:59.900775 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:13:59.899962 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:01.193255 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:01.193225 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:01.849893 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:01.849848 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:01.891517 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:01.891486 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f81908-z8dt5"] Oct 30 02:14:01.896017 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:01.895993 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f81908-z8dt5"] Oct 30 02:14:02.043007 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:02.042967 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:02.466906 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:02.466876 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:02.544366 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:02.544332 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72ff6c4a-7156-401a-a334-8b11b5d3c460" path="/var/lib/kubelet/pods/72ff6c4a-7156-401a-a334-8b11b5d3c460/volumes" Oct 30 02:14:03.875768 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:03.875518 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:05.515788 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:05.515735 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:07.117776 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:07.112138 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:08.521290 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:08.521243 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:08.741209 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:08.741170 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst"] Oct 30 02:14:08.741468 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:08.741441 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" podUID="ee995a27-1255-4c9c-88c4-c6e4eb58a77d" containerName="istio-proxy" containerID="cri-o://62afb4d5e249e315edbcb3f4fff9df302ef2ea37a401bf2d1fea6e47542ec553" gracePeriod=30 Oct 30 02:14:08.745812 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:08.745789 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:09.010483 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:09.010447 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:09.226877 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:09.226839 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt"] Oct 30 02:14:09.227200 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:09.227150 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" podUID="fe01e89d-ffd9-40bc-8720-eee65f4a42eb" containerName="istio-proxy" containerID="cri-o://12996ff0bc936e2d0a017b435104cbe551e061b1ddb1c7b7b24d3ac1c0bd2e35" gracePeriod=30 Oct 30 02:14:09.350828 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:09.350721 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2eae39-6n5sl"] Oct 30 02:14:09.351510 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:09.351495 6572 cpu_manager.go:414] "containerMap: removing container" podUID="58622924-77ea-4ecb-b876-6d43d54997d1" containerName="revoke-keys" Oct 30 02:14:09.351557 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:09.351511 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58622924-77ea-4ecb-b876-6d43d54997d1" containerName="revoke-keys" Oct 30 02:14:09.351557 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:09.351530 6572 cpu_manager.go:414] "containerMap: removing container" podUID="e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497" containerName="payload-pre-processing" Oct 30 02:14:09.351557 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:09.351535 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497" containerName="payload-pre-processing" Oct 30 02:14:09.351663 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:09.351565 6572 cpu_manager.go:414] "containerMap: removing container" podUID="9c2e3d3a-c68b-4dd6-94ea-e17f286b5738" containerName="payload-pre-processing" Oct 30 02:14:09.351663 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:09.351570 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c2e3d3a-c68b-4dd6-94ea-e17f286b5738" containerName="payload-pre-processing" Oct 30 02:14:09.351663 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:09.351577 6572 cpu_manager.go:414] "containerMap: removing container" podUID="ddf6fa1c-dc69-45e9-b865-46d8b020731a" containerName="payload-processing" Oct 30 02:14:09.351663 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:09.351581 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddf6fa1c-dc69-45e9-b865-46d8b020731a" containerName="payload-processing" Oct 30 02:14:09.351852 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:09.351690 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="58622924-77ea-4ecb-b876-6d43d54997d1" containerName="revoke-keys" Oct 30 02:14:09.351852 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:09.351702 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="e29b33d1-d0ec-4ad3-a2ac-7c27b90c2497" containerName="payload-pre-processing" Oct 30 02:14:09.351852 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:09.351712 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c2e3d3a-c68b-4dd6-94ea-e17f286b5738" containerName="payload-pre-processing" Oct 30 02:14:09.351852 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:09.351716 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddf6fa1c-dc69-45e9-b865-46d8b020731a" containerName="payload-processing" Oct 30 02:14:09.357339 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:09.357316 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2eae39-6n5sl" Oct 30 02:14:09.371182 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:09.371159 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:14:09.371354 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:09.371333 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:14:09.373161 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:09.373140 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2eae39-6n5sl"] Oct 30 02:14:09.385397 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:09.385376 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:14:09.386997 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:09.386973 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:14:09.505109 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:09.505079 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66sdv\" (UniqueName: \"kubernetes.io/projected/613abc91-29df-44d4-9bec-d37e815e9c86-kube-api-access-66sdv\") pod \"maas-api-revoke-keys-e2e-extmod-a-2eae39-6n5sl\" (UID: \"613abc91-29df-44d4-9bec-d37e815e9c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2eae39-6n5sl" Oct 30 02:14:09.505282 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:09.505247 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/613abc91-29df-44d4-9bec-d37e815e9c86-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2eae39-6n5sl\" (UID: \"613abc91-29df-44d4-9bec-d37e815e9c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2eae39-6n5sl" Oct 30 02:14:09.505282 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:09.505270 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/613abc91-29df-44d4-9bec-d37e815e9c86-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2eae39-6n5sl\" (UID: \"613abc91-29df-44d4-9bec-d37e815e9c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2eae39-6n5sl" Oct 30 02:14:09.607217 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:09.607134 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66sdv\" (UniqueName: \"kubernetes.io/projected/613abc91-29df-44d4-9bec-d37e815e9c86-kube-api-access-66sdv\") pod \"maas-api-revoke-keys-e2e-extmod-a-2eae39-6n5sl\" (UID: \"613abc91-29df-44d4-9bec-d37e815e9c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2eae39-6n5sl" Oct 30 02:14:09.607581 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:09.607351 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/613abc91-29df-44d4-9bec-d37e815e9c86-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2eae39-6n5sl\" (UID: \"613abc91-29df-44d4-9bec-d37e815e9c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2eae39-6n5sl" Oct 30 02:14:09.607581 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:09.607390 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/613abc91-29df-44d4-9bec-d37e815e9c86-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2eae39-6n5sl\" (UID: \"613abc91-29df-44d4-9bec-d37e815e9c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2eae39-6n5sl" Oct 30 02:14:09.608119 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:09.608097 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/613abc91-29df-44d4-9bec-d37e815e9c86-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2eae39-6n5sl\" (UID: \"613abc91-29df-44d4-9bec-d37e815e9c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2eae39-6n5sl" Oct 30 02:14:09.609722 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:09.609699 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/613abc91-29df-44d4-9bec-d37e815e9c86-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2eae39-6n5sl\" (UID: \"613abc91-29df-44d4-9bec-d37e815e9c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2eae39-6n5sl" Oct 30 02:14:09.625276 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:09.625254 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66sdv\" (UniqueName: \"kubernetes.io/projected/613abc91-29df-44d4-9bec-d37e815e9c86-kube-api-access-66sdv\") pod \"maas-api-revoke-keys-e2e-extmod-a-2eae39-6n5sl\" (UID: \"613abc91-29df-44d4-9bec-d37e815e9c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2eae39-6n5sl" Oct 30 02:14:09.673184 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:09.673162 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2eae39-6n5sl" Oct 30 02:14:09.996840 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:14:09.996792 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod613abc91_29df_44d4_9bec_d37e815e9c86.slice/crio-19d5fafe2be927985ac2610fd2692738e9d3abac10f0d6897c79b09a727ec9a7 WatchSource:0}: Error finding container 19d5fafe2be927985ac2610fd2692738e9d3abac10f0d6897c79b09a727ec9a7: Status 404 returned error can't find the container with id 19d5fafe2be927985ac2610fd2692738e9d3abac10f0d6897c79b09a727ec9a7 Oct 30 02:14:09.998663 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:09.998567 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2eae39-6n5sl"] Oct 30 02:14:10.520919 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:10.520880 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:10.866635 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:10.866550 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2eae39-6n5sl" event={"ID":"613abc91-29df-44d4-9bec-d37e815e9c86","Type":"ContainerStarted","Data":"ace2cc1ea69c1a546fec74f78528f9a27c384c067ef9f13dd04c0ebd40e5b58e"} Oct 30 02:14:10.866635 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:10.866614 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2eae39-6n5sl" event={"ID":"613abc91-29df-44d4-9bec-d37e815e9c86","Type":"ContainerStarted","Data":"19d5fafe2be927985ac2610fd2692738e9d3abac10f0d6897c79b09a727ec9a7"} Oct 30 02:14:10.906942 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:10.906890 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2eae39-6n5sl" podStartSLOduration=1.906877564 podStartE2EDuration="1.906877564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:14:10.905562287 +0000 UTC m=+3376.481987696" watchObservedRunningTime="2026-10-30 02:14:10.906877564 +0000 UTC m=+3376.483302966" Oct 30 02:14:11.783993 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:11.783954 6572 patch_prober.go:29] interesting pod/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:14:11.784166 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:11.784007 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" podUID="ee995a27-1255-4c9c-88c4-c6e4eb58a77d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:14:11.877230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:11.877196 6572 generic.go:356] "Generic (PLEG): container finished" podID="613abc91-29df-44d4-9bec-d37e815e9c86" containerID="ace2cc1ea69c1a546fec74f78528f9a27c384c067ef9f13dd04c0ebd40e5b58e" exitCode=0 Oct 30 02:14:11.877699 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:11.877281 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2eae39-6n5sl" event={"ID":"613abc91-29df-44d4-9bec-d37e815e9c86","Type":"ContainerDied","Data":"ace2cc1ea69c1a546fec74f78528f9a27c384c067ef9f13dd04c0ebd40e5b58e"} Oct 30 02:14:12.006788 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:12.006705 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:13.063382 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:13.063353 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2eae39-6n5sl" Oct 30 02:14:13.185438 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:13.185410 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/613abc91-29df-44d4-9bec-d37e815e9c86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/613abc91-29df-44d4-9bec-d37e815e9c86-maas-api-openshift-ca\") pod \"613abc91-29df-44d4-9bec-d37e815e9c86\" (UID: \"613abc91-29df-44d4-9bec-d37e815e9c86\") " Oct 30 02:14:13.185664 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:13.185634 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/613abc91-29df-44d4-9bec-d37e815e9c86-kube-api-access-66sdv\" (UniqueName: \"kubernetes.io/projected/613abc91-29df-44d4-9bec-d37e815e9c86-kube-api-access-66sdv\") pod \"613abc91-29df-44d4-9bec-d37e815e9c86\" (UID: \"613abc91-29df-44d4-9bec-d37e815e9c86\") " Oct 30 02:14:13.185837 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:13.185703 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/613abc91-29df-44d4-9bec-d37e815e9c86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/613abc91-29df-44d4-9bec-d37e815e9c86-maas-api-xks-ca\") pod \"613abc91-29df-44d4-9bec-d37e815e9c86\" (UID: \"613abc91-29df-44d4-9bec-d37e815e9c86\") " Oct 30 02:14:13.185837 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:13.185814 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/613abc91-29df-44d4-9bec-d37e815e9c86-maas-api-openshift-ca" pod "613abc91-29df-44d4-9bec-d37e815e9c86" (UID: "613abc91-29df-44d4-9bec-d37e815e9c86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:14:13.186313 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:13.186292 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/613abc91-29df-44d4-9bec-d37e815e9c86-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:13.187761 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:13.187727 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/613abc91-29df-44d4-9bec-d37e815e9c86-maas-api-xks-ca" pod "613abc91-29df-44d4-9bec-d37e815e9c86" (UID: "613abc91-29df-44d4-9bec-d37e815e9c86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:14:13.187886 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:13.187861 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/613abc91-29df-44d4-9bec-d37e815e9c86-kube-api-access-66sdv" pod "613abc91-29df-44d4-9bec-d37e815e9c86" (UID: "613abc91-29df-44d4-9bec-d37e815e9c86"). InnerVolumeSpecName "kube-api-access-66sdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:13.288433 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:13.288400 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66sdv\" (UniqueName: \"kubernetes.io/projected/613abc91-29df-44d4-9bec-d37e815e9c86-kube-api-access-66sdv\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:13.288433 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:13.288429 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/613abc91-29df-44d4-9bec-d37e815e9c86-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:13.564212 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:13.564119 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:13.677861 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:13.677825 6572 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:14:13.678147 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:13.678110 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" podUID="fe01e89d-ffd9-40bc-8720-eee65f4a42eb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:14:13.777089 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:13.777058 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:13.898587 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:13.898555 6572 generic.go:356] "Generic (PLEG): container finished" podID="ee995a27-1255-4c9c-88c4-c6e4eb58a77d" containerID="62afb4d5e249e315edbcb3f4fff9df302ef2ea37a401bf2d1fea6e47542ec553" exitCode=0 Oct 30 02:14:13.898813 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:13.898632 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" event={"ID":"ee995a27-1255-4c9c-88c4-c6e4eb58a77d","Type":"ContainerDied","Data":"62afb4d5e249e315edbcb3f4fff9df302ef2ea37a401bf2d1fea6e47542ec553"} Oct 30 02:14:13.898813 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:13.898705 6572 scope.go:122] "RemoveContainer" containerID="e12add2d220062db183af8b624107420e07785b347a77a42019f4da8c03eff4c" Oct 30 02:14:13.900650 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:13.900628 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2eae39-6n5sl" Oct 30 02:14:13.900650 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:13.900637 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2eae39-6n5sl" event={"ID":"613abc91-29df-44d4-9bec-d37e815e9c86","Type":"ContainerDied","Data":"19d5fafe2be927985ac2610fd2692738e9d3abac10f0d6897c79b09a727ec9a7"} Oct 30 02:14:13.900848 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:13.900677 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19d5fafe2be927985ac2610fd2692738e9d3abac10f0d6897c79b09a727ec9a7" Oct 30 02:14:14.015625 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:14.015596 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:14:14.106866 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:14.106782 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-istio-podinfo\") pod \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\" (UID: \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\") " Oct 30 02:14:14.106866 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:14.106866 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-kube-api-access-7l4p9\" (UniqueName: \"kubernetes.io/projected/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-kube-api-access-7l4p9\") pod \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\" (UID: \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\") " Oct 30 02:14:14.107432 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:14.106953 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-credential-socket\") pod \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\" (UID: \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\") " Oct 30 02:14:14.107432 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:14.107061 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-istio-data\") pod \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\" (UID: \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\") " Oct 30 02:14:14.107432 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:14.107180 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-istio-token\" (UniqueName: \"kubernetes.io/projected/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-istio-token\") pod \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\" (UID: \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\") " Oct 30 02:14:14.107432 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:14.107210 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-workload-certs\") pod \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\" (UID: \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\") " Oct 30 02:14:14.107432 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:14.107256 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-credential-socket" pod "ee995a27-1255-4c9c-88c4-c6e4eb58a77d" (UID: "ee995a27-1255-4c9c-88c4-c6e4eb58a77d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:14.107432 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:14.107270 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-istio-envoy\") pod \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\" (UID: \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\") " Oct 30 02:14:14.107432 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:14.107317 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-workload-socket\") pod \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\" (UID: \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\") " Oct 30 02:14:14.107432 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:14.107374 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-istiod-ca-cert\") pod \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\" (UID: \"ee995a27-1255-4c9c-88c4-c6e4eb58a77d\") " Oct 30 02:14:14.107896 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:14.107451 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-workload-certs" pod "ee995a27-1255-4c9c-88c4-c6e4eb58a77d" (UID: "ee995a27-1255-4c9c-88c4-c6e4eb58a77d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:14.107896 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:14.107607 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-istio-data" pod "ee995a27-1255-4c9c-88c4-c6e4eb58a77d" (UID: "ee995a27-1255-4c9c-88c4-c6e4eb58a77d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:14.108038 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:14.107937 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-istiod-ca-cert" pod "ee995a27-1255-4c9c-88c4-c6e4eb58a77d" (UID: "ee995a27-1255-4c9c-88c4-c6e4eb58a77d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:14:14.108230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:14.108205 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:14.108326 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:14.108235 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:14.108326 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:14.108248 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:14.108326 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:14.108267 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:14.108777 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:14.108736 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-workload-socket" pod "ee995a27-1255-4c9c-88c4-c6e4eb58a77d" (UID: "ee995a27-1255-4c9c-88c4-c6e4eb58a77d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:14.109716 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:14.109693 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-kube-api-access-7l4p9" pod "ee995a27-1255-4c9c-88c4-c6e4eb58a77d" (UID: "ee995a27-1255-4c9c-88c4-c6e4eb58a77d"). InnerVolumeSpecName "kube-api-access-7l4p9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:14.109997 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:14.109976 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-istio-podinfo" pod "ee995a27-1255-4c9c-88c4-c6e4eb58a77d" (UID: "ee995a27-1255-4c9c-88c4-c6e4eb58a77d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:14:14.110226 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:14.110201 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-istio-envoy" pod "ee995a27-1255-4c9c-88c4-c6e4eb58a77d" (UID: "ee995a27-1255-4c9c-88c4-c6e4eb58a77d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:14.111285 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:14.111259 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-istio-token" pod "ee995a27-1255-4c9c-88c4-c6e4eb58a77d" (UID: "ee995a27-1255-4c9c-88c4-c6e4eb58a77d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:14.209655 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:14.209618 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:14.209655 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:14.209649 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:14.209655 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:14.209658 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:14.209967 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:14.209668 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:14.209967 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:14.209678 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l4p9\" (UniqueName: \"kubernetes.io/projected/ee995a27-1255-4c9c-88c4-c6e4eb58a77d-kube-api-access-7l4p9\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:14.677859 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:14:14.677825 6572 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee995a27_1255_4c9c_88c4_c6e4eb58a77d.slice\": RecentStats: unable to find data in memory cache]" Oct 30 02:14:14.693526 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:14:14.693496 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:14:14.914235 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:14.913978 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" Oct 30 02:14:14.914652 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:14.914629 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst" event={"ID":"ee995a27-1255-4c9c-88c4-c6e4eb58a77d","Type":"ContainerDied","Data":"bf711748e1edb4ac0df162a153deffecfe286b5d63cd82f773ec5636fa717f36"} Oct 30 02:14:14.914792 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:14.914708 6572 scope.go:122] "RemoveContainer" containerID="62afb4d5e249e315edbcb3f4fff9df302ef2ea37a401bf2d1fea6e47542ec553" Oct 30 02:14:14.917131 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:14.917097 6572 generic.go:356] "Generic (PLEG): container finished" podID="fe01e89d-ffd9-40bc-8720-eee65f4a42eb" containerID="12996ff0bc936e2d0a017b435104cbe551e061b1ddb1c7b7b24d3ac1c0bd2e35" exitCode=0 Oct 30 02:14:14.917279 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:14.917168 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" event={"ID":"fe01e89d-ffd9-40bc-8720-eee65f4a42eb","Type":"ContainerDied","Data":"12996ff0bc936e2d0a017b435104cbe551e061b1ddb1c7b7b24d3ac1c0bd2e35"} Oct 30 02:14:15.001769 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:14.998799 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" Oct 30 02:14:15.024769 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:15.017153 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst"] Oct 30 02:14:15.027053 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:15.026714 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-zwjst"] Oct 30 02:14:15.128481 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:15.128449 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-workload-certs\") pod \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\" (UID: \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\") " Oct 30 02:14:15.128892 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:15.128609 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-istiod-ca-cert\") pod \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\" (UID: \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\") " Oct 30 02:14:15.128892 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:15.128697 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-credential-socket\") pod \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\" (UID: \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\") " Oct 30 02:14:15.128892 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:15.128734 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-kube-api-access-pzndw\" (UniqueName: \"kubernetes.io/projected/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-kube-api-access-pzndw\") pod \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\" (UID: \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\") " Oct 30 02:14:15.128892 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:15.128834 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-istio-podinfo\") pod \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\" (UID: \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\") " Oct 30 02:14:15.128892 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:15.128728 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-workload-certs" pod "fe01e89d-ffd9-40bc-8720-eee65f4a42eb" (UID: "fe01e89d-ffd9-40bc-8720-eee65f4a42eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:15.128892 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:15.128889 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-istio-data\") pod \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\" (UID: \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\") " Oct 30 02:14:15.129273 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:15.128933 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-istio-token\" (UniqueName: \"kubernetes.io/projected/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-istio-token\") pod \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\" (UID: \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\") " Oct 30 02:14:15.129273 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:15.129009 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-istio-envoy\") pod \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\" (UID: \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\") " Oct 30 02:14:15.129273 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:15.129025 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-credential-socket" pod "fe01e89d-ffd9-40bc-8720-eee65f4a42eb" (UID: "fe01e89d-ffd9-40bc-8720-eee65f4a42eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:15.129273 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:15.129038 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-workload-socket\") pod \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\" (UID: \"fe01e89d-ffd9-40bc-8720-eee65f4a42eb\") " Oct 30 02:14:15.129273 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:15.129094 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-istiod-ca-cert" pod "fe01e89d-ffd9-40bc-8720-eee65f4a42eb" (UID: "fe01e89d-ffd9-40bc-8720-eee65f4a42eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:14:15.129273 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:15.129176 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-istio-data" pod "fe01e89d-ffd9-40bc-8720-eee65f4a42eb" (UID: "fe01e89d-ffd9-40bc-8720-eee65f4a42eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:15.129550 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:15.129431 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-workload-socket" pod "fe01e89d-ffd9-40bc-8720-eee65f4a42eb" (UID: "fe01e89d-ffd9-40bc-8720-eee65f4a42eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:15.129712 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:15.129696 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:15.129798 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:15.129714 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:15.129798 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:15.129724 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:15.129798 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:15.129734 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:15.129798 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:15.129742 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:15.131483 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:15.131459 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-istio-token" pod "fe01e89d-ffd9-40bc-8720-eee65f4a42eb" (UID: "fe01e89d-ffd9-40bc-8720-eee65f4a42eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:15.131598 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:15.131487 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-kube-api-access-pzndw" pod "fe01e89d-ffd9-40bc-8720-eee65f4a42eb" (UID: "fe01e89d-ffd9-40bc-8720-eee65f4a42eb"). InnerVolumeSpecName "kube-api-access-pzndw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:15.131598 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:15.131560 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-istio-envoy" pod "fe01e89d-ffd9-40bc-8720-eee65f4a42eb" (UID: "fe01e89d-ffd9-40bc-8720-eee65f4a42eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:15.131695 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:15.131623 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-istio-podinfo" pod "fe01e89d-ffd9-40bc-8720-eee65f4a42eb" (UID: "fe01e89d-ffd9-40bc-8720-eee65f4a42eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:14:15.231501 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:15.231424 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:15.231501 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:15.231450 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:15.231501 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:15.231461 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:15.231501 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:15.231470 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzndw\" (UniqueName: \"kubernetes.io/projected/fe01e89d-ffd9-40bc-8720-eee65f4a42eb-kube-api-access-pzndw\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:15.378903 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:15.375632 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:15.928634 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:15.928606 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" Oct 30 02:14:15.928842 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:15.928625 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt" event={"ID":"fe01e89d-ffd9-40bc-8720-eee65f4a42eb","Type":"ContainerDied","Data":"0102f470f963359de5cb964ca9a04c4f66eccaacf528ad8010119deb35b1ecfa"} Oct 30 02:14:15.928842 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:15.928690 6572 scope.go:122] "RemoveContainer" containerID="12996ff0bc936e2d0a017b435104cbe551e061b1ddb1c7b7b24d3ac1c0bd2e35" Oct 30 02:14:15.994226 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:15.994195 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt"] Oct 30 02:14:16.005012 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:16.004980 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ldtmt"] Oct 30 02:14:16.545427 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:16.545393 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee995a27-1255-4c9c-88c4-c6e4eb58a77d" path="/var/lib/kubelet/pods/ee995a27-1255-4c9c-88c4-c6e4eb58a77d/volumes" Oct 30 02:14:16.546136 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:16.546116 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe01e89d-ffd9-40bc-8720-eee65f4a42eb" path="/var/lib/kubelet/pods/fe01e89d-ffd9-40bc-8720-eee65f4a42eb/volumes" Oct 30 02:14:16.889895 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:16.889805 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:18.378770 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:18.376298 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:19.815070 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:19.815035 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2eae39-7989c5cb95-k9lsj"] Oct 30 02:14:19.999609 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:19.999572 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:20.187249 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:20.187215 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:21.709913 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:21.709817 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:23.155148 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:23.155109 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:24.768192 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:24.768153 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:24.815435 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:24.815376 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2eae39-7989c5cb95-k9lsj" podUID="5387b465-e2a2-44dd-aa7e-42a8447a7639" containerName="payload-pre-processing" containerID="cri-o://4bb34ab80172d72178911912feabdedc1d842d78e58fe1568c10ea20adac234b" gracePeriod=25 Oct 30 02:14:25.026009 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:25.025930 6572 generic.go:356] "Generic (PLEG): container finished" podID="5387b465-e2a2-44dd-aa7e-42a8447a7639" containerID="4bb34ab80172d72178911912feabdedc1d842d78e58fe1568c10ea20adac234b" exitCode=0 Oct 30 02:14:25.026155 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:25.026002 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2eae39-7989c5cb95-k9lsj" event={"ID":"5387b465-e2a2-44dd-aa7e-42a8447a7639","Type":"ContainerDied","Data":"4bb34ab80172d72178911912feabdedc1d842d78e58fe1568c10ea20adac234b"} Oct 30 02:14:25.100320 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:25.100289 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2eae39-7989c5cb95-k9lsj" Oct 30 02:14:25.252676 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:25.252647 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5387b465-e2a2-44dd-aa7e-42a8447a7639-kube-api-access-p4ngj\" (UniqueName: \"kubernetes.io/projected/5387b465-e2a2-44dd-aa7e-42a8447a7639-kube-api-access-p4ngj\") pod \"5387b465-e2a2-44dd-aa7e-42a8447a7639\" (UID: \"5387b465-e2a2-44dd-aa7e-42a8447a7639\") " Oct 30 02:14:25.252936 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:25.252834 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5387b465-e2a2-44dd-aa7e-42a8447a7639-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5387b465-e2a2-44dd-aa7e-42a8447a7639-plugins-config-volume\") pod \"5387b465-e2a2-44dd-aa7e-42a8447a7639\" (UID: \"5387b465-e2a2-44dd-aa7e-42a8447a7639\") " Oct 30 02:14:25.253275 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:25.253247 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5387b465-e2a2-44dd-aa7e-42a8447a7639-plugins-config-volume" pod "5387b465-e2a2-44dd-aa7e-42a8447a7639" (UID: "5387b465-e2a2-44dd-aa7e-42a8447a7639"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:14:25.253563 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:25.253543 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5387b465-e2a2-44dd-aa7e-42a8447a7639-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:25.254983 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:25.254960 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5387b465-e2a2-44dd-aa7e-42a8447a7639-kube-api-access-p4ngj" pod "5387b465-e2a2-44dd-aa7e-42a8447a7639" (UID: "5387b465-e2a2-44dd-aa7e-42a8447a7639"). InnerVolumeSpecName "kube-api-access-p4ngj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:25.355582 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:25.355494 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4ngj\" (UniqueName: \"kubernetes.io/projected/5387b465-e2a2-44dd-aa7e-42a8447a7639-kube-api-access-p4ngj\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:26.037306 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:26.037278 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2eae39-7989c5cb95-k9lsj" Oct 30 02:14:26.037794 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:26.037320 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2eae39-7989c5cb95-k9lsj" event={"ID":"5387b465-e2a2-44dd-aa7e-42a8447a7639","Type":"ContainerDied","Data":"c46a04c80f945818c3fee74268d17bef98bb4102d90641d652cc97057b3ed359"} Oct 30 02:14:26.037794 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:26.037375 6572 scope.go:122] "RemoveContainer" containerID="4bb34ab80172d72178911912feabdedc1d842d78e58fe1568c10ea20adac234b" Oct 30 02:14:26.087765 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:26.087727 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2eae39-7989c5cb95-k9lsj"] Oct 30 02:14:26.098214 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:26.098191 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2eae39-7989c5cb95-k9lsj"] Oct 30 02:14:26.298306 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:26.298196 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:26.544955 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:26.544918 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5387b465-e2a2-44dd-aa7e-42a8447a7639" path="/var/lib/kubelet/pods/5387b465-e2a2-44dd-aa7e-42a8447a7639/volumes" Oct 30 02:14:27.648518 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:27.648477 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:28.307627 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:28.307482 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:28.413843 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:28.413810 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f6e11e-fg5tj"] Oct 30 02:14:28.423852 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:28.423826 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f6e11e-fg5tj"] Oct 30 02:14:28.492146 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:28.492104 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:28.591465 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:28.591379 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58622924-77ea-4ecb-b876-6d43d54997d1" path="/var/lib/kubelet/pods/58622924-77ea-4ecb-b876-6d43d54997d1/volumes" Oct 30 02:14:28.690941 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:28.690912 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:29.018606 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:29.018571 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:30.282975 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:30.282941 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:31.745645 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:31.745568 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:33.333775 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:33.330416 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:34.950547 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:34.950510 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:35.133370 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:35.133337 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn"] Oct 30 02:14:35.133598 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:35.133571 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" podUID="a341c867-33db-494c-b3e4-873415967cf0" containerName="istio-proxy" containerID="cri-o://d9440f31cdf646bc27031e114d9b9f724b896f8a03b03b44d33b71beff6a6797" gracePeriod=30 Oct 30 02:14:35.160879 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:35.160840 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:35.373830 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:35.373289 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:35.734791 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:35.730392 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-292018-fjhhq"] Oct 30 02:14:35.734791 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:35.733713 6572 cpu_manager.go:414] "containerMap: removing container" podUID="fe01e89d-ffd9-40bc-8720-eee65f4a42eb" containerName="istio-proxy" Oct 30 02:14:35.734791 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:35.733742 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe01e89d-ffd9-40bc-8720-eee65f4a42eb" containerName="istio-proxy" Oct 30 02:14:35.734791 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:35.733825 6572 cpu_manager.go:414] "containerMap: removing container" podUID="ee995a27-1255-4c9c-88c4-c6e4eb58a77d" containerName="istio-proxy" Oct 30 02:14:35.734791 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:35.733834 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee995a27-1255-4c9c-88c4-c6e4eb58a77d" containerName="istio-proxy" Oct 30 02:14:35.734791 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:35.733863 6572 cpu_manager.go:414] "containerMap: removing container" podUID="613abc91-29df-44d4-9bec-d37e815e9c86" containerName="revoke-keys" Oct 30 02:14:35.734791 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:35.733872 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="613abc91-29df-44d4-9bec-d37e815e9c86" containerName="revoke-keys" Oct 30 02:14:35.734791 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:35.733946 6572 cpu_manager.go:414] "containerMap: removing container" podUID="5387b465-e2a2-44dd-aa7e-42a8447a7639" containerName="payload-pre-processing" Oct 30 02:14:35.734791 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:35.733955 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5387b465-e2a2-44dd-aa7e-42a8447a7639" containerName="payload-pre-processing" Oct 30 02:14:35.734791 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:35.734139 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="5387b465-e2a2-44dd-aa7e-42a8447a7639" containerName="payload-pre-processing" Oct 30 02:14:35.734791 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:35.734157 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe01e89d-ffd9-40bc-8720-eee65f4a42eb" containerName="istio-proxy" Oct 30 02:14:35.734791 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:35.734169 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee995a27-1255-4c9c-88c4-c6e4eb58a77d" containerName="istio-proxy" Oct 30 02:14:35.734791 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:35.734181 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="613abc91-29df-44d4-9bec-d37e815e9c86" containerName="revoke-keys" Oct 30 02:14:35.740556 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:35.740527 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-292018-fjhhq" Oct 30 02:14:35.745241 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:35.745214 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-292018-fjhhq"] Oct 30 02:14:35.767998 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:35.767963 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:14:35.768199 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:35.767992 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:14:35.831183 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:35.831138 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:14:35.831383 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:35.831281 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:14:35.838128 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:35.838091 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wqrc\" (UniqueName: \"kubernetes.io/projected/a238ef55-f32f-42c5-9e67-a478bf8eeb8c-kube-api-access-6wqrc\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-292018-fjhhq\" (UID: \"a238ef55-f32f-42c5-9e67-a478bf8eeb8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-292018-fjhhq" Oct 30 02:14:35.838259 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:35.838159 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a238ef55-f32f-42c5-9e67-a478bf8eeb8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-292018-fjhhq\" (UID: \"a238ef55-f32f-42c5-9e67-a478bf8eeb8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-292018-fjhhq" Oct 30 02:14:35.838313 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:35.838277 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a238ef55-f32f-42c5-9e67-a478bf8eeb8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-292018-fjhhq\" (UID: \"a238ef55-f32f-42c5-9e67-a478bf8eeb8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-292018-fjhhq" Oct 30 02:14:35.939636 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:35.939595 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wqrc\" (UniqueName: \"kubernetes.io/projected/a238ef55-f32f-42c5-9e67-a478bf8eeb8c-kube-api-access-6wqrc\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-292018-fjhhq\" (UID: \"a238ef55-f32f-42c5-9e67-a478bf8eeb8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-292018-fjhhq" Oct 30 02:14:35.939854 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:35.939653 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a238ef55-f32f-42c5-9e67-a478bf8eeb8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-292018-fjhhq\" (UID: \"a238ef55-f32f-42c5-9e67-a478bf8eeb8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-292018-fjhhq" Oct 30 02:14:35.939854 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:35.939695 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a238ef55-f32f-42c5-9e67-a478bf8eeb8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-292018-fjhhq\" (UID: \"a238ef55-f32f-42c5-9e67-a478bf8eeb8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-292018-fjhhq" Oct 30 02:14:35.940486 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:35.940458 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a238ef55-f32f-42c5-9e67-a478bf8eeb8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-292018-fjhhq\" (UID: \"a238ef55-f32f-42c5-9e67-a478bf8eeb8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-292018-fjhhq" Oct 30 02:14:35.942094 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:35.942068 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a238ef55-f32f-42c5-9e67-a478bf8eeb8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-292018-fjhhq\" (UID: \"a238ef55-f32f-42c5-9e67-a478bf8eeb8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-292018-fjhhq" Oct 30 02:14:35.958242 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:35.958219 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wqrc\" (UniqueName: \"kubernetes.io/projected/a238ef55-f32f-42c5-9e67-a478bf8eeb8c-kube-api-access-6wqrc\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-292018-fjhhq\" (UID: \"a238ef55-f32f-42c5-9e67-a478bf8eeb8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-292018-fjhhq" Oct 30 02:14:36.060459 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:36.060361 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-292018-fjhhq" Oct 30 02:14:36.393267 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:14:36.393235 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda238ef55_f32f_42c5_9e67_a478bf8eeb8c.slice/crio-2af32a3bfe04e8fa5d2eecba675a876c06262eda4d2f0cb59dcbf7c6e916f8cd WatchSource:0}: Error finding container 2af32a3bfe04e8fa5d2eecba675a876c06262eda4d2f0cb59dcbf7c6e916f8cd: Status 404 returned error can't find the container with id 2af32a3bfe04e8fa5d2eecba675a876c06262eda4d2f0cb59dcbf7c6e916f8cd Oct 30 02:14:36.394960 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:36.394933 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-292018-fjhhq"] Oct 30 02:14:36.879721 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:36.879379 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:37.265525 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:37.265436 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-292018-fjhhq" event={"ID":"a238ef55-f32f-42c5-9e67-a478bf8eeb8c","Type":"ContainerStarted","Data":"64d073a7bb553ff2ad0b96082836e87142299b8974a9c3161ad4e03929bad220"} Oct 30 02:14:37.265525 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:37.265490 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-292018-fjhhq" event={"ID":"a238ef55-f32f-42c5-9e67-a478bf8eeb8c","Type":"ContainerStarted","Data":"2af32a3bfe04e8fa5d2eecba675a876c06262eda4d2f0cb59dcbf7c6e916f8cd"} Oct 30 02:14:37.325783 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:37.322628 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-292018-fjhhq" podStartSLOduration=2.322602329 podStartE2EDuration="2.322602329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:14:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:14:37.30263752 +0000 UTC m=+3402.879062922" watchObservedRunningTime="2026-10-30 02:14:37.322602329 +0000 UTC m=+3402.899027733" Oct 30 02:14:38.276259 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:38.276222 6572 generic.go:356] "Generic (PLEG): container finished" podID="a238ef55-f32f-42c5-9e67-a478bf8eeb8c" containerID="64d073a7bb553ff2ad0b96082836e87142299b8974a9c3161ad4e03929bad220" exitCode=0 Oct 30 02:14:38.276684 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:38.276260 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-292018-fjhhq" event={"ID":"a238ef55-f32f-42c5-9e67-a478bf8eeb8c","Type":"ContainerDied","Data":"64d073a7bb553ff2ad0b96082836e87142299b8974a9c3161ad4e03929bad220"} Oct 30 02:14:38.464427 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:38.464390 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:39.529772 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:39.528941 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-292018-fjhhq" Oct 30 02:14:39.700079 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:39.700036 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a238ef55-f32f-42c5-9e67-a478bf8eeb8c-kube-api-access-6wqrc\" (UniqueName: \"kubernetes.io/projected/a238ef55-f32f-42c5-9e67-a478bf8eeb8c-kube-api-access-6wqrc\") pod \"a238ef55-f32f-42c5-9e67-a478bf8eeb8c\" (UID: \"a238ef55-f32f-42c5-9e67-a478bf8eeb8c\") " Oct 30 02:14:39.700281 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:39.700093 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a238ef55-f32f-42c5-9e67-a478bf8eeb8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a238ef55-f32f-42c5-9e67-a478bf8eeb8c-maas-api-xks-ca\") pod \"a238ef55-f32f-42c5-9e67-a478bf8eeb8c\" (UID: \"a238ef55-f32f-42c5-9e67-a478bf8eeb8c\") " Oct 30 02:14:39.700281 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:39.700240 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a238ef55-f32f-42c5-9e67-a478bf8eeb8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a238ef55-f32f-42c5-9e67-a478bf8eeb8c-maas-api-openshift-ca\") pod \"a238ef55-f32f-42c5-9e67-a478bf8eeb8c\" (UID: \"a238ef55-f32f-42c5-9e67-a478bf8eeb8c\") " Oct 30 02:14:39.700680 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:39.700646 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a238ef55-f32f-42c5-9e67-a478bf8eeb8c-maas-api-openshift-ca" pod "a238ef55-f32f-42c5-9e67-a478bf8eeb8c" (UID: "a238ef55-f32f-42c5-9e67-a478bf8eeb8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:14:39.700972 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:39.700953 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a238ef55-f32f-42c5-9e67-a478bf8eeb8c-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:39.702405 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:39.702377 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a238ef55-f32f-42c5-9e67-a478bf8eeb8c-maas-api-xks-ca" pod "a238ef55-f32f-42c5-9e67-a478bf8eeb8c" (UID: "a238ef55-f32f-42c5-9e67-a478bf8eeb8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:14:39.702502 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:39.702422 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a238ef55-f32f-42c5-9e67-a478bf8eeb8c-kube-api-access-6wqrc" pod "a238ef55-f32f-42c5-9e67-a478bf8eeb8c" (UID: "a238ef55-f32f-42c5-9e67-a478bf8eeb8c"). InnerVolumeSpecName "kube-api-access-6wqrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:39.802258 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:39.802223 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wqrc\" (UniqueName: \"kubernetes.io/projected/a238ef55-f32f-42c5-9e67-a478bf8eeb8c-kube-api-access-6wqrc\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:39.802258 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:39.802253 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a238ef55-f32f-42c5-9e67-a478bf8eeb8c-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:40.089232 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:40.087693 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:40.298659 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:40.298618 6572 generic.go:356] "Generic (PLEG): container finished" podID="a341c867-33db-494c-b3e4-873415967cf0" containerID="d9440f31cdf646bc27031e114d9b9f724b896f8a03b03b44d33b71beff6a6797" exitCode=0 Oct 30 02:14:40.298877 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:40.298792 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" event={"ID":"a341c867-33db-494c-b3e4-873415967cf0","Type":"ContainerDied","Data":"d9440f31cdf646bc27031e114d9b9f724b896f8a03b03b44d33b71beff6a6797"} Oct 30 02:14:40.298877 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:40.298861 6572 scope.go:122] "RemoveContainer" containerID="ad35146aaa1dd21e8cd4038b8e3ad2d4a22358aa3ac01d1cc3f633cb654cf713" Oct 30 02:14:40.302240 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:40.302194 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-292018-fjhhq" event={"ID":"a238ef55-f32f-42c5-9e67-a478bf8eeb8c","Type":"ContainerDied","Data":"2af32a3bfe04e8fa5d2eecba675a876c06262eda4d2f0cb59dcbf7c6e916f8cd"} Oct 30 02:14:40.302357 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:40.302261 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2af32a3bfe04e8fa5d2eecba675a876c06262eda4d2f0cb59dcbf7c6e916f8cd" Oct 30 02:14:40.302357 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:40.302345 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-292018-fjhhq" Oct 30 02:14:40.499045 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:40.499014 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 02:14:40.553225 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:40.553192 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a341c867-33db-494c-b3e4-873415967cf0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a341c867-33db-494c-b3e4-873415967cf0-istio-data\") pod \"a341c867-33db-494c-b3e4-873415967cf0\" (UID: \"a341c867-33db-494c-b3e4-873415967cf0\") " Oct 30 02:14:40.553225 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:40.553230 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a341c867-33db-494c-b3e4-873415967cf0-kube-api-access-5g6tf\" (UniqueName: \"kubernetes.io/projected/a341c867-33db-494c-b3e4-873415967cf0-kube-api-access-5g6tf\") pod \"a341c867-33db-494c-b3e4-873415967cf0\" (UID: \"a341c867-33db-494c-b3e4-873415967cf0\") " Oct 30 02:14:40.553763 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:40.553258 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a341c867-33db-494c-b3e4-873415967cf0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a341c867-33db-494c-b3e4-873415967cf0-istiod-ca-cert\") pod \"a341c867-33db-494c-b3e4-873415967cf0\" (UID: \"a341c867-33db-494c-b3e4-873415967cf0\") " Oct 30 02:14:40.553763 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:40.553338 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a341c867-33db-494c-b3e4-873415967cf0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a341c867-33db-494c-b3e4-873415967cf0-workload-socket\") pod \"a341c867-33db-494c-b3e4-873415967cf0\" (UID: \"a341c867-33db-494c-b3e4-873415967cf0\") " Oct 30 02:14:40.553763 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:40.553386 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a341c867-33db-494c-b3e4-873415967cf0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a341c867-33db-494c-b3e4-873415967cf0-istio-envoy\") pod \"a341c867-33db-494c-b3e4-873415967cf0\" (UID: \"a341c867-33db-494c-b3e4-873415967cf0\") " Oct 30 02:14:40.553763 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:40.553418 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a341c867-33db-494c-b3e4-873415967cf0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a341c867-33db-494c-b3e4-873415967cf0-istio-podinfo\") pod \"a341c867-33db-494c-b3e4-873415967cf0\" (UID: \"a341c867-33db-494c-b3e4-873415967cf0\") " Oct 30 02:14:40.553763 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:40.553455 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a341c867-33db-494c-b3e4-873415967cf0-istio-data" pod "a341c867-33db-494c-b3e4-873415967cf0" (UID: "a341c867-33db-494c-b3e4-873415967cf0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:40.553763 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:40.553483 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a341c867-33db-494c-b3e4-873415967cf0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a341c867-33db-494c-b3e4-873415967cf0-credential-socket\") pod \"a341c867-33db-494c-b3e4-873415967cf0\" (UID: \"a341c867-33db-494c-b3e4-873415967cf0\") " Oct 30 02:14:40.553763 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:40.553546 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a341c867-33db-494c-b3e4-873415967cf0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a341c867-33db-494c-b3e4-873415967cf0-workload-certs\") pod \"a341c867-33db-494c-b3e4-873415967cf0\" (UID: \"a341c867-33db-494c-b3e4-873415967cf0\") " Oct 30 02:14:40.553763 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:40.553659 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a341c867-33db-494c-b3e4-873415967cf0-istio-token\" (UniqueName: \"kubernetes.io/projected/a341c867-33db-494c-b3e4-873415967cf0-istio-token\") pod \"a341c867-33db-494c-b3e4-873415967cf0\" (UID: \"a341c867-33db-494c-b3e4-873415967cf0\") " Oct 30 02:14:40.553763 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:40.553715 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a341c867-33db-494c-b3e4-873415967cf0-istiod-ca-cert" pod "a341c867-33db-494c-b3e4-873415967cf0" (UID: "a341c867-33db-494c-b3e4-873415967cf0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:14:40.553763 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:40.553718 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a341c867-33db-494c-b3e4-873415967cf0-workload-socket" pod "a341c867-33db-494c-b3e4-873415967cf0" (UID: "a341c867-33db-494c-b3e4-873415967cf0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:40.554318 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:40.553882 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a341c867-33db-494c-b3e4-873415967cf0-workload-certs" pod "a341c867-33db-494c-b3e4-873415967cf0" (UID: "a341c867-33db-494c-b3e4-873415967cf0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:40.554318 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:40.554263 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a341c867-33db-494c-b3e4-873415967cf0-credential-socket" pod "a341c867-33db-494c-b3e4-873415967cf0" (UID: "a341c867-33db-494c-b3e4-873415967cf0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:40.555012 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:40.554984 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a341c867-33db-494c-b3e4-873415967cf0-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:40.555150 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:40.555015 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a341c867-33db-494c-b3e4-873415967cf0-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:40.555150 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:40.555032 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a341c867-33db-494c-b3e4-873415967cf0-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:40.555150 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:40.555047 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a341c867-33db-494c-b3e4-873415967cf0-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:40.555150 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:40.555064 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a341c867-33db-494c-b3e4-873415967cf0-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:40.556112 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:40.556087 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a341c867-33db-494c-b3e4-873415967cf0-istio-token" pod "a341c867-33db-494c-b3e4-873415967cf0" (UID: "a341c867-33db-494c-b3e4-873415967cf0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:40.558202 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:40.556174 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a341c867-33db-494c-b3e4-873415967cf0-istio-envoy" pod "a341c867-33db-494c-b3e4-873415967cf0" (UID: "a341c867-33db-494c-b3e4-873415967cf0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:40.558202 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:40.556385 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a341c867-33db-494c-b3e4-873415967cf0-kube-api-access-5g6tf" pod "a341c867-33db-494c-b3e4-873415967cf0" (UID: "a341c867-33db-494c-b3e4-873415967cf0"). InnerVolumeSpecName "kube-api-access-5g6tf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:40.558202 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:40.557845 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a341c867-33db-494c-b3e4-873415967cf0-istio-podinfo" pod "a341c867-33db-494c-b3e4-873415967cf0" (UID: "a341c867-33db-494c-b3e4-873415967cf0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:14:40.656192 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:40.656102 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a341c867-33db-494c-b3e4-873415967cf0-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:40.656192 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:40.656134 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a341c867-33db-494c-b3e4-873415967cf0-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:40.656192 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:40.656144 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a341c867-33db-494c-b3e4-873415967cf0-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:40.656192 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:40.656153 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g6tf\" (UniqueName: \"kubernetes.io/projected/a341c867-33db-494c-b3e4-873415967cf0-kube-api-access-5g6tf\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:41.312995 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:41.312964 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" Oct 30 02:14:41.313165 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:41.312995 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn" event={"ID":"a341c867-33db-494c-b3e4-873415967cf0","Type":"ContainerDied","Data":"c17d7848b23d31220c9671a5319059ae031dc544e838eacf82cac5a9be2c165a"} Oct 30 02:14:41.313165 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:41.313060 6572 scope.go:122] "RemoveContainer" containerID="d9440f31cdf646bc27031e114d9b9f724b896f8a03b03b44d33b71beff6a6797" Oct 30 02:14:41.378596 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:41.378559 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn"] Oct 30 02:14:41.392576 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:41.392492 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-f6e11e-openshift-default-c89fd7dd5-lrjfn"] Oct 30 02:14:41.695610 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:41.679684 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:42.547912 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:42.547870 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a341c867-33db-494c-b3e4-873415967cf0" path="/var/lib/kubelet/pods/a341c867-33db-494c-b3e4-873415967cf0/volumes" Oct 30 02:14:43.188985 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:43.188942 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:44.691734 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:14:44.691692 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:14:44.883488 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:44.883447 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:46.139654 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:46.139609 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-292018-6fc6bf57c6-nslxv"] Oct 30 02:14:46.402594 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:46.169804 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-292018-f884cbb67-mvk86"] Oct 30 02:14:46.402594 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:46.355004 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:46.552679 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:46.552603 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:46.751348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:46.751278 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:47.425251 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:47.425218 6572 generic.go:356] "Generic (PLEG): container finished" podID="0e662617-4bbb-4021-8bae-ba979960bb94" containerID="275743f8bf29b2b7143c534cfc570df9a9e1ca985baed8835a56ac070c838cb9" exitCode=0 Oct 30 02:14:47.425816 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:47.425297 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" event={"ID":"0e662617-4bbb-4021-8bae-ba979960bb94","Type":"ContainerDied","Data":"275743f8bf29b2b7143c534cfc570df9a9e1ca985baed8835a56ac070c838cb9"} Oct 30 02:14:47.425922 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:47.425903 6572 scope.go:122] "RemoveContainer" containerID="275743f8bf29b2b7143c534cfc570df9a9e1ca985baed8835a56ac070c838cb9" Oct 30 02:14:47.427853 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:47.427826 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:14:47.427937 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:47.427900 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:14:47.427978 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:47.427935 6572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 02:14:48.083495 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:48.083461 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:48.442536 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:48.442499 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" event={"ID":"0e662617-4bbb-4021-8bae-ba979960bb94","Type":"ContainerStarted","Data":"487304269ac77b713e363b4327e1c9b8dc649bba358bcb82b512c4e79430b6c5"} Oct 30 02:14:49.426774 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:49.423117 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:14:49.430991 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:49.430943 6572 patch_prober.go:29] interesting pod/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.113:15021/healthz/ready\": dial tcp 10.133.1.113:15021: connect: connection refused" start-of-body= Oct 30 02:14:49.431186 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:49.431041 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" podUID="0e662617-4bbb-4021-8bae-ba979960bb94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.113:15021/healthz/ready\": dial tcp 10.133.1.113:15021: connect: connection refused" Oct 30 02:14:49.717824 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:49.716786 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:50.426773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:50.423247 6572 patch_prober.go:29] interesting pod/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.113:15021/healthz/ready\": dial tcp 10.133.1.113:15021: connect: connection refused" start-of-body= Oct 30 02:14:50.426773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:50.423319 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" podUID="0e662617-4bbb-4021-8bae-ba979960bb94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.113:15021/healthz/ready\": dial tcp 10.133.1.113:15021: connect: connection refused" Oct 30 02:14:51.140240 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:51.140177 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-292018-6fc6bf57c6-nslxv" podUID="cd10ed97-4e22-4efe-a0c6-7cff1ca2f157" containerName="payload-processing" containerID="cri-o://2afe31e5495d6d4ce4a3f9834325d50ea2852c2fcc83681af8d02959cb59482a" gracePeriod=25 Oct 30 02:14:51.170435 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:51.170368 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-292018-f884cbb67-mvk86" podUID="d0a98e88-a93c-462b-b2f5-68c61b386789" containerName="payload-pre-processing" containerID="cri-o://4625e5617338c40526823aca86dbe2fb3e8217e4e4c1c41f1bb25b44c70ac2a2" gracePeriod=25 Oct 30 02:14:51.314216 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:51.312302 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:51.425086 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:51.424464 6572 patch_prober.go:29] interesting pod/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.113:15021/healthz/ready\": dial tcp 10.133.1.113:15021: connect: connection refused" start-of-body= Oct 30 02:14:51.425086 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:51.424541 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" podUID="0e662617-4bbb-4021-8bae-ba979960bb94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.113:15021/healthz/ready\": dial tcp 10.133.1.113:15021: connect: connection refused" Oct 30 02:14:52.428773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:52.421718 6572 patch_prober.go:29] interesting pod/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.113:15021/healthz/ready\": dial tcp 10.133.1.113:15021: connect: connection refused" start-of-body= Oct 30 02:14:52.428773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:52.421812 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" podUID="0e662617-4bbb-4021-8bae-ba979960bb94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.113:15021/healthz/ready\": dial tcp 10.133.1.113:15021: connect: connection refused" Oct 30 02:14:52.465965 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:52.465928 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2"] Oct 30 02:14:52.466163 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:52.466147 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" podUID="5f3448e6-c22d-466c-a485-fbe2e676c250" containerName="istio-proxy" containerID="cri-o://0635c8e2fdad88fff1b57a81a5208e77d93cfa9be156bb2af249114f0913534e" gracePeriod=30 Oct 30 02:14:52.478295 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:52.478264 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct"] Oct 30 02:14:52.478518 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:52.478493 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" podUID="dcda177c-9d37-4f05-ac01-d99f9139f87e" containerName="istio-proxy" containerID="cri-o://9ebd16028c44463f7b7e12509b4614b5d1d14125135eb71a7f73a677aa136a73" gracePeriod=30 Oct 30 02:14:52.483282 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:52.483252 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw"] Oct 30 02:14:52.483494 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:52.483466 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" podUID="230f0407-5c25-44aa-9a97-1de91305e4f3" containerName="istio-proxy" containerID="cri-o://c2d23000006f5624e239b5fb1e018bd43cb1905d4614a9bd0f09a4ba5f9da268" gracePeriod=30 Oct 30 02:14:52.489365 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:52.489341 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6"] Oct 30 02:14:52.489569 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:52.489548 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" podUID="e1f0afe7-d54e-4577-857a-da30b84237de" containerName="istio-proxy" containerID="cri-o://9dd72b963ccf9883ba203df86886469c88ff25bdd02ee0930d07e87428914ba1" gracePeriod=30 Oct 30 02:14:52.543777 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:52.542305 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-tznct container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:14:52.543777 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:52.542356 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" podUID="dcda177c-9d37-4f05-ac01-d99f9139f87e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:14:52.543777 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:52.542410 6572 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-zsgv6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:14:52.543777 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:52.542427 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" podUID="e1f0afe7-d54e-4577-857a-da30b84237de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:14:52.890823 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:52.889726 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:53.430778 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:53.427247 6572 patch_prober.go:29] interesting pod/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.113:15021/healthz/ready\": dial tcp 10.133.1.113:15021: connect: connection refused" start-of-body= Oct 30 02:14:53.430778 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:53.427318 6572 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" podUID="0e662617-4bbb-4021-8bae-ba979960bb94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.113:15021/healthz/ready\": dial tcp 10.133.1.113:15021: connect: connection refused" Oct 30 02:14:53.865152 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:53.865050 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:14:54.403038 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:54.402995 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:54.412388 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:54.412343 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-292018-6fc6bf57c6-nslxv" podUID="cd10ed97-4e22-4efe-a0c6-7cff1ca2f157" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:14:54.412643 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:54.412565 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-292018-f884cbb67-mvk86" podUID="d0a98e88-a93c-462b-b2f5-68c61b386789" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:14:54.426153 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:54.426121 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:14:54.506243 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:54.506210 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:14:55.938841 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:55.936653 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:57.203840 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:57.203802 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:57.825256 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:57.825122 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:57.863662 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:57.863492 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2eae39-6n5sl"] Oct 30 02:14:57.872445 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:57.872411 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2eae39-6n5sl"] Oct 30 02:14:57.989847 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:57.961362 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" Oct 30 02:14:57.989847 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:57.986953 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" Oct 30 02:14:58.030450 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.030407 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:58.093845 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.093806 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f3448e6-c22d-466c-a485-fbe2e676c250-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f3448e6-c22d-466c-a485-fbe2e676c250-workload-certs\") pod \"5f3448e6-c22d-466c-a485-fbe2e676c250\" (UID: \"5f3448e6-c22d-466c-a485-fbe2e676c250\") " Oct 30 02:14:58.094132 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.093880 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/230f0407-5c25-44aa-9a97-1de91305e4f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/230f0407-5c25-44aa-9a97-1de91305e4f3-istio-data\") pod \"230f0407-5c25-44aa-9a97-1de91305e4f3\" (UID: \"230f0407-5c25-44aa-9a97-1de91305e4f3\") " Oct 30 02:14:58.094132 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.093927 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f3448e6-c22d-466c-a485-fbe2e676c250-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f3448e6-c22d-466c-a485-fbe2e676c250-istio-podinfo\") pod \"5f3448e6-c22d-466c-a485-fbe2e676c250\" (UID: \"5f3448e6-c22d-466c-a485-fbe2e676c250\") " Oct 30 02:14:58.094132 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.093990 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/230f0407-5c25-44aa-9a97-1de91305e4f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/230f0407-5c25-44aa-9a97-1de91305e4f3-workload-certs\") pod \"230f0407-5c25-44aa-9a97-1de91305e4f3\" (UID: \"230f0407-5c25-44aa-9a97-1de91305e4f3\") " Oct 30 02:14:58.094132 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.094075 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/230f0407-5c25-44aa-9a97-1de91305e4f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/230f0407-5c25-44aa-9a97-1de91305e4f3-istio-podinfo\") pod \"230f0407-5c25-44aa-9a97-1de91305e4f3\" (UID: \"230f0407-5c25-44aa-9a97-1de91305e4f3\") " Oct 30 02:14:58.094422 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.094145 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f3448e6-c22d-466c-a485-fbe2e676c250-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f3448e6-c22d-466c-a485-fbe2e676c250-credential-socket\") pod \"5f3448e6-c22d-466c-a485-fbe2e676c250\" (UID: \"5f3448e6-c22d-466c-a485-fbe2e676c250\") " Oct 30 02:14:58.094422 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.094185 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f3448e6-c22d-466c-a485-fbe2e676c250-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f3448e6-c22d-466c-a485-fbe2e676c250-istio-data\") pod \"5f3448e6-c22d-466c-a485-fbe2e676c250\" (UID: \"5f3448e6-c22d-466c-a485-fbe2e676c250\") " Oct 30 02:14:58.094422 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.094251 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f3448e6-c22d-466c-a485-fbe2e676c250-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f3448e6-c22d-466c-a485-fbe2e676c250-workload-socket\") pod \"5f3448e6-c22d-466c-a485-fbe2e676c250\" (UID: \"5f3448e6-c22d-466c-a485-fbe2e676c250\") " Oct 30 02:14:58.094422 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.094338 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/230f0407-5c25-44aa-9a97-1de91305e4f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/230f0407-5c25-44aa-9a97-1de91305e4f3-credential-socket\") pod \"230f0407-5c25-44aa-9a97-1de91305e4f3\" (UID: \"230f0407-5c25-44aa-9a97-1de91305e4f3\") " Oct 30 02:14:58.094422 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.094389 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/230f0407-5c25-44aa-9a97-1de91305e4f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/230f0407-5c25-44aa-9a97-1de91305e4f3-workload-socket\") pod \"230f0407-5c25-44aa-9a97-1de91305e4f3\" (UID: \"230f0407-5c25-44aa-9a97-1de91305e4f3\") " Oct 30 02:14:58.094703 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.094419 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/230f0407-5c25-44aa-9a97-1de91305e4f3-kube-api-access-w49f8\" (UniqueName: \"kubernetes.io/projected/230f0407-5c25-44aa-9a97-1de91305e4f3-kube-api-access-w49f8\") pod \"230f0407-5c25-44aa-9a97-1de91305e4f3\" (UID: \"230f0407-5c25-44aa-9a97-1de91305e4f3\") " Oct 30 02:14:58.094703 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.094474 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/230f0407-5c25-44aa-9a97-1de91305e4f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/230f0407-5c25-44aa-9a97-1de91305e4f3-istiod-ca-cert\") pod \"230f0407-5c25-44aa-9a97-1de91305e4f3\" (UID: \"230f0407-5c25-44aa-9a97-1de91305e4f3\") " Oct 30 02:14:58.094703 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.094544 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/230f0407-5c25-44aa-9a97-1de91305e4f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/230f0407-5c25-44aa-9a97-1de91305e4f3-istio-envoy\") pod \"230f0407-5c25-44aa-9a97-1de91305e4f3\" (UID: \"230f0407-5c25-44aa-9a97-1de91305e4f3\") " Oct 30 02:14:58.094703 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.094594 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f3448e6-c22d-466c-a485-fbe2e676c250-istio-token\" (UniqueName: \"kubernetes.io/projected/5f3448e6-c22d-466c-a485-fbe2e676c250-istio-token\") pod \"5f3448e6-c22d-466c-a485-fbe2e676c250\" (UID: \"5f3448e6-c22d-466c-a485-fbe2e676c250\") " Oct 30 02:14:58.094703 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.094644 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f3448e6-c22d-466c-a485-fbe2e676c250-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f3448e6-c22d-466c-a485-fbe2e676c250-istio-envoy\") pod \"5f3448e6-c22d-466c-a485-fbe2e676c250\" (UID: \"5f3448e6-c22d-466c-a485-fbe2e676c250\") " Oct 30 02:14:58.095129 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.094732 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f3448e6-c22d-466c-a485-fbe2e676c250-kube-api-access-mvbbv\" (UniqueName: \"kubernetes.io/projected/5f3448e6-c22d-466c-a485-fbe2e676c250-kube-api-access-mvbbv\") pod \"5f3448e6-c22d-466c-a485-fbe2e676c250\" (UID: \"5f3448e6-c22d-466c-a485-fbe2e676c250\") " Oct 30 02:14:58.095129 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.094791 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/230f0407-5c25-44aa-9a97-1de91305e4f3-istio-token\" (UniqueName: \"kubernetes.io/projected/230f0407-5c25-44aa-9a97-1de91305e4f3-istio-token\") pod \"230f0407-5c25-44aa-9a97-1de91305e4f3\" (UID: \"230f0407-5c25-44aa-9a97-1de91305e4f3\") " Oct 30 02:14:58.095129 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.094879 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f3448e6-c22d-466c-a485-fbe2e676c250-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f3448e6-c22d-466c-a485-fbe2e676c250-istiod-ca-cert\") pod \"5f3448e6-c22d-466c-a485-fbe2e676c250\" (UID: \"5f3448e6-c22d-466c-a485-fbe2e676c250\") " Oct 30 02:14:58.097953 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.097200 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230f0407-5c25-44aa-9a97-1de91305e4f3-credential-socket" pod "230f0407-5c25-44aa-9a97-1de91305e4f3" (UID: "230f0407-5c25-44aa-9a97-1de91305e4f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:58.097953 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.097439 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f3448e6-c22d-466c-a485-fbe2e676c250-workload-certs" pod "5f3448e6-c22d-466c-a485-fbe2e676c250" (UID: "5f3448e6-c22d-466c-a485-fbe2e676c250"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:58.097953 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.097604 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230f0407-5c25-44aa-9a97-1de91305e4f3-istio-data" pod "230f0407-5c25-44aa-9a97-1de91305e4f3" (UID: "230f0407-5c25-44aa-9a97-1de91305e4f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:58.099537 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.099006 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f3448e6-c22d-466c-a485-fbe2e676c250-credential-socket" pod "5f3448e6-c22d-466c-a485-fbe2e676c250" (UID: "5f3448e6-c22d-466c-a485-fbe2e676c250"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:58.099537 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.099209 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230f0407-5c25-44aa-9a97-1de91305e4f3-workload-certs" pod "230f0407-5c25-44aa-9a97-1de91305e4f3" (UID: "230f0407-5c25-44aa-9a97-1de91305e4f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:58.099537 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.099288 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f3448e6-c22d-466c-a485-fbe2e676c250-istiod-ca-cert" pod "5f3448e6-c22d-466c-a485-fbe2e676c250" (UID: "5f3448e6-c22d-466c-a485-fbe2e676c250"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:14:58.103985 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.103945 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230f0407-5c25-44aa-9a97-1de91305e4f3-istio-envoy" pod "230f0407-5c25-44aa-9a97-1de91305e4f3" (UID: "230f0407-5c25-44aa-9a97-1de91305e4f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:58.104234 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.104209 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230f0407-5c25-44aa-9a97-1de91305e4f3-workload-socket" pod "230f0407-5c25-44aa-9a97-1de91305e4f3" (UID: "230f0407-5c25-44aa-9a97-1de91305e4f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:58.108180 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.107827 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f3448e6-c22d-466c-a485-fbe2e676c250-istio-data" pod "5f3448e6-c22d-466c-a485-fbe2e676c250" (UID: "5f3448e6-c22d-466c-a485-fbe2e676c250"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:58.108180 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.108075 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f3448e6-c22d-466c-a485-fbe2e676c250-workload-socket" pod "5f3448e6-c22d-466c-a485-fbe2e676c250" (UID: "5f3448e6-c22d-466c-a485-fbe2e676c250"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:58.108782 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.108739 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/230f0407-5c25-44aa-9a97-1de91305e4f3-istiod-ca-cert" pod "230f0407-5c25-44aa-9a97-1de91305e4f3" (UID: "230f0407-5c25-44aa-9a97-1de91305e4f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:14:58.112026 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.111992 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f3448e6-c22d-466c-a485-fbe2e676c250-kube-api-access-mvbbv" pod "5f3448e6-c22d-466c-a485-fbe2e676c250" (UID: "5f3448e6-c22d-466c-a485-fbe2e676c250"). InnerVolumeSpecName "kube-api-access-mvbbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:58.113048 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.112504 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230f0407-5c25-44aa-9a97-1de91305e4f3-istio-token" pod "230f0407-5c25-44aa-9a97-1de91305e4f3" (UID: "230f0407-5c25-44aa-9a97-1de91305e4f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:58.114401 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.114330 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f3448e6-c22d-466c-a485-fbe2e676c250-istio-envoy" pod "5f3448e6-c22d-466c-a485-fbe2e676c250" (UID: "5f3448e6-c22d-466c-a485-fbe2e676c250"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:58.118867 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.116263 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" Oct 30 02:14:58.122619 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.122576 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/230f0407-5c25-44aa-9a97-1de91305e4f3-istio-podinfo" pod "230f0407-5c25-44aa-9a97-1de91305e4f3" (UID: "230f0407-5c25-44aa-9a97-1de91305e4f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:14:58.122619 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.122604 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f3448e6-c22d-466c-a485-fbe2e676c250-istio-podinfo" pod "5f3448e6-c22d-466c-a485-fbe2e676c250" (UID: "5f3448e6-c22d-466c-a485-fbe2e676c250"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:14:58.122826 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.122631 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f3448e6-c22d-466c-a485-fbe2e676c250-istio-token" pod "5f3448e6-c22d-466c-a485-fbe2e676c250" (UID: "5f3448e6-c22d-466c-a485-fbe2e676c250"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:58.123870 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.123844 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" Oct 30 02:14:58.125073 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.125048 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230f0407-5c25-44aa-9a97-1de91305e4f3-kube-api-access-w49f8" pod "230f0407-5c25-44aa-9a97-1de91305e4f3" (UID: "230f0407-5c25-44aa-9a97-1de91305e4f3"). InnerVolumeSpecName "kube-api-access-w49f8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:58.197512 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.197470 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1f0afe7-d54e-4577-857a-da30b84237de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1f0afe7-d54e-4577-857a-da30b84237de-istiod-ca-cert\") pod \"e1f0afe7-d54e-4577-857a-da30b84237de\" (UID: \"e1f0afe7-d54e-4577-857a-da30b84237de\") " Oct 30 02:14:58.197710 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.197543 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcda177c-9d37-4f05-ac01-d99f9139f87e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcda177c-9d37-4f05-ac01-d99f9139f87e-istio-data\") pod \"dcda177c-9d37-4f05-ac01-d99f9139f87e\" (UID: \"dcda177c-9d37-4f05-ac01-d99f9139f87e\") " Oct 30 02:14:58.197710 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.197579 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1f0afe7-d54e-4577-857a-da30b84237de-istio-token\" (UniqueName: \"kubernetes.io/projected/e1f0afe7-d54e-4577-857a-da30b84237de-istio-token\") pod \"e1f0afe7-d54e-4577-857a-da30b84237de\" (UID: \"e1f0afe7-d54e-4577-857a-da30b84237de\") " Oct 30 02:14:58.197710 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.197679 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcda177c-9d37-4f05-ac01-d99f9139f87e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcda177c-9d37-4f05-ac01-d99f9139f87e-credential-socket\") pod \"dcda177c-9d37-4f05-ac01-d99f9139f87e\" (UID: \"dcda177c-9d37-4f05-ac01-d99f9139f87e\") " Oct 30 02:14:58.197877 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.197715 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcda177c-9d37-4f05-ac01-d99f9139f87e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcda177c-9d37-4f05-ac01-d99f9139f87e-istio-envoy\") pod \"dcda177c-9d37-4f05-ac01-d99f9139f87e\" (UID: \"dcda177c-9d37-4f05-ac01-d99f9139f87e\") " Oct 30 02:14:58.197877 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.197784 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1f0afe7-d54e-4577-857a-da30b84237de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1f0afe7-d54e-4577-857a-da30b84237de-istio-podinfo\") pod \"e1f0afe7-d54e-4577-857a-da30b84237de\" (UID: \"e1f0afe7-d54e-4577-857a-da30b84237de\") " Oct 30 02:14:58.197955 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.197871 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1f0afe7-d54e-4577-857a-da30b84237de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1f0afe7-d54e-4577-857a-da30b84237de-istio-data\") pod \"e1f0afe7-d54e-4577-857a-da30b84237de\" (UID: \"e1f0afe7-d54e-4577-857a-da30b84237de\") " Oct 30 02:14:58.197955 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.197907 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcda177c-9d37-4f05-ac01-d99f9139f87e-istio-data" pod "dcda177c-9d37-4f05-ac01-d99f9139f87e" (UID: "dcda177c-9d37-4f05-ac01-d99f9139f87e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:58.197955 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.197921 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1f0afe7-d54e-4577-857a-da30b84237de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f0afe7-d54e-4577-857a-da30b84237de-credential-socket\") pod \"e1f0afe7-d54e-4577-857a-da30b84237de\" (UID: \"e1f0afe7-d54e-4577-857a-da30b84237de\") " Oct 30 02:14:58.198171 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.197959 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1f0afe7-d54e-4577-857a-da30b84237de-kube-api-access-gnptz\" (UniqueName: \"kubernetes.io/projected/e1f0afe7-d54e-4577-857a-da30b84237de-kube-api-access-gnptz\") pod \"e1f0afe7-d54e-4577-857a-da30b84237de\" (UID: \"e1f0afe7-d54e-4577-857a-da30b84237de\") " Oct 30 02:14:58.198171 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.198000 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1f0afe7-d54e-4577-857a-da30b84237de-istiod-ca-cert" pod "e1f0afe7-d54e-4577-857a-da30b84237de" (UID: "e1f0afe7-d54e-4577-857a-da30b84237de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:14:58.198171 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.198029 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1f0afe7-d54e-4577-857a-da30b84237de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1f0afe7-d54e-4577-857a-da30b84237de-workload-certs\") pod \"e1f0afe7-d54e-4577-857a-da30b84237de\" (UID: \"e1f0afe7-d54e-4577-857a-da30b84237de\") " Oct 30 02:14:58.198171 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.198101 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1f0afe7-d54e-4577-857a-da30b84237de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1f0afe7-d54e-4577-857a-da30b84237de-istio-envoy\") pod \"e1f0afe7-d54e-4577-857a-da30b84237de\" (UID: \"e1f0afe7-d54e-4577-857a-da30b84237de\") " Oct 30 02:14:58.198410 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.198207 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcda177c-9d37-4f05-ac01-d99f9139f87e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcda177c-9d37-4f05-ac01-d99f9139f87e-workload-certs\") pod \"dcda177c-9d37-4f05-ac01-d99f9139f87e\" (UID: \"dcda177c-9d37-4f05-ac01-d99f9139f87e\") " Oct 30 02:14:58.198410 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.198226 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f0afe7-d54e-4577-857a-da30b84237de-istio-data" pod "e1f0afe7-d54e-4577-857a-da30b84237de" (UID: "e1f0afe7-d54e-4577-857a-da30b84237de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:58.198410 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.198268 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcda177c-9d37-4f05-ac01-d99f9139f87e-istio-token\" (UniqueName: \"kubernetes.io/projected/dcda177c-9d37-4f05-ac01-d99f9139f87e-istio-token\") pod \"dcda177c-9d37-4f05-ac01-d99f9139f87e\" (UID: \"dcda177c-9d37-4f05-ac01-d99f9139f87e\") " Oct 30 02:14:58.198410 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.198301 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f0afe7-d54e-4577-857a-da30b84237de-credential-socket" pod "e1f0afe7-d54e-4577-857a-da30b84237de" (UID: "e1f0afe7-d54e-4577-857a-da30b84237de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:58.198410 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.198331 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1f0afe7-d54e-4577-857a-da30b84237de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f0afe7-d54e-4577-857a-da30b84237de-workload-socket\") pod \"e1f0afe7-d54e-4577-857a-da30b84237de\" (UID: \"e1f0afe7-d54e-4577-857a-da30b84237de\") " Oct 30 02:14:58.198410 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.198373 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcda177c-9d37-4f05-ac01-d99f9139f87e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcda177c-9d37-4f05-ac01-d99f9139f87e-istio-podinfo\") pod \"dcda177c-9d37-4f05-ac01-d99f9139f87e\" (UID: \"dcda177c-9d37-4f05-ac01-d99f9139f87e\") " Oct 30 02:14:58.198760 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.198416 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcda177c-9d37-4f05-ac01-d99f9139f87e-kube-api-access-9wg7h\" (UniqueName: \"kubernetes.io/projected/dcda177c-9d37-4f05-ac01-d99f9139f87e-kube-api-access-9wg7h\") pod \"dcda177c-9d37-4f05-ac01-d99f9139f87e\" (UID: \"dcda177c-9d37-4f05-ac01-d99f9139f87e\") " Oct 30 02:14:58.198760 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.198455 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcda177c-9d37-4f05-ac01-d99f9139f87e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcda177c-9d37-4f05-ac01-d99f9139f87e-istiod-ca-cert\") pod \"dcda177c-9d37-4f05-ac01-d99f9139f87e\" (UID: \"dcda177c-9d37-4f05-ac01-d99f9139f87e\") " Oct 30 02:14:58.198760 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.198518 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcda177c-9d37-4f05-ac01-d99f9139f87e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcda177c-9d37-4f05-ac01-d99f9139f87e-workload-socket\") pod \"dcda177c-9d37-4f05-ac01-d99f9139f87e\" (UID: \"dcda177c-9d37-4f05-ac01-d99f9139f87e\") " Oct 30 02:14:58.199361 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.199331 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1f0afe7-d54e-4577-857a-da30b84237de-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.199481 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.198329 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcda177c-9d37-4f05-ac01-d99f9139f87e-credential-socket" pod "dcda177c-9d37-4f05-ac01-d99f9139f87e" (UID: "dcda177c-9d37-4f05-ac01-d99f9139f87e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:58.199481 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.198450 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f0afe7-d54e-4577-857a-da30b84237de-workload-certs" pod "e1f0afe7-d54e-4577-857a-da30b84237de" (UID: "e1f0afe7-d54e-4577-857a-da30b84237de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:58.199481 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.198582 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcda177c-9d37-4f05-ac01-d99f9139f87e-workload-certs" pod "dcda177c-9d37-4f05-ac01-d99f9139f87e" (UID: "dcda177c-9d37-4f05-ac01-d99f9139f87e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:58.199481 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.199000 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f0afe7-d54e-4577-857a-da30b84237de-workload-socket" pod "e1f0afe7-d54e-4577-857a-da30b84237de" (UID: "e1f0afe7-d54e-4577-857a-da30b84237de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:58.199481 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.199366 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f3448e6-c22d-466c-a485-fbe2e676c250-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.199481 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.199440 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcda177c-9d37-4f05-ac01-d99f9139f87e-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.199481 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.199458 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f3448e6-c22d-466c-a485-fbe2e676c250-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.199481 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.199472 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/230f0407-5c25-44aa-9a97-1de91305e4f3-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.199911 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.199485 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f3448e6-c22d-466c-a485-fbe2e676c250-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.199911 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.199498 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/230f0407-5c25-44aa-9a97-1de91305e4f3-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.199911 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.199603 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/230f0407-5c25-44aa-9a97-1de91305e4f3-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.199911 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.199616 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1f0afe7-d54e-4577-857a-da30b84237de-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.199911 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.199613 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcda177c-9d37-4f05-ac01-d99f9139f87e-workload-socket" pod "dcda177c-9d37-4f05-ac01-d99f9139f87e" (UID: "dcda177c-9d37-4f05-ac01-d99f9139f87e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:58.199911 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.199631 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f0afe7-d54e-4577-857a-da30b84237de-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.199911 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.199648 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f3448e6-c22d-466c-a485-fbe2e676c250-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.199911 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.199662 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f3448e6-c22d-466c-a485-fbe2e676c250-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.199911 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.199675 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f3448e6-c22d-466c-a485-fbe2e676c250-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.199911 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.199688 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/230f0407-5c25-44aa-9a97-1de91305e4f3-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.199911 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.199693 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcda177c-9d37-4f05-ac01-d99f9139f87e-istiod-ca-cert" pod "dcda177c-9d37-4f05-ac01-d99f9139f87e" (UID: "dcda177c-9d37-4f05-ac01-d99f9139f87e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:14:58.199911 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.199703 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/230f0407-5c25-44aa-9a97-1de91305e4f3-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.199911 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.199740 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w49f8\" (UniqueName: \"kubernetes.io/projected/230f0407-5c25-44aa-9a97-1de91305e4f3-kube-api-access-w49f8\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.199911 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.199808 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/230f0407-5c25-44aa-9a97-1de91305e4f3-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.199911 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.199824 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/230f0407-5c25-44aa-9a97-1de91305e4f3-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.199911 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.199840 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f3448e6-c22d-466c-a485-fbe2e676c250-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.199911 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.199854 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f3448e6-c22d-466c-a485-fbe2e676c250-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.199911 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.199867 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvbbv\" (UniqueName: \"kubernetes.io/projected/5f3448e6-c22d-466c-a485-fbe2e676c250-kube-api-access-mvbbv\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.199911 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.199881 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/230f0407-5c25-44aa-9a97-1de91305e4f3-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.201135 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.201097 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f0afe7-d54e-4577-857a-da30b84237de-istio-envoy" pod "e1f0afe7-d54e-4577-857a-da30b84237de" (UID: "e1f0afe7-d54e-4577-857a-da30b84237de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:58.201249 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.201199 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1f0afe7-d54e-4577-857a-da30b84237de-istio-token" pod "e1f0afe7-d54e-4577-857a-da30b84237de" (UID: "e1f0afe7-d54e-4577-857a-da30b84237de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:58.201249 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.201200 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcda177c-9d37-4f05-ac01-d99f9139f87e-istio-envoy" pod "dcda177c-9d37-4f05-ac01-d99f9139f87e" (UID: "dcda177c-9d37-4f05-ac01-d99f9139f87e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:58.201626 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.201607 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1f0afe7-d54e-4577-857a-da30b84237de-kube-api-access-gnptz" pod "e1f0afe7-d54e-4577-857a-da30b84237de" (UID: "e1f0afe7-d54e-4577-857a-da30b84237de"). InnerVolumeSpecName "kube-api-access-gnptz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:58.201865 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.201838 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcda177c-9d37-4f05-ac01-d99f9139f87e-istio-podinfo" pod "dcda177c-9d37-4f05-ac01-d99f9139f87e" (UID: "dcda177c-9d37-4f05-ac01-d99f9139f87e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:14:58.202292 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.202269 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcda177c-9d37-4f05-ac01-d99f9139f87e-istio-token" pod "dcda177c-9d37-4f05-ac01-d99f9139f87e" (UID: "dcda177c-9d37-4f05-ac01-d99f9139f87e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:58.202432 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.202413 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1f0afe7-d54e-4577-857a-da30b84237de-istio-podinfo" pod "e1f0afe7-d54e-4577-857a-da30b84237de" (UID: "e1f0afe7-d54e-4577-857a-da30b84237de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:14:58.203229 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.203190 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcda177c-9d37-4f05-ac01-d99f9139f87e-kube-api-access-9wg7h" pod "dcda177c-9d37-4f05-ac01-d99f9139f87e" (UID: "dcda177c-9d37-4f05-ac01-d99f9139f87e"). InnerVolumeSpecName "kube-api-access-9wg7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:58.301307 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.301269 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnptz\" (UniqueName: \"kubernetes.io/projected/e1f0afe7-d54e-4577-857a-da30b84237de-kube-api-access-gnptz\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.301307 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.301307 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1f0afe7-d54e-4577-857a-da30b84237de-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.301802 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.301322 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1f0afe7-d54e-4577-857a-da30b84237de-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.301802 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.301337 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcda177c-9d37-4f05-ac01-d99f9139f87e-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.301802 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.301350 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcda177c-9d37-4f05-ac01-d99f9139f87e-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.301802 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.301363 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f0afe7-d54e-4577-857a-da30b84237de-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.301802 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.301376 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcda177c-9d37-4f05-ac01-d99f9139f87e-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.301802 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.301390 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wg7h\" (UniqueName: \"kubernetes.io/projected/dcda177c-9d37-4f05-ac01-d99f9139f87e-kube-api-access-9wg7h\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.301802 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.301405 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcda177c-9d37-4f05-ac01-d99f9139f87e-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.301802 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.301418 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcda177c-9d37-4f05-ac01-d99f9139f87e-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.301802 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.301431 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1f0afe7-d54e-4577-857a-da30b84237de-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.301802 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.301444 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcda177c-9d37-4f05-ac01-d99f9139f87e-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.301802 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.301457 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcda177c-9d37-4f05-ac01-d99f9139f87e-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.301802 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.301471 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1f0afe7-d54e-4577-857a-da30b84237de-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.442360 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.442326 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:58.545437 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.545395 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="613abc91-29df-44d4-9bec-d37e815e9c86" path="/var/lib/kubelet/pods/613abc91-29df-44d4-9bec-d37e815e9c86/volumes" Oct 30 02:14:58.565681 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.565650 6572 generic.go:356] "Generic (PLEG): container finished" podID="e1f0afe7-d54e-4577-857a-da30b84237de" containerID="9dd72b963ccf9883ba203df86886469c88ff25bdd02ee0930d07e87428914ba1" exitCode=0 Oct 30 02:14:58.565847 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.565707 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" Oct 30 02:14:58.565847 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.565740 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" event={"ID":"e1f0afe7-d54e-4577-857a-da30b84237de","Type":"ContainerDied","Data":"9dd72b963ccf9883ba203df86886469c88ff25bdd02ee0930d07e87428914ba1"} Oct 30 02:14:58.565847 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.565816 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6" event={"ID":"e1f0afe7-d54e-4577-857a-da30b84237de","Type":"ContainerDied","Data":"4aeebf8a79ab6e8b11bf735eda4a8a7be01fac1a60a06d792a73608a3b1b41e9"} Oct 30 02:14:58.565847 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.565837 6572 scope.go:122] "RemoveContainer" containerID="9dd72b963ccf9883ba203df86886469c88ff25bdd02ee0930d07e87428914ba1" Oct 30 02:14:58.567678 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.567654 6572 generic.go:356] "Generic (PLEG): container finished" podID="230f0407-5c25-44aa-9a97-1de91305e4f3" containerID="c2d23000006f5624e239b5fb1e018bd43cb1905d4614a9bd0f09a4ba5f9da268" exitCode=0 Oct 30 02:14:58.568967 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.567702 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" event={"ID":"230f0407-5c25-44aa-9a97-1de91305e4f3","Type":"ContainerDied","Data":"c2d23000006f5624e239b5fb1e018bd43cb1905d4614a9bd0f09a4ba5f9da268"} Oct 30 02:14:58.569093 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.568998 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" event={"ID":"230f0407-5c25-44aa-9a97-1de91305e4f3","Type":"ContainerDied","Data":"8876f28a99416f33b6f854397e1d5bf50f4567f3d6f82f3dd077d40ea8c96910"} Oct 30 02:14:58.569093 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.567809 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw" Oct 30 02:14:58.570675 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.570649 6572 generic.go:356] "Generic (PLEG): container finished" podID="dcda177c-9d37-4f05-ac01-d99f9139f87e" containerID="9ebd16028c44463f7b7e12509b4614b5d1d14125135eb71a7f73a677aa136a73" exitCode=0 Oct 30 02:14:58.570835 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.570698 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" Oct 30 02:14:58.570835 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.570735 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" event={"ID":"dcda177c-9d37-4f05-ac01-d99f9139f87e","Type":"ContainerDied","Data":"9ebd16028c44463f7b7e12509b4614b5d1d14125135eb71a7f73a677aa136a73"} Oct 30 02:14:58.570963 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.570837 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct" event={"ID":"dcda177c-9d37-4f05-ac01-d99f9139f87e","Type":"ContainerDied","Data":"1a7d136a8accebbad4e9a549da9791ced4e2d574b2880967b013f70ada82efa2"} Oct 30 02:14:58.572431 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.572410 6572 generic.go:356] "Generic (PLEG): container finished" podID="5f3448e6-c22d-466c-a485-fbe2e676c250" containerID="0635c8e2fdad88fff1b57a81a5208e77d93cfa9be156bb2af249114f0913534e" exitCode=0 Oct 30 02:14:58.572530 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.572453 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" event={"ID":"5f3448e6-c22d-466c-a485-fbe2e676c250","Type":"ContainerDied","Data":"0635c8e2fdad88fff1b57a81a5208e77d93cfa9be156bb2af249114f0913534e"} Oct 30 02:14:58.572530 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.572471 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" Oct 30 02:14:58.572530 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.572491 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2" event={"ID":"5f3448e6-c22d-466c-a485-fbe2e676c250","Type":"ContainerDied","Data":"b5fa13bdadc6ab30b79f4001275e85ee681fcb877ab3ff13a8ef837b28cf47ca"} Oct 30 02:14:58.605203 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.605173 6572 scope.go:122] "RemoveContainer" containerID="9dd72b963ccf9883ba203df86886469c88ff25bdd02ee0930d07e87428914ba1" Oct 30 02:14:58.605594 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:14:58.605572 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dd72b963ccf9883ba203df86886469c88ff25bdd02ee0930d07e87428914ba1\": container with ID starting with 9dd72b963ccf9883ba203df86886469c88ff25bdd02ee0930d07e87428914ba1 not found: ID does not exist" containerID="9dd72b963ccf9883ba203df86886469c88ff25bdd02ee0930d07e87428914ba1" Oct 30 02:14:58.605694 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.605602 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dd72b963ccf9883ba203df86886469c88ff25bdd02ee0930d07e87428914ba1"} err="failed to get container status \"9dd72b963ccf9883ba203df86886469c88ff25bdd02ee0930d07e87428914ba1\": rpc error: code = NotFound desc = could not find container \"9dd72b963ccf9883ba203df86886469c88ff25bdd02ee0930d07e87428914ba1\": container with ID starting with 9dd72b963ccf9883ba203df86886469c88ff25bdd02ee0930d07e87428914ba1 not found: ID does not exist" Oct 30 02:14:58.605694 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.605625 6572 scope.go:122] "RemoveContainer" containerID="c2d23000006f5624e239b5fb1e018bd43cb1905d4614a9bd0f09a4ba5f9da268" Oct 30 02:14:58.620031 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.619862 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6"] Oct 30 02:14:58.620124 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.620036 6572 scope.go:122] "RemoveContainer" containerID="c2d23000006f5624e239b5fb1e018bd43cb1905d4614a9bd0f09a4ba5f9da268" Oct 30 02:14:58.620458 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:14:58.620440 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2d23000006f5624e239b5fb1e018bd43cb1905d4614a9bd0f09a4ba5f9da268\": container with ID starting with c2d23000006f5624e239b5fb1e018bd43cb1905d4614a9bd0f09a4ba5f9da268 not found: ID does not exist" containerID="c2d23000006f5624e239b5fb1e018bd43cb1905d4614a9bd0f09a4ba5f9da268" Oct 30 02:14:58.620513 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.620466 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2d23000006f5624e239b5fb1e018bd43cb1905d4614a9bd0f09a4ba5f9da268"} err="failed to get container status \"c2d23000006f5624e239b5fb1e018bd43cb1905d4614a9bd0f09a4ba5f9da268\": rpc error: code = NotFound desc = could not find container \"c2d23000006f5624e239b5fb1e018bd43cb1905d4614a9bd0f09a4ba5f9da268\": container with ID starting with c2d23000006f5624e239b5fb1e018bd43cb1905d4614a9bd0f09a4ba5f9da268 not found: ID does not exist" Oct 30 02:14:58.620513 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.620484 6572 scope.go:122] "RemoveContainer" containerID="9ebd16028c44463f7b7e12509b4614b5d1d14125135eb71a7f73a677aa136a73" Oct 30 02:14:58.624781 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.624735 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zsgv6"] Oct 30 02:14:58.636065 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.636038 6572 scope.go:122] "RemoveContainer" containerID="9ebd16028c44463f7b7e12509b4614b5d1d14125135eb71a7f73a677aa136a73" Oct 30 02:14:58.636620 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:14:58.636591 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ebd16028c44463f7b7e12509b4614b5d1d14125135eb71a7f73a677aa136a73\": container with ID starting with 9ebd16028c44463f7b7e12509b4614b5d1d14125135eb71a7f73a677aa136a73 not found: ID does not exist" containerID="9ebd16028c44463f7b7e12509b4614b5d1d14125135eb71a7f73a677aa136a73" Oct 30 02:14:58.636728 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.636633 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ebd16028c44463f7b7e12509b4614b5d1d14125135eb71a7f73a677aa136a73"} err="failed to get container status \"9ebd16028c44463f7b7e12509b4614b5d1d14125135eb71a7f73a677aa136a73\": rpc error: code = NotFound desc = could not find container \"9ebd16028c44463f7b7e12509b4614b5d1d14125135eb71a7f73a677aa136a73\": container with ID starting with 9ebd16028c44463f7b7e12509b4614b5d1d14125135eb71a7f73a677aa136a73 not found: ID does not exist" Oct 30 02:14:58.636728 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.636657 6572 scope.go:122] "RemoveContainer" containerID="0635c8e2fdad88fff1b57a81a5208e77d93cfa9be156bb2af249114f0913534e" Oct 30 02:14:58.648872 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.648845 6572 scope.go:122] "RemoveContainer" containerID="0635c8e2fdad88fff1b57a81a5208e77d93cfa9be156bb2af249114f0913534e" Oct 30 02:14:58.649204 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:14:58.649183 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0635c8e2fdad88fff1b57a81a5208e77d93cfa9be156bb2af249114f0913534e\": container with ID starting with 0635c8e2fdad88fff1b57a81a5208e77d93cfa9be156bb2af249114f0913534e not found: ID does not exist" containerID="0635c8e2fdad88fff1b57a81a5208e77d93cfa9be156bb2af249114f0913534e" Oct 30 02:14:58.649259 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.649213 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0635c8e2fdad88fff1b57a81a5208e77d93cfa9be156bb2af249114f0913534e"} err="failed to get container status \"0635c8e2fdad88fff1b57a81a5208e77d93cfa9be156bb2af249114f0913534e\": rpc error: code = NotFound desc = could not find container \"0635c8e2fdad88fff1b57a81a5208e77d93cfa9be156bb2af249114f0913534e\": container with ID starting with 0635c8e2fdad88fff1b57a81a5208e77d93cfa9be156bb2af249114f0913534e not found: ID does not exist" Oct 30 02:14:58.702908 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.702800 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw"] Oct 30 02:14:58.708159 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.708122 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tzxqw"] Oct 30 02:14:58.744296 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.744263 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2"] Oct 30 02:14:58.748691 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.748661 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rl4q2"] Oct 30 02:14:58.778829 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.778792 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct"] Oct 30 02:14:58.786424 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:58.786383 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-tznct"] Oct 30 02:14:59.087453 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:59.087371 6572 scope.go:122] "RemoveContainer" containerID="a18dac67dd224c926a3a13e29cc6c9000a720e5a2c34a47dcb1ecbaed651d19b" Oct 30 02:14:59.125078 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:59.125056 6572 scope.go:122] "RemoveContainer" containerID="ad8b6ba999d4e6c37f44f4539621a51987e497a3e485bda1ad10c4dc627b218d" Oct 30 02:14:59.698881 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:14:59.698839 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:00.158813 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.158704 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l"] Oct 30 02:15:00.159684 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.159658 6572 cpu_manager.go:414] "containerMap: removing container" podUID="ee995a27-1255-4c9c-88c4-c6e4eb58a77d" containerName="istio-proxy" Oct 30 02:15:00.159684 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.159682 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee995a27-1255-4c9c-88c4-c6e4eb58a77d" containerName="istio-proxy" Oct 30 02:15:00.159915 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.159716 6572 cpu_manager.go:414] "containerMap: removing container" podUID="a341c867-33db-494c-b3e4-873415967cf0" containerName="istio-proxy" Oct 30 02:15:00.159915 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.159723 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a341c867-33db-494c-b3e4-873415967cf0" containerName="istio-proxy" Oct 30 02:15:00.159915 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.159740 6572 cpu_manager.go:414] "containerMap: removing container" podUID="e1f0afe7-d54e-4577-857a-da30b84237de" containerName="istio-proxy" Oct 30 02:15:00.159915 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.159763 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1f0afe7-d54e-4577-857a-da30b84237de" containerName="istio-proxy" Oct 30 02:15:00.159915 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.159793 6572 cpu_manager.go:414] "containerMap: removing container" podUID="a341c867-33db-494c-b3e4-873415967cf0" containerName="istio-proxy" Oct 30 02:15:00.159915 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.159799 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a341c867-33db-494c-b3e4-873415967cf0" containerName="istio-proxy" Oct 30 02:15:00.159915 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.159819 6572 cpu_manager.go:414] "containerMap: removing container" podUID="dcda177c-9d37-4f05-ac01-d99f9139f87e" containerName="istio-proxy" Oct 30 02:15:00.159915 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.159826 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcda177c-9d37-4f05-ac01-d99f9139f87e" containerName="istio-proxy" Oct 30 02:15:00.159915 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.159847 6572 cpu_manager.go:414] "containerMap: removing container" podUID="a238ef55-f32f-42c5-9e67-a478bf8eeb8c" containerName="revoke-keys" Oct 30 02:15:00.159915 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.159854 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a238ef55-f32f-42c5-9e67-a478bf8eeb8c" containerName="revoke-keys" Oct 30 02:15:00.159915 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.159879 6572 cpu_manager.go:414] "containerMap: removing container" podUID="230f0407-5c25-44aa-9a97-1de91305e4f3" containerName="istio-proxy" Oct 30 02:15:00.159915 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.159886 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="230f0407-5c25-44aa-9a97-1de91305e4f3" containerName="istio-proxy" Oct 30 02:15:00.159915 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.159917 6572 cpu_manager.go:414] "containerMap: removing container" podUID="5f3448e6-c22d-466c-a485-fbe2e676c250" containerName="istio-proxy" Oct 30 02:15:00.160587 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.159924 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f3448e6-c22d-466c-a485-fbe2e676c250" containerName="istio-proxy" Oct 30 02:15:00.160587 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.160104 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1f0afe7-d54e-4577-857a-da30b84237de" containerName="istio-proxy" Oct 30 02:15:00.160587 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.160121 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="a341c867-33db-494c-b3e4-873415967cf0" containerName="istio-proxy" Oct 30 02:15:00.160587 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.160132 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcda177c-9d37-4f05-ac01-d99f9139f87e" containerName="istio-proxy" Oct 30 02:15:00.160587 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.160146 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="a238ef55-f32f-42c5-9e67-a478bf8eeb8c" containerName="revoke-keys" Oct 30 02:15:00.160587 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.160157 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="230f0407-5c25-44aa-9a97-1de91305e4f3" containerName="istio-proxy" Oct 30 02:15:00.160587 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.160170 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f3448e6-c22d-466c-a485-fbe2e676c250" containerName="istio-proxy" Oct 30 02:15:00.160587 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.160179 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee995a27-1255-4c9c-88c4-c6e4eb58a77d" containerName="istio-proxy" Oct 30 02:15:00.166920 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.166891 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888775-zd7gh"] Oct 30 02:15:00.167094 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.167017 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l" Oct 30 02:15:00.168179 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.168148 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="a341c867-33db-494c-b3e4-873415967cf0" containerName="istio-proxy" Oct 30 02:15:00.171912 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.171886 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888775-rbx7m"] Oct 30 02:15:00.172052 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.172011 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888775-zd7gh" Oct 30 02:15:00.177279 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.177244 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888775-ttncp"] Oct 30 02:15:00.177467 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.177423 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888775-rbx7m" Oct 30 02:15:00.182924 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.182894 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888775-qr45g"] Oct 30 02:15:00.183097 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.183017 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888775-ttncp" Oct 30 02:15:00.183685 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.183651 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:15:00.190237 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.188379 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888775-wrhfw"] Oct 30 02:15:00.190237 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.188636 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888775-qr45g" Oct 30 02:15:00.195812 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.195783 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:15:00.197582 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.197536 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d5aa9-29888775-9pqsc"] Oct 30 02:15:00.203700 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.198739 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888775-wrhfw" Oct 30 02:15:00.203700 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.201197 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:15:00.233135 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.233034 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f641ac83-f4e6-4003-ae9c-be79e4ca0337-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888775-ttncp\" (UID: \"f641ac83-f4e6-4003-ae9c-be79e4ca0337\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888775-ttncp" Oct 30 02:15:00.233312 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.233186 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d66a87ee-dc89-4aba-845b-17b58bc6f4a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888775-zd7gh\" (UID: \"d66a87ee-dc89-4aba-845b-17b58bc6f4a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888775-zd7gh" Oct 30 02:15:00.233400 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.233307 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl7vc\" (UniqueName: \"kubernetes.io/projected/1f68c408-6bab-42c7-b574-35d5d61c7544-kube-api-access-gl7vc\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888775-qr45g\" (UID: \"1f68c408-6bab-42c7-b574-35d5d61c7544\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888775-qr45g" Oct 30 02:15:00.233461 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.233398 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxml8\" (UniqueName: \"kubernetes.io/projected/3f9f8331-2090-4262-96a0-c3ea360e4375-kube-api-access-mxml8\") pod \"maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l\" (UID: \"3f9f8331-2090-4262-96a0-c3ea360e4375\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l" Oct 30 02:15:00.233513 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.233461 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc59p\" (UniqueName: \"kubernetes.io/projected/d66a87ee-dc89-4aba-845b-17b58bc6f4a6-kube-api-access-vc59p\") pod \"maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888775-zd7gh\" (UID: \"d66a87ee-dc89-4aba-845b-17b58bc6f4a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888775-zd7gh" Oct 30 02:15:00.233699 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.233668 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47xwk\" (UniqueName: \"kubernetes.io/projected/0dc3a619-596a-4e28-9845-9d1e2fc7bb06-kube-api-access-47xwk\") pod \"maas-api-key-cleanup-e2e-extmod-a-d44413-29888775-rbx7m\" (UID: \"0dc3a619-596a-4e28-9845-9d1e2fc7bb06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888775-rbx7m" Oct 30 02:15:00.233812 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.233795 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f68c408-6bab-42c7-b574-35d5d61c7544-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888775-qr45g\" (UID: \"1f68c408-6bab-42c7-b574-35d5d61c7544\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888775-qr45g" Oct 30 02:15:00.233881 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.233867 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f9f8331-2090-4262-96a0-c3ea360e4375-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l\" (UID: \"3f9f8331-2090-4262-96a0-c3ea360e4375\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l" Oct 30 02:15:00.233933 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.233893 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l"] Oct 30 02:15:00.233933 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.233918 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2bd8bc3-6252-42fc-9055-ca393b6f45d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-cd8173-29888775-wrhfw\" (UID: \"c2bd8bc3-6252-42fc-9055-ca393b6f45d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888775-wrhfw" Oct 30 02:15:00.234031 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.233948 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888775-rbx7m"] Oct 30 02:15:00.234031 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.233961 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dc3a619-596a-4e28-9845-9d1e2fc7bb06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d44413-29888775-rbx7m\" (UID: \"0dc3a619-596a-4e28-9845-9d1e2fc7bb06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888775-rbx7m" Oct 30 02:15:00.234031 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.233986 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888775-r46v9"] Oct 30 02:15:00.234181 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.234046 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px6wh\" (UniqueName: \"kubernetes.io/projected/c2bd8bc3-6252-42fc-9055-ca393b6f45d9-kube-api-access-px6wh\") pod \"maas-api-key-cleanup-e2e-extmod-b-cd8173-29888775-wrhfw\" (UID: \"c2bd8bc3-6252-42fc-9055-ca393b6f45d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888775-wrhfw" Oct 30 02:15:00.234181 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.234144 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glsdv\" (UniqueName: \"kubernetes.io/projected/f641ac83-f4e6-4003-ae9c-be79e4ca0337-kube-api-access-glsdv\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888775-ttncp\" (UID: \"f641ac83-f4e6-4003-ae9c-be79e4ca0337\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888775-ttncp" Oct 30 02:15:00.235275 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.235246 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d5aa9-29888775-9pqsc" Oct 30 02:15:00.270941 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.270906 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888775-r46v9" Oct 30 02:15:00.272603 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.272312 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888775-p4j7s"] Oct 30 02:15:00.320403 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.320372 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888775-8tzkk"] Oct 30 02:15:00.321414 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.320699 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888775-p4j7s" Oct 30 02:15:00.333003 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.332972 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888775-zd7gh"] Oct 30 02:15:00.333252 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.333237 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888775-ttncp"] Oct 30 02:15:00.333719 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.333692 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888775-8tzkk" Oct 30 02:15:00.335944 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.335883 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2bd8bc3-6252-42fc-9055-ca393b6f45d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-cd8173-29888775-wrhfw\" (UID: \"c2bd8bc3-6252-42fc-9055-ca393b6f45d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888775-wrhfw" Oct 30 02:15:00.336133 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.336112 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dc3a619-596a-4e28-9845-9d1e2fc7bb06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d44413-29888775-rbx7m\" (UID: \"0dc3a619-596a-4e28-9845-9d1e2fc7bb06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888775-rbx7m" Oct 30 02:15:00.336264 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.336246 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6edd5565-a917-481f-9302-95011124898b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-803472-29888775-r46v9\" (UID: \"6edd5565-a917-481f-9302-95011124898b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888775-r46v9" Oct 30 02:15:00.336399 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.336384 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px6wh\" (UniqueName: \"kubernetes.io/projected/c2bd8bc3-6252-42fc-9055-ca393b6f45d9-kube-api-access-px6wh\") pod \"maas-api-key-cleanup-e2e-extmod-b-cd8173-29888775-wrhfw\" (UID: \"c2bd8bc3-6252-42fc-9055-ca393b6f45d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888775-wrhfw" Oct 30 02:15:00.336559 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.336540 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh9rx\" (UniqueName: \"kubernetes.io/projected/8968e7e0-04c1-4469-9062-f7d31ee6dbf2-kube-api-access-bh9rx\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888775-p4j7s\" (UID: \"8968e7e0-04c1-4469-9062-f7d31ee6dbf2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888775-p4j7s" Oct 30 02:15:00.336731 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.336712 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glsdv\" (UniqueName: \"kubernetes.io/projected/f641ac83-f4e6-4003-ae9c-be79e4ca0337-kube-api-access-glsdv\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888775-ttncp\" (UID: \"f641ac83-f4e6-4003-ae9c-be79e4ca0337\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888775-ttncp" Oct 30 02:15:00.340522 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.336896 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdf23f37-2870-4b9c-b558-9fea4452f4e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888775-8tzkk\" (UID: \"bdf23f37-2870-4b9c-b558-9fea4452f4e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888775-8tzkk" Oct 30 02:15:00.342824 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.337128 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2bd8bc3-6252-42fc-9055-ca393b6f45d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-cd8173-29888775-wrhfw\" (UID: \"c2bd8bc3-6252-42fc-9055-ca393b6f45d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888775-wrhfw" Oct 30 02:15:00.342824 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.338066 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dc3a619-596a-4e28-9845-9d1e2fc7bb06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d44413-29888775-rbx7m\" (UID: \"0dc3a619-596a-4e28-9845-9d1e2fc7bb06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888775-rbx7m" Oct 30 02:15:00.343088 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.342939 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888775-qr45g"] Oct 30 02:15:00.343222 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.343191 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec1f4d7-877e-4d7c-88d2-f2365469869a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-2d5aa9-29888775-9pqsc\" (UID: \"fec1f4d7-877e-4d7c-88d2-f2365469869a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d5aa9-29888775-9pqsc" Oct 30 02:15:00.344305 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.344282 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f641ac83-f4e6-4003-ae9c-be79e4ca0337-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888775-ttncp\" (UID: \"f641ac83-f4e6-4003-ae9c-be79e4ca0337\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888775-ttncp" Oct 30 02:15:00.345211 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.345193 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d66a87ee-dc89-4aba-845b-17b58bc6f4a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888775-zd7gh\" (UID: \"d66a87ee-dc89-4aba-845b-17b58bc6f4a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888775-zd7gh" Oct 30 02:15:00.346045 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.345889 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjgfp\" (UniqueName: \"kubernetes.io/projected/6edd5565-a917-481f-9302-95011124898b-kube-api-access-tjgfp\") pod \"maas-api-key-cleanup-e2e-worker-w4-803472-29888775-r46v9\" (UID: \"6edd5565-a917-481f-9302-95011124898b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888775-r46v9" Oct 30 02:15:00.346045 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.345939 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl7vc\" (UniqueName: \"kubernetes.io/projected/1f68c408-6bab-42c7-b574-35d5d61c7544-kube-api-access-gl7vc\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888775-qr45g\" (UID: \"1f68c408-6bab-42c7-b574-35d5d61c7544\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888775-qr45g" Oct 30 02:15:00.346045 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.345984 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d66a87ee-dc89-4aba-845b-17b58bc6f4a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888775-zd7gh\" (UID: \"d66a87ee-dc89-4aba-845b-17b58bc6f4a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888775-zd7gh" Oct 30 02:15:00.346045 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.345998 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxml8\" (UniqueName: \"kubernetes.io/projected/3f9f8331-2090-4262-96a0-c3ea360e4375-kube-api-access-mxml8\") pod \"maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l\" (UID: \"3f9f8331-2090-4262-96a0-c3ea360e4375\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l" Oct 30 02:15:00.346045 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.345143 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f641ac83-f4e6-4003-ae9c-be79e4ca0337-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888775-ttncp\" (UID: \"f641ac83-f4e6-4003-ae9c-be79e4ca0337\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888775-ttncp" Oct 30 02:15:00.346499 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.346479 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc59p\" (UniqueName: \"kubernetes.io/projected/d66a87ee-dc89-4aba-845b-17b58bc6f4a6-kube-api-access-vc59p\") pod \"maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888775-zd7gh\" (UID: \"d66a87ee-dc89-4aba-845b-17b58bc6f4a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888775-zd7gh" Oct 30 02:15:00.346689 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.346673 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8968e7e0-04c1-4469-9062-f7d31ee6dbf2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888775-p4j7s\" (UID: \"8968e7e0-04c1-4469-9062-f7d31ee6dbf2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888775-p4j7s" Oct 30 02:15:00.346854 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.346837 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmdt2\" (UniqueName: \"kubernetes.io/projected/bdf23f37-2870-4b9c-b558-9fea4452f4e5-kube-api-access-fmdt2\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888775-8tzkk\" (UID: \"bdf23f37-2870-4b9c-b558-9fea4452f4e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888775-8tzkk" Oct 30 02:15:00.347235 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.347215 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47xwk\" (UniqueName: \"kubernetes.io/projected/0dc3a619-596a-4e28-9845-9d1e2fc7bb06-kube-api-access-47xwk\") pod \"maas-api-key-cleanup-e2e-extmod-a-d44413-29888775-rbx7m\" (UID: \"0dc3a619-596a-4e28-9845-9d1e2fc7bb06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888775-rbx7m" Oct 30 02:15:00.347461 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.347442 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f68c408-6bab-42c7-b574-35d5d61c7544-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888775-qr45g\" (UID: \"1f68c408-6bab-42c7-b574-35d5d61c7544\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888775-qr45g" Oct 30 02:15:00.350641 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.350620 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tx8z\" (UniqueName: \"kubernetes.io/projected/fec1f4d7-877e-4d7c-88d2-f2365469869a-kube-api-access-5tx8z\") pod \"maas-api-key-cleanup-e2e-inf-b-2d5aa9-29888775-9pqsc\" (UID: \"fec1f4d7-877e-4d7c-88d2-f2365469869a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d5aa9-29888775-9pqsc" Oct 30 02:15:00.350850 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.350833 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f9f8331-2090-4262-96a0-c3ea360e4375-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l\" (UID: \"3f9f8331-2090-4262-96a0-c3ea360e4375\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l" Oct 30 02:15:00.351887 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.348385 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f68c408-6bab-42c7-b574-35d5d61c7544-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888775-qr45g\" (UID: \"1f68c408-6bab-42c7-b574-35d5d61c7544\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888775-qr45g" Oct 30 02:15:00.353063 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.350319 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888775-wrhfw"] Oct 30 02:15:00.353217 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.351665 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f9f8331-2090-4262-96a0-c3ea360e4375-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l\" (UID: \"3f9f8331-2090-4262-96a0-c3ea360e4375\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l" Oct 30 02:15:00.357024 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.356999 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888775-sv8kf"] Oct 30 02:15:00.364125 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.364096 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888775-sv8kf" Oct 30 02:15:00.364950 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.364892 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glsdv\" (UniqueName: \"kubernetes.io/projected/f641ac83-f4e6-4003-ae9c-be79e4ca0337-kube-api-access-glsdv\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888775-ttncp\" (UID: \"f641ac83-f4e6-4003-ae9c-be79e4ca0337\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888775-ttncp" Oct 30 02:15:00.365375 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.365261 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px6wh\" (UniqueName: \"kubernetes.io/projected/c2bd8bc3-6252-42fc-9055-ca393b6f45d9-kube-api-access-px6wh\") pod \"maas-api-key-cleanup-e2e-extmod-b-cd8173-29888775-wrhfw\" (UID: \"c2bd8bc3-6252-42fc-9055-ca393b6f45d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888775-wrhfw" Oct 30 02:15:00.368395 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.368359 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl7vc\" (UniqueName: \"kubernetes.io/projected/1f68c408-6bab-42c7-b574-35d5d61c7544-kube-api-access-gl7vc\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888775-qr45g\" (UID: \"1f68c408-6bab-42c7-b574-35d5d61c7544\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888775-qr45g" Oct 30 02:15:00.369646 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.369621 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxml8\" (UniqueName: \"kubernetes.io/projected/3f9f8331-2090-4262-96a0-c3ea360e4375-kube-api-access-mxml8\") pod \"maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l\" (UID: \"3f9f8331-2090-4262-96a0-c3ea360e4375\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l" Oct 30 02:15:00.370864 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.370847 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888775-p4j7s"] Oct 30 02:15:00.371057 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.370976 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47xwk\" (UniqueName: \"kubernetes.io/projected/0dc3a619-596a-4e28-9845-9d1e2fc7bb06-kube-api-access-47xwk\") pod \"maas-api-key-cleanup-e2e-extmod-a-d44413-29888775-rbx7m\" (UID: \"0dc3a619-596a-4e28-9845-9d1e2fc7bb06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888775-rbx7m" Oct 30 02:15:00.371145 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.371123 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc59p\" (UniqueName: \"kubernetes.io/projected/d66a87ee-dc89-4aba-845b-17b58bc6f4a6-kube-api-access-vc59p\") pod \"maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888775-zd7gh\" (UID: \"d66a87ee-dc89-4aba-845b-17b58bc6f4a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888775-zd7gh" Oct 30 02:15:00.378343 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.378302 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888775-djsrl"] Oct 30 02:15:00.384951 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.384921 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888775-djsrl" Oct 30 02:15:00.385829 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.385767 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888775-289bf"] Oct 30 02:15:00.392821 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.392783 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888775-289bf" Oct 30 02:15:00.394095 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.394073 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888775-vcqdq"] Oct 30 02:15:00.401802 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.400204 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888775-vcqdq" Oct 30 02:15:00.401802 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.401634 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888775-8tzkk"] Oct 30 02:15:00.407315 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.407282 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d5aa9-29888775-9pqsc"] Oct 30 02:15:00.412195 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.412170 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888775-r46v9"] Oct 30 02:15:00.417547 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.417516 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888775-sv8kf"] Oct 30 02:15:00.422707 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.422676 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888775-brm4t"] Oct 30 02:15:00.427836 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.427807 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888775-djsrl"] Oct 30 02:15:00.428026 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.427925 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888775-brm4t" Oct 30 02:15:00.432588 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.432564 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888775-8xvvv"] Oct 30 02:15:00.437569 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.437546 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-8xvvv" Oct 30 02:15:00.440488 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.438347 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888775-vcqdq"] Oct 30 02:15:00.451203 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.451175 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888775-fxr88"] Oct 30 02:15:00.453523 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.453495 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-429l8\" (UniqueName: \"kubernetes.io/projected/ba0ac388-9127-47e0-8985-dd48570c8d27-kube-api-access-429l8\") pod \"maas-api-key-cleanup-e2e-extmod-b-093d23-29888775-vcqdq\" (UID: \"ba0ac388-9127-47e0-8985-dd48570c8d27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888775-vcqdq" Oct 30 02:15:00.453677 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.453572 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdf23f37-2870-4b9c-b558-9fea4452f4e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888775-8tzkk\" (UID: \"bdf23f37-2870-4b9c-b558-9fea4452f4e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888775-8tzkk" Oct 30 02:15:00.453677 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.453649 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec1f4d7-877e-4d7c-88d2-f2365469869a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-2d5aa9-29888775-9pqsc\" (UID: \"fec1f4d7-877e-4d7c-88d2-f2365469869a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d5aa9-29888775-9pqsc" Oct 30 02:15:00.453778 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.453700 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs7rx\" (UniqueName: \"kubernetes.io/projected/99cb9b6a-51ec-41be-a85d-dd1de997fceb-kube-api-access-bs7rx\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888775-sv8kf\" (UID: \"99cb9b6a-51ec-41be-a85d-dd1de997fceb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888775-sv8kf" Oct 30 02:15:00.453822 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.453801 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8jwh\" (UniqueName: \"kubernetes.io/projected/1c7f35b4-a13a-4be3-be70-135dcc096f49-kube-api-access-g8jwh\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888775-brm4t\" (UID: \"1c7f35b4-a13a-4be3-be70-135dcc096f49\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888775-brm4t" Oct 30 02:15:00.454738 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.453864 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjgfp\" (UniqueName: \"kubernetes.io/projected/6edd5565-a917-481f-9302-95011124898b-kube-api-access-tjgfp\") pod \"maas-api-key-cleanup-e2e-worker-w4-803472-29888775-r46v9\" (UID: \"6edd5565-a917-481f-9302-95011124898b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888775-r46v9" Oct 30 02:15:00.454738 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.453922 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99cb9b6a-51ec-41be-a85d-dd1de997fceb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888775-sv8kf\" (UID: \"99cb9b6a-51ec-41be-a85d-dd1de997fceb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888775-sv8kf" Oct 30 02:15:00.454738 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.453951 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcf5437e-0075-467b-a4fd-82b5dc563489-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-292018-29888775-289bf\" (UID: \"bcf5437e-0075-467b-a4fd-82b5dc563489\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888775-289bf" Oct 30 02:15:00.454738 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.454002 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8968e7e0-04c1-4469-9062-f7d31ee6dbf2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888775-p4j7s\" (UID: \"8968e7e0-04c1-4469-9062-f7d31ee6dbf2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888775-p4j7s" Oct 30 02:15:00.454738 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.454033 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmdt2\" (UniqueName: \"kubernetes.io/projected/bdf23f37-2870-4b9c-b558-9fea4452f4e5-kube-api-access-fmdt2\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888775-8tzkk\" (UID: \"bdf23f37-2870-4b9c-b558-9fea4452f4e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888775-8tzkk" Oct 30 02:15:00.454738 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.454140 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9vdw\" (UniqueName: \"kubernetes.io/projected/bcf5437e-0075-467b-a4fd-82b5dc563489-kube-api-access-k9vdw\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-292018-29888775-289bf\" (UID: \"bcf5437e-0075-467b-a4fd-82b5dc563489\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888775-289bf" Oct 30 02:15:00.454738 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.454199 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbttd\" (UniqueName: \"kubernetes.io/projected/4ed6af0f-eb0e-41f8-8b21-654babbb98b5-kube-api-access-dbttd\") pod \"maas-api-key-cleanup-e2e-worker-w3-558327-29888775-djsrl\" (UID: \"4ed6af0f-eb0e-41f8-8b21-654babbb98b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888775-djsrl" Oct 30 02:15:00.454738 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.454263 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba0ac388-9127-47e0-8985-dd48570c8d27-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-093d23-29888775-vcqdq\" (UID: \"ba0ac388-9127-47e0-8985-dd48570c8d27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888775-vcqdq" Oct 30 02:15:00.454738 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.454295 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tx8z\" (UniqueName: \"kubernetes.io/projected/fec1f4d7-877e-4d7c-88d2-f2365469869a-kube-api-access-5tx8z\") pod \"maas-api-key-cleanup-e2e-inf-b-2d5aa9-29888775-9pqsc\" (UID: \"fec1f4d7-877e-4d7c-88d2-f2365469869a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d5aa9-29888775-9pqsc" Oct 30 02:15:00.454738 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.454346 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6edd5565-a917-481f-9302-95011124898b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-803472-29888775-r46v9\" (UID: \"6edd5565-a917-481f-9302-95011124898b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888775-r46v9" Oct 30 02:15:00.454738 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.454369 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7f35b4-a13a-4be3-be70-135dcc096f49-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888775-brm4t\" (UID: \"1c7f35b4-a13a-4be3-be70-135dcc096f49\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888775-brm4t" Oct 30 02:15:00.454738 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.454413 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ed6af0f-eb0e-41f8-8b21-654babbb98b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-558327-29888775-djsrl\" (UID: \"4ed6af0f-eb0e-41f8-8b21-654babbb98b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888775-djsrl" Oct 30 02:15:00.454738 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.454433 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh9rx\" (UniqueName: \"kubernetes.io/projected/8968e7e0-04c1-4469-9062-f7d31ee6dbf2-kube-api-access-bh9rx\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888775-p4j7s\" (UID: \"8968e7e0-04c1-4469-9062-f7d31ee6dbf2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888775-p4j7s" Oct 30 02:15:00.454738 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.454625 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdf23f37-2870-4b9c-b558-9fea4452f4e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888775-8tzkk\" (UID: \"bdf23f37-2870-4b9c-b558-9fea4452f4e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888775-8tzkk" Oct 30 02:15:00.454738 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.454625 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec1f4d7-877e-4d7c-88d2-f2365469869a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-2d5aa9-29888775-9pqsc\" (UID: \"fec1f4d7-877e-4d7c-88d2-f2365469869a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d5aa9-29888775-9pqsc" Oct 30 02:15:00.456766 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.456117 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8968e7e0-04c1-4469-9062-f7d31ee6dbf2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888775-p4j7s\" (UID: \"8968e7e0-04c1-4469-9062-f7d31ee6dbf2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888775-p4j7s" Oct 30 02:15:00.456766 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.456732 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6edd5565-a917-481f-9302-95011124898b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-803472-29888775-r46v9\" (UID: \"6edd5565-a917-481f-9302-95011124898b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888775-r46v9" Oct 30 02:15:00.459132 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.459109 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888775-82nz4"] Oct 30 02:15:00.459479 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.459455 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888775-fxr88" Oct 30 02:15:00.464100 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.464077 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888775-brm4t"] Oct 30 02:15:00.464310 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.464177 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888775-82nz4" Oct 30 02:15:00.467263 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.467235 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-340ae6-29888775-8dfms"] Oct 30 02:15:00.473305 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.473283 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888775-fxr88"] Oct 30 02:15:00.473305 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.473398 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-340ae6-29888775-8dfms" Oct 30 02:15:00.474671 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.474645 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmdt2\" (UniqueName: \"kubernetes.io/projected/bdf23f37-2870-4b9c-b558-9fea4452f4e5-kube-api-access-fmdt2\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888775-8tzkk\" (UID: \"bdf23f37-2870-4b9c-b558-9fea4452f4e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888775-8tzkk" Oct 30 02:15:00.474816 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.474794 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh9rx\" (UniqueName: \"kubernetes.io/projected/8968e7e0-04c1-4469-9062-f7d31ee6dbf2-kube-api-access-bh9rx\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888775-p4j7s\" (UID: \"8968e7e0-04c1-4469-9062-f7d31ee6dbf2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888775-p4j7s" Oct 30 02:15:00.474875 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.474797 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjgfp\" (UniqueName: \"kubernetes.io/projected/6edd5565-a917-481f-9302-95011124898b-kube-api-access-tjgfp\") pod \"maas-api-key-cleanup-e2e-worker-w4-803472-29888775-r46v9\" (UID: \"6edd5565-a917-481f-9302-95011124898b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888775-r46v9" Oct 30 02:15:00.475247 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.475225 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tx8z\" (UniqueName: \"kubernetes.io/projected/fec1f4d7-877e-4d7c-88d2-f2365469869a-kube-api-access-5tx8z\") pod \"maas-api-key-cleanup-e2e-inf-b-2d5aa9-29888775-9pqsc\" (UID: \"fec1f4d7-877e-4d7c-88d2-f2365469869a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d5aa9-29888775-9pqsc" Oct 30 02:15:00.477052 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.477028 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f89697-29888775-9m9lw"] Oct 30 02:15:00.482228 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.482205 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888775-82nz4"] Oct 30 02:15:00.482347 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.482294 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f89697-29888775-9m9lw" Oct 30 02:15:00.487151 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.487124 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l" Oct 30 02:15:00.490525 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.490306 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-340ae6-29888775-8dfms"] Oct 30 02:15:00.495676 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.495645 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888775-289bf"] Oct 30 02:15:00.500314 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.500284 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888775-8xvvv"] Oct 30 02:15:00.504705 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.504684 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f89697-29888775-9m9lw"] Oct 30 02:15:00.506846 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.506823 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888775-zd7gh" Oct 30 02:15:00.560228 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.559351 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-429l8\" (UniqueName: \"kubernetes.io/projected/ba0ac388-9127-47e0-8985-dd48570c8d27-kube-api-access-429l8\") pod \"maas-api-key-cleanup-e2e-extmod-b-093d23-29888775-vcqdq\" (UID: \"ba0ac388-9127-47e0-8985-dd48570c8d27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888775-vcqdq" Oct 30 02:15:00.560228 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.559459 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a1834c-2859-4e9c-9d32-80a6f7247460-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888775-82nz4\" (UID: \"75a1834c-2859-4e9c-9d32-80a6f7247460\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888775-82nz4" Oct 30 02:15:00.560228 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.559527 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs7rx\" (UniqueName: \"kubernetes.io/projected/99cb9b6a-51ec-41be-a85d-dd1de997fceb-kube-api-access-bs7rx\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888775-sv8kf\" (UID: \"99cb9b6a-51ec-41be-a85d-dd1de997fceb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888775-sv8kf" Oct 30 02:15:00.560228 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.559579 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8ghh\" (UniqueName: \"kubernetes.io/projected/75a1834c-2859-4e9c-9d32-80a6f7247460-kube-api-access-j8ghh\") pod \"maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888775-82nz4\" (UID: \"75a1834c-2859-4e9c-9d32-80a6f7247460\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888775-82nz4" Oct 30 02:15:00.560228 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.559613 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8jwh\" (UniqueName: \"kubernetes.io/projected/1c7f35b4-a13a-4be3-be70-135dcc096f49-kube-api-access-g8jwh\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888775-brm4t\" (UID: \"1c7f35b4-a13a-4be3-be70-135dcc096f49\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888775-brm4t" Oct 30 02:15:00.560228 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.559641 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888775-fxr88\" (UID: \"2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888775-fxr88" Oct 30 02:15:00.560228 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.559742 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99cb9b6a-51ec-41be-a85d-dd1de997fceb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888775-sv8kf\" (UID: \"99cb9b6a-51ec-41be-a85d-dd1de997fceb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888775-sv8kf" Oct 30 02:15:00.560228 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.559799 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcf5437e-0075-467b-a4fd-82b5dc563489-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-292018-29888775-289bf\" (UID: \"bcf5437e-0075-467b-a4fd-82b5dc563489\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888775-289bf" Oct 30 02:15:00.560228 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.559890 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gpqh\" (UniqueName: \"kubernetes.io/projected/39c584cd-b462-4d52-aa9d-786e9a0b8e94-kube-api-access-5gpqh\") pod \"maas-api-key-cleanup-e2e-inf-a-340ae6-29888775-8dfms\" (UID: \"39c584cd-b462-4d52-aa9d-786e9a0b8e94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-340ae6-29888775-8dfms" Oct 30 02:15:00.560228 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.559942 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgqdv\" (UniqueName: \"kubernetes.io/projected/2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90-kube-api-access-qgqdv\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888775-fxr88\" (UID: \"2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888775-fxr88" Oct 30 02:15:00.560228 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.560002 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9vdw\" (UniqueName: \"kubernetes.io/projected/bcf5437e-0075-467b-a4fd-82b5dc563489-kube-api-access-k9vdw\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-292018-29888775-289bf\" (UID: \"bcf5437e-0075-467b-a4fd-82b5dc563489\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888775-289bf" Oct 30 02:15:00.560228 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.560071 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbttd\" (UniqueName: \"kubernetes.io/projected/4ed6af0f-eb0e-41f8-8b21-654babbb98b5-kube-api-access-dbttd\") pod \"maas-api-key-cleanup-e2e-worker-w3-558327-29888775-djsrl\" (UID: \"4ed6af0f-eb0e-41f8-8b21-654babbb98b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888775-djsrl" Oct 30 02:15:00.560228 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.560100 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da67c0ab-cf74-4396-9987-b1bb46eb802b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888775-8xvvv\" (UID: \"da67c0ab-cf74-4396-9987-b1bb46eb802b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-8xvvv" Oct 30 02:15:00.560228 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.560142 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba0ac388-9127-47e0-8985-dd48570c8d27-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-093d23-29888775-vcqdq\" (UID: \"ba0ac388-9127-47e0-8985-dd48570c8d27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888775-vcqdq" Oct 30 02:15:00.560228 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.560204 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f89697-29888775-9m9lw\" (UID: \"63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f89697-29888775-9m9lw" Oct 30 02:15:00.561196 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.560251 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c584cd-b462-4d52-aa9d-786e9a0b8e94-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-340ae6-29888775-8dfms\" (UID: \"39c584cd-b462-4d52-aa9d-786e9a0b8e94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-340ae6-29888775-8dfms" Oct 30 02:15:00.561196 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.560322 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c62p2\" (UniqueName: \"kubernetes.io/projected/da67c0ab-cf74-4396-9987-b1bb46eb802b-kube-api-access-c62p2\") pod \"maas-api-key-cleanup-29888775-8xvvv\" (UID: \"da67c0ab-cf74-4396-9987-b1bb46eb802b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-8xvvv" Oct 30 02:15:00.561196 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.560355 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7f35b4-a13a-4be3-be70-135dcc096f49-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888775-brm4t\" (UID: \"1c7f35b4-a13a-4be3-be70-135dcc096f49\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888775-brm4t" Oct 30 02:15:00.561196 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.560411 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr8vv\" (UniqueName: \"kubernetes.io/projected/63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4-kube-api-access-nr8vv\") pod \"maas-api-key-cleanup-e2e-inf-a-f89697-29888775-9m9lw\" (UID: \"63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f89697-29888775-9m9lw" Oct 30 02:15:00.561196 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.560452 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ed6af0f-eb0e-41f8-8b21-654babbb98b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-558327-29888775-djsrl\" (UID: \"4ed6af0f-eb0e-41f8-8b21-654babbb98b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888775-djsrl" Oct 30 02:15:00.562359 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.562227 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcf5437e-0075-467b-a4fd-82b5dc563489-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-292018-29888775-289bf\" (UID: \"bcf5437e-0075-467b-a4fd-82b5dc563489\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888775-289bf" Oct 30 02:15:00.563007 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.562979 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ed6af0f-eb0e-41f8-8b21-654babbb98b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-558327-29888775-djsrl\" (UID: \"4ed6af0f-eb0e-41f8-8b21-654babbb98b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888775-djsrl" Oct 30 02:15:00.563240 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.563202 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7f35b4-a13a-4be3-be70-135dcc096f49-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888775-brm4t\" (UID: \"1c7f35b4-a13a-4be3-be70-135dcc096f49\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888775-brm4t" Oct 30 02:15:00.563677 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.563656 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba0ac388-9127-47e0-8985-dd48570c8d27-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-093d23-29888775-vcqdq\" (UID: \"ba0ac388-9127-47e0-8985-dd48570c8d27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888775-vcqdq" Oct 30 02:15:00.568122 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.567969 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888775-rbx7m" Oct 30 02:15:00.570360 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.570328 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99cb9b6a-51ec-41be-a85d-dd1de997fceb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888775-sv8kf\" (UID: \"99cb9b6a-51ec-41be-a85d-dd1de997fceb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888775-sv8kf" Oct 30 02:15:00.572974 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.572941 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="230f0407-5c25-44aa-9a97-1de91305e4f3" path="/var/lib/kubelet/pods/230f0407-5c25-44aa-9a97-1de91305e4f3/volumes" Oct 30 02:15:00.574501 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.574474 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f3448e6-c22d-466c-a485-fbe2e676c250" path="/var/lib/kubelet/pods/5f3448e6-c22d-466c-a485-fbe2e676c250/volumes" Oct 30 02:15:00.575776 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.575733 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcda177c-9d37-4f05-ac01-d99f9139f87e" path="/var/lib/kubelet/pods/dcda177c-9d37-4f05-ac01-d99f9139f87e/volumes" Oct 30 02:15:00.578280 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.576723 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1f0afe7-d54e-4577-857a-da30b84237de" path="/var/lib/kubelet/pods/e1f0afe7-d54e-4577-857a-da30b84237de/volumes" Oct 30 02:15:00.622136 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.621738 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-429l8\" (UniqueName: \"kubernetes.io/projected/ba0ac388-9127-47e0-8985-dd48570c8d27-kube-api-access-429l8\") pod \"maas-api-key-cleanup-e2e-extmod-b-093d23-29888775-vcqdq\" (UID: \"ba0ac388-9127-47e0-8985-dd48570c8d27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888775-vcqdq" Oct 30 02:15:00.635993 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.634540 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbttd\" (UniqueName: \"kubernetes.io/projected/4ed6af0f-eb0e-41f8-8b21-654babbb98b5-kube-api-access-dbttd\") pod \"maas-api-key-cleanup-e2e-worker-w3-558327-29888775-djsrl\" (UID: \"4ed6af0f-eb0e-41f8-8b21-654babbb98b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888775-djsrl" Oct 30 02:15:00.635993 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.635076 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888775-qr45g" Oct 30 02:15:00.635993 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.635198 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9vdw\" (UniqueName: \"kubernetes.io/projected/bcf5437e-0075-467b-a4fd-82b5dc563489-kube-api-access-k9vdw\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-292018-29888775-289bf\" (UID: \"bcf5437e-0075-467b-a4fd-82b5dc563489\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888775-289bf" Oct 30 02:15:00.635993 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.635242 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs7rx\" (UniqueName: \"kubernetes.io/projected/99cb9b6a-51ec-41be-a85d-dd1de997fceb-kube-api-access-bs7rx\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888775-sv8kf\" (UID: \"99cb9b6a-51ec-41be-a85d-dd1de997fceb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888775-sv8kf" Oct 30 02:15:00.635993 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.635844 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8jwh\" (UniqueName: \"kubernetes.io/projected/1c7f35b4-a13a-4be3-be70-135dcc096f49-kube-api-access-g8jwh\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888775-brm4t\" (UID: \"1c7f35b4-a13a-4be3-be70-135dcc096f49\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888775-brm4t" Oct 30 02:15:00.635993 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.635900 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888775-ttncp" Oct 30 02:15:00.646117 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.646051 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888775-wrhfw" Oct 30 02:15:00.662957 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.662077 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d5aa9-29888775-9pqsc" Oct 30 02:15:00.662957 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.662217 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gpqh\" (UniqueName: \"kubernetes.io/projected/39c584cd-b462-4d52-aa9d-786e9a0b8e94-kube-api-access-5gpqh\") pod \"maas-api-key-cleanup-e2e-inf-a-340ae6-29888775-8dfms\" (UID: \"39c584cd-b462-4d52-aa9d-786e9a0b8e94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-340ae6-29888775-8dfms" Oct 30 02:15:00.662957 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.662276 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgqdv\" (UniqueName: \"kubernetes.io/projected/2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90-kube-api-access-qgqdv\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888775-fxr88\" (UID: \"2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888775-fxr88" Oct 30 02:15:00.662957 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.662360 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da67c0ab-cf74-4396-9987-b1bb46eb802b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888775-8xvvv\" (UID: \"da67c0ab-cf74-4396-9987-b1bb46eb802b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-8xvvv" Oct 30 02:15:00.662957 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.662415 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f89697-29888775-9m9lw\" (UID: \"63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f89697-29888775-9m9lw" Oct 30 02:15:00.662957 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.662450 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c584cd-b462-4d52-aa9d-786e9a0b8e94-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-340ae6-29888775-8dfms\" (UID: \"39c584cd-b462-4d52-aa9d-786e9a0b8e94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-340ae6-29888775-8dfms" Oct 30 02:15:00.662957 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.662509 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c62p2\" (UniqueName: \"kubernetes.io/projected/da67c0ab-cf74-4396-9987-b1bb46eb802b-kube-api-access-c62p2\") pod \"maas-api-key-cleanup-29888775-8xvvv\" (UID: \"da67c0ab-cf74-4396-9987-b1bb46eb802b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-8xvvv" Oct 30 02:15:00.662957 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.662575 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr8vv\" (UniqueName: \"kubernetes.io/projected/63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4-kube-api-access-nr8vv\") pod \"maas-api-key-cleanup-e2e-inf-a-f89697-29888775-9m9lw\" (UID: \"63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f89697-29888775-9m9lw" Oct 30 02:15:00.662957 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.662676 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a1834c-2859-4e9c-9d32-80a6f7247460-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888775-82nz4\" (UID: \"75a1834c-2859-4e9c-9d32-80a6f7247460\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888775-82nz4" Oct 30 02:15:00.662957 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.662812 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8ghh\" (UniqueName: \"kubernetes.io/projected/75a1834c-2859-4e9c-9d32-80a6f7247460-kube-api-access-j8ghh\") pod \"maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888775-82nz4\" (UID: \"75a1834c-2859-4e9c-9d32-80a6f7247460\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888775-82nz4" Oct 30 02:15:00.662957 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.662853 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888775-fxr88\" (UID: \"2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888775-fxr88" Oct 30 02:15:00.663711 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.663456 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f89697-29888775-9m9lw\" (UID: \"63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f89697-29888775-9m9lw" Oct 30 02:15:00.663817 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.663742 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888775-fxr88\" (UID: \"2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888775-fxr88" Oct 30 02:15:00.664160 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.663919 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da67c0ab-cf74-4396-9987-b1bb46eb802b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888775-8xvvv\" (UID: \"da67c0ab-cf74-4396-9987-b1bb46eb802b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-8xvvv" Oct 30 02:15:00.664636 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.664581 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c584cd-b462-4d52-aa9d-786e9a0b8e94-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-340ae6-29888775-8dfms\" (UID: \"39c584cd-b462-4d52-aa9d-786e9a0b8e94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-340ae6-29888775-8dfms" Oct 30 02:15:00.665181 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.665123 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a1834c-2859-4e9c-9d32-80a6f7247460-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888775-82nz4\" (UID: \"75a1834c-2859-4e9c-9d32-80a6f7247460\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888775-82nz4" Oct 30 02:15:00.683725 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.683678 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gpqh\" (UniqueName: \"kubernetes.io/projected/39c584cd-b462-4d52-aa9d-786e9a0b8e94-kube-api-access-5gpqh\") pod \"maas-api-key-cleanup-e2e-inf-a-340ae6-29888775-8dfms\" (UID: \"39c584cd-b462-4d52-aa9d-786e9a0b8e94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-340ae6-29888775-8dfms" Oct 30 02:15:00.686283 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.686180 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8ghh\" (UniqueName: \"kubernetes.io/projected/75a1834c-2859-4e9c-9d32-80a6f7247460-kube-api-access-j8ghh\") pod \"maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888775-82nz4\" (UID: \"75a1834c-2859-4e9c-9d32-80a6f7247460\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888775-82nz4" Oct 30 02:15:00.708712 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.708680 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c62p2\" (UniqueName: \"kubernetes.io/projected/da67c0ab-cf74-4396-9987-b1bb46eb802b-kube-api-access-c62p2\") pod \"maas-api-key-cleanup-29888775-8xvvv\" (UID: \"da67c0ab-cf74-4396-9987-b1bb46eb802b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-8xvvv" Oct 30 02:15:00.710528 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.710491 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888775-r46v9" Oct 30 02:15:00.710847 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.710824 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgqdv\" (UniqueName: \"kubernetes.io/projected/2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90-kube-api-access-qgqdv\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888775-fxr88\" (UID: \"2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888775-fxr88" Oct 30 02:15:00.714224 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.712357 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr8vv\" (UniqueName: \"kubernetes.io/projected/63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4-kube-api-access-nr8vv\") pod \"maas-api-key-cleanup-e2e-inf-a-f89697-29888775-9m9lw\" (UID: \"63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f89697-29888775-9m9lw" Oct 30 02:15:00.724091 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.724052 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888775-p4j7s" Oct 30 02:15:00.728298 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.728186 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888775-8tzkk" Oct 30 02:15:00.739481 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.739408 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888775-sv8kf" Oct 30 02:15:00.747585 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.747495 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888775-djsrl" Oct 30 02:15:00.759596 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.759339 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888775-289bf" Oct 30 02:15:00.772969 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.772508 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888775-vcqdq" Oct 30 02:15:00.779998 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.779815 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888775-brm4t" Oct 30 02:15:00.808812 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.808491 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-8xvvv" Oct 30 02:15:00.822052 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.822027 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l"] Oct 30 02:15:00.874161 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.873989 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888775-fxr88" Oct 30 02:15:00.953265 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.952175 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888775-82nz4" Oct 30 02:15:00.953265 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.952664 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-340ae6-29888775-8dfms" Oct 30 02:15:00.966926 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:00.966893 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f89697-29888775-9m9lw" Oct 30 02:15:01.083689 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:01.081585 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888775-zd7gh"] Oct 30 02:15:01.155635 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:01.155580 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888775-rbx7m"] Oct 30 02:15:01.182225 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:15:01.182184 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd66a87ee_dc89_4aba_845b_17b58bc6f4a6.slice/crio-2f7e2f9ebead493b271ecd7fe9b8088c4b738a589927d851fae515a519dcfb60 WatchSource:0}: Error finding container 2f7e2f9ebead493b271ecd7fe9b8088c4b738a589927d851fae515a519dcfb60: Status 404 returned error can't find the container with id 2f7e2f9ebead493b271ecd7fe9b8088c4b738a589927d851fae515a519dcfb60 Oct 30 02:15:01.241167 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:01.241133 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888775-qr45g"] Oct 30 02:15:01.250237 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:01.250148 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888775-wrhfw"] Oct 30 02:15:01.304307 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:15:01.304211 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2bd8bc3_6252_42fc_9055_ca393b6f45d9.slice/crio-ec71cf41ebcc993fe1be28e8f90b4bcaf5eae31b989edeba11e3dd9cd35c4805 WatchSource:0}: Error finding container ec71cf41ebcc993fe1be28e8f90b4bcaf5eae31b989edeba11e3dd9cd35c4805: Status 404 returned error can't find the container with id ec71cf41ebcc993fe1be28e8f90b4bcaf5eae31b989edeba11e3dd9cd35c4805 Oct 30 02:15:01.424351 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:01.424309 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d5aa9-29888775-9pqsc"] Oct 30 02:15:01.424701 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:01.424665 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888775-r46v9"] Oct 30 02:15:01.461735 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:01.461579 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:01.487237 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:15:01.487202 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec1f4d7_877e_4d7c_88d2_f2365469869a.slice/crio-4e47374c2d0eac041e284c717257c2ac6c06abf3571d8e9195fd23922e876c97 WatchSource:0}: Error finding container 4e47374c2d0eac041e284c717257c2ac6c06abf3571d8e9195fd23922e876c97: Status 404 returned error can't find the container with id 4e47374c2d0eac041e284c717257c2ac6c06abf3571d8e9195fd23922e876c97 Oct 30 02:15:01.551953 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:01.551927 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888775-ttncp"] Oct 30 02:15:01.558872 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:15:01.558837 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf641ac83_f4e6_4003_ae9c_be79e4ca0337.slice/crio-2345a6bc6e87c1c1af36c9f81f0dc171a37b7888e8ff92182213681d0ea93f3e WatchSource:0}: Error finding container 2345a6bc6e87c1c1af36c9f81f0dc171a37b7888e8ff92182213681d0ea93f3e: Status 404 returned error can't find the container with id 2345a6bc6e87c1c1af36c9f81f0dc171a37b7888e8ff92182213681d0ea93f3e Oct 30 02:15:01.654761 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:01.654723 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888775-289bf"] Oct 30 02:15:01.657799 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:01.657727 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d5aa9-29888775-9pqsc" event={"ID":"fec1f4d7-877e-4d7c-88d2-f2365469869a","Type":"ContainerStarted","Data":"4e47374c2d0eac041e284c717257c2ac6c06abf3571d8e9195fd23922e876c97"} Oct 30 02:15:01.659344 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:01.659316 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888775-ttncp" event={"ID":"f641ac83-f4e6-4003-ae9c-be79e4ca0337","Type":"ContainerStarted","Data":"2345a6bc6e87c1c1af36c9f81f0dc171a37b7888e8ff92182213681d0ea93f3e"} Oct 30 02:15:01.659918 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:01.659896 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888775-sv8kf"] Oct 30 02:15:01.660638 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:01.660613 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888775-r46v9" event={"ID":"6edd5565-a917-481f-9302-95011124898b","Type":"ContainerStarted","Data":"717afdbbc6f03ed9048ec080972cab4bd269e8bd1a4693f41ca170f087a02d54"} Oct 30 02:15:01.662052 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:01.662027 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888775-zd7gh" event={"ID":"d66a87ee-dc89-4aba-845b-17b58bc6f4a6","Type":"ContainerStarted","Data":"2f7e2f9ebead493b271ecd7fe9b8088c4b738a589927d851fae515a519dcfb60"} Oct 30 02:15:01.663431 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:01.663405 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888775-rbx7m" event={"ID":"0dc3a619-596a-4e28-9845-9d1e2fc7bb06","Type":"ContainerStarted","Data":"e7d2c5bba25f429f6a1beb16c42dbe23129791b7f1ad14d01c21c90beb1b8fdd"} Oct 30 02:15:01.664824 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:01.664803 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888775-wrhfw" event={"ID":"c2bd8bc3-6252-42fc-9055-ca393b6f45d9","Type":"ContainerStarted","Data":"ec71cf41ebcc993fe1be28e8f90b4bcaf5eae31b989edeba11e3dd9cd35c4805"} Oct 30 02:15:01.666221 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:01.666194 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888775-qr45g" event={"ID":"1f68c408-6bab-42c7-b574-35d5d61c7544","Type":"ContainerStarted","Data":"9e54a53dbe0ef8f9b289bf176e03d43cb9c25cd7e54da999e134d42aee01f510"} Oct 30 02:15:01.667608 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:01.667584 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l" event={"ID":"3f9f8331-2090-4262-96a0-c3ea360e4375","Type":"ContainerStarted","Data":"1fd8f2534ee04aca8b594031bf5cb98952724ec4cef18b7432af606aae45e65e"} Oct 30 02:15:01.719655 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:01.719620 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888775-p4j7s"] Oct 30 02:15:01.730470 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:01.730312 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888775-djsrl"] Oct 30 02:15:01.778249 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:01.778212 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888775-8tzkk"] Oct 30 02:15:01.783851 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:01.783818 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888775-vcqdq"] Oct 30 02:15:01.814442 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:15:01.814102 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99cb9b6a_51ec_41be_a85d_dd1de997fceb.slice/crio-88319ab8fe3cd6d70a99a1fc44e10ef5ed3463212188c659195199abca4e3825 WatchSource:0}: Error finding container 88319ab8fe3cd6d70a99a1fc44e10ef5ed3463212188c659195199abca4e3825: Status 404 returned error can't find the container with id 88319ab8fe3cd6d70a99a1fc44e10ef5ed3463212188c659195199abca4e3825 Oct 30 02:15:01.816173 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:15:01.815511 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8968e7e0_04c1_4469_9062_f7d31ee6dbf2.slice/crio-09bd292cbadc3f620fca3ab083ecdd96e764b1430a3aa7c16e6774be0915aacf WatchSource:0}: Error finding container 09bd292cbadc3f620fca3ab083ecdd96e764b1430a3aa7c16e6774be0915aacf: Status 404 returned error can't find the container with id 09bd292cbadc3f620fca3ab083ecdd96e764b1430a3aa7c16e6774be0915aacf Oct 30 02:15:01.829500 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:15:01.829461 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ed6af0f_eb0e_41f8_8b21_654babbb98b5.slice/crio-fdb4778917a6ea1ac9c1c6c8bbff9bbc02c85aebf8cfe87d28c11f53919f80bb WatchSource:0}: Error finding container fdb4778917a6ea1ac9c1c6c8bbff9bbc02c85aebf8cfe87d28c11f53919f80bb: Status 404 returned error can't find the container with id fdb4778917a6ea1ac9c1c6c8bbff9bbc02c85aebf8cfe87d28c11f53919f80bb Oct 30 02:15:01.876643 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:01.876604 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-340ae6-29888775-8dfms"] Oct 30 02:15:01.889634 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:01.889603 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888775-brm4t"] Oct 30 02:15:01.894704 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:01.894674 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888775-8xvvv"] Oct 30 02:15:01.902849 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:01.902175 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888775-82nz4"] Oct 30 02:15:01.922791 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:15:01.922710 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdf23f37_2870_4b9c_b558_9fea4452f4e5.slice/crio-e9ef79542a91f5025ef25f8360f6842ede7843bb33c7c6bd526433c48c0a50dc WatchSource:0}: Error finding container e9ef79542a91f5025ef25f8360f6842ede7843bb33c7c6bd526433c48c0a50dc: Status 404 returned error can't find the container with id e9ef79542a91f5025ef25f8360f6842ede7843bb33c7c6bd526433c48c0a50dc Oct 30 02:15:01.925311 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:01.925279 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888775-fxr88"] Oct 30 02:15:01.959441 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:01.959402 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f89697-29888775-9m9lw"] Oct 30 02:15:02.678991 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:02.678948 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888775-fxr88" event={"ID":"2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90","Type":"ContainerStarted","Data":"a65f7cac80d0eba8df9535e5a73b788f2cc54b13911f08dc9aaa3726850336ed"} Oct 30 02:15:02.680724 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:02.680685 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d5aa9-29888775-9pqsc" event={"ID":"fec1f4d7-877e-4d7c-88d2-f2365469869a","Type":"ContainerStarted","Data":"a276b3d2e8c7d8b8955d74d52c10bde6cdd982a3fe8338d70cc2de73fb2cca81"} Oct 30 02:15:02.682230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:02.682204 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888775-p4j7s" event={"ID":"8968e7e0-04c1-4469-9062-f7d31ee6dbf2","Type":"ContainerStarted","Data":"09bd292cbadc3f620fca3ab083ecdd96e764b1430a3aa7c16e6774be0915aacf"} Oct 30 02:15:02.683827 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:02.683801 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888775-djsrl" event={"ID":"4ed6af0f-eb0e-41f8-8b21-654babbb98b5","Type":"ContainerStarted","Data":"fe0e2f6e7c80e7b7dcac15924ec8e19f781456bcc6695d72b688df839d541189"} Oct 30 02:15:02.683923 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:02.683842 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888775-djsrl" event={"ID":"4ed6af0f-eb0e-41f8-8b21-654babbb98b5","Type":"ContainerStarted","Data":"fdb4778917a6ea1ac9c1c6c8bbff9bbc02c85aebf8cfe87d28c11f53919f80bb"} Oct 30 02:15:02.685527 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:02.685499 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888775-ttncp" event={"ID":"f641ac83-f4e6-4003-ae9c-be79e4ca0337","Type":"ContainerStarted","Data":"12bd7bbe8963240ffe8948b0ebe2f27ecb470350c34adbed0a4d4f82b76222f3"} Oct 30 02:15:02.687036 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:02.687013 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888775-r46v9" event={"ID":"6edd5565-a917-481f-9302-95011124898b","Type":"ContainerStarted","Data":"4ba50b3f1a15067ef0ccd3b01d1921bdc8ca94d827c6cbe2488b718a1b1cf210"} Oct 30 02:15:02.688665 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:02.688643 6572 generic.go:356] "Generic (PLEG): container finished" podID="d66a87ee-dc89-4aba-845b-17b58bc6f4a6" containerID="6f04373be7f7c6bcb6bca493602d4d5e6a45d83bd8dd5f00685bd55544daa115" exitCode=60 Oct 30 02:15:02.688823 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:02.688726 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888775-zd7gh" event={"ID":"d66a87ee-dc89-4aba-845b-17b58bc6f4a6","Type":"ContainerDied","Data":"6f04373be7f7c6bcb6bca493602d4d5e6a45d83bd8dd5f00685bd55544daa115"} Oct 30 02:15:02.688981 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:02.688963 6572 scope.go:122] "RemoveContainer" containerID="6f04373be7f7c6bcb6bca493602d4d5e6a45d83bd8dd5f00685bd55544daa115" Oct 30 02:15:02.690202 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:02.690179 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888775-82nz4" event={"ID":"75a1834c-2859-4e9c-9d32-80a6f7247460","Type":"ContainerStarted","Data":"4b756c8f54c4132b4d4fc77b83ebc5c6f39857aa48d98e409b80813a60e5c2a0"} Oct 30 02:15:02.691626 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:02.691604 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-8xvvv" event={"ID":"da67c0ab-cf74-4396-9987-b1bb46eb802b","Type":"ContainerStarted","Data":"71096c36965ec8e0269cb84d48fd7651da2cf9fa7b077d93068d5d61d858a3cb"} Oct 30 02:15:02.693327 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:02.693293 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888775-289bf" event={"ID":"bcf5437e-0075-467b-a4fd-82b5dc563489","Type":"ContainerStarted","Data":"04a6361f8f76d35b905d3f42e91e672dfb96996087b3b21a53956635d01c23c7"} Oct 30 02:15:02.693442 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:02.693337 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888775-289bf" event={"ID":"bcf5437e-0075-467b-a4fd-82b5dc563489","Type":"ContainerStarted","Data":"94458f05692f600a1874158107980c98a5cf5c3ccae02ae8a0d474bf7bc8a67d"} Oct 30 02:15:02.695095 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:02.695071 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888775-rbx7m" event={"ID":"0dc3a619-596a-4e28-9845-9d1e2fc7bb06","Type":"ContainerStarted","Data":"36fa10c7c9130d463f5577e0e1e829cabe1fbf8e4c172891a482e8351943439f"} Oct 30 02:15:02.696831 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:02.696808 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888775-wrhfw" event={"ID":"c2bd8bc3-6252-42fc-9055-ca393b6f45d9","Type":"ContainerStarted","Data":"a90d95d04c5ce67c97e5bbd29437b4d4fcf32cf96776f0e858dbe685ba44f32f"} Oct 30 02:15:02.698557 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:02.698531 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888775-sv8kf" event={"ID":"99cb9b6a-51ec-41be-a85d-dd1de997fceb","Type":"ContainerStarted","Data":"88319ab8fe3cd6d70a99a1fc44e10ef5ed3463212188c659195199abca4e3825"} Oct 30 02:15:02.717119 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:02.713475 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f89697-29888775-9m9lw" event={"ID":"63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4","Type":"ContainerStarted","Data":"0bcdb372955770ff64b167fe858eda04982d2e3d04340d8b1f78a3df98e090e2"} Oct 30 02:15:02.724252 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:02.724213 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888775-qr45g" event={"ID":"1f68c408-6bab-42c7-b574-35d5d61c7544","Type":"ContainerStarted","Data":"6b7733459bee22eee391b66479f95617de1a6b67c3790371243022827576c3a3"} Oct 30 02:15:02.726184 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:02.725980 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d5aa9-29888775-9pqsc" podStartSLOduration=2.725957794 podStartE2EDuration="2.725957794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:02.721923422 +0000 UTC m=+3428.298348824" watchObservedRunningTime="2026-10-30 02:15:02.725957794 +0000 UTC m=+3428.302383199" Oct 30 02:15:02.728815 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:02.728642 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888775-brm4t" event={"ID":"1c7f35b4-a13a-4be3-be70-135dcc096f49","Type":"ContainerStarted","Data":"e855dd0ae04414dc7176c2eeb4d24551dd761eb42e21625ae4a42ed61e8147d8"} Oct 30 02:15:02.730878 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:02.730834 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888775-vcqdq" event={"ID":"ba0ac388-9127-47e0-8985-dd48570c8d27","Type":"ContainerStarted","Data":"33f70d3bbb5dea11936ce04f726110c3bd79072ad1c1a2f9dc95c2cb2dbc81c9"} Oct 30 02:15:02.886713 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:02.884563 6572 generic.go:356] "Generic (PLEG): container finished" podID="3f9f8331-2090-4262-96a0-c3ea360e4375" containerID="73b42970353506c9e1bd367e02f592fb0f50124a0aacc8a6b82083357638c266" exitCode=60 Oct 30 02:15:02.886713 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:02.884664 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l" event={"ID":"3f9f8331-2090-4262-96a0-c3ea360e4375","Type":"ContainerDied","Data":"73b42970353506c9e1bd367e02f592fb0f50124a0aacc8a6b82083357638c266"} Oct 30 02:15:02.886713 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:02.885269 6572 scope.go:122] "RemoveContainer" containerID="73b42970353506c9e1bd367e02f592fb0f50124a0aacc8a6b82083357638c266" Oct 30 02:15:02.892978 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:02.891803 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-340ae6-29888775-8dfms" event={"ID":"39c584cd-b462-4d52-aa9d-786e9a0b8e94","Type":"ContainerStarted","Data":"99db59d61ccb475091549443699d5a892a46f82bef091b1f36b0dbedeb707422"} Oct 30 02:15:02.982900 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:02.934896 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888775-ttncp" podStartSLOduration=2.934875703 podStartE2EDuration="2.934875703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:02.801741536 +0000 UTC m=+3428.378166939" watchObservedRunningTime="2026-10-30 02:15:02.934875703 +0000 UTC m=+3428.511301110" Oct 30 02:15:02.982900 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:02.938629 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888775-8tzkk" event={"ID":"bdf23f37-2870-4b9c-b558-9fea4452f4e5","Type":"ContainerStarted","Data":"e9ef79542a91f5025ef25f8360f6842ede7843bb33c7c6bd526433c48c0a50dc"} Oct 30 02:15:02.983473 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:02.983396 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888775-rbx7m" podStartSLOduration=2.98337655 podStartE2EDuration="2.98337655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:02.872174409 +0000 UTC m=+3428.448599812" watchObservedRunningTime="2026-10-30 02:15:02.98337655 +0000 UTC m=+3428.559801958" Oct 30 02:15:02.990543 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:02.990491 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888775-289bf" podStartSLOduration=2.99047558 podStartE2EDuration="2.99047558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:02.915500261 +0000 UTC m=+3428.491925664" watchObservedRunningTime="2026-10-30 02:15:02.99047558 +0000 UTC m=+3428.566900982" Oct 30 02:15:03.001053 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.000996 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888775-r46v9" podStartSLOduration=3.000978454 podStartE2EDuration="3.000978454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:02.9799772 +0000 UTC m=+3428.556402602" watchObservedRunningTime="2026-10-30 02:15:03.000978454 +0000 UTC m=+3428.577403898" Oct 30 02:15:03.034006 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.029689 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888775-wrhfw" podStartSLOduration=3.029666989 podStartE2EDuration="3.029666989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:03.028642748 +0000 UTC m=+3428.605068152" watchObservedRunningTime="2026-10-30 02:15:03.029666989 +0000 UTC m=+3428.606092392" Oct 30 02:15:03.063493 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.063397 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:03.134541 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.134471 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888775-djsrl" podStartSLOduration=3.134446311 podStartE2EDuration="3.134446311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:03.125501938 +0000 UTC m=+3428.701927342" watchObservedRunningTime="2026-10-30 02:15:03.134446311 +0000 UTC m=+3428.710871717" Oct 30 02:15:03.167312 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.167184 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888775-qr45g" podStartSLOduration=3.167163127 podStartE2EDuration="3.167163127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:03.166673513 +0000 UTC m=+3428.743098919" watchObservedRunningTime="2026-10-30 02:15:03.167163127 +0000 UTC m=+3428.743588531" Oct 30 02:15:03.951039 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.951006 6572 generic.go:356] "Generic (PLEG): container finished" podID="c2bd8bc3-6252-42fc-9055-ca393b6f45d9" containerID="a90d95d04c5ce67c97e5bbd29437b4d4fcf32cf96776f0e858dbe685ba44f32f" exitCode=60 Oct 30 02:15:03.951560 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.951084 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888775-wrhfw" event={"ID":"c2bd8bc3-6252-42fc-9055-ca393b6f45d9","Type":"ContainerDied","Data":"a90d95d04c5ce67c97e5bbd29437b4d4fcf32cf96776f0e858dbe685ba44f32f"} Oct 30 02:15:03.951560 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.951412 6572 scope.go:122] "RemoveContainer" containerID="a90d95d04c5ce67c97e5bbd29437b4d4fcf32cf96776f0e858dbe685ba44f32f" Oct 30 02:15:03.953023 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.952995 6572 generic.go:356] "Generic (PLEG): container finished" podID="99cb9b6a-51ec-41be-a85d-dd1de997fceb" containerID="c6558b2ef9872a2b27c682d668c8ffd383757ec0c5b0ddf9fef552e06c8e2221" exitCode=60 Oct 30 02:15:03.953152 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.953072 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888775-sv8kf" event={"ID":"99cb9b6a-51ec-41be-a85d-dd1de997fceb","Type":"ContainerDied","Data":"c6558b2ef9872a2b27c682d668c8ffd383757ec0c5b0ddf9fef552e06c8e2221"} Oct 30 02:15:03.953341 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.953317 6572 scope.go:122] "RemoveContainer" containerID="c6558b2ef9872a2b27c682d668c8ffd383757ec0c5b0ddf9fef552e06c8e2221" Oct 30 02:15:03.959314 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.956367 6572 generic.go:356] "Generic (PLEG): container finished" podID="1f68c408-6bab-42c7-b574-35d5d61c7544" containerID="6b7733459bee22eee391b66479f95617de1a6b67c3790371243022827576c3a3" exitCode=60 Oct 30 02:15:03.959314 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.956584 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888775-qr45g" event={"ID":"1f68c408-6bab-42c7-b574-35d5d61c7544","Type":"ContainerDied","Data":"6b7733459bee22eee391b66479f95617de1a6b67c3790371243022827576c3a3"} Oct 30 02:15:03.959314 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.957158 6572 scope.go:122] "RemoveContainer" containerID="6b7733459bee22eee391b66479f95617de1a6b67c3790371243022827576c3a3" Oct 30 02:15:03.968084 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.962679 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l" event={"ID":"3f9f8331-2090-4262-96a0-c3ea360e4375","Type":"ContainerStarted","Data":"5e7cce123f5ce292cfeb86b365cefa77af1040bb35f2fee03af09fcfada18ec1"} Oct 30 02:15:03.973244 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.973205 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-340ae6-29888775-8dfms" event={"ID":"39c584cd-b462-4d52-aa9d-786e9a0b8e94","Type":"ContainerStarted","Data":"b71da30be4bf2256d5903fd0bcff383816f1fa933d7d8c503438a7d19c7cf60a"} Oct 30 02:15:03.976527 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.975792 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888775-8tzkk" event={"ID":"bdf23f37-2870-4b9c-b558-9fea4452f4e5","Type":"ContainerStarted","Data":"f44b76b6541726c9bc7153dd8fefeee3ceee298a5edbcfcd6f2014ea1a275f44"} Oct 30 02:15:03.978110 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.978079 6572 generic.go:356] "Generic (PLEG): container finished" podID="fec1f4d7-877e-4d7c-88d2-f2365469869a" containerID="a276b3d2e8c7d8b8955d74d52c10bde6cdd982a3fe8338d70cc2de73fb2cca81" exitCode=60 Oct 30 02:15:03.978256 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.978154 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d5aa9-29888775-9pqsc" event={"ID":"fec1f4d7-877e-4d7c-88d2-f2365469869a","Type":"ContainerDied","Data":"a276b3d2e8c7d8b8955d74d52c10bde6cdd982a3fe8338d70cc2de73fb2cca81"} Oct 30 02:15:03.978422 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.978404 6572 scope.go:122] "RemoveContainer" containerID="a276b3d2e8c7d8b8955d74d52c10bde6cdd982a3fe8338d70cc2de73fb2cca81" Oct 30 02:15:03.980091 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.980065 6572 generic.go:356] "Generic (PLEG): container finished" podID="8968e7e0-04c1-4469-9062-f7d31ee6dbf2" containerID="3534e89f4bcf1c0cf11400a615614eebf65760dc201a09b380761383079c811b" exitCode=60 Oct 30 02:15:03.980201 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.980151 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888775-p4j7s" event={"ID":"8968e7e0-04c1-4469-9062-f7d31ee6dbf2","Type":"ContainerDied","Data":"3534e89f4bcf1c0cf11400a615614eebf65760dc201a09b380761383079c811b"} Oct 30 02:15:03.980421 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.980405 6572 scope.go:122] "RemoveContainer" containerID="3534e89f4bcf1c0cf11400a615614eebf65760dc201a09b380761383079c811b" Oct 30 02:15:03.982556 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.982536 6572 generic.go:356] "Generic (PLEG): container finished" podID="4ed6af0f-eb0e-41f8-8b21-654babbb98b5" containerID="fe0e2f6e7c80e7b7dcac15924ec8e19f781456bcc6695d72b688df839d541189" exitCode=60 Oct 30 02:15:03.982657 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.982624 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888775-djsrl" event={"ID":"4ed6af0f-eb0e-41f8-8b21-654babbb98b5","Type":"ContainerDied","Data":"fe0e2f6e7c80e7b7dcac15924ec8e19f781456bcc6695d72b688df839d541189"} Oct 30 02:15:03.983221 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.983198 6572 scope.go:122] "RemoveContainer" containerID="fe0e2f6e7c80e7b7dcac15924ec8e19f781456bcc6695d72b688df839d541189" Oct 30 02:15:03.984556 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.984534 6572 generic.go:356] "Generic (PLEG): container finished" podID="f641ac83-f4e6-4003-ae9c-be79e4ca0337" containerID="12bd7bbe8963240ffe8948b0ebe2f27ecb470350c34adbed0a4d4f82b76222f3" exitCode=60 Oct 30 02:15:03.984663 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.984607 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888775-ttncp" event={"ID":"f641ac83-f4e6-4003-ae9c-be79e4ca0337","Type":"ContainerDied","Data":"12bd7bbe8963240ffe8948b0ebe2f27ecb470350c34adbed0a4d4f82b76222f3"} Oct 30 02:15:03.984868 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.984853 6572 scope.go:122] "RemoveContainer" containerID="12bd7bbe8963240ffe8948b0ebe2f27ecb470350c34adbed0a4d4f82b76222f3" Oct 30 02:15:03.987810 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.987780 6572 generic.go:356] "Generic (PLEG): container finished" podID="6edd5565-a917-481f-9302-95011124898b" containerID="4ba50b3f1a15067ef0ccd3b01d1921bdc8ca94d827c6cbe2488b718a1b1cf210" exitCode=60 Oct 30 02:15:03.987937 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.987822 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888775-r46v9" event={"ID":"6edd5565-a917-481f-9302-95011124898b","Type":"ContainerDied","Data":"4ba50b3f1a15067ef0ccd3b01d1921bdc8ca94d827c6cbe2488b718a1b1cf210"} Oct 30 02:15:03.988138 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.988119 6572 scope.go:122] "RemoveContainer" containerID="4ba50b3f1a15067ef0ccd3b01d1921bdc8ca94d827c6cbe2488b718a1b1cf210" Oct 30 02:15:03.991239 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.991206 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-8xvvv" event={"ID":"da67c0ab-cf74-4396-9987-b1bb46eb802b","Type":"ContainerStarted","Data":"09d8f11bf5adb588ca64b685653e818d9a216c17e52385faf11b5cbec7f6845a"} Oct 30 02:15:03.993412 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.993384 6572 generic.go:356] "Generic (PLEG): container finished" podID="bcf5437e-0075-467b-a4fd-82b5dc563489" containerID="04a6361f8f76d35b905d3f42e91e672dfb96996087b3b21a53956635d01c23c7" exitCode=60 Oct 30 02:15:03.993543 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.993481 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888775-289bf" event={"ID":"bcf5437e-0075-467b-a4fd-82b5dc563489","Type":"ContainerDied","Data":"04a6361f8f76d35b905d3f42e91e672dfb96996087b3b21a53956635d01c23c7"} Oct 30 02:15:03.993862 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.993841 6572 scope.go:122] "RemoveContainer" containerID="04a6361f8f76d35b905d3f42e91e672dfb96996087b3b21a53956635d01c23c7" Oct 30 02:15:03.995395 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.995366 6572 generic.go:356] "Generic (PLEG): container finished" podID="0dc3a619-596a-4e28-9845-9d1e2fc7bb06" containerID="36fa10c7c9130d463f5577e0e1e829cabe1fbf8e4c172891a482e8351943439f" exitCode=60 Oct 30 02:15:03.995513 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.995428 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888775-rbx7m" event={"ID":"0dc3a619-596a-4e28-9845-9d1e2fc7bb06","Type":"ContainerDied","Data":"36fa10c7c9130d463f5577e0e1e829cabe1fbf8e4c172891a482e8351943439f"} Oct 30 02:15:03.995714 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:03.995695 6572 scope.go:122] "RemoveContainer" containerID="36fa10c7c9130d463f5577e0e1e829cabe1fbf8e4c172891a482e8351943439f" Oct 30 02:15:04.035241 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:04.035179 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888775-8tzkk" podStartSLOduration=4.035161529 podStartE2EDuration="4.035161529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:04.032280126 +0000 UTC m=+3429.608705530" watchObservedRunningTime="2026-10-30 02:15:04.035161529 +0000 UTC m=+3429.611586932" Oct 30 02:15:04.151872 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:04.151811 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-340ae6-29888775-8dfms" podStartSLOduration=4.151790279 podStartE2EDuration="4.151790279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:04.148720548 +0000 UTC m=+3429.725145951" watchObservedRunningTime="2026-10-30 02:15:04.151790279 +0000 UTC m=+3429.728215682" Oct 30 02:15:04.189670 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:04.189557 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l" podStartSLOduration=4.18954115 podStartE2EDuration="4.18954115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:04.185816398 +0000 UTC m=+3429.762241800" watchObservedRunningTime="2026-10-30 02:15:04.18954115 +0000 UTC m=+3429.765966552" Oct 30 02:15:04.415829 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:04.415788 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-292018-6fc6bf57c6-nslxv" podUID="cd10ed97-4e22-4efe-a0c6-7cff1ca2f157" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:15:04.416122 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:04.416094 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-292018-f884cbb67-mvk86" podUID="d0a98e88-a93c-462b-b2f5-68c61b386789" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:15:04.456070 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:04.455960 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-8xvvv" podStartSLOduration=4.455942875 podStartE2EDuration="4.455942875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:04.452052663 +0000 UTC m=+3430.028478067" watchObservedRunningTime="2026-10-30 02:15:04.455942875 +0000 UTC m=+3430.032368255" Oct 30 02:15:04.877418 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:04.877383 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:05.105276 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.097826 6572 generic.go:356] "Generic (PLEG): container finished" podID="75a1834c-2859-4e9c-9d32-80a6f7247460" containerID="98a99b3c07c1a577805f139b2f30ef7997d82d2ea47808d6564af9591cc6886a" exitCode=60 Oct 30 02:15:05.105276 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.097919 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888775-82nz4" event={"ID":"75a1834c-2859-4e9c-9d32-80a6f7247460","Type":"ContainerDied","Data":"98a99b3c07c1a577805f139b2f30ef7997d82d2ea47808d6564af9591cc6886a"} Oct 30 02:15:05.105276 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.098875 6572 scope.go:122] "RemoveContainer" containerID="98a99b3c07c1a577805f139b2f30ef7997d82d2ea47808d6564af9591cc6886a" Oct 30 02:15:05.134279 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.133966 6572 generic.go:356] "Generic (PLEG): container finished" podID="da67c0ab-cf74-4396-9987-b1bb46eb802b" containerID="09d8f11bf5adb588ca64b685653e818d9a216c17e52385faf11b5cbec7f6845a" exitCode=60 Oct 30 02:15:05.134279 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.134089 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-8xvvv" event={"ID":"da67c0ab-cf74-4396-9987-b1bb46eb802b","Type":"ContainerDied","Data":"09d8f11bf5adb588ca64b685653e818d9a216c17e52385faf11b5cbec7f6845a"} Oct 30 02:15:05.138622 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.138176 6572 scope.go:122] "RemoveContainer" containerID="09d8f11bf5adb588ca64b685653e818d9a216c17e52385faf11b5cbec7f6845a" Oct 30 02:15:05.143796 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.142578 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888775-rbx7m" event={"ID":"0dc3a619-596a-4e28-9845-9d1e2fc7bb06","Type":"ContainerStarted","Data":"bab6ab85395f733507b5656677bac5397befb0f93542e970af90a9490fb4703d"} Oct 30 02:15:05.146626 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.146587 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888775-wrhfw" event={"ID":"c2bd8bc3-6252-42fc-9055-ca393b6f45d9","Type":"ContainerStarted","Data":"e340d270403c3cf077a0e66934bf3be9d218fdc4735f7e2bef43a669c5c8be00"} Oct 30 02:15:05.156455 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.155161 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888775-sv8kf" event={"ID":"99cb9b6a-51ec-41be-a85d-dd1de997fceb","Type":"ContainerStarted","Data":"a493c551af32b3caca9c6c4b4e73e247853f149efbce0fdda32ed34d5b805e33"} Oct 30 02:15:05.177306 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.177275 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:05.178146 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.178116 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f89697-29888775-9m9lw" event={"ID":"63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4","Type":"ContainerStarted","Data":"fe7f238664d1f4e7062a3083dfba44e980f9ac51b6aec896926c2e660a52d8f8"} Oct 30 02:15:05.188168 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.188132 6572 scope.go:122] "RemoveContainer" containerID="fe7f238664d1f4e7062a3083dfba44e980f9ac51b6aec896926c2e660a52d8f8" Oct 30 02:15:05.210093 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.203641 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888775-qr45g" event={"ID":"1f68c408-6bab-42c7-b574-35d5d61c7544","Type":"ContainerStarted","Data":"ed4d3e7a496e64a0b66df34ed26e51728754ae27ab02c2506dbe9e0fbdc2b307"} Oct 30 02:15:05.213844 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.213785 6572 generic.go:356] "Generic (PLEG): container finished" podID="1c7f35b4-a13a-4be3-be70-135dcc096f49" containerID="973011bb7c95b9dbcd60fc27719b7defb5dc94c0181752d445b1ed6046f25320" exitCode=60 Oct 30 02:15:05.214095 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.213969 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888775-brm4t" event={"ID":"1c7f35b4-a13a-4be3-be70-135dcc096f49","Type":"ContainerDied","Data":"973011bb7c95b9dbcd60fc27719b7defb5dc94c0181752d445b1ed6046f25320"} Oct 30 02:15:05.219984 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.219934 6572 generic.go:356] "Generic (PLEG): container finished" podID="ba0ac388-9127-47e0-8985-dd48570c8d27" containerID="af2099fb81f1ea2f4bff51224dedbed85874910bce7282b79befcb4a40b664a5" exitCode=60 Oct 30 02:15:05.220412 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.220353 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888775-vcqdq" event={"ID":"ba0ac388-9127-47e0-8985-dd48570c8d27","Type":"ContainerDied","Data":"af2099fb81f1ea2f4bff51224dedbed85874910bce7282b79befcb4a40b664a5"} Oct 30 02:15:05.220769 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.220731 6572 scope.go:122] "RemoveContainer" containerID="973011bb7c95b9dbcd60fc27719b7defb5dc94c0181752d445b1ed6046f25320" Oct 30 02:15:05.221996 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.221966 6572 scope.go:122] "RemoveContainer" containerID="af2099fb81f1ea2f4bff51224dedbed85874910bce7282b79befcb4a40b664a5" Oct 30 02:15:05.231937 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.231905 6572 generic.go:356] "Generic (PLEG): container finished" podID="3f9f8331-2090-4262-96a0-c3ea360e4375" containerID="5e7cce123f5ce292cfeb86b365cefa77af1040bb35f2fee03af09fcfada18ec1" exitCode=60 Oct 30 02:15:05.232023 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.231992 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l" event={"ID":"3f9f8331-2090-4262-96a0-c3ea360e4375","Type":"ContainerDied","Data":"5e7cce123f5ce292cfeb86b365cefa77af1040bb35f2fee03af09fcfada18ec1"} Oct 30 02:15:05.232071 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.232047 6572 scope.go:122] "RemoveContainer" containerID="73b42970353506c9e1bd367e02f592fb0f50124a0aacc8a6b82083357638c266" Oct 30 02:15:05.232547 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.232530 6572 scope.go:122] "RemoveContainer" containerID="5e7cce123f5ce292cfeb86b365cefa77af1040bb35f2fee03af09fcfada18ec1" Oct 30 02:15:05.235472 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.235438 6572 generic.go:356] "Generic (PLEG): container finished" podID="39c584cd-b462-4d52-aa9d-786e9a0b8e94" containerID="b71da30be4bf2256d5903fd0bcff383816f1fa933d7d8c503438a7d19c7cf60a" exitCode=60 Oct 30 02:15:05.235570 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.235517 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-340ae6-29888775-8dfms" event={"ID":"39c584cd-b462-4d52-aa9d-786e9a0b8e94","Type":"ContainerDied","Data":"b71da30be4bf2256d5903fd0bcff383816f1fa933d7d8c503438a7d19c7cf60a"} Oct 30 02:15:05.235864 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.235843 6572 scope.go:122] "RemoveContainer" containerID="b71da30be4bf2256d5903fd0bcff383816f1fa933d7d8c503438a7d19c7cf60a" Oct 30 02:15:05.237705 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:05.237680 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l_odh-ai-gateway-infra(3f9f8331-2090-4262-96a0-c3ea360e4375)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l" podUID="3f9f8331-2090-4262-96a0-c3ea360e4375" Oct 30 02:15:05.237845 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.237699 6572 generic.go:356] "Generic (PLEG): container finished" podID="bdf23f37-2870-4b9c-b558-9fea4452f4e5" containerID="f44b76b6541726c9bc7153dd8fefeee3ceee298a5edbcfcd6f2014ea1a275f44" exitCode=60 Oct 30 02:15:05.237845 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.237830 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888775-8tzkk" event={"ID":"bdf23f37-2870-4b9c-b558-9fea4452f4e5","Type":"ContainerDied","Data":"f44b76b6541726c9bc7153dd8fefeee3ceee298a5edbcfcd6f2014ea1a275f44"} Oct 30 02:15:05.238244 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.238222 6572 scope.go:122] "RemoveContainer" containerID="f44b76b6541726c9bc7153dd8fefeee3ceee298a5edbcfcd6f2014ea1a275f44" Oct 30 02:15:05.245637 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.245606 6572 generic.go:356] "Generic (PLEG): container finished" podID="2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90" containerID="99fb55d279c5cdcb2990e4d4744533d8b26b1f86acfa1cac12142e818f2ad75f" exitCode=60 Oct 30 02:15:05.245799 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.245769 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888775-fxr88" event={"ID":"2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90","Type":"ContainerDied","Data":"99fb55d279c5cdcb2990e4d4744533d8b26b1f86acfa1cac12142e818f2ad75f"} Oct 30 02:15:05.246152 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.246136 6572 scope.go:122] "RemoveContainer" containerID="99fb55d279c5cdcb2990e4d4744533d8b26b1f86acfa1cac12142e818f2ad75f" Oct 30 02:15:05.250903 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.250871 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888775-ttncp" event={"ID":"f641ac83-f4e6-4003-ae9c-be79e4ca0337","Type":"ContainerStarted","Data":"d0017525943fb3768bac58c0ce27a376501c0e35f5d9bca7e7408a59e4aa6f44"} Oct 30 02:15:05.261031 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.259971 6572 generic.go:356] "Generic (PLEG): container finished" podID="d66a87ee-dc89-4aba-845b-17b58bc6f4a6" containerID="392a0347ee1fed48949fd4623cd4c4923c4ed2707a60657e1dab6dff8b86f43e" exitCode=60 Oct 30 02:15:05.261031 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.260051 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888775-zd7gh" event={"ID":"d66a87ee-dc89-4aba-845b-17b58bc6f4a6","Type":"ContainerDied","Data":"392a0347ee1fed48949fd4623cd4c4923c4ed2707a60657e1dab6dff8b86f43e"} Oct 30 02:15:05.261031 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.260534 6572 scope.go:122] "RemoveContainer" containerID="392a0347ee1fed48949fd4623cd4c4923c4ed2707a60657e1dab6dff8b86f43e" Oct 30 02:15:05.261031 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:05.260953 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888775-zd7gh_odh-ai-gateway-infra(d66a87ee-dc89-4aba-845b-17b58bc6f4a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888775-zd7gh" podUID="d66a87ee-dc89-4aba-845b-17b58bc6f4a6" Oct 30 02:15:05.347642 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.347601 6572 scope.go:122] "RemoveContainer" containerID="6f04373be7f7c6bcb6bca493602d4d5e6a45d83bd8dd5f00685bd55544daa115" Oct 30 02:15:05.399652 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.399607 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:05.523380 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.523300 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888775-sv8kf" podStartSLOduration=5.523277508 podStartE2EDuration="5.523277508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:05.467209047 +0000 UTC m=+3431.043634451" watchObservedRunningTime="2026-10-30 02:15:05.523277508 +0000 UTC m=+3431.099702912" Oct 30 02:15:05.649206 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.649165 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:05.963716 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.963671 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-cd8173-zd4ct"] Oct 30 02:15:05.968676 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.968651 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-cd8173-zd4ct" Oct 30 02:15:05.978244 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.978216 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r64f\" (UniqueName: \"kubernetes.io/projected/c8e0de10-d72a-4a21-9f72-4d27d10c1623-kube-api-access-6r64f\") pod \"maas-api-revoke-keys-e2e-extmod-b-cd8173-zd4ct\" (UID: \"c8e0de10-d72a-4a21-9f72-4d27d10c1623\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-cd8173-zd4ct" Oct 30 02:15:05.978401 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.978347 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8e0de10-d72a-4a21-9f72-4d27d10c1623-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-cd8173-zd4ct\" (UID: \"c8e0de10-d72a-4a21-9f72-4d27d10c1623\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-cd8173-zd4ct" Oct 30 02:15:05.978401 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.978376 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8e0de10-d72a-4a21-9f72-4d27d10c1623-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-cd8173-zd4ct\" (UID: \"c8e0de10-d72a-4a21-9f72-4d27d10c1623\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-cd8173-zd4ct" Oct 30 02:15:05.990742 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:05.990705 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-cd8173-zd4ct"] Oct 30 02:15:06.050465 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.050417 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:15:06.080031 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.079995 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r64f\" (UniqueName: \"kubernetes.io/projected/c8e0de10-d72a-4a21-9f72-4d27d10c1623-kube-api-access-6r64f\") pod \"maas-api-revoke-keys-e2e-extmod-b-cd8173-zd4ct\" (UID: \"c8e0de10-d72a-4a21-9f72-4d27d10c1623\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-cd8173-zd4ct" Oct 30 02:15:06.080226 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.080144 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8e0de10-d72a-4a21-9f72-4d27d10c1623-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-cd8173-zd4ct\" (UID: \"c8e0de10-d72a-4a21-9f72-4d27d10c1623\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-cd8173-zd4ct" Oct 30 02:15:06.080226 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.080190 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8e0de10-d72a-4a21-9f72-4d27d10c1623-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-cd8173-zd4ct\" (UID: \"c8e0de10-d72a-4a21-9f72-4d27d10c1623\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-cd8173-zd4ct" Oct 30 02:15:06.080911 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.080893 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8e0de10-d72a-4a21-9f72-4d27d10c1623-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-cd8173-zd4ct\" (UID: \"c8e0de10-d72a-4a21-9f72-4d27d10c1623\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-cd8173-zd4ct" Oct 30 02:15:06.109130 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.109093 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8e0de10-d72a-4a21-9f72-4d27d10c1623-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-cd8173-zd4ct\" (UID: \"c8e0de10-d72a-4a21-9f72-4d27d10c1623\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-cd8173-zd4ct" Oct 30 02:15:06.113903 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.110309 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r64f\" (UniqueName: \"kubernetes.io/projected/c8e0de10-d72a-4a21-9f72-4d27d10c1623-kube-api-access-6r64f\") pod \"maas-api-revoke-keys-e2e-extmod-b-cd8173-zd4ct\" (UID: \"c8e0de10-d72a-4a21-9f72-4d27d10c1623\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-cd8173-zd4ct" Oct 30 02:15:06.272913 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.272876 6572 generic.go:356] "Generic (PLEG): container finished" podID="f641ac83-f4e6-4003-ae9c-be79e4ca0337" containerID="d0017525943fb3768bac58c0ce27a376501c0e35f5d9bca7e7408a59e4aa6f44" exitCode=60 Oct 30 02:15:06.273121 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.272956 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888775-ttncp" event={"ID":"f641ac83-f4e6-4003-ae9c-be79e4ca0337","Type":"ContainerDied","Data":"d0017525943fb3768bac58c0ce27a376501c0e35f5d9bca7e7408a59e4aa6f44"} Oct 30 02:15:06.273121 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.273020 6572 scope.go:122] "RemoveContainer" containerID="12bd7bbe8963240ffe8948b0ebe2f27ecb470350c34adbed0a4d4f82b76222f3" Oct 30 02:15:06.273346 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.273327 6572 scope.go:122] "RemoveContainer" containerID="d0017525943fb3768bac58c0ce27a376501c0e35f5d9bca7e7408a59e4aa6f44" Oct 30 02:15:06.273826 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:06.273793 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888775-ttncp_odh-ai-gateway-infra(f641ac83-f4e6-4003-ae9c-be79e4ca0337)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888775-ttncp" podUID="f641ac83-f4e6-4003-ae9c-be79e4ca0337" Oct 30 02:15:06.275294 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.275257 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888775-r46v9" event={"ID":"6edd5565-a917-481f-9302-95011124898b","Type":"ContainerStarted","Data":"572bc92b6afa1a7231e116cc191382befbe1b48f25867fbbab15015c0b1ce2ab"} Oct 30 02:15:06.277554 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.277532 6572 scope.go:122] "RemoveContainer" containerID="392a0347ee1fed48949fd4623cd4c4923c4ed2707a60657e1dab6dff8b86f43e" Oct 30 02:15:06.277925 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:06.277903 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888775-zd7gh_odh-ai-gateway-infra(d66a87ee-dc89-4aba-845b-17b58bc6f4a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888775-zd7gh" podUID="d66a87ee-dc89-4aba-845b-17b58bc6f4a6" Oct 30 02:15:06.279690 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.279621 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888775-289bf" event={"ID":"bcf5437e-0075-467b-a4fd-82b5dc563489","Type":"ContainerStarted","Data":"11379d21f289dc1837deb3ee9c67074732cd20c8a1623f99fba3153cbdd40546"} Oct 30 02:15:06.281827 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.281801 6572 generic.go:356] "Generic (PLEG): container finished" podID="0dc3a619-596a-4e28-9845-9d1e2fc7bb06" containerID="bab6ab85395f733507b5656677bac5397befb0f93542e970af90a9490fb4703d" exitCode=60 Oct 30 02:15:06.281932 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.281915 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888775-rbx7m" event={"ID":"0dc3a619-596a-4e28-9845-9d1e2fc7bb06","Type":"ContainerDied","Data":"bab6ab85395f733507b5656677bac5397befb0f93542e970af90a9490fb4703d"} Oct 30 02:15:06.282122 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.282104 6572 scope.go:122] "RemoveContainer" containerID="bab6ab85395f733507b5656677bac5397befb0f93542e970af90a9490fb4703d" Oct 30 02:15:06.282435 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:06.282409 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d44413-29888775-rbx7m_odh-ai-gateway-infra(0dc3a619-596a-4e28-9845-9d1e2fc7bb06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888775-rbx7m" podUID="0dc3a619-596a-4e28-9845-9d1e2fc7bb06" Oct 30 02:15:06.284337 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.284310 6572 generic.go:356] "Generic (PLEG): container finished" podID="c2bd8bc3-6252-42fc-9055-ca393b6f45d9" containerID="e340d270403c3cf077a0e66934bf3be9d218fdc4735f7e2bef43a669c5c8be00" exitCode=60 Oct 30 02:15:06.284431 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.284385 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888775-wrhfw" event={"ID":"c2bd8bc3-6252-42fc-9055-ca393b6f45d9","Type":"ContainerDied","Data":"e340d270403c3cf077a0e66934bf3be9d218fdc4735f7e2bef43a669c5c8be00"} Oct 30 02:15:06.284603 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.284589 6572 scope.go:122] "RemoveContainer" containerID="e340d270403c3cf077a0e66934bf3be9d218fdc4735f7e2bef43a669c5c8be00" Oct 30 02:15:06.284891 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:06.284869 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-cd8173-29888775-wrhfw_odh-ai-gateway-infra(c2bd8bc3-6252-42fc-9055-ca393b6f45d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888775-wrhfw" podUID="c2bd8bc3-6252-42fc-9055-ca393b6f45d9" Oct 30 02:15:06.286506 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.286482 6572 generic.go:356] "Generic (PLEG): container finished" podID="99cb9b6a-51ec-41be-a85d-dd1de997fceb" containerID="a493c551af32b3caca9c6c4b4e73e247853f149efbce0fdda32ed34d5b805e33" exitCode=60 Oct 30 02:15:06.286630 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.286566 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888775-sv8kf" event={"ID":"99cb9b6a-51ec-41be-a85d-dd1de997fceb","Type":"ContainerDied","Data":"a493c551af32b3caca9c6c4b4e73e247853f149efbce0fdda32ed34d5b805e33"} Oct 30 02:15:06.286831 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.286801 6572 scope.go:122] "RemoveContainer" containerID="a493c551af32b3caca9c6c4b4e73e247853f149efbce0fdda32ed34d5b805e33" Oct 30 02:15:06.287068 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:06.287050 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888775-sv8kf_odh-ai-gateway-infra(99cb9b6a-51ec-41be-a85d-dd1de997fceb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888775-sv8kf" podUID="99cb9b6a-51ec-41be-a85d-dd1de997fceb" Oct 30 02:15:06.287162 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.287149 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-cd8173-zd4ct" Oct 30 02:15:06.288725 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.288701 6572 generic.go:356] "Generic (PLEG): container finished" podID="63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4" containerID="fe7f238664d1f4e7062a3083dfba44e980f9ac51b6aec896926c2e660a52d8f8" exitCode=60 Oct 30 02:15:06.288851 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.288817 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f89697-29888775-9m9lw" event={"ID":"63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4","Type":"ContainerDied","Data":"fe7f238664d1f4e7062a3083dfba44e980f9ac51b6aec896926c2e660a52d8f8"} Oct 30 02:15:06.291169 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.291145 6572 generic.go:356] "Generic (PLEG): container finished" podID="1f68c408-6bab-42c7-b574-35d5d61c7544" containerID="ed4d3e7a496e64a0b66df34ed26e51728754ae27ab02c2506dbe9e0fbdc2b307" exitCode=60 Oct 30 02:15:06.291291 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.291187 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888775-qr45g" event={"ID":"1f68c408-6bab-42c7-b574-35d5d61c7544","Type":"ContainerDied","Data":"ed4d3e7a496e64a0b66df34ed26e51728754ae27ab02c2506dbe9e0fbdc2b307"} Oct 30 02:15:06.291587 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.291567 6572 scope.go:122] "RemoveContainer" containerID="ed4d3e7a496e64a0b66df34ed26e51728754ae27ab02c2506dbe9e0fbdc2b307" Oct 30 02:15:06.291949 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:06.291928 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888775-qr45g_odh-ai-gateway-infra(1f68c408-6bab-42c7-b574-35d5d61c7544)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888775-qr45g" podUID="1f68c408-6bab-42c7-b574-35d5d61c7544" Oct 30 02:15:06.293804 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.293761 6572 scope.go:122] "RemoveContainer" containerID="5e7cce123f5ce292cfeb86b365cefa77af1040bb35f2fee03af09fcfada18ec1" Oct 30 02:15:06.294185 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:06.294158 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l_odh-ai-gateway-infra(3f9f8331-2090-4262-96a0-c3ea360e4375)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l" podUID="3f9f8331-2090-4262-96a0-c3ea360e4375" Oct 30 02:15:06.295563 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.295530 6572 generic.go:356] "Generic (PLEG): container finished" podID="fec1f4d7-877e-4d7c-88d2-f2365469869a" containerID="a763a8188134fed66f2082d810679e403c93177f313eea4218187fcf6b39d917" exitCode=60 Oct 30 02:15:06.295670 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.295619 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d5aa9-29888775-9pqsc" event={"ID":"fec1f4d7-877e-4d7c-88d2-f2365469869a","Type":"ContainerDied","Data":"a763a8188134fed66f2082d810679e403c93177f313eea4218187fcf6b39d917"} Oct 30 02:15:06.296022 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.296001 6572 scope.go:122] "RemoveContainer" containerID="a763a8188134fed66f2082d810679e403c93177f313eea4218187fcf6b39d917" Oct 30 02:15:06.296405 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:06.296379 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-2d5aa9-29888775-9pqsc_odh-ai-gateway-infra(fec1f4d7-877e-4d7c-88d2-f2365469869a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d5aa9-29888775-9pqsc" podUID="fec1f4d7-877e-4d7c-88d2-f2365469869a" Oct 30 02:15:06.305152 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.305116 6572 generic.go:356] "Generic (PLEG): container finished" podID="8968e7e0-04c1-4469-9062-f7d31ee6dbf2" containerID="83f4a31bc2d5df42e6985efa8edefebb5160b6e612bd4c8575a82f0d9d8b8770" exitCode=60 Oct 30 02:15:06.305342 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.305215 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888775-p4j7s" event={"ID":"8968e7e0-04c1-4469-9062-f7d31ee6dbf2","Type":"ContainerDied","Data":"83f4a31bc2d5df42e6985efa8edefebb5160b6e612bd4c8575a82f0d9d8b8770"} Oct 30 02:15:06.305649 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.305631 6572 scope.go:122] "RemoveContainer" containerID="83f4a31bc2d5df42e6985efa8edefebb5160b6e612bd4c8575a82f0d9d8b8770" Oct 30 02:15:06.306125 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:06.306088 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888775-p4j7s_odh-ai-gateway-infra(8968e7e0-04c1-4469-9062-f7d31ee6dbf2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888775-p4j7s" podUID="8968e7e0-04c1-4469-9062-f7d31ee6dbf2" Oct 30 02:15:06.315608 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.313808 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888775-djsrl" event={"ID":"4ed6af0f-eb0e-41f8-8b21-654babbb98b5","Type":"ContainerStarted","Data":"e0af46bda0dd9b0b72a23b3071d9c559492e76bbcf9c15a837d8d9d97589276a"} Oct 30 02:15:06.337530 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.337500 6572 scope.go:122] "RemoveContainer" containerID="36fa10c7c9130d463f5577e0e1e829cabe1fbf8e4c172891a482e8351943439f" Oct 30 02:15:06.357368 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.357339 6572 scope.go:122] "RemoveContainer" containerID="a90d95d04c5ce67c97e5bbd29437b4d4fcf32cf96776f0e858dbe685ba44f32f" Oct 30 02:15:06.379893 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.379844 6572 scope.go:122] "RemoveContainer" containerID="c6558b2ef9872a2b27c682d668c8ffd383757ec0c5b0ddf9fef552e06c8e2221" Oct 30 02:15:06.414048 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.414016 6572 scope.go:122] "RemoveContainer" containerID="6b7733459bee22eee391b66479f95617de1a6b67c3790371243022827576c3a3" Oct 30 02:15:06.620004 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.619976 6572 scope.go:122] "RemoveContainer" containerID="a276b3d2e8c7d8b8955d74d52c10bde6cdd982a3fe8338d70cc2de73fb2cca81" Oct 30 02:15:06.637662 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.637640 6572 scope.go:122] "RemoveContainer" containerID="3534e89f4bcf1c0cf11400a615614eebf65760dc201a09b380761383079c811b" Oct 30 02:15:06.697827 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.696220 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:06.701181 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:15:06.701146 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e0de10_d72a_4a21_9f72_4d27d10c1623.slice/crio-dd35771ca05e259c2c0044097a6c3e81c067a081a6cea348ca6cab62687645b3 WatchSource:0}: Error finding container dd35771ca05e259c2c0044097a6c3e81c067a081a6cea348ca6cab62687645b3: Status 404 returned error can't find the container with id dd35771ca05e259c2c0044097a6c3e81c067a081a6cea348ca6cab62687645b3 Oct 30 02:15:06.706224 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:06.706196 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-cd8173-zd4ct"] Oct 30 02:15:07.328894 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.328778 6572 scope.go:122] "RemoveContainer" containerID="d0017525943fb3768bac58c0ce27a376501c0e35f5d9bca7e7408a59e4aa6f44" Oct 30 02:15:07.329316 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:07.329186 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888775-ttncp_odh-ai-gateway-infra(f641ac83-f4e6-4003-ae9c-be79e4ca0337)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888775-ttncp" podUID="f641ac83-f4e6-4003-ae9c-be79e4ca0337" Oct 30 02:15:07.330800 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.330777 6572 generic.go:356] "Generic (PLEG): container finished" podID="6edd5565-a917-481f-9302-95011124898b" containerID="572bc92b6afa1a7231e116cc191382befbe1b48f25867fbbab15015c0b1ce2ab" exitCode=60 Oct 30 02:15:07.330924 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.330872 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888775-r46v9" event={"ID":"6edd5565-a917-481f-9302-95011124898b","Type":"ContainerDied","Data":"572bc92b6afa1a7231e116cc191382befbe1b48f25867fbbab15015c0b1ce2ab"} Oct 30 02:15:07.330979 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.330945 6572 scope.go:122] "RemoveContainer" containerID="4ba50b3f1a15067ef0ccd3b01d1921bdc8ca94d827c6cbe2488b718a1b1cf210" Oct 30 02:15:07.331187 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.331167 6572 scope.go:122] "RemoveContainer" containerID="572bc92b6afa1a7231e116cc191382befbe1b48f25867fbbab15015c0b1ce2ab" Oct 30 02:15:07.331543 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:07.331524 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-803472-29888775-r46v9_odh-ai-gateway-infra(6edd5565-a917-481f-9302-95011124898b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888775-r46v9" podUID="6edd5565-a917-481f-9302-95011124898b" Oct 30 02:15:07.333097 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.333072 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-cd8173-zd4ct" event={"ID":"c8e0de10-d72a-4a21-9f72-4d27d10c1623","Type":"ContainerStarted","Data":"f59bf48939839aa69b7b3e1c944925c8521f5751d9a8ab03054cb8926e99c8c8"} Oct 30 02:15:07.333233 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.333129 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-cd8173-zd4ct" event={"ID":"c8e0de10-d72a-4a21-9f72-4d27d10c1623","Type":"ContainerStarted","Data":"dd35771ca05e259c2c0044097a6c3e81c067a081a6cea348ca6cab62687645b3"} Oct 30 02:15:07.335142 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.335118 6572 generic.go:356] "Generic (PLEG): container finished" podID="75a1834c-2859-4e9c-9d32-80a6f7247460" containerID="9ef8956dbc2e3fa923dfd2fdb2676d58f0c50bf801af6822838ef48d50eee5e3" exitCode=60 Oct 30 02:15:07.335233 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.335184 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888775-82nz4" event={"ID":"75a1834c-2859-4e9c-9d32-80a6f7247460","Type":"ContainerDied","Data":"9ef8956dbc2e3fa923dfd2fdb2676d58f0c50bf801af6822838ef48d50eee5e3"} Oct 30 02:15:07.335545 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.335527 6572 scope.go:122] "RemoveContainer" containerID="9ef8956dbc2e3fa923dfd2fdb2676d58f0c50bf801af6822838ef48d50eee5e3" Oct 30 02:15:07.335945 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:07.335919 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888775-82nz4_odh-ai-gateway-infra(75a1834c-2859-4e9c-9d32-80a6f7247460)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888775-82nz4" podUID="75a1834c-2859-4e9c-9d32-80a6f7247460" Oct 30 02:15:07.337370 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.337351 6572 generic.go:356] "Generic (PLEG): container finished" podID="da67c0ab-cf74-4396-9987-b1bb46eb802b" containerID="f8ddef1ef53f32698e43625da2e4630c8fdcf4f9e0f286db9e493723421130f5" exitCode=60 Oct 30 02:15:07.337454 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.337414 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-8xvvv" event={"ID":"da67c0ab-cf74-4396-9987-b1bb46eb802b","Type":"ContainerDied","Data":"f8ddef1ef53f32698e43625da2e4630c8fdcf4f9e0f286db9e493723421130f5"} Oct 30 02:15:07.337787 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.337765 6572 scope.go:122] "RemoveContainer" containerID="f8ddef1ef53f32698e43625da2e4630c8fdcf4f9e0f286db9e493723421130f5" Oct 30 02:15:07.338176 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:07.338131 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888775-8xvvv_odh-ai-gateway-infra(da67c0ab-cf74-4396-9987-b1bb46eb802b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-8xvvv" podUID="da67c0ab-cf74-4396-9987-b1bb46eb802b" Oct 30 02:15:07.339414 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.339392 6572 generic.go:356] "Generic (PLEG): container finished" podID="bcf5437e-0075-467b-a4fd-82b5dc563489" containerID="11379d21f289dc1837deb3ee9c67074732cd20c8a1623f99fba3153cbdd40546" exitCode=60 Oct 30 02:15:07.339519 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.339474 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888775-289bf" event={"ID":"bcf5437e-0075-467b-a4fd-82b5dc563489","Type":"ContainerDied","Data":"11379d21f289dc1837deb3ee9c67074732cd20c8a1623f99fba3153cbdd40546"} Oct 30 02:15:07.339703 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.339688 6572 scope.go:122] "RemoveContainer" containerID="11379d21f289dc1837deb3ee9c67074732cd20c8a1623f99fba3153cbdd40546" Oct 30 02:15:07.340056 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:07.340036 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-292018-29888775-289bf_odh-ai-gateway-infra(bcf5437e-0075-467b-a4fd-82b5dc563489)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888775-289bf" podUID="bcf5437e-0075-467b-a4fd-82b5dc563489" Oct 30 02:15:07.341556 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.341539 6572 scope.go:122] "RemoveContainer" containerID="bab6ab85395f733507b5656677bac5397befb0f93542e970af90a9490fb4703d" Oct 30 02:15:07.341861 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:07.341842 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d44413-29888775-rbx7m_odh-ai-gateway-infra(0dc3a619-596a-4e28-9845-9d1e2fc7bb06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888775-rbx7m" podUID="0dc3a619-596a-4e28-9845-9d1e2fc7bb06" Oct 30 02:15:07.343486 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.343461 6572 scope.go:122] "RemoveContainer" containerID="e340d270403c3cf077a0e66934bf3be9d218fdc4735f7e2bef43a669c5c8be00" Oct 30 02:15:07.343833 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:07.343806 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-cd8173-29888775-wrhfw_odh-ai-gateway-infra(c2bd8bc3-6252-42fc-9055-ca393b6f45d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888775-wrhfw" podUID="c2bd8bc3-6252-42fc-9055-ca393b6f45d9" Oct 30 02:15:07.345376 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.345348 6572 scope.go:122] "RemoveContainer" containerID="a493c551af32b3caca9c6c4b4e73e247853f149efbce0fdda32ed34d5b805e33" Oct 30 02:15:07.345658 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:07.345636 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888775-sv8kf_odh-ai-gateway-infra(99cb9b6a-51ec-41be-a85d-dd1de997fceb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888775-sv8kf" podUID="99cb9b6a-51ec-41be-a85d-dd1de997fceb" Oct 30 02:15:07.347057 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.347037 6572 generic.go:356] "Generic (PLEG): container finished" podID="63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4" containerID="d810fa8660a72b72e23cb870a5b52ed29fd0d1a7547d6b82a8c08b7b756e178a" exitCode=60 Oct 30 02:15:07.347141 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.347118 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f89697-29888775-9m9lw" event={"ID":"63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4","Type":"ContainerDied","Data":"d810fa8660a72b72e23cb870a5b52ed29fd0d1a7547d6b82a8c08b7b756e178a"} Oct 30 02:15:07.347355 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.347338 6572 scope.go:122] "RemoveContainer" containerID="d810fa8660a72b72e23cb870a5b52ed29fd0d1a7547d6b82a8c08b7b756e178a" Oct 30 02:15:07.347629 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:07.347608 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f89697-29888775-9m9lw_odh-ai-gateway-infra(63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f89697-29888775-9m9lw" podUID="63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4" Oct 30 02:15:07.349239 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.349129 6572 scope.go:122] "RemoveContainer" containerID="ed4d3e7a496e64a0b66df34ed26e51728754ae27ab02c2506dbe9e0fbdc2b307" Oct 30 02:15:07.349432 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:07.349412 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888775-qr45g_odh-ai-gateway-infra(1f68c408-6bab-42c7-b574-35d5d61c7544)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888775-qr45g" podUID="1f68c408-6bab-42c7-b574-35d5d61c7544" Oct 30 02:15:07.351008 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.350986 6572 generic.go:356] "Generic (PLEG): container finished" podID="1c7f35b4-a13a-4be3-be70-135dcc096f49" containerID="92d1b42bf8acdcf4b40d283f0f3c7ff3f3cb9f620db40c3360dd21f49a9741ef" exitCode=60 Oct 30 02:15:07.351114 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.351092 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888775-brm4t" event={"ID":"1c7f35b4-a13a-4be3-be70-135dcc096f49","Type":"ContainerDied","Data":"92d1b42bf8acdcf4b40d283f0f3c7ff3f3cb9f620db40c3360dd21f49a9741ef"} Oct 30 02:15:07.351446 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.351430 6572 scope.go:122] "RemoveContainer" containerID="92d1b42bf8acdcf4b40d283f0f3c7ff3f3cb9f620db40c3360dd21f49a9741ef" Oct 30 02:15:07.351693 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:07.351676 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888775-brm4t_odh-ai-gateway-infra(1c7f35b4-a13a-4be3-be70-135dcc096f49)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888775-brm4t" podUID="1c7f35b4-a13a-4be3-be70-135dcc096f49" Oct 30 02:15:07.353781 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.353757 6572 generic.go:356] "Generic (PLEG): container finished" podID="ba0ac388-9127-47e0-8985-dd48570c8d27" containerID="0119beb5bd480c062e45ae1cf0e3a2d81e48d25eb0a8aefae9e143a6dfbfdaf9" exitCode=60 Oct 30 02:15:07.353887 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.353837 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888775-vcqdq" event={"ID":"ba0ac388-9127-47e0-8985-dd48570c8d27","Type":"ContainerDied","Data":"0119beb5bd480c062e45ae1cf0e3a2d81e48d25eb0a8aefae9e143a6dfbfdaf9"} Oct 30 02:15:07.354103 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.354087 6572 scope.go:122] "RemoveContainer" containerID="0119beb5bd480c062e45ae1cf0e3a2d81e48d25eb0a8aefae9e143a6dfbfdaf9" Oct 30 02:15:07.354424 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:07.354405 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-093d23-29888775-vcqdq_odh-ai-gateway-infra(ba0ac388-9127-47e0-8985-dd48570c8d27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888775-vcqdq" podUID="ba0ac388-9127-47e0-8985-dd48570c8d27" Oct 30 02:15:07.355723 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.355699 6572 generic.go:356] "Generic (PLEG): container finished" podID="39c584cd-b462-4d52-aa9d-786e9a0b8e94" containerID="e0656f18299892b23fe64ecd9b7550fefa33b0c01294b7c6bdf10f73b1efc0e7" exitCode=60 Oct 30 02:15:07.355839 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.355821 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-340ae6-29888775-8dfms" event={"ID":"39c584cd-b462-4d52-aa9d-786e9a0b8e94","Type":"ContainerDied","Data":"e0656f18299892b23fe64ecd9b7550fefa33b0c01294b7c6bdf10f73b1efc0e7"} Oct 30 02:15:07.356061 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.356045 6572 scope.go:122] "RemoveContainer" containerID="e0656f18299892b23fe64ecd9b7550fefa33b0c01294b7c6bdf10f73b1efc0e7" Oct 30 02:15:07.356379 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:07.356350 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-340ae6-29888775-8dfms_odh-ai-gateway-infra(39c584cd-b462-4d52-aa9d-786e9a0b8e94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-340ae6-29888775-8dfms" podUID="39c584cd-b462-4d52-aa9d-786e9a0b8e94" Oct 30 02:15:07.357684 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.357659 6572 generic.go:356] "Generic (PLEG): container finished" podID="bdf23f37-2870-4b9c-b558-9fea4452f4e5" containerID="b0e7bb5695e7d324c0e5fc21d0bdd3c727cf2a31719d2359f8b1118428921980" exitCode=60 Oct 30 02:15:07.357787 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.357737 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888775-8tzkk" event={"ID":"bdf23f37-2870-4b9c-b558-9fea4452f4e5","Type":"ContainerDied","Data":"b0e7bb5695e7d324c0e5fc21d0bdd3c727cf2a31719d2359f8b1118428921980"} Oct 30 02:15:07.358021 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.358005 6572 scope.go:122] "RemoveContainer" containerID="b0e7bb5695e7d324c0e5fc21d0bdd3c727cf2a31719d2359f8b1118428921980" Oct 30 02:15:07.358289 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:07.358267 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888775-8tzkk_odh-ai-gateway-infra(bdf23f37-2870-4b9c-b558-9fea4452f4e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888775-8tzkk" podUID="bdf23f37-2870-4b9c-b558-9fea4452f4e5" Oct 30 02:15:07.359841 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.359816 6572 generic.go:356] "Generic (PLEG): container finished" podID="2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90" containerID="23982c8af5c99d930e4e498d1e13ed07b06d4b35ad2bc90979d24058b0d09719" exitCode=60 Oct 30 02:15:07.359926 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.359860 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888775-fxr88" event={"ID":"2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90","Type":"ContainerDied","Data":"23982c8af5c99d930e4e498d1e13ed07b06d4b35ad2bc90979d24058b0d09719"} Oct 30 02:15:07.360142 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.360125 6572 scope.go:122] "RemoveContainer" containerID="23982c8af5c99d930e4e498d1e13ed07b06d4b35ad2bc90979d24058b0d09719" Oct 30 02:15:07.360444 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:07.360418 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888775-fxr88_odh-ai-gateway-infra(2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888775-fxr88" podUID="2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90" Oct 30 02:15:07.362335 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.362313 6572 scope.go:122] "RemoveContainer" containerID="a763a8188134fed66f2082d810679e403c93177f313eea4218187fcf6b39d917" Oct 30 02:15:07.362608 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:07.362587 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-2d5aa9-29888775-9pqsc_odh-ai-gateway-infra(fec1f4d7-877e-4d7c-88d2-f2365469869a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d5aa9-29888775-9pqsc" podUID="fec1f4d7-877e-4d7c-88d2-f2365469869a" Oct 30 02:15:07.364080 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.364064 6572 scope.go:122] "RemoveContainer" containerID="83f4a31bc2d5df42e6985efa8edefebb5160b6e612bd4c8575a82f0d9d8b8770" Oct 30 02:15:07.364351 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:07.364329 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888775-p4j7s_odh-ai-gateway-infra(8968e7e0-04c1-4469-9062-f7d31ee6dbf2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888775-p4j7s" podUID="8968e7e0-04c1-4469-9062-f7d31ee6dbf2" Oct 30 02:15:07.365839 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.365815 6572 generic.go:356] "Generic (PLEG): container finished" podID="4ed6af0f-eb0e-41f8-8b21-654babbb98b5" containerID="e0af46bda0dd9b0b72a23b3071d9c559492e76bbcf9c15a837d8d9d97589276a" exitCode=60 Oct 30 02:15:07.365952 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.365845 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888775-djsrl" event={"ID":"4ed6af0f-eb0e-41f8-8b21-654babbb98b5","Type":"ContainerDied","Data":"e0af46bda0dd9b0b72a23b3071d9c559492e76bbcf9c15a837d8d9d97589276a"} Oct 30 02:15:07.366118 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.366104 6572 scope.go:122] "RemoveContainer" containerID="e0af46bda0dd9b0b72a23b3071d9c559492e76bbcf9c15a837d8d9d97589276a" Oct 30 02:15:07.366361 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:07.366343 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-558327-29888775-djsrl_odh-ai-gateway-infra(4ed6af0f-eb0e-41f8-8b21-654babbb98b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888775-djsrl" podUID="4ed6af0f-eb0e-41f8-8b21-654babbb98b5" Oct 30 02:15:07.421593 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.421567 6572 scope.go:122] "RemoveContainer" containerID="98a99b3c07c1a577805f139b2f30ef7997d82d2ea47808d6564af9591cc6886a" Oct 30 02:15:07.437426 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.437401 6572 scope.go:122] "RemoveContainer" containerID="09d8f11bf5adb588ca64b685653e818d9a216c17e52385faf11b5cbec7f6845a" Oct 30 02:15:07.457530 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.457504 6572 scope.go:122] "RemoveContainer" containerID="04a6361f8f76d35b905d3f42e91e672dfb96996087b3b21a53956635d01c23c7" Oct 30 02:15:07.474179 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.474154 6572 scope.go:122] "RemoveContainer" containerID="fe7f238664d1f4e7062a3083dfba44e980f9ac51b6aec896926c2e660a52d8f8" Oct 30 02:15:07.490637 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.490606 6572 scope.go:122] "RemoveContainer" containerID="973011bb7c95b9dbcd60fc27719b7defb5dc94c0181752d445b1ed6046f25320" Oct 30 02:15:07.507459 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.507437 6572 scope.go:122] "RemoveContainer" containerID="af2099fb81f1ea2f4bff51224dedbed85874910bce7282b79befcb4a40b664a5" Oct 30 02:15:07.520736 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.520605 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-cd8173-zd4ct" podStartSLOduration=2.520586369 podStartE2EDuration="2.520586369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:07.51563402 +0000 UTC m=+3433.092059423" watchObservedRunningTime="2026-10-30 02:15:07.520586369 +0000 UTC m=+3433.097011771" Oct 30 02:15:07.528416 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.528387 6572 scope.go:122] "RemoveContainer" containerID="b71da30be4bf2256d5903fd0bcff383816f1fa933d7d8c503438a7d19c7cf60a" Oct 30 02:15:07.546130 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.546101 6572 scope.go:122] "RemoveContainer" containerID="f44b76b6541726c9bc7153dd8fefeee3ceee298a5edbcfcd6f2014ea1a275f44" Oct 30 02:15:07.567941 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.567918 6572 scope.go:122] "RemoveContainer" containerID="99fb55d279c5cdcb2990e4d4744533d8b26b1f86acfa1cac12142e818f2ad75f" Oct 30 02:15:07.588931 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.588763 6572 scope.go:122] "RemoveContainer" containerID="fe0e2f6e7c80e7b7dcac15924ec8e19f781456bcc6695d72b688df839d541189" Oct 30 02:15:07.715680 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:07.715641 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:08.314318 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:08.314285 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:08.378086 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:08.378055 6572 scope.go:122] "RemoveContainer" containerID="0119beb5bd480c062e45ae1cf0e3a2d81e48d25eb0a8aefae9e143a6dfbfdaf9" Oct 30 02:15:08.378536 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:08.378474 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-093d23-29888775-vcqdq_odh-ai-gateway-infra(ba0ac388-9127-47e0-8985-dd48570c8d27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888775-vcqdq" podUID="ba0ac388-9127-47e0-8985-dd48570c8d27" Oct 30 02:15:08.380030 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:08.380006 6572 scope.go:122] "RemoveContainer" containerID="e0656f18299892b23fe64ecd9b7550fefa33b0c01294b7c6bdf10f73b1efc0e7" Oct 30 02:15:08.380395 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:08.380367 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-340ae6-29888775-8dfms_odh-ai-gateway-infra(39c584cd-b462-4d52-aa9d-786e9a0b8e94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-340ae6-29888775-8dfms" podUID="39c584cd-b462-4d52-aa9d-786e9a0b8e94" Oct 30 02:15:08.381908 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:08.381886 6572 scope.go:122] "RemoveContainer" containerID="b0e7bb5695e7d324c0e5fc21d0bdd3c727cf2a31719d2359f8b1118428921980" Oct 30 02:15:08.382257 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:08.382230 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888775-8tzkk_odh-ai-gateway-infra(bdf23f37-2870-4b9c-b558-9fea4452f4e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888775-8tzkk" podUID="bdf23f37-2870-4b9c-b558-9fea4452f4e5" Oct 30 02:15:08.383654 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:08.383634 6572 scope.go:122] "RemoveContainer" containerID="23982c8af5c99d930e4e498d1e13ed07b06d4b35ad2bc90979d24058b0d09719" Oct 30 02:15:08.383975 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:08.383953 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888775-fxr88_odh-ai-gateway-infra(2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888775-fxr88" podUID="2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90" Oct 30 02:15:08.385410 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:08.385393 6572 scope.go:122] "RemoveContainer" containerID="e0af46bda0dd9b0b72a23b3071d9c559492e76bbcf9c15a837d8d9d97589276a" Oct 30 02:15:08.385667 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:08.385644 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-558327-29888775-djsrl_odh-ai-gateway-infra(4ed6af0f-eb0e-41f8-8b21-654babbb98b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888775-djsrl" podUID="4ed6af0f-eb0e-41f8-8b21-654babbb98b5" Oct 30 02:15:08.387396 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:08.387359 6572 scope.go:122] "RemoveContainer" containerID="572bc92b6afa1a7231e116cc191382befbe1b48f25867fbbab15015c0b1ce2ab" Oct 30 02:15:08.387666 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:08.387645 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-803472-29888775-r46v9_odh-ai-gateway-infra(6edd5565-a917-481f-9302-95011124898b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888775-r46v9" podUID="6edd5565-a917-481f-9302-95011124898b" Oct 30 02:15:08.388821 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:08.388798 6572 generic.go:356] "Generic (PLEG): container finished" podID="c8e0de10-d72a-4a21-9f72-4d27d10c1623" containerID="f59bf48939839aa69b7b3e1c944925c8521f5751d9a8ab03054cb8926e99c8c8" exitCode=0 Oct 30 02:15:08.388918 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:08.388874 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-cd8173-zd4ct" event={"ID":"c8e0de10-d72a-4a21-9f72-4d27d10c1623","Type":"ContainerDied","Data":"f59bf48939839aa69b7b3e1c944925c8521f5751d9a8ab03054cb8926e99c8c8"} Oct 30 02:15:08.390711 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:08.390694 6572 scope.go:122] "RemoveContainer" containerID="9ef8956dbc2e3fa923dfd2fdb2676d58f0c50bf801af6822838ef48d50eee5e3" Oct 30 02:15:08.390993 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:08.390967 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888775-82nz4_odh-ai-gateway-infra(75a1834c-2859-4e9c-9d32-80a6f7247460)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888775-82nz4" podUID="75a1834c-2859-4e9c-9d32-80a6f7247460" Oct 30 02:15:08.392366 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:08.392348 6572 scope.go:122] "RemoveContainer" containerID="f8ddef1ef53f32698e43625da2e4630c8fdcf4f9e0f286db9e493723421130f5" Oct 30 02:15:08.392627 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:08.392606 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888775-8xvvv_odh-ai-gateway-infra(da67c0ab-cf74-4396-9987-b1bb46eb802b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-8xvvv" podUID="da67c0ab-cf74-4396-9987-b1bb46eb802b" Oct 30 02:15:08.394517 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:08.394500 6572 scope.go:122] "RemoveContainer" containerID="11379d21f289dc1837deb3ee9c67074732cd20c8a1623f99fba3153cbdd40546" Oct 30 02:15:08.394864 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:08.394840 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-292018-29888775-289bf_odh-ai-gateway-infra(bcf5437e-0075-467b-a4fd-82b5dc563489)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888775-289bf" podUID="bcf5437e-0075-467b-a4fd-82b5dc563489" Oct 30 02:15:08.396364 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:08.396347 6572 scope.go:122] "RemoveContainer" containerID="d810fa8660a72b72e23cb870a5b52ed29fd0d1a7547d6b82a8c08b7b756e178a" Oct 30 02:15:08.396574 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:08.396546 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f89697-29888775-9m9lw_odh-ai-gateway-infra(63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f89697-29888775-9m9lw" podUID="63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4" Oct 30 02:15:08.398905 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:08.398889 6572 scope.go:122] "RemoveContainer" containerID="92d1b42bf8acdcf4b40d283f0f3c7ff3f3cb9f620db40c3360dd21f49a9741ef" Oct 30 02:15:08.399101 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:08.399081 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888775-brm4t_odh-ai-gateway-infra(1c7f35b4-a13a-4be3-be70-135dcc096f49)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888775-brm4t" podUID="1c7f35b4-a13a-4be3-be70-135dcc096f49" Oct 30 02:15:08.879554 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:08.879514 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:09.589910 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.589879 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-cd8173-zd4ct" Oct 30 02:15:09.646406 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.646367 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8e0de10-d72a-4a21-9f72-4d27d10c1623-kube-api-access-6r64f\" (UniqueName: \"kubernetes.io/projected/c8e0de10-d72a-4a21-9f72-4d27d10c1623-kube-api-access-6r64f\") pod \"c8e0de10-d72a-4a21-9f72-4d27d10c1623\" (UID: \"c8e0de10-d72a-4a21-9f72-4d27d10c1623\") " Oct 30 02:15:09.646621 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.646576 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8e0de10-d72a-4a21-9f72-4d27d10c1623-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8e0de10-d72a-4a21-9f72-4d27d10c1623-maas-api-xks-ca\") pod \"c8e0de10-d72a-4a21-9f72-4d27d10c1623\" (UID: \"c8e0de10-d72a-4a21-9f72-4d27d10c1623\") " Oct 30 02:15:09.646769 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.646724 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8e0de10-d72a-4a21-9f72-4d27d10c1623-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8e0de10-d72a-4a21-9f72-4d27d10c1623-maas-api-openshift-ca\") pod \"c8e0de10-d72a-4a21-9f72-4d27d10c1623\" (UID: \"c8e0de10-d72a-4a21-9f72-4d27d10c1623\") " Oct 30 02:15:09.647563 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.647533 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8e0de10-d72a-4a21-9f72-4d27d10c1623-maas-api-openshift-ca" pod "c8e0de10-d72a-4a21-9f72-4d27d10c1623" (UID: "c8e0de10-d72a-4a21-9f72-4d27d10c1623"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:09.649306 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.649267 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8e0de10-d72a-4a21-9f72-4d27d10c1623-kube-api-access-6r64f" pod "c8e0de10-d72a-4a21-9f72-4d27d10c1623" (UID: "c8e0de10-d72a-4a21-9f72-4d27d10c1623"). InnerVolumeSpecName "kube-api-access-6r64f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:09.649635 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.649600 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8e0de10-d72a-4a21-9f72-4d27d10c1623-maas-api-xks-ca" pod "c8e0de10-d72a-4a21-9f72-4d27d10c1623" (UID: "c8e0de10-d72a-4a21-9f72-4d27d10c1623"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:15:09.682182 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.682136 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv"] Oct 30 02:15:09.686196 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.683588 6572 cpu_manager.go:414] "containerMap: removing container" podUID="c8e0de10-d72a-4a21-9f72-4d27d10c1623" containerName="revoke-keys" Oct 30 02:15:09.686196 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.683617 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8e0de10-d72a-4a21-9f72-4d27d10c1623" containerName="revoke-keys" Oct 30 02:15:09.686196 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.683967 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8e0de10-d72a-4a21-9f72-4d27d10c1623" containerName="revoke-keys" Oct 30 02:15:09.689470 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.689442 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv" Oct 30 02:15:09.705210 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.705178 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-340ae6\"/\"default-dockercfg-gt4wq\"" Oct 30 02:15:09.705526 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.705503 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-340ae6\"/\"openshift-service-ca.crt\"" Oct 30 02:15:09.705733 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.705701 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-340ae6\"/\"kube-root-ca.crt\"" Oct 30 02:15:09.706014 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.705987 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-340ae6\"/\"test-model-340ae6-kserve-self-signed-certs\"" Oct 30 02:15:09.713486 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.713445 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv"] Oct 30 02:15:09.749682 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.749627 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c789c\" (UniqueName: \"kubernetes.io/projected/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-kube-api-access-c789c\") pod \"test-model-340ae6-kserve-6bc96674b5-z2bcv\" (UID: \"dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea\") " pod="ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv" Oct 30 02:15:09.749902 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.749769 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-model-cache\") pod \"test-model-340ae6-kserve-6bc96674b5-z2bcv\" (UID: \"dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea\") " pod="ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv" Oct 30 02:15:09.749967 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.749918 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-home\") pod \"test-model-340ae6-kserve-6bc96674b5-z2bcv\" (UID: \"dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea\") " pod="ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv" Oct 30 02:15:09.750093 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.750068 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-dshm\") pod \"test-model-340ae6-kserve-6bc96674b5-z2bcv\" (UID: \"dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea\") " pod="ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv" Oct 30 02:15:09.750314 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.750289 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-tmp-dir\") pod \"test-model-340ae6-kserve-6bc96674b5-z2bcv\" (UID: \"dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea\") " pod="ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv" Oct 30 02:15:09.750399 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.750334 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-tls-certs\") pod \"test-model-340ae6-kserve-6bc96674b5-z2bcv\" (UID: \"dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea\") " pod="ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv" Oct 30 02:15:09.750495 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.750481 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8e0de10-d72a-4a21-9f72-4d27d10c1623-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:09.750558 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.750503 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8e0de10-d72a-4a21-9f72-4d27d10c1623-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:09.750558 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.750519 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r64f\" (UniqueName: \"kubernetes.io/projected/c8e0de10-d72a-4a21-9f72-4d27d10c1623-kube-api-access-6r64f\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:09.832991 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.832950 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:09.851815 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.851730 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-dshm\") pod \"test-model-340ae6-kserve-6bc96674b5-z2bcv\" (UID: \"dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea\") " pod="ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv" Oct 30 02:15:09.852122 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.852003 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-tmp-dir\") pod \"test-model-340ae6-kserve-6bc96674b5-z2bcv\" (UID: \"dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea\") " pod="ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv" Oct 30 02:15:09.852122 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.852042 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-tls-certs\") pod \"test-model-340ae6-kserve-6bc96674b5-z2bcv\" (UID: \"dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea\") " pod="ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv" Oct 30 02:15:09.852122 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.852108 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c789c\" (UniqueName: \"kubernetes.io/projected/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-kube-api-access-c789c\") pod \"test-model-340ae6-kserve-6bc96674b5-z2bcv\" (UID: \"dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea\") " pod="ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv" Oct 30 02:15:09.852324 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.852133 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-model-cache\") pod \"test-model-340ae6-kserve-6bc96674b5-z2bcv\" (UID: \"dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea\") " pod="ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv" Oct 30 02:15:09.852324 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.852223 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-home\") pod \"test-model-340ae6-kserve-6bc96674b5-z2bcv\" (UID: \"dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea\") " pod="ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv" Oct 30 02:15:09.852509 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.852469 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-tmp-dir\") pod \"test-model-340ae6-kserve-6bc96674b5-z2bcv\" (UID: \"dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea\") " pod="ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv" Oct 30 02:15:09.852605 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.852563 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-model-cache\") pod \"test-model-340ae6-kserve-6bc96674b5-z2bcv\" (UID: \"dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea\") " pod="ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv" Oct 30 02:15:09.852681 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.852653 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-home\") pod \"test-model-340ae6-kserve-6bc96674b5-z2bcv\" (UID: \"dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea\") " pod="ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv" Oct 30 02:15:09.854681 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.854653 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-dshm\") pod \"test-model-340ae6-kserve-6bc96674b5-z2bcv\" (UID: \"dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea\") " pod="ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv" Oct 30 02:15:09.855011 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.854992 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-tls-certs\") pod \"test-model-340ae6-kserve-6bc96674b5-z2bcv\" (UID: \"dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea\") " pod="ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv" Oct 30 02:15:09.871600 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:09.871564 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c789c\" (UniqueName: \"kubernetes.io/projected/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-kube-api-access-c789c\") pod \"test-model-340ae6-kserve-6bc96674b5-z2bcv\" (UID: \"dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea\") " pod="ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv" Oct 30 02:15:10.010828 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:10.010250 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv" Oct 30 02:15:10.028139 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:10.023575 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:10.350683 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:15:10.350651 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddac0f19b_d0bd_45d7_9e36_a6e0ddf7e7ea.slice/crio-cd0913bfa41d89bb751ca6e423bba3869a6b0e6b66097d8891a71e9ebfe88d14 WatchSource:0}: Error finding container cd0913bfa41d89bb751ca6e423bba3869a6b0e6b66097d8891a71e9ebfe88d14: Status 404 returned error can't find the container with id cd0913bfa41d89bb751ca6e423bba3869a6b0e6b66097d8891a71e9ebfe88d14 Oct 30 02:15:10.351058 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:10.351030 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv"] Oct 30 02:15:10.420402 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:10.420371 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-cd8173-zd4ct" Oct 30 02:15:10.420402 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:10.420390 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-cd8173-zd4ct" event={"ID":"c8e0de10-d72a-4a21-9f72-4d27d10c1623","Type":"ContainerDied","Data":"dd35771ca05e259c2c0044097a6c3e81c067a081a6cea348ca6cab62687645b3"} Oct 30 02:15:10.420622 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:10.420468 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd35771ca05e259c2c0044097a6c3e81c067a081a6cea348ca6cab62687645b3" Oct 30 02:15:10.422129 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:10.422098 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv" event={"ID":"dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea","Type":"ContainerStarted","Data":"cd0913bfa41d89bb751ca6e423bba3869a6b0e6b66097d8891a71e9ebfe88d14"} Oct 30 02:15:11.440392 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:11.440353 6572 generic.go:356] "Generic (PLEG): container finished" podID="d0a98e88-a93c-462b-b2f5-68c61b386789" containerID="4625e5617338c40526823aca86dbe2fb3e8217e4e4c1c41f1bb25b44c70ac2a2" exitCode=0 Oct 30 02:15:11.441008 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:11.440561 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-292018-f884cbb67-mvk86" event={"ID":"d0a98e88-a93c-462b-b2f5-68c61b386789","Type":"ContainerDied","Data":"4625e5617338c40526823aca86dbe2fb3e8217e4e4c1c41f1bb25b44c70ac2a2"} Oct 30 02:15:11.445513 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:11.444560 6572 generic.go:356] "Generic (PLEG): container finished" podID="cd10ed97-4e22-4efe-a0c6-7cff1ca2f157" containerID="2afe31e5495d6d4ce4a3f9834325d50ea2852c2fcc83681af8d02959cb59482a" exitCode=0 Oct 30 02:15:11.445513 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:11.444676 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-292018-6fc6bf57c6-nslxv" event={"ID":"cd10ed97-4e22-4efe-a0c6-7cff1ca2f157","Type":"ContainerDied","Data":"2afe31e5495d6d4ce4a3f9834325d50ea2852c2fcc83681af8d02959cb59482a"} Oct 30 02:15:11.451068 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:11.448096 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv" event={"ID":"dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea","Type":"ContainerStarted","Data":"bdb43767b55fc3e3e3f5d543d7899171d470a11ea3bc3c68b55d107ce68b3f6e"} Oct 30 02:15:11.481022 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:11.480993 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-292018-6fc6bf57c6-nslxv" Oct 30 02:15:11.491964 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:11.491893 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv" podStartSLOduration=2.49187274 podStartE2EDuration="2.49187274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:11.49044781 +0000 UTC m=+3437.066873209" watchObservedRunningTime="2026-10-30 02:15:11.49187274 +0000 UTC m=+3437.068298144" Oct 30 02:15:11.522120 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:11.522094 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-292018-f884cbb67-mvk86" Oct 30 02:15:11.592144 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:11.592100 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0a98e88-a93c-462b-b2f5-68c61b386789-kube-api-access-h49cm\" (UniqueName: \"kubernetes.io/projected/d0a98e88-a93c-462b-b2f5-68c61b386789-kube-api-access-h49cm\") pod \"d0a98e88-a93c-462b-b2f5-68c61b386789\" (UID: \"d0a98e88-a93c-462b-b2f5-68c61b386789\") " Oct 30 02:15:11.592335 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:11.592156 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd10ed97-4e22-4efe-a0c6-7cff1ca2f157-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd10ed97-4e22-4efe-a0c6-7cff1ca2f157-plugins-config-volume\") pod \"cd10ed97-4e22-4efe-a0c6-7cff1ca2f157\" (UID: \"cd10ed97-4e22-4efe-a0c6-7cff1ca2f157\") " Oct 30 02:15:11.592401 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:11.592357 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0a98e88-a93c-462b-b2f5-68c61b386789-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a98e88-a93c-462b-b2f5-68c61b386789-plugins-config-volume\") pod \"d0a98e88-a93c-462b-b2f5-68c61b386789\" (UID: \"d0a98e88-a93c-462b-b2f5-68c61b386789\") " Oct 30 02:15:11.592551 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:11.592528 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd10ed97-4e22-4efe-a0c6-7cff1ca2f157-kube-api-access-6d7br\" (UniqueName: \"kubernetes.io/projected/cd10ed97-4e22-4efe-a0c6-7cff1ca2f157-kube-api-access-6d7br\") pod \"cd10ed97-4e22-4efe-a0c6-7cff1ca2f157\" (UID: \"cd10ed97-4e22-4efe-a0c6-7cff1ca2f157\") " Oct 30 02:15:11.592709 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:11.592683 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd10ed97-4e22-4efe-a0c6-7cff1ca2f157-plugins-config-volume" pod "cd10ed97-4e22-4efe-a0c6-7cff1ca2f157" (UID: "cd10ed97-4e22-4efe-a0c6-7cff1ca2f157"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:11.592823 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:11.592803 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a98e88-a93c-462b-b2f5-68c61b386789-plugins-config-volume" pod "d0a98e88-a93c-462b-b2f5-68c61b386789" (UID: "d0a98e88-a93c-462b-b2f5-68c61b386789"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:11.593487 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:11.593460 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd10ed97-4e22-4efe-a0c6-7cff1ca2f157-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:11.593487 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:11.593488 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a98e88-a93c-462b-b2f5-68c61b386789-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:11.595196 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:11.595160 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a98e88-a93c-462b-b2f5-68c61b386789-kube-api-access-h49cm" pod "d0a98e88-a93c-462b-b2f5-68c61b386789" (UID: "d0a98e88-a93c-462b-b2f5-68c61b386789"). InnerVolumeSpecName "kube-api-access-h49cm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:11.595683 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:11.595661 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd10ed97-4e22-4efe-a0c6-7cff1ca2f157-kube-api-access-6d7br" pod "cd10ed97-4e22-4efe-a0c6-7cff1ca2f157" (UID: "cd10ed97-4e22-4efe-a0c6-7cff1ca2f157"). InnerVolumeSpecName "kube-api-access-6d7br". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:11.638717 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:11.638613 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:11.694994 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:11.694909 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d7br\" (UniqueName: \"kubernetes.io/projected/cd10ed97-4e22-4efe-a0c6-7cff1ca2f157-kube-api-access-6d7br\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:11.694994 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:11.694941 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h49cm\" (UniqueName: \"kubernetes.io/projected/d0a98e88-a93c-462b-b2f5-68c61b386789-kube-api-access-h49cm\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:12.011248 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:12.011160 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv" Oct 30 02:15:12.460992 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:12.460954 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-292018-f884cbb67-mvk86" Oct 30 02:15:12.461498 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:12.461030 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-292018-f884cbb67-mvk86" event={"ID":"d0a98e88-a93c-462b-b2f5-68c61b386789","Type":"ContainerDied","Data":"90ffc21ab3b5915567c2024b8831c1ae5928c4f017a905d5cc439b1fd2199838"} Oct 30 02:15:12.461498 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:12.461099 6572 scope.go:122] "RemoveContainer" containerID="4625e5617338c40526823aca86dbe2fb3e8217e4e4c1c41f1bb25b44c70ac2a2" Oct 30 02:15:12.462913 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:12.462894 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-292018-6fc6bf57c6-nslxv" Oct 30 02:15:12.463053 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:12.462924 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-292018-6fc6bf57c6-nslxv" event={"ID":"cd10ed97-4e22-4efe-a0c6-7cff1ca2f157","Type":"ContainerDied","Data":"6eef2ec45c3ff2268700e34182f2aa69bed060ef22ad6e7e6dc77073eb73e7c8"} Oct 30 02:15:12.476200 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:12.476175 6572 scope.go:122] "RemoveContainer" containerID="2afe31e5495d6d4ce4a3f9834325d50ea2852c2fcc83681af8d02959cb59482a" Oct 30 02:15:12.513623 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:12.513578 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-292018-6fc6bf57c6-nslxv"] Oct 30 02:15:12.518678 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:12.518645 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-292018-6fc6bf57c6-nslxv"] Oct 30 02:15:12.545656 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:12.545596 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd10ed97-4e22-4efe-a0c6-7cff1ca2f157" path="/var/lib/kubelet/pods/cd10ed97-4e22-4efe-a0c6-7cff1ca2f157/volumes" Oct 30 02:15:12.549434 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:12.549405 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-292018-f884cbb67-mvk86"] Oct 30 02:15:12.554500 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:12.554467 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-292018-f884cbb67-mvk86"] Oct 30 02:15:13.089938 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:13.089896 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:14.547070 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:14.547026 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0a98e88-a93c-462b-b2f5-68c61b386789" path="/var/lib/kubelet/pods/d0a98e88-a93c-462b-b2f5-68c61b386789/volumes" Oct 30 02:15:14.604813 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:14.603915 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:14.694304 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:14.694260 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:15:15.535597 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:15.535555 6572 scope.go:122] "RemoveContainer" containerID="5e7cce123f5ce292cfeb86b365cefa77af1040bb35f2fee03af09fcfada18ec1" Oct 30 02:15:16.273278 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:16.273242 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:16.383418 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:16.383378 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-cd8173-8445cbb698-7wtm5"] Oct 30 02:15:16.546379 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:16.545721 6572 scope.go:122] "RemoveContainer" containerID="d0017525943fb3768bac58c0ce27a376501c0e35f5d9bca7e7408a59e4aa6f44" Oct 30 02:15:16.558341 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:16.558302 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l" event={"ID":"3f9f8331-2090-4262-96a0-c3ea360e4375","Type":"ContainerStarted","Data":"256270db4f75a501f3f070b66c7183368af3b05914e7e72d6e3b6626ae6f21ea"} Oct 30 02:15:16.558867 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:16.558842 6572 scope.go:122] "RemoveContainer" containerID="256270db4f75a501f3f070b66c7183368af3b05914e7e72d6e3b6626ae6f21ea" Oct 30 02:15:16.559300 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:16.559273 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l_odh-ai-gateway-infra(3f9f8331-2090-4262-96a0-c3ea360e4375)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l" podUID="3f9f8331-2090-4262-96a0-c3ea360e4375" Oct 30 02:15:16.603419 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:16.603384 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:16.801360 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:16.801264 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:17.535937 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:17.535845 6572 scope.go:122] "RemoveContainer" containerID="d810fa8660a72b72e23cb870a5b52ed29fd0d1a7547d6b82a8c08b7b756e178a" Oct 30 02:15:17.536428 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:17.536038 6572 scope.go:122] "RemoveContainer" containerID="23982c8af5c99d930e4e498d1e13ed07b06d4b35ad2bc90979d24058b0d09719" Oct 30 02:15:17.536551 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:17.536528 6572 scope.go:122] "RemoveContainer" containerID="a493c551af32b3caca9c6c4b4e73e247853f149efbce0fdda32ed34d5b805e33" Oct 30 02:15:17.537222 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:17.537162 6572 scope.go:122] "RemoveContainer" containerID="bab6ab85395f733507b5656677bac5397befb0f93542e970af90a9490fb4703d" Oct 30 02:15:17.537222 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:17.537191 6572 scope.go:122] "RemoveContainer" containerID="f8ddef1ef53f32698e43625da2e4630c8fdcf4f9e0f286db9e493723421130f5" Oct 30 02:15:17.539182 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:17.539013 6572 scope.go:122] "RemoveContainer" containerID="392a0347ee1fed48949fd4623cd4c4923c4ed2707a60657e1dab6dff8b86f43e" Oct 30 02:15:17.565023 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:17.564841 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888775-ttncp" event={"ID":"f641ac83-f4e6-4003-ae9c-be79e4ca0337","Type":"ContainerStarted","Data":"6c50384620024323388a9d0d5697c6fa7e741aa8a4847e7b589fc4a2a4b1769a"} Oct 30 02:15:17.569344 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:17.568886 6572 generic.go:356] "Generic (PLEG): container finished" podID="3f9f8331-2090-4262-96a0-c3ea360e4375" containerID="256270db4f75a501f3f070b66c7183368af3b05914e7e72d6e3b6626ae6f21ea" exitCode=60 Oct 30 02:15:17.569344 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:17.568982 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l" event={"ID":"3f9f8331-2090-4262-96a0-c3ea360e4375","Type":"ContainerDied","Data":"256270db4f75a501f3f070b66c7183368af3b05914e7e72d6e3b6626ae6f21ea"} Oct 30 02:15:17.569344 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:17.569051 6572 scope.go:122] "RemoveContainer" containerID="5e7cce123f5ce292cfeb86b365cefa77af1040bb35f2fee03af09fcfada18ec1" Oct 30 02:15:17.569578 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:17.569517 6572 scope.go:122] "RemoveContainer" containerID="256270db4f75a501f3f070b66c7183368af3b05914e7e72d6e3b6626ae6f21ea" Oct 30 02:15:17.570105 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:17.569990 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l_odh-ai-gateway-infra(3f9f8331-2090-4262-96a0-c3ea360e4375)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l" podUID="3f9f8331-2090-4262-96a0-c3ea360e4375" Oct 30 02:15:17.617523 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:17.617426 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l"] Oct 30 02:15:17.913027 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:17.912995 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:18.536976 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:18.536937 6572 scope.go:122] "RemoveContainer" containerID="9ef8956dbc2e3fa923dfd2fdb2676d58f0c50bf801af6822838ef48d50eee5e3" Oct 30 02:15:18.539639 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:18.539201 6572 scope.go:122] "RemoveContainer" containerID="a763a8188134fed66f2082d810679e403c93177f313eea4218187fcf6b39d917" Oct 30 02:15:18.540404 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:18.540377 6572 scope.go:122] "RemoveContainer" containerID="ed4d3e7a496e64a0b66df34ed26e51728754ae27ab02c2506dbe9e0fbdc2b307" Oct 30 02:15:18.549615 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:18.549578 6572 scope.go:122] "RemoveContainer" containerID="11379d21f289dc1837deb3ee9c67074732cd20c8a1623f99fba3153cbdd40546" Oct 30 02:15:18.551594 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:18.551147 6572 scope.go:122] "RemoveContainer" containerID="572bc92b6afa1a7231e116cc191382befbe1b48f25867fbbab15015c0b1ce2ab" Oct 30 02:15:18.552036 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:18.551603 6572 scope.go:122] "RemoveContainer" containerID="e0af46bda0dd9b0b72a23b3071d9c559492e76bbcf9c15a837d8d9d97589276a" Oct 30 02:15:18.552239 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:18.552217 6572 scope.go:122] "RemoveContainer" containerID="e340d270403c3cf077a0e66934bf3be9d218fdc4735f7e2bef43a669c5c8be00" Oct 30 02:15:18.621097 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:18.616240 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888775-sv8kf" event={"ID":"99cb9b6a-51ec-41be-a85d-dd1de997fceb","Type":"ContainerStarted","Data":"710659788f7adfc407225b5e79d65afa893cdd8d0f3614dd1a8bd519ca76ebb8"} Oct 30 02:15:18.630241 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:18.630195 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f89697-29888775-9m9lw" event={"ID":"63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4","Type":"ContainerStarted","Data":"935bb42a2fb91f97c1e205afeb511e1f54147ed092734ccb87dbc0f2eea03478"} Oct 30 02:15:18.632052 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:18.632022 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888775-ttncp"] Oct 30 02:15:18.638332 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:18.638296 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888775-fxr88" event={"ID":"2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90","Type":"ContainerStarted","Data":"dbc2b148c876eef7b3ff0cf558f72e9f5b75ec2d5f19ff4d202753951a91d7de"} Oct 30 02:15:18.663123 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:18.662655 6572 generic.go:356] "Generic (PLEG): container finished" podID="f641ac83-f4e6-4003-ae9c-be79e4ca0337" containerID="6c50384620024323388a9d0d5697c6fa7e741aa8a4847e7b589fc4a2a4b1769a" exitCode=60 Oct 30 02:15:18.663123 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:18.662804 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888775-ttncp" event={"ID":"f641ac83-f4e6-4003-ae9c-be79e4ca0337","Type":"ContainerDied","Data":"6c50384620024323388a9d0d5697c6fa7e741aa8a4847e7b589fc4a2a4b1769a"} Oct 30 02:15:18.663123 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:18.662868 6572 scope.go:122] "RemoveContainer" containerID="d0017525943fb3768bac58c0ce27a376501c0e35f5d9bca7e7408a59e4aa6f44" Oct 30 02:15:18.680867 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:18.680826 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888775-zd7gh" event={"ID":"d66a87ee-dc89-4aba-845b-17b58bc6f4a6","Type":"ContainerStarted","Data":"8a4f95baa6b8608c9ee9b854b658522f9161d3460c2f9d349c5f24744623269b"} Oct 30 02:15:18.734045 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:18.733913 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888775-fxr88" podStartSLOduration=18.733890876 podStartE2EDuration="18.733890876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:18.72990875 +0000 UTC m=+3444.306334152" watchObservedRunningTime="2026-10-30 02:15:18.733890876 +0000 UTC m=+3444.310316281" Oct 30 02:15:18.751434 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:18.751384 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-8xvvv" event={"ID":"da67c0ab-cf74-4396-9987-b1bb46eb802b","Type":"ContainerStarted","Data":"68f6fca115aeb984ef152f007aac03e8c81f8cc9236d589c44ca188616159d4b"} Oct 30 02:15:18.759155 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:18.759079 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888775-rbx7m" event={"ID":"0dc3a619-596a-4e28-9845-9d1e2fc7bb06","Type":"ContainerStarted","Data":"a1ca0a954508fa5fa5463aeb5d8d6385f8a51fa84445ff01213b2f74855c3693"} Oct 30 02:15:19.310730 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.310671 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f89697-29888775-9m9lw" podStartSLOduration=19.310653431 podStartE2EDuration="19.310653431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:18.787053287 +0000 UTC m=+3444.363478690" watchObservedRunningTime="2026-10-30 02:15:19.310653431 +0000 UTC m=+3444.887078835" Oct 30 02:15:19.335657 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.335577 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888775-zd7gh" podStartSLOduration=19.335554449 podStartE2EDuration="19.335554449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:18.967143817 +0000 UTC m=+3444.543569243" watchObservedRunningTime="2026-10-30 02:15:19.335554449 +0000 UTC m=+3444.911979853" Oct 30 02:15:19.435019 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.434983 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l" Oct 30 02:15:19.518155 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.518115 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f9f8331-2090-4262-96a0-c3ea360e4375-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f9f8331-2090-4262-96a0-c3ea360e4375-maas-api-service-ca\") pod \"3f9f8331-2090-4262-96a0-c3ea360e4375\" (UID: \"3f9f8331-2090-4262-96a0-c3ea360e4375\") " Oct 30 02:15:19.518502 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.518481 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f9f8331-2090-4262-96a0-c3ea360e4375-kube-api-access-mxml8\" (UniqueName: \"kubernetes.io/projected/3f9f8331-2090-4262-96a0-c3ea360e4375-kube-api-access-mxml8\") pod \"3f9f8331-2090-4262-96a0-c3ea360e4375\" (UID: \"3f9f8331-2090-4262-96a0-c3ea360e4375\") " Oct 30 02:15:19.519801 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.519768 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f9f8331-2090-4262-96a0-c3ea360e4375-maas-api-service-ca" pod "3f9f8331-2090-4262-96a0-c3ea360e4375" (UID: "3f9f8331-2090-4262-96a0-c3ea360e4375"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:19.522193 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.522155 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f9f8331-2090-4262-96a0-c3ea360e4375-kube-api-access-mxml8" pod "3f9f8331-2090-4262-96a0-c3ea360e4375" (UID: "3f9f8331-2090-4262-96a0-c3ea360e4375"). InnerVolumeSpecName "kube-api-access-mxml8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:19.537859 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.536603 6572 scope.go:122] "RemoveContainer" containerID="92d1b42bf8acdcf4b40d283f0f3c7ff3f3cb9f620db40c3360dd21f49a9741ef" Oct 30 02:15:19.590391 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.590334 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:19.621821 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.621789 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxml8\" (UniqueName: \"kubernetes.io/projected/3f9f8331-2090-4262-96a0-c3ea360e4375-kube-api-access-mxml8\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:19.621821 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.621821 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f9f8331-2090-4262-96a0-c3ea360e4375-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:19.674057 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.674008 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888775-sv8kf"] Oct 30 02:15:19.714755 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.714702 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888775-fxr88"] Oct 30 02:15:19.773115 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.773078 6572 generic.go:356] "Generic (PLEG): container finished" podID="2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90" containerID="dbc2b148c876eef7b3ff0cf558f72e9f5b75ec2d5f19ff4d202753951a91d7de" exitCode=60 Oct 30 02:15:19.773340 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.773165 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888775-fxr88" event={"ID":"2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90","Type":"ContainerDied","Data":"dbc2b148c876eef7b3ff0cf558f72e9f5b75ec2d5f19ff4d202753951a91d7de"} Oct 30 02:15:19.773340 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.773229 6572 scope.go:122] "RemoveContainer" containerID="23982c8af5c99d930e4e498d1e13ed07b06d4b35ad2bc90979d24058b0d09719" Oct 30 02:15:19.775442 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.775414 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888775-ttncp" event={"ID":"f641ac83-f4e6-4003-ae9c-be79e4ca0337","Type":"ContainerDied","Data":"2345a6bc6e87c1c1af36c9f81f0dc171a37b7888e8ff92182213681d0ea93f3e"} Oct 30 02:15:19.775611 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.775471 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2345a6bc6e87c1c1af36c9f81f0dc171a37b7888e8ff92182213681d0ea93f3e" Oct 30 02:15:19.778055 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.778021 6572 generic.go:356] "Generic (PLEG): container finished" podID="d66a87ee-dc89-4aba-845b-17b58bc6f4a6" containerID="8a4f95baa6b8608c9ee9b854b658522f9161d3460c2f9d349c5f24744623269b" exitCode=60 Oct 30 02:15:19.778187 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.778085 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888775-zd7gh" event={"ID":"d66a87ee-dc89-4aba-845b-17b58bc6f4a6","Type":"ContainerDied","Data":"8a4f95baa6b8608c9ee9b854b658522f9161d3460c2f9d349c5f24744623269b"} Oct 30 02:15:19.778561 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.778544 6572 scope.go:122] "RemoveContainer" containerID="8a4f95baa6b8608c9ee9b854b658522f9161d3460c2f9d349c5f24744623269b" Oct 30 02:15:19.779014 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:19.778989 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888775-zd7gh_odh-ai-gateway-infra(d66a87ee-dc89-4aba-845b-17b58bc6f4a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888775-zd7gh" podUID="d66a87ee-dc89-4aba-845b-17b58bc6f4a6" Oct 30 02:15:19.780694 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.780666 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888775-82nz4" event={"ID":"75a1834c-2859-4e9c-9d32-80a6f7247460","Type":"ContainerStarted","Data":"a9461b9db3d0ecfd709551e2a184495ee35d555d96a2d9ce2dcce1b8bcbca98e"} Oct 30 02:15:19.783143 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.783121 6572 generic.go:356] "Generic (PLEG): container finished" podID="da67c0ab-cf74-4396-9987-b1bb46eb802b" containerID="68f6fca115aeb984ef152f007aac03e8c81f8cc9236d589c44ca188616159d4b" exitCode=60 Oct 30 02:15:19.783256 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.783192 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-8xvvv" event={"ID":"da67c0ab-cf74-4396-9987-b1bb46eb802b","Type":"ContainerDied","Data":"68f6fca115aeb984ef152f007aac03e8c81f8cc9236d589c44ca188616159d4b"} Oct 30 02:15:19.783613 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.783596 6572 scope.go:122] "RemoveContainer" containerID="68f6fca115aeb984ef152f007aac03e8c81f8cc9236d589c44ca188616159d4b" Oct 30 02:15:19.784064 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:19.784041 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888775-8xvvv_odh-ai-gateway-infra(da67c0ab-cf74-4396-9987-b1bb46eb802b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-8xvvv" podUID="da67c0ab-cf74-4396-9987-b1bb46eb802b" Oct 30 02:15:19.785444 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.785424 6572 generic.go:356] "Generic (PLEG): container finished" podID="0dc3a619-596a-4e28-9845-9d1e2fc7bb06" containerID="a1ca0a954508fa5fa5463aeb5d8d6385f8a51fa84445ff01213b2f74855c3693" exitCode=60 Oct 30 02:15:19.785559 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.785498 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888775-rbx7m" event={"ID":"0dc3a619-596a-4e28-9845-9d1e2fc7bb06","Type":"ContainerDied","Data":"a1ca0a954508fa5fa5463aeb5d8d6385f8a51fa84445ff01213b2f74855c3693"} Oct 30 02:15:19.785931 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.785913 6572 scope.go:122] "RemoveContainer" containerID="a1ca0a954508fa5fa5463aeb5d8d6385f8a51fa84445ff01213b2f74855c3693" Oct 30 02:15:19.786269 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:19.786247 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d44413-29888775-rbx7m_odh-ai-gateway-infra(0dc3a619-596a-4e28-9845-9d1e2fc7bb06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888775-rbx7m" podUID="0dc3a619-596a-4e28-9845-9d1e2fc7bb06" Oct 30 02:15:19.787485 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.787461 6572 generic.go:356] "Generic (PLEG): container finished" podID="99cb9b6a-51ec-41be-a85d-dd1de997fceb" containerID="710659788f7adfc407225b5e79d65afa893cdd8d0f3614dd1a8bd519ca76ebb8" exitCode=60 Oct 30 02:15:19.787578 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.787548 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888775-sv8kf" event={"ID":"99cb9b6a-51ec-41be-a85d-dd1de997fceb","Type":"ContainerDied","Data":"710659788f7adfc407225b5e79d65afa893cdd8d0f3614dd1a8bd519ca76ebb8"} Oct 30 02:15:19.789975 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.789949 6572 generic.go:356] "Generic (PLEG): container finished" podID="63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4" containerID="935bb42a2fb91f97c1e205afeb511e1f54147ed092734ccb87dbc0f2eea03478" exitCode=60 Oct 30 02:15:19.790085 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.789977 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f89697-29888775-9m9lw" event={"ID":"63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4","Type":"ContainerDied","Data":"935bb42a2fb91f97c1e205afeb511e1f54147ed092734ccb87dbc0f2eea03478"} Oct 30 02:15:19.790435 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.790411 6572 scope.go:122] "RemoveContainer" containerID="935bb42a2fb91f97c1e205afeb511e1f54147ed092734ccb87dbc0f2eea03478" Oct 30 02:15:19.790853 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:19.790823 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f89697-29888775-9m9lw_odh-ai-gateway-infra(63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f89697-29888775-9m9lw" podUID="63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4" Oct 30 02:15:19.792600 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.792573 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888775-qr45g" event={"ID":"1f68c408-6bab-42c7-b574-35d5d61c7544","Type":"ContainerStarted","Data":"0795ab88ac3c3d3ff2f53f2e5addf302e7358b0aebde45619d4d80614be755f1"} Oct 30 02:15:19.794104 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.794079 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l" event={"ID":"3f9f8331-2090-4262-96a0-c3ea360e4375","Type":"ContainerDied","Data":"1fd8f2534ee04aca8b594031bf5cb98952724ec4cef18b7432af606aae45e65e"} Oct 30 02:15:19.794104 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.794094 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l" Oct 30 02:15:19.811792 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.811722 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f89697-29888775-9m9lw"] Oct 30 02:15:19.869643 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.869617 6572 scope.go:122] "RemoveContainer" containerID="392a0347ee1fed48949fd4623cd4c4923c4ed2707a60657e1dab6dff8b86f43e" Oct 30 02:15:19.894970 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.894867 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888775-ttncp" Oct 30 02:15:19.900627 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.900593 6572 scope.go:122] "RemoveContainer" containerID="f8ddef1ef53f32698e43625da2e4630c8fdcf4f9e0f286db9e493723421130f5" Oct 30 02:15:19.966006 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:19.965950 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888775-zd7gh"] Oct 30 02:15:20.024795 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.013774 6572 scope.go:122] "RemoveContainer" containerID="bab6ab85395f733507b5656677bac5397befb0f93542e970af90a9490fb4703d" Oct 30 02:15:20.024795 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.013916 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv" Oct 30 02:15:20.030925 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.029149 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f641ac83-f4e6-4003-ae9c-be79e4ca0337-kube-api-access-glsdv\" (UniqueName: \"kubernetes.io/projected/f641ac83-f4e6-4003-ae9c-be79e4ca0337-kube-api-access-glsdv\") pod \"f641ac83-f4e6-4003-ae9c-be79e4ca0337\" (UID: \"f641ac83-f4e6-4003-ae9c-be79e4ca0337\") " Oct 30 02:15:20.030925 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.029206 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f641ac83-f4e6-4003-ae9c-be79e4ca0337-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f641ac83-f4e6-4003-ae9c-be79e4ca0337-maas-api-service-ca\") pod \"f641ac83-f4e6-4003-ae9c-be79e4ca0337\" (UID: \"f641ac83-f4e6-4003-ae9c-be79e4ca0337\") " Oct 30 02:15:20.031194 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.031151 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f641ac83-f4e6-4003-ae9c-be79e4ca0337-maas-api-service-ca" pod "f641ac83-f4e6-4003-ae9c-be79e4ca0337" (UID: "f641ac83-f4e6-4003-ae9c-be79e4ca0337"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:20.034678 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.034639 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f641ac83-f4e6-4003-ae9c-be79e4ca0337-kube-api-access-glsdv" pod "f641ac83-f4e6-4003-ae9c-be79e4ca0337" (UID: "f641ac83-f4e6-4003-ae9c-be79e4ca0337"). InnerVolumeSpecName "kube-api-access-glsdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:20.047653 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.047629 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888775-sv8kf" Oct 30 02:15:20.053271 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.053244 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv" Oct 30 02:15:20.059237 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.059039 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888775-8xvvv"] Oct 30 02:15:20.064173 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.063050 6572 scope.go:122] "RemoveContainer" containerID="a493c551af32b3caca9c6c4b4e73e247853f149efbce0fdda32ed34d5b805e33" Oct 30 02:15:20.096016 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.095987 6572 scope.go:122] "RemoveContainer" containerID="d810fa8660a72b72e23cb870a5b52ed29fd0d1a7547d6b82a8c08b7b756e178a" Oct 30 02:15:20.100316 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.100287 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888775-fxr88" Oct 30 02:15:20.131074 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.131036 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99cb9b6a-51ec-41be-a85d-dd1de997fceb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99cb9b6a-51ec-41be-a85d-dd1de997fceb-maas-api-service-ca\") pod \"99cb9b6a-51ec-41be-a85d-dd1de997fceb\" (UID: \"99cb9b6a-51ec-41be-a85d-dd1de997fceb\") " Oct 30 02:15:20.131241 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.131219 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99cb9b6a-51ec-41be-a85d-dd1de997fceb-kube-api-access-bs7rx\" (UniqueName: \"kubernetes.io/projected/99cb9b6a-51ec-41be-a85d-dd1de997fceb-kube-api-access-bs7rx\") pod \"99cb9b6a-51ec-41be-a85d-dd1de997fceb\" (UID: \"99cb9b6a-51ec-41be-a85d-dd1de997fceb\") " Oct 30 02:15:20.132080 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.132058 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glsdv\" (UniqueName: \"kubernetes.io/projected/f641ac83-f4e6-4003-ae9c-be79e4ca0337-kube-api-access-glsdv\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:20.132125 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.132088 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f641ac83-f4e6-4003-ae9c-be79e4ca0337-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:20.133607 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.133578 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99cb9b6a-51ec-41be-a85d-dd1de997fceb-maas-api-service-ca" pod "99cb9b6a-51ec-41be-a85d-dd1de997fceb" (UID: "99cb9b6a-51ec-41be-a85d-dd1de997fceb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:20.135300 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.135270 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99cb9b6a-51ec-41be-a85d-dd1de997fceb-kube-api-access-bs7rx" pod "99cb9b6a-51ec-41be-a85d-dd1de997fceb" (UID: "99cb9b6a-51ec-41be-a85d-dd1de997fceb"). InnerVolumeSpecName "kube-api-access-bs7rx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:20.158720 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.158684 6572 scope.go:122] "RemoveContainer" containerID="256270db4f75a501f3f070b66c7183368af3b05914e7e72d6e3b6626ae6f21ea" Oct 30 02:15:20.202419 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.202362 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888775-82nz4" podStartSLOduration=20.202346904 podStartE2EDuration="20.202346904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:20.199297872 +0000 UTC m=+3445.775723273" watchObservedRunningTime="2026-10-30 02:15:20.202346904 +0000 UTC m=+3445.778772305" Oct 30 02:15:20.233201 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.233164 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90-maas-api-service-ca\") pod \"2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90\" (UID: \"2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90\") " Oct 30 02:15:20.233408 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.233326 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90-kube-api-access-qgqdv\" (UniqueName: \"kubernetes.io/projected/2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90-kube-api-access-qgqdv\") pod \"2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90\" (UID: \"2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90\") " Oct 30 02:15:20.233553 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.233529 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90-maas-api-service-ca" pod "2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90" (UID: "2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:20.233843 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.233826 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:20.233914 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.233846 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99cb9b6a-51ec-41be-a85d-dd1de997fceb-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:20.233914 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.233855 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs7rx\" (UniqueName: \"kubernetes.io/projected/99cb9b6a-51ec-41be-a85d-dd1de997fceb-kube-api-access-bs7rx\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:20.236218 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.236188 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90-kube-api-access-qgqdv" pod "2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90" (UID: "2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90"). InnerVolumeSpecName "kube-api-access-qgqdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:20.275805 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.275742 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888775-rbx7m"] Oct 30 02:15:20.336093 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.336003 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgqdv\" (UniqueName: \"kubernetes.io/projected/2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90-kube-api-access-qgqdv\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:20.351835 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.351767 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l"] Oct 30 02:15:20.357602 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.357565 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-92f0cc-29888775-nxg6l"] Oct 30 02:15:20.536044 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.536000 6572 scope.go:122] "RemoveContainer" containerID="b0e7bb5695e7d324c0e5fc21d0bdd3c727cf2a31719d2359f8b1118428921980" Oct 30 02:15:20.536044 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.536051 6572 scope.go:122] "RemoveContainer" containerID="0119beb5bd480c062e45ae1cf0e3a2d81e48d25eb0a8aefae9e143a6dfbfdaf9" Oct 30 02:15:20.536330 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.536275 6572 scope.go:122] "RemoveContainer" containerID="83f4a31bc2d5df42e6985efa8edefebb5160b6e612bd4c8575a82f0d9d8b8770" Oct 30 02:15:20.549499 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.549458 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f9f8331-2090-4262-96a0-c3ea360e4375" path="/var/lib/kubelet/pods/3f9f8331-2090-4262-96a0-c3ea360e4375/volumes" Oct 30 02:15:20.810481 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.810453 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888775-fxr88" Oct 30 02:15:20.810672 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.810457 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888775-fxr88" event={"ID":"2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90","Type":"ContainerDied","Data":"a65f7cac80d0eba8df9535e5a73b788f2cc54b13911f08dc9aaa3726850336ed"} Oct 30 02:15:20.810672 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.810555 6572 scope.go:122] "RemoveContainer" containerID="dbc2b148c876eef7b3ff0cf558f72e9f5b75ec2d5f19ff4d202753951a91d7de" Oct 30 02:15:20.814101 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.814069 6572 generic.go:356] "Generic (PLEG): container finished" podID="fec1f4d7-877e-4d7c-88d2-f2365469869a" containerID="b1dc089566b84b3ba2fe877796b41869b9d148e141aabbb4eb92b139dfc93e0f" exitCode=60 Oct 30 02:15:20.814256 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.814103 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d5aa9-29888775-9pqsc" event={"ID":"fec1f4d7-877e-4d7c-88d2-f2365469869a","Type":"ContainerDied","Data":"b1dc089566b84b3ba2fe877796b41869b9d148e141aabbb4eb92b139dfc93e0f"} Oct 30 02:15:20.814669 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.814647 6572 scope.go:122] "RemoveContainer" containerID="b1dc089566b84b3ba2fe877796b41869b9d148e141aabbb4eb92b139dfc93e0f" Oct 30 02:15:20.815157 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:20.815124 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-2d5aa9-29888775-9pqsc_odh-ai-gateway-infra(fec1f4d7-877e-4d7c-88d2-f2365469869a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d5aa9-29888775-9pqsc" podUID="fec1f4d7-877e-4d7c-88d2-f2365469869a" Oct 30 02:15:20.816920 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.816893 6572 generic.go:356] "Generic (PLEG): container finished" podID="4ed6af0f-eb0e-41f8-8b21-654babbb98b5" containerID="695cb097363f84039dce4463719635b96607a6110f772bca106f63613460d2e9" exitCode=60 Oct 30 02:15:20.817050 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.816969 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888775-djsrl" event={"ID":"4ed6af0f-eb0e-41f8-8b21-654babbb98b5","Type":"ContainerDied","Data":"695cb097363f84039dce4463719635b96607a6110f772bca106f63613460d2e9"} Oct 30 02:15:20.817516 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.817493 6572 scope.go:122] "RemoveContainer" containerID="695cb097363f84039dce4463719635b96607a6110f772bca106f63613460d2e9" Oct 30 02:15:20.817986 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:20.817963 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-558327-29888775-djsrl_odh-ai-gateway-infra(4ed6af0f-eb0e-41f8-8b21-654babbb98b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888775-djsrl" podUID="4ed6af0f-eb0e-41f8-8b21-654babbb98b5" Oct 30 02:15:20.819571 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.819544 6572 generic.go:356] "Generic (PLEG): container finished" podID="6edd5565-a917-481f-9302-95011124898b" containerID="3dac0819fc1b19b5eac50802539337aa6c171b2b7ce466375721536fa1204d97" exitCode=60 Oct 30 02:15:20.819674 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.819620 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888775-r46v9" event={"ID":"6edd5565-a917-481f-9302-95011124898b","Type":"ContainerDied","Data":"3dac0819fc1b19b5eac50802539337aa6c171b2b7ce466375721536fa1204d97"} Oct 30 02:15:20.820163 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.820144 6572 scope.go:122] "RemoveContainer" containerID="3dac0819fc1b19b5eac50802539337aa6c171b2b7ce466375721536fa1204d97" Oct 30 02:15:20.820557 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:20.820529 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-803472-29888775-r46v9_odh-ai-gateway-infra(6edd5565-a917-481f-9302-95011124898b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888775-r46v9" podUID="6edd5565-a917-481f-9302-95011124898b" Oct 30 02:15:20.824182 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.824155 6572 generic.go:356] "Generic (PLEG): container finished" podID="75a1834c-2859-4e9c-9d32-80a6f7247460" containerID="a9461b9db3d0ecfd709551e2a184495ee35d555d96a2d9ce2dcce1b8bcbca98e" exitCode=60 Oct 30 02:15:20.824299 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.824231 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888775-82nz4" event={"ID":"75a1834c-2859-4e9c-9d32-80a6f7247460","Type":"ContainerDied","Data":"a9461b9db3d0ecfd709551e2a184495ee35d555d96a2d9ce2dcce1b8bcbca98e"} Oct 30 02:15:20.824725 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.824706 6572 scope.go:122] "RemoveContainer" containerID="a9461b9db3d0ecfd709551e2a184495ee35d555d96a2d9ce2dcce1b8bcbca98e" Oct 30 02:15:20.825194 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:20.825166 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888775-82nz4_odh-ai-gateway-infra(75a1834c-2859-4e9c-9d32-80a6f7247460)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888775-82nz4" podUID="75a1834c-2859-4e9c-9d32-80a6f7247460" Oct 30 02:15:20.829170 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.829131 6572 generic.go:356] "Generic (PLEG): container finished" podID="bcf5437e-0075-467b-a4fd-82b5dc563489" containerID="b20392f5d5f27d99b13cb0bfb23a1ec6efc816065423b1c3bd75660cf0fab805" exitCode=60 Oct 30 02:15:20.829260 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.829197 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888775-289bf" event={"ID":"bcf5437e-0075-467b-a4fd-82b5dc563489","Type":"ContainerDied","Data":"b20392f5d5f27d99b13cb0bfb23a1ec6efc816065423b1c3bd75660cf0fab805"} Oct 30 02:15:20.829696 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.829678 6572 scope.go:122] "RemoveContainer" containerID="b20392f5d5f27d99b13cb0bfb23a1ec6efc816065423b1c3bd75660cf0fab805" Oct 30 02:15:20.830172 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:20.830148 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-292018-29888775-289bf_odh-ai-gateway-infra(bcf5437e-0075-467b-a4fd-82b5dc563489)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888775-289bf" podUID="bcf5437e-0075-467b-a4fd-82b5dc563489" Oct 30 02:15:20.834120 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.834088 6572 generic.go:356] "Generic (PLEG): container finished" podID="c2bd8bc3-6252-42fc-9055-ca393b6f45d9" containerID="ae59a74838f7af9804b2f0319fbfaf489508f09c429125e18f8ee7653fb210d5" exitCode=60 Oct 30 02:15:20.834243 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.834226 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888775-wrhfw" event={"ID":"c2bd8bc3-6252-42fc-9055-ca393b6f45d9","Type":"ContainerDied","Data":"ae59a74838f7af9804b2f0319fbfaf489508f09c429125e18f8ee7653fb210d5"} Oct 30 02:15:20.834722 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.834697 6572 scope.go:122] "RemoveContainer" containerID="ae59a74838f7af9804b2f0319fbfaf489508f09c429125e18f8ee7653fb210d5" Oct 30 02:15:20.835192 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:20.835158 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-cd8173-29888775-wrhfw_odh-ai-gateway-infra(c2bd8bc3-6252-42fc-9055-ca393b6f45d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888775-wrhfw" podUID="c2bd8bc3-6252-42fc-9055-ca393b6f45d9" Oct 30 02:15:20.836151 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.836134 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888775-sv8kf" Oct 30 02:15:20.836722 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.836702 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888775-sv8kf" event={"ID":"99cb9b6a-51ec-41be-a85d-dd1de997fceb","Type":"ContainerDied","Data":"88319ab8fe3cd6d70a99a1fc44e10ef5ed3463212188c659195199abca4e3825"} Oct 30 02:15:20.868061 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.868021 6572 generic.go:356] "Generic (PLEG): container finished" podID="1f68c408-6bab-42c7-b574-35d5d61c7544" containerID="0795ab88ac3c3d3ff2f53f2e5addf302e7358b0aebde45619d4d80614be755f1" exitCode=60 Oct 30 02:15:20.869403 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.869367 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888775-qr45g" event={"ID":"1f68c408-6bab-42c7-b574-35d5d61c7544","Type":"ContainerDied","Data":"0795ab88ac3c3d3ff2f53f2e5addf302e7358b0aebde45619d4d80614be755f1"} Oct 30 02:15:20.871505 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.871476 6572 scope.go:122] "RemoveContainer" containerID="0795ab88ac3c3d3ff2f53f2e5addf302e7358b0aebde45619d4d80614be755f1" Oct 30 02:15:20.873370 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:20.872335 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888775-qr45g_odh-ai-gateway-infra(1f68c408-6bab-42c7-b574-35d5d61c7544)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888775-qr45g" podUID="1f68c408-6bab-42c7-b574-35d5d61c7544" Oct 30 02:15:20.878640 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.878610 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888775-ttncp" Oct 30 02:15:20.878816 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.878489 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888775-brm4t" event={"ID":"1c7f35b4-a13a-4be3-be70-135dcc096f49","Type":"ContainerStarted","Data":"1fbe6340af431b8c51dae8396cedcdc3c25254b55f2bfc8d81a0c3f055ec516b"} Oct 30 02:15:20.880230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.880201 6572 scope.go:122] "RemoveContainer" containerID="1fbe6340af431b8c51dae8396cedcdc3c25254b55f2bfc8d81a0c3f055ec516b" Oct 30 02:15:20.881288 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:20.881244 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888775-brm4t_odh-ai-gateway-infra(1c7f35b4-a13a-4be3-be70-135dcc096f49)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888775-brm4t" podUID="1c7f35b4-a13a-4be3-be70-135dcc096f49" Oct 30 02:15:20.899731 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:20.899699 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv" Oct 30 02:15:21.085161 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.084349 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888775-289bf"] Oct 30 02:15:21.091088 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.091056 6572 scope.go:122] "RemoveContainer" containerID="a763a8188134fed66f2082d810679e403c93177f313eea4218187fcf6b39d917" Oct 30 02:15:21.108932 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.108680 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-8xvvv" Oct 30 02:15:21.120731 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.120647 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888775-zd7gh" Oct 30 02:15:21.133508 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.133481 6572 scope.go:122] "RemoveContainer" containerID="e0af46bda0dd9b0b72a23b3071d9c559492e76bbcf9c15a837d8d9d97589276a" Oct 30 02:15:21.133736 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.133702 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:21.188847 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.188812 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888775-qr45g"] Oct 30 02:15:21.197725 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.197487 6572 scope.go:122] "RemoveContainer" containerID="572bc92b6afa1a7231e116cc191382befbe1b48f25867fbbab15015c0b1ce2ab" Oct 30 02:15:21.252796 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.235937 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888775-rbx7m" Oct 30 02:15:21.277644 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.277595 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da67c0ab-cf74-4396-9987-b1bb46eb802b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da67c0ab-cf74-4396-9987-b1bb46eb802b-maas-api-service-ca\") pod \"da67c0ab-cf74-4396-9987-b1bb46eb802b\" (UID: \"da67c0ab-cf74-4396-9987-b1bb46eb802b\") " Oct 30 02:15:21.278142 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.278117 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d66a87ee-dc89-4aba-845b-17b58bc6f4a6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d66a87ee-dc89-4aba-845b-17b58bc6f4a6-maas-api-service-ca\") pod \"d66a87ee-dc89-4aba-845b-17b58bc6f4a6\" (UID: \"d66a87ee-dc89-4aba-845b-17b58bc6f4a6\") " Oct 30 02:15:21.278334 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.278317 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da67c0ab-cf74-4396-9987-b1bb46eb802b-kube-api-access-c62p2\" (UniqueName: \"kubernetes.io/projected/da67c0ab-cf74-4396-9987-b1bb46eb802b-kube-api-access-c62p2\") pod \"da67c0ab-cf74-4396-9987-b1bb46eb802b\" (UID: \"da67c0ab-cf74-4396-9987-b1bb46eb802b\") " Oct 30 02:15:21.278564 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.278550 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d66a87ee-dc89-4aba-845b-17b58bc6f4a6-kube-api-access-vc59p\" (UniqueName: \"kubernetes.io/projected/d66a87ee-dc89-4aba-845b-17b58bc6f4a6-kube-api-access-vc59p\") pod \"d66a87ee-dc89-4aba-845b-17b58bc6f4a6\" (UID: \"d66a87ee-dc89-4aba-845b-17b58bc6f4a6\") " Oct 30 02:15:21.281468 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.281346 6572 scope.go:122] "RemoveContainer" containerID="9ef8956dbc2e3fa923dfd2fdb2676d58f0c50bf801af6822838ef48d50eee5e3" Oct 30 02:15:21.281468 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.281438 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da67c0ab-cf74-4396-9987-b1bb46eb802b-maas-api-service-ca" pod "da67c0ab-cf74-4396-9987-b1bb46eb802b" (UID: "da67c0ab-cf74-4396-9987-b1bb46eb802b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:21.281811 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.281792 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d66a87ee-dc89-4aba-845b-17b58bc6f4a6-maas-api-service-ca" pod "d66a87ee-dc89-4aba-845b-17b58bc6f4a6" (UID: "d66a87ee-dc89-4aba-845b-17b58bc6f4a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:21.289992 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.289900 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f89697-29888775-9m9lw" Oct 30 02:15:21.336000 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.335892 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888775-82nz4"] Oct 30 02:15:21.405853 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.381800 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4-kube-api-access-nr8vv\" (UniqueName: \"kubernetes.io/projected/63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4-kube-api-access-nr8vv\") pod \"63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4\" (UID: \"63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4\") " Oct 30 02:15:21.405853 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.381852 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dc3a619-596a-4e28-9845-9d1e2fc7bb06-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dc3a619-596a-4e28-9845-9d1e2fc7bb06-maas-api-service-ca\") pod \"0dc3a619-596a-4e28-9845-9d1e2fc7bb06\" (UID: \"0dc3a619-596a-4e28-9845-9d1e2fc7bb06\") " Oct 30 02:15:21.405853 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.381958 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4-maas-api-service-ca\") pod \"63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4\" (UID: \"63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4\") " Oct 30 02:15:21.405853 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.382296 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dc3a619-596a-4e28-9845-9d1e2fc7bb06-kube-api-access-47xwk\" (UniqueName: \"kubernetes.io/projected/0dc3a619-596a-4e28-9845-9d1e2fc7bb06-kube-api-access-47xwk\") pod \"0dc3a619-596a-4e28-9845-9d1e2fc7bb06\" (UID: \"0dc3a619-596a-4e28-9845-9d1e2fc7bb06\") " Oct 30 02:15:21.445027 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.444543 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d66a87ee-dc89-4aba-845b-17b58bc6f4a6-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:21.445027 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.444578 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da67c0ab-cf74-4396-9987-b1bb46eb802b-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:21.447554 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.447521 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dc3a619-596a-4e28-9845-9d1e2fc7bb06-maas-api-service-ca" pod "0dc3a619-596a-4e28-9845-9d1e2fc7bb06" (UID: "0dc3a619-596a-4e28-9845-9d1e2fc7bb06"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:21.448147 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.448122 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4-maas-api-service-ca" pod "63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4" (UID: "63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:21.515686 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.515641 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-cd8173-8445cbb698-7wtm5" podUID="909b1459-46ea-4c3c-bc2e-1fb29527200a" containerName="payload-processing" containerID="cri-o://62ea3c1253f7e97159102d95e5784501cc79b28778bed95989c3ae698979a14a" gracePeriod=25 Oct 30 02:15:21.568086 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.543587 6572 scope.go:122] "RemoveContainer" containerID="e0656f18299892b23fe64ecd9b7550fefa33b0c01294b7c6bdf10f73b1efc0e7" Oct 30 02:15:21.572804 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.546263 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:21.579320 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.579280 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dc3a619-596a-4e28-9845-9d1e2fc7bb06-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:21.633326 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.633283 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da67c0ab-cf74-4396-9987-b1bb46eb802b-kube-api-access-c62p2" pod "da67c0ab-cf74-4396-9987-b1bb46eb802b" (UID: "da67c0ab-cf74-4396-9987-b1bb46eb802b"). InnerVolumeSpecName "kube-api-access-c62p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:21.638570 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.638524 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dc3a619-596a-4e28-9845-9d1e2fc7bb06-kube-api-access-47xwk" pod "0dc3a619-596a-4e28-9845-9d1e2fc7bb06" (UID: "0dc3a619-596a-4e28-9845-9d1e2fc7bb06"). InnerVolumeSpecName "kube-api-access-47xwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:21.638696 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.638589 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d66a87ee-dc89-4aba-845b-17b58bc6f4a6-kube-api-access-vc59p" pod "d66a87ee-dc89-4aba-845b-17b58bc6f4a6" (UID: "d66a87ee-dc89-4aba-845b-17b58bc6f4a6"). InnerVolumeSpecName "kube-api-access-vc59p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:21.660351 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.659189 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4-kube-api-access-nr8vv" pod "63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4" (UID: "63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4"). InnerVolumeSpecName "kube-api-access-nr8vv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:21.681225 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.680971 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47xwk\" (UniqueName: \"kubernetes.io/projected/0dc3a619-596a-4e28-9845-9d1e2fc7bb06-kube-api-access-47xwk\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:21.681225 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.681009 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c62p2\" (UniqueName: \"kubernetes.io/projected/da67c0ab-cf74-4396-9987-b1bb46eb802b-kube-api-access-c62p2\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:21.681225 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.681024 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr8vv\" (UniqueName: \"kubernetes.io/projected/63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4-kube-api-access-nr8vv\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:21.681225 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.681043 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc59p\" (UniqueName: \"kubernetes.io/projected/d66a87ee-dc89-4aba-845b-17b58bc6f4a6-kube-api-access-vc59p\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:21.706091 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.705832 6572 scope.go:122] "RemoveContainer" containerID="11379d21f289dc1837deb3ee9c67074732cd20c8a1623f99fba3153cbdd40546" Oct 30 02:15:21.758766 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.758629 6572 scope.go:122] "RemoveContainer" containerID="e340d270403c3cf077a0e66934bf3be9d218fdc4735f7e2bef43a669c5c8be00" Oct 30 02:15:21.767944 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.767905 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:21.769280 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.769232 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888775-ttncp"] Oct 30 02:15:21.834968 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.801425 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-657fba-29888775-ttncp"] Oct 30 02:15:21.835554 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.835519 6572 scope.go:122] "RemoveContainer" containerID="710659788f7adfc407225b5e79d65afa893cdd8d0f3614dd1a8bd519ca76ebb8" Oct 30 02:15:21.843240 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.843173 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888775-fxr88"] Oct 30 02:15:21.855463 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.855406 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-264bdd-29888775-fxr88"] Oct 30 02:15:21.859551 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.859525 6572 scope.go:122] "RemoveContainer" containerID="ed4d3e7a496e64a0b66df34ed26e51728754ae27ab02c2506dbe9e0fbdc2b307" Oct 30 02:15:21.876720 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.876553 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-292018-fjhhq"] Oct 30 02:15:21.887789 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.881605 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888775-sv8kf"] Oct 30 02:15:21.909794 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.890043 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-292018-fjhhq"] Oct 30 02:15:21.941694 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.941549 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d376-29888775-sv8kf"] Oct 30 02:15:21.957497 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.957461 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888775-vcqdq" event={"ID":"ba0ac388-9127-47e0-8985-dd48570c8d27","Type":"ContainerStarted","Data":"2c22c0e37f8ea7985e18a766f62ef4ab229a28b6e9b21a6096e443273e1989de"} Oct 30 02:15:21.968501 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.968458 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888775-8tzkk" event={"ID":"bdf23f37-2870-4b9c-b558-9fea4452f4e5","Type":"ContainerStarted","Data":"39a696c948ceba59a65a157bd3afe843261f3a9b0bd4b5d07353300afaaa7292"} Oct 30 02:15:21.974340 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.974309 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d5aa9-29888775-9pqsc"] Oct 30 02:15:21.988409 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.988365 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888775-p4j7s" event={"ID":"8968e7e0-04c1-4469-9062-f7d31ee6dbf2","Type":"ContainerStarted","Data":"4ca16cea826c1086899ce766a8c4a731d60267e36fccede601a0fb7c3dcd019a"} Oct 30 02:15:21.993008 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:21.992975 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:22.002966 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.002927 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888775-zd7gh" event={"ID":"d66a87ee-dc89-4aba-845b-17b58bc6f4a6","Type":"ContainerDied","Data":"2f7e2f9ebead493b271ecd7fe9b8088c4b738a589927d851fae515a519dcfb60"} Oct 30 02:15:22.003171 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.003020 6572 scope.go:122] "RemoveContainer" containerID="8a4f95baa6b8608c9ee9b854b658522f9161d3460c2f9d349c5f24744623269b" Oct 30 02:15:22.003261 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.003175 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888775-zd7gh" Oct 30 02:15:22.017562 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.017535 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-8xvvv" Oct 30 02:15:22.023285 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.021896 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888775-djsrl"] Oct 30 02:15:22.023285 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.021974 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-8xvvv" event={"ID":"da67c0ab-cf74-4396-9987-b1bb46eb802b","Type":"ContainerDied","Data":"71096c36965ec8e0269cb84d48fd7651da2cf9fa7b077d93068d5d61d858a3cb"} Oct 30 02:15:22.030297 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.025737 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888775-vcqdq" podStartSLOduration=22.025719436 podStartE2EDuration="22.025719436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:21.999730403 +0000 UTC m=+3447.576155806" watchObservedRunningTime="2026-10-30 02:15:22.025719436 +0000 UTC m=+3447.602144839" Oct 30 02:15:22.069233 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.067188 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888775-rbx7m" Oct 30 02:15:22.078060 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.078013 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888775-rbx7m" event={"ID":"0dc3a619-596a-4e28-9845-9d1e2fc7bb06","Type":"ContainerDied","Data":"e7d2c5bba25f429f6a1beb16c42dbe23129791b7f1ad14d01c21c90beb1b8fdd"} Oct 30 02:15:22.079795 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.079764 6572 scope.go:122] "RemoveContainer" containerID="68f6fca115aeb984ef152f007aac03e8c81f8cc9236d589c44ca188616159d4b" Oct 30 02:15:22.088224 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.086923 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f89697-29888775-9m9lw" event={"ID":"63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4","Type":"ContainerDied","Data":"0bcdb372955770ff64b167fe858eda04982d2e3d04340d8b1f78a3df98e090e2"} Oct 30 02:15:22.088224 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.087073 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f89697-29888775-9m9lw" Oct 30 02:15:22.111563 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.110283 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888775-r46v9"] Oct 30 02:15:22.118114 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.117910 6572 generic.go:356] "Generic (PLEG): container finished" podID="1c7f35b4-a13a-4be3-be70-135dcc096f49" containerID="1fbe6340af431b8c51dae8396cedcdc3c25254b55f2bfc8d81a0c3f055ec516b" exitCode=60 Oct 30 02:15:22.118973 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.118937 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888775-brm4t" event={"ID":"1c7f35b4-a13a-4be3-be70-135dcc096f49","Type":"ContainerDied","Data":"1fbe6340af431b8c51dae8396cedcdc3c25254b55f2bfc8d81a0c3f055ec516b"} Oct 30 02:15:22.120235 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.120207 6572 scope.go:122] "RemoveContainer" containerID="1fbe6340af431b8c51dae8396cedcdc3c25254b55f2bfc8d81a0c3f055ec516b" Oct 30 02:15:22.142445 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:22.121467 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888775-brm4t_odh-ai-gateway-infra(1c7f35b4-a13a-4be3-be70-135dcc096f49)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888775-brm4t" podUID="1c7f35b4-a13a-4be3-be70-135dcc096f49" Oct 30 02:15:22.142445 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.135320 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888775-p4j7s" podStartSLOduration=22.135297241 podStartE2EDuration="22.135297241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:22.092482323 +0000 UTC m=+3447.668907742" watchObservedRunningTime="2026-10-30 02:15:22.135297241 +0000 UTC m=+3447.711722643" Oct 30 02:15:22.209208 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.208077 6572 scope.go:122] "RemoveContainer" containerID="a1ca0a954508fa5fa5463aeb5d8d6385f8a51fa84445ff01213b2f74855c3693" Oct 30 02:15:22.319271 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.319236 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888775-wrhfw"] Oct 30 02:15:22.321825 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.321741 6572 scope.go:122] "RemoveContainer" containerID="935bb42a2fb91f97c1e205afeb511e1f54147ed092734ccb87dbc0f2eea03478" Oct 30 02:15:22.346035 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.345879 6572 scope.go:122] "RemoveContainer" containerID="92d1b42bf8acdcf4b40d283f0f3c7ff3f3cb9f620db40c3360dd21f49a9741ef" Oct 30 02:15:22.352333 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.352241 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d5aa9-29888775-9pqsc" Oct 30 02:15:22.363144 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.363103 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888775-brm4t"] Oct 30 02:15:22.368097 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.368026 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fec1f4d7-877e-4d7c-88d2-f2365469869a-maas-api-service-ca" pod "fec1f4d7-877e-4d7c-88d2-f2365469869a" (UID: "fec1f4d7-877e-4d7c-88d2-f2365469869a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:22.370027 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.369814 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:22.370274 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.370249 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fec1f4d7-877e-4d7c-88d2-f2365469869a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec1f4d7-877e-4d7c-88d2-f2365469869a-maas-api-service-ca\") pod \"fec1f4d7-877e-4d7c-88d2-f2365469869a\" (UID: \"fec1f4d7-877e-4d7c-88d2-f2365469869a\") " Oct 30 02:15:22.370935 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.370907 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fec1f4d7-877e-4d7c-88d2-f2365469869a-kube-api-access-5tx8z\" (UniqueName: \"kubernetes.io/projected/fec1f4d7-877e-4d7c-88d2-f2365469869a-kube-api-access-5tx8z\") pod \"fec1f4d7-877e-4d7c-88d2-f2365469869a\" (UID: \"fec1f4d7-877e-4d7c-88d2-f2365469869a\") " Oct 30 02:15:22.372288 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.372261 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec1f4d7-877e-4d7c-88d2-f2365469869a-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:22.372880 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.372856 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888775-zd7gh"] Oct 30 02:15:22.383879 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.382935 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888775-zd7gh"] Oct 30 02:15:22.424340 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.423530 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec1f4d7-877e-4d7c-88d2-f2365469869a-kube-api-access-5tx8z" pod "fec1f4d7-877e-4d7c-88d2-f2365469869a" (UID: "fec1f4d7-877e-4d7c-88d2-f2365469869a"). InnerVolumeSpecName "kube-api-access-5tx8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:22.462392 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.461494 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888775-8xvvv"] Oct 30 02:15:22.468939 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.468857 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888775-8xvvv"] Oct 30 02:15:22.474131 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.473945 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tx8z\" (UniqueName: \"kubernetes.io/projected/fec1f4d7-877e-4d7c-88d2-f2365469869a-kube-api-access-5tx8z\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:22.497002 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.496928 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888775-rbx7m"] Oct 30 02:15:22.505630 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.505594 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d44413-29888775-rbx7m"] Oct 30 02:15:22.527462 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.523535 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888775-djsrl" Oct 30 02:15:22.559566 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.559532 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888775-r46v9" Oct 30 02:15:22.561166 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.561138 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dc3a619-596a-4e28-9845-9d1e2fc7bb06" path="/var/lib/kubelet/pods/0dc3a619-596a-4e28-9845-9d1e2fc7bb06/volumes" Oct 30 02:15:22.561970 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.561948 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90" path="/var/lib/kubelet/pods/2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90/volumes" Oct 30 02:15:22.562728 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.562708 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99cb9b6a-51ec-41be-a85d-dd1de997fceb" path="/var/lib/kubelet/pods/99cb9b6a-51ec-41be-a85d-dd1de997fceb/volumes" Oct 30 02:15:22.563812 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.563791 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a238ef55-f32f-42c5-9e67-a478bf8eeb8c" path="/var/lib/kubelet/pods/a238ef55-f32f-42c5-9e67-a478bf8eeb8c/volumes" Oct 30 02:15:22.564277 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.564258 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d66a87ee-dc89-4aba-845b-17b58bc6f4a6" path="/var/lib/kubelet/pods/d66a87ee-dc89-4aba-845b-17b58bc6f4a6/volumes" Oct 30 02:15:22.564734 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.564717 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da67c0ab-cf74-4396-9987-b1bb46eb802b" path="/var/lib/kubelet/pods/da67c0ab-cf74-4396-9987-b1bb46eb802b/volumes" Oct 30 02:15:22.565090 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.565064 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888775-82nz4" Oct 30 02:15:22.565733 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.565715 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f641ac83-f4e6-4003-ae9c-be79e4ca0337" path="/var/lib/kubelet/pods/f641ac83-f4e6-4003-ae9c-be79e4ca0337/volumes" Oct 30 02:15:22.578907 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.578858 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ed6af0f-eb0e-41f8-8b21-654babbb98b5-maas-api-service-ca" pod "4ed6af0f-eb0e-41f8-8b21-654babbb98b5" (UID: "4ed6af0f-eb0e-41f8-8b21-654babbb98b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:22.579696 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.578298 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ed6af0f-eb0e-41f8-8b21-654babbb98b5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ed6af0f-eb0e-41f8-8b21-654babbb98b5-maas-api-service-ca\") pod \"4ed6af0f-eb0e-41f8-8b21-654babbb98b5\" (UID: \"4ed6af0f-eb0e-41f8-8b21-654babbb98b5\") " Oct 30 02:15:22.579696 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.579652 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ed6af0f-eb0e-41f8-8b21-654babbb98b5-kube-api-access-dbttd\" (UniqueName: \"kubernetes.io/projected/4ed6af0f-eb0e-41f8-8b21-654babbb98b5-kube-api-access-dbttd\") pod \"4ed6af0f-eb0e-41f8-8b21-654babbb98b5\" (UID: \"4ed6af0f-eb0e-41f8-8b21-654babbb98b5\") " Oct 30 02:15:22.579886 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.579780 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6edd5565-a917-481f-9302-95011124898b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6edd5565-a917-481f-9302-95011124898b-maas-api-service-ca\") pod \"6edd5565-a917-481f-9302-95011124898b\" (UID: \"6edd5565-a917-481f-9302-95011124898b\") " Oct 30 02:15:22.579886 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.579840 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6edd5565-a917-481f-9302-95011124898b-kube-api-access-tjgfp\" (UniqueName: \"kubernetes.io/projected/6edd5565-a917-481f-9302-95011124898b-kube-api-access-tjgfp\") pod \"6edd5565-a917-481f-9302-95011124898b\" (UID: \"6edd5565-a917-481f-9302-95011124898b\") " Oct 30 02:15:22.580800 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.580465 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6edd5565-a917-481f-9302-95011124898b-maas-api-service-ca" pod "6edd5565-a917-481f-9302-95011124898b" (UID: "6edd5565-a917-481f-9302-95011124898b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:22.581172 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.581141 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6edd5565-a917-481f-9302-95011124898b-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:22.581275 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.581173 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ed6af0f-eb0e-41f8-8b21-654babbb98b5-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:22.582438 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.582410 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ed6af0f-eb0e-41f8-8b21-654babbb98b5-kube-api-access-dbttd" pod "4ed6af0f-eb0e-41f8-8b21-654babbb98b5" (UID: "4ed6af0f-eb0e-41f8-8b21-654babbb98b5"). InnerVolumeSpecName "kube-api-access-dbttd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:22.587076 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.586965 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6edd5565-a917-481f-9302-95011124898b-kube-api-access-tjgfp" pod "6edd5565-a917-481f-9302-95011124898b" (UID: "6edd5565-a917-481f-9302-95011124898b"). InnerVolumeSpecName "kube-api-access-tjgfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:22.588805 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.588696 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:22.683108 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.683074 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75a1834c-2859-4e9c-9d32-80a6f7247460-kube-api-access-j8ghh\" (UniqueName: \"kubernetes.io/projected/75a1834c-2859-4e9c-9d32-80a6f7247460-kube-api-access-j8ghh\") pod \"75a1834c-2859-4e9c-9d32-80a6f7247460\" (UID: \"75a1834c-2859-4e9c-9d32-80a6f7247460\") " Oct 30 02:15:22.683369 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.683350 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75a1834c-2859-4e9c-9d32-80a6f7247460-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a1834c-2859-4e9c-9d32-80a6f7247460-maas-api-service-ca\") pod \"75a1834c-2859-4e9c-9d32-80a6f7247460\" (UID: \"75a1834c-2859-4e9c-9d32-80a6f7247460\") " Oct 30 02:15:22.684806 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.684708 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbttd\" (UniqueName: \"kubernetes.io/projected/4ed6af0f-eb0e-41f8-8b21-654babbb98b5-kube-api-access-dbttd\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:22.685049 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.685031 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjgfp\" (UniqueName: \"kubernetes.io/projected/6edd5565-a917-481f-9302-95011124898b-kube-api-access-tjgfp\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:22.685791 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.685762 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75a1834c-2859-4e9c-9d32-80a6f7247460-maas-api-service-ca" pod "75a1834c-2859-4e9c-9d32-80a6f7247460" (UID: "75a1834c-2859-4e9c-9d32-80a6f7247460"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:22.690560 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.690509 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a1834c-2859-4e9c-9d32-80a6f7247460-kube-api-access-j8ghh" pod "75a1834c-2859-4e9c-9d32-80a6f7247460" (UID: "75a1834c-2859-4e9c-9d32-80a6f7247460"). InnerVolumeSpecName "kube-api-access-j8ghh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:22.712186 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.712114 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f89697-29888775-9m9lw"] Oct 30 02:15:22.720092 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.720000 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f89697-29888775-9m9lw"] Oct 30 02:15:22.746002 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.745967 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888775-289bf" Oct 30 02:15:22.750362 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.750335 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888775-qr45g" Oct 30 02:15:22.764482 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.764450 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888775-wrhfw" Oct 30 02:15:22.786170 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.786127 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f68c408-6bab-42c7-b574-35d5d61c7544-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f68c408-6bab-42c7-b574-35d5d61c7544-maas-api-service-ca\") pod \"1f68c408-6bab-42c7-b574-35d5d61c7544\" (UID: \"1f68c408-6bab-42c7-b574-35d5d61c7544\") " Oct 30 02:15:22.786406 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.786209 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2bd8bc3-6252-42fc-9055-ca393b6f45d9-kube-api-access-px6wh\" (UniqueName: \"kubernetes.io/projected/c2bd8bc3-6252-42fc-9055-ca393b6f45d9-kube-api-access-px6wh\") pod \"c2bd8bc3-6252-42fc-9055-ca393b6f45d9\" (UID: \"c2bd8bc3-6252-42fc-9055-ca393b6f45d9\") " Oct 30 02:15:22.786406 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.786367 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2bd8bc3-6252-42fc-9055-ca393b6f45d9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2bd8bc3-6252-42fc-9055-ca393b6f45d9-maas-api-service-ca\") pod \"c2bd8bc3-6252-42fc-9055-ca393b6f45d9\" (UID: \"c2bd8bc3-6252-42fc-9055-ca393b6f45d9\") " Oct 30 02:15:22.786547 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.786411 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcf5437e-0075-467b-a4fd-82b5dc563489-kube-api-access-k9vdw\" (UniqueName: \"kubernetes.io/projected/bcf5437e-0075-467b-a4fd-82b5dc563489-kube-api-access-k9vdw\") pod \"bcf5437e-0075-467b-a4fd-82b5dc563489\" (UID: \"bcf5437e-0075-467b-a4fd-82b5dc563489\") " Oct 30 02:15:22.786547 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.786453 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f68c408-6bab-42c7-b574-35d5d61c7544-kube-api-access-gl7vc\" (UniqueName: \"kubernetes.io/projected/1f68c408-6bab-42c7-b574-35d5d61c7544-kube-api-access-gl7vc\") pod \"1f68c408-6bab-42c7-b574-35d5d61c7544\" (UID: \"1f68c408-6bab-42c7-b574-35d5d61c7544\") " Oct 30 02:15:22.786673 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.786586 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f68c408-6bab-42c7-b574-35d5d61c7544-maas-api-service-ca" pod "1f68c408-6bab-42c7-b574-35d5d61c7544" (UID: "1f68c408-6bab-42c7-b574-35d5d61c7544"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:22.788180 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.786927 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcf5437e-0075-467b-a4fd-82b5dc563489-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcf5437e-0075-467b-a4fd-82b5dc563489-maas-api-service-ca\") pod \"bcf5437e-0075-467b-a4fd-82b5dc563489\" (UID: \"bcf5437e-0075-467b-a4fd-82b5dc563489\") " Oct 30 02:15:22.788180 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.787859 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f68c408-6bab-42c7-b574-35d5d61c7544-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:22.788180 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.787885 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8ghh\" (UniqueName: \"kubernetes.io/projected/75a1834c-2859-4e9c-9d32-80a6f7247460-kube-api-access-j8ghh\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:22.788180 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.787901 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a1834c-2859-4e9c-9d32-80a6f7247460-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:22.788692 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.788662 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcf5437e-0075-467b-a4fd-82b5dc563489-maas-api-service-ca" pod "bcf5437e-0075-467b-a4fd-82b5dc563489" (UID: "bcf5437e-0075-467b-a4fd-82b5dc563489"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:22.788887 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.788855 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2bd8bc3-6252-42fc-9055-ca393b6f45d9-maas-api-service-ca" pod "c2bd8bc3-6252-42fc-9055-ca393b6f45d9" (UID: "c2bd8bc3-6252-42fc-9055-ca393b6f45d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:22.791419 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.791392 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcf5437e-0075-467b-a4fd-82b5dc563489-kube-api-access-k9vdw" pod "bcf5437e-0075-467b-a4fd-82b5dc563489" (UID: "bcf5437e-0075-467b-a4fd-82b5dc563489"). InnerVolumeSpecName "kube-api-access-k9vdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:22.791539 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.791434 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f68c408-6bab-42c7-b574-35d5d61c7544-kube-api-access-gl7vc" pod "1f68c408-6bab-42c7-b574-35d5d61c7544" (UID: "1f68c408-6bab-42c7-b574-35d5d61c7544"). InnerVolumeSpecName "kube-api-access-gl7vc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:22.791925 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.791895 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2bd8bc3-6252-42fc-9055-ca393b6f45d9-kube-api-access-px6wh" pod "c2bd8bc3-6252-42fc-9055-ca393b6f45d9" (UID: "c2bd8bc3-6252-42fc-9055-ca393b6f45d9"). InnerVolumeSpecName "kube-api-access-px6wh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:22.889869 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.889828 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2bd8bc3-6252-42fc-9055-ca393b6f45d9-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:22.890058 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.889874 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9vdw\" (UniqueName: \"kubernetes.io/projected/bcf5437e-0075-467b-a4fd-82b5dc563489-kube-api-access-k9vdw\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:22.890058 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.889894 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl7vc\" (UniqueName: \"kubernetes.io/projected/1f68c408-6bab-42c7-b574-35d5d61c7544-kube-api-access-gl7vc\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:22.890058 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.889909 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcf5437e-0075-467b-a4fd-82b5dc563489-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:22.890058 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:22.889924 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px6wh\" (UniqueName: \"kubernetes.io/projected/c2bd8bc3-6252-42fc-9055-ca393b6f45d9-kube-api-access-px6wh\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:23.015256 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.015143 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888775-vcqdq"] Oct 30 02:15:23.063081 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.063035 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888775-8tzkk"] Oct 30 02:15:23.104672 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.104629 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888775-p4j7s"] Oct 30 02:15:23.130763 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.130719 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d5aa9-29888775-9pqsc" Oct 30 02:15:23.130763 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.130730 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d5aa9-29888775-9pqsc" event={"ID":"fec1f4d7-877e-4d7c-88d2-f2365469869a","Type":"ContainerDied","Data":"4e47374c2d0eac041e284c717257c2ac6c06abf3571d8e9195fd23922e876c97"} Oct 30 02:15:23.131009 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.130816 6572 scope.go:122] "RemoveContainer" containerID="b1dc089566b84b3ba2fe877796b41869b9d148e141aabbb4eb92b139dfc93e0f" Oct 30 02:15:23.133209 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.133176 6572 generic.go:356] "Generic (PLEG): container finished" podID="8968e7e0-04c1-4469-9062-f7d31ee6dbf2" containerID="4ca16cea826c1086899ce766a8c4a731d60267e36fccede601a0fb7c3dcd019a" exitCode=60 Oct 30 02:15:23.133379 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.133318 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888775-p4j7s" event={"ID":"8968e7e0-04c1-4469-9062-f7d31ee6dbf2","Type":"ContainerDied","Data":"4ca16cea826c1086899ce766a8c4a731d60267e36fccede601a0fb7c3dcd019a"} Oct 30 02:15:23.136289 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.136266 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888775-djsrl" Oct 30 02:15:23.136443 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.136296 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888775-djsrl" event={"ID":"4ed6af0f-eb0e-41f8-8b21-654babbb98b5","Type":"ContainerDied","Data":"fdb4778917a6ea1ac9c1c6c8bbff9bbc02c85aebf8cfe87d28c11f53919f80bb"} Oct 30 02:15:23.142169 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.142129 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888775-r46v9" event={"ID":"6edd5565-a917-481f-9302-95011124898b","Type":"ContainerDied","Data":"717afdbbc6f03ed9048ec080972cab4bd269e8bd1a4693f41ca170f087a02d54"} Oct 30 02:15:23.142317 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.142251 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888775-r46v9" Oct 30 02:15:23.147497 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.147465 6572 scope.go:122] "RemoveContainer" containerID="83f4a31bc2d5df42e6985efa8edefebb5160b6e612bd4c8575a82f0d9d8b8770" Oct 30 02:15:23.152712 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.152683 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888775-82nz4" event={"ID":"75a1834c-2859-4e9c-9d32-80a6f7247460","Type":"ContainerDied","Data":"4b756c8f54c4132b4d4fc77b83ebc5c6f39857aa48d98e409b80813a60e5c2a0"} Oct 30 02:15:23.152712 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.152701 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888775-82nz4" Oct 30 02:15:23.158279 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.158172 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888775-289bf" Oct 30 02:15:23.158279 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.158191 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888775-289bf" event={"ID":"bcf5437e-0075-467b-a4fd-82b5dc563489","Type":"ContainerDied","Data":"94458f05692f600a1874158107980c98a5cf5c3ccae02ae8a0d474bf7bc8a67d"} Oct 30 02:15:23.165531 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.164117 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888775-wrhfw" Oct 30 02:15:23.165531 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.164173 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888775-wrhfw" event={"ID":"c2bd8bc3-6252-42fc-9055-ca393b6f45d9","Type":"ContainerDied","Data":"ec71cf41ebcc993fe1be28e8f90b4bcaf5eae31b989edeba11e3dd9cd35c4805"} Oct 30 02:15:23.168361 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.168336 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888775-qr45g" Oct 30 02:15:23.168595 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.168338 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888775-qr45g" event={"ID":"1f68c408-6bab-42c7-b574-35d5d61c7544","Type":"ContainerDied","Data":"9e54a53dbe0ef8f9b289bf176e03d43cb9c25cd7e54da999e134d42aee01f510"} Oct 30 02:15:23.172446 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.172421 6572 scope.go:122] "RemoveContainer" containerID="695cb097363f84039dce4463719635b96607a6110f772bca106f63613460d2e9" Oct 30 02:15:23.181405 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.181376 6572 generic.go:356] "Generic (PLEG): container finished" podID="ba0ac388-9127-47e0-8985-dd48570c8d27" containerID="2c22c0e37f8ea7985e18a766f62ef4ab229a28b6e9b21a6096e443273e1989de" exitCode=60 Oct 30 02:15:23.181739 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.181715 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888775-vcqdq" event={"ID":"ba0ac388-9127-47e0-8985-dd48570c8d27","Type":"ContainerDied","Data":"2c22c0e37f8ea7985e18a766f62ef4ab229a28b6e9b21a6096e443273e1989de"} Oct 30 02:15:23.186038 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.186006 6572 generic.go:356] "Generic (PLEG): container finished" podID="39c584cd-b462-4d52-aa9d-786e9a0b8e94" containerID="0e88e2b8c8c7a93b4e5be8d6b7402d59ef69a675bb95d992335684840cac4ba9" exitCode=60 Oct 30 02:15:23.186301 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.186278 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-340ae6-29888775-8dfms" event={"ID":"39c584cd-b462-4d52-aa9d-786e9a0b8e94","Type":"ContainerDied","Data":"0e88e2b8c8c7a93b4e5be8d6b7402d59ef69a675bb95d992335684840cac4ba9"} Oct 30 02:15:23.187022 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.186980 6572 scope.go:122] "RemoveContainer" containerID="0e88e2b8c8c7a93b4e5be8d6b7402d59ef69a675bb95d992335684840cac4ba9" Oct 30 02:15:23.187739 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:23.187714 6572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-340ae6-29888775-8dfms_odh-ai-gateway-infra(39c584cd-b462-4d52-aa9d-786e9a0b8e94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-340ae6-29888775-8dfms" podUID="39c584cd-b462-4d52-aa9d-786e9a0b8e94" Oct 30 02:15:23.196500 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.196162 6572 scope.go:122] "RemoveContainer" containerID="3dac0819fc1b19b5eac50802539337aa6c171b2b7ce466375721536fa1204d97" Oct 30 02:15:23.199509 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.199457 6572 generic.go:356] "Generic (PLEG): container finished" podID="bdf23f37-2870-4b9c-b558-9fea4452f4e5" containerID="39a696c948ceba59a65a157bd3afe843261f3a9b0bd4b5d07353300afaaa7292" exitCode=60 Oct 30 02:15:23.199813 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.199713 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888775-8tzkk" event={"ID":"bdf23f37-2870-4b9c-b558-9fea4452f4e5","Type":"ContainerDied","Data":"39a696c948ceba59a65a157bd3afe843261f3a9b0bd4b5d07353300afaaa7292"} Oct 30 02:15:23.217282 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.217153 6572 scope.go:122] "RemoveContainer" containerID="a9461b9db3d0ecfd709551e2a184495ee35d555d96a2d9ce2dcce1b8bcbca98e" Oct 30 02:15:23.267432 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.255812 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d5aa9-29888775-9pqsc"] Oct 30 02:15:23.267432 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.261034 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2d5aa9-29888775-9pqsc"] Oct 30 02:15:23.311194 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.303010 6572 scope.go:122] "RemoveContainer" containerID="b20392f5d5f27d99b13cb0bfb23a1ec6efc816065423b1c3bd75660cf0fab805" Oct 30 02:15:23.341777 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.341730 6572 scope.go:122] "RemoveContainer" containerID="ae59a74838f7af9804b2f0319fbfaf489508f09c429125e18f8ee7653fb210d5" Oct 30 02:15:23.364089 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.363968 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888775-289bf"] Oct 30 02:15:23.368585 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.368498 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888775-p4j7s" Oct 30 02:15:23.371573 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.371540 6572 scope.go:122] "RemoveContainer" containerID="0795ab88ac3c3d3ff2f53f2e5addf302e7358b0aebde45619d4d80614be755f1" Oct 30 02:15:23.372125 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.372104 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-292018-29888775-289bf"] Oct 30 02:15:23.389367 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.389337 6572 scope.go:122] "RemoveContainer" containerID="0119beb5bd480c062e45ae1cf0e3a2d81e48d25eb0a8aefae9e143a6dfbfdaf9" Oct 30 02:15:23.410915 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.410826 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888775-r46v9"] Oct 30 02:15:23.421713 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.421670 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8968e7e0-04c1-4469-9062-f7d31ee6dbf2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8968e7e0-04c1-4469-9062-f7d31ee6dbf2-maas-api-service-ca\") pod \"8968e7e0-04c1-4469-9062-f7d31ee6dbf2\" (UID: \"8968e7e0-04c1-4469-9062-f7d31ee6dbf2\") " Oct 30 02:15:23.421935 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.421775 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8968e7e0-04c1-4469-9062-f7d31ee6dbf2-kube-api-access-bh9rx\" (UniqueName: \"kubernetes.io/projected/8968e7e0-04c1-4469-9062-f7d31ee6dbf2-kube-api-access-bh9rx\") pod \"8968e7e0-04c1-4469-9062-f7d31ee6dbf2\" (UID: \"8968e7e0-04c1-4469-9062-f7d31ee6dbf2\") " Oct 30 02:15:23.423478 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.423444 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8968e7e0-04c1-4469-9062-f7d31ee6dbf2-maas-api-service-ca" pod "8968e7e0-04c1-4469-9062-f7d31ee6dbf2" (UID: "8968e7e0-04c1-4469-9062-f7d31ee6dbf2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:23.425607 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.425564 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8968e7e0-04c1-4469-9062-f7d31ee6dbf2-kube-api-access-bh9rx" pod "8968e7e0-04c1-4469-9062-f7d31ee6dbf2" (UID: "8968e7e0-04c1-4469-9062-f7d31ee6dbf2"). InnerVolumeSpecName "kube-api-access-bh9rx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:23.430520 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.430487 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888775-r46v9"] Oct 30 02:15:23.443695 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.443653 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888775-wrhfw"] Oct 30 02:15:23.447870 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.447845 6572 scope.go:122] "RemoveContainer" containerID="e0656f18299892b23fe64ecd9b7550fefa33b0c01294b7c6bdf10f73b1efc0e7" Oct 30 02:15:23.468071 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.468034 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888775-wrhfw"] Oct 30 02:15:23.471217 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.471178 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888775-brm4t" Oct 30 02:15:23.483847 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.483815 6572 scope.go:122] "RemoveContainer" containerID="b0e7bb5695e7d324c0e5fc21d0bdd3c727cf2a31719d2359f8b1118428921980" Oct 30 02:15:23.522538 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.522468 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888775-8tzkk" Oct 30 02:15:23.523189 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.523159 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c7f35b4-a13a-4be3-be70-135dcc096f49-kube-api-access-g8jwh\" (UniqueName: \"kubernetes.io/projected/1c7f35b4-a13a-4be3-be70-135dcc096f49-kube-api-access-g8jwh\") pod \"1c7f35b4-a13a-4be3-be70-135dcc096f49\" (UID: \"1c7f35b4-a13a-4be3-be70-135dcc096f49\") " Oct 30 02:15:23.523357 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.523228 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c7f35b4-a13a-4be3-be70-135dcc096f49-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7f35b4-a13a-4be3-be70-135dcc096f49-maas-api-service-ca\") pod \"1c7f35b4-a13a-4be3-be70-135dcc096f49\" (UID: \"1c7f35b4-a13a-4be3-be70-135dcc096f49\") " Oct 30 02:15:23.523826 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.523801 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c7f35b4-a13a-4be3-be70-135dcc096f49-maas-api-service-ca" pod "1c7f35b4-a13a-4be3-be70-135dcc096f49" (UID: "1c7f35b4-a13a-4be3-be70-135dcc096f49"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:23.524016 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.523998 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7f35b4-a13a-4be3-be70-135dcc096f49-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:23.524123 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.524022 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8968e7e0-04c1-4469-9062-f7d31ee6dbf2-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:23.524123 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.524038 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh9rx\" (UniqueName: \"kubernetes.io/projected/8968e7e0-04c1-4469-9062-f7d31ee6dbf2-kube-api-access-bh9rx\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:23.525928 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.525902 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c7f35b4-a13a-4be3-be70-135dcc096f49-kube-api-access-g8jwh" pod "1c7f35b4-a13a-4be3-be70-135dcc096f49" (UID: "1c7f35b4-a13a-4be3-be70-135dcc096f49"). InnerVolumeSpecName "kube-api-access-g8jwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:23.526877 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.526855 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888775-vcqdq" Oct 30 02:15:23.560978 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.560938 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888775-djsrl"] Oct 30 02:15:23.565601 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.565562 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888775-djsrl"] Oct 30 02:15:23.625162 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.625128 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba0ac388-9127-47e0-8985-dd48570c8d27-kube-api-access-429l8\" (UniqueName: \"kubernetes.io/projected/ba0ac388-9127-47e0-8985-dd48570c8d27-kube-api-access-429l8\") pod \"ba0ac388-9127-47e0-8985-dd48570c8d27\" (UID: \"ba0ac388-9127-47e0-8985-dd48570c8d27\") " Oct 30 02:15:23.625669 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.625251 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdf23f37-2870-4b9c-b558-9fea4452f4e5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdf23f37-2870-4b9c-b558-9fea4452f4e5-maas-api-service-ca\") pod \"bdf23f37-2870-4b9c-b558-9fea4452f4e5\" (UID: \"bdf23f37-2870-4b9c-b558-9fea4452f4e5\") " Oct 30 02:15:23.625669 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.625434 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdf23f37-2870-4b9c-b558-9fea4452f4e5-kube-api-access-fmdt2\" (UniqueName: \"kubernetes.io/projected/bdf23f37-2870-4b9c-b558-9fea4452f4e5-kube-api-access-fmdt2\") pod \"bdf23f37-2870-4b9c-b558-9fea4452f4e5\" (UID: \"bdf23f37-2870-4b9c-b558-9fea4452f4e5\") " Oct 30 02:15:23.625669 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.625496 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba0ac388-9127-47e0-8985-dd48570c8d27-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba0ac388-9127-47e0-8985-dd48570c8d27-maas-api-service-ca\") pod \"ba0ac388-9127-47e0-8985-dd48570c8d27\" (UID: \"ba0ac388-9127-47e0-8985-dd48570c8d27\") " Oct 30 02:15:23.626033 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.625669 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdf23f37-2870-4b9c-b558-9fea4452f4e5-maas-api-service-ca" pod "bdf23f37-2870-4b9c-b558-9fea4452f4e5" (UID: "bdf23f37-2870-4b9c-b558-9fea4452f4e5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:23.626225 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.626185 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba0ac388-9127-47e0-8985-dd48570c8d27-maas-api-service-ca" pod "ba0ac388-9127-47e0-8985-dd48570c8d27" (UID: "ba0ac388-9127-47e0-8985-dd48570c8d27"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:23.626549 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.626522 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba0ac388-9127-47e0-8985-dd48570c8d27-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:23.626549 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.626552 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8jwh\" (UniqueName: \"kubernetes.io/projected/1c7f35b4-a13a-4be3-be70-135dcc096f49-kube-api-access-g8jwh\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:23.626737 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.626571 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdf23f37-2870-4b9c-b558-9fea4452f4e5-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:23.628015 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.627978 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba0ac388-9127-47e0-8985-dd48570c8d27-kube-api-access-429l8" pod "ba0ac388-9127-47e0-8985-dd48570c8d27" (UID: "ba0ac388-9127-47e0-8985-dd48570c8d27"). InnerVolumeSpecName "kube-api-access-429l8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:23.628145 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.628120 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdf23f37-2870-4b9c-b558-9fea4452f4e5-kube-api-access-fmdt2" pod "bdf23f37-2870-4b9c-b558-9fea4452f4e5" (UID: "bdf23f37-2870-4b9c-b558-9fea4452f4e5"). InnerVolumeSpecName "kube-api-access-fmdt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:23.643562 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.643417 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888775-qr45g"] Oct 30 02:15:23.651074 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.651034 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888775-qr45g"] Oct 30 02:15:23.683873 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.683818 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888775-82nz4"] Oct 30 02:15:23.688581 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.688547 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888775-82nz4"] Oct 30 02:15:23.728090 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.728048 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmdt2\" (UniqueName: \"kubernetes.io/projected/bdf23f37-2870-4b9c-b558-9fea4452f4e5-kube-api-access-fmdt2\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:23.728467 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.728443 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-429l8\" (UniqueName: \"kubernetes.io/projected/ba0ac388-9127-47e0-8985-dd48570c8d27-kube-api-access-429l8\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:23.785880 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:23.785528 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:24.213536 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.213495 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888775-brm4t" Oct 30 02:15:24.213803 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.213543 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888775-brm4t" event={"ID":"1c7f35b4-a13a-4be3-be70-135dcc096f49","Type":"ContainerDied","Data":"e855dd0ae04414dc7176c2eeb4d24551dd761eb42e21625ae4a42ed61e8147d8"} Oct 30 02:15:24.213803 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.213618 6572 scope.go:122] "RemoveContainer" containerID="1fbe6340af431b8c51dae8396cedcdc3c25254b55f2bfc8d81a0c3f055ec516b" Oct 30 02:15:24.215912 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.215885 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888775-vcqdq" Oct 30 02:15:24.215912 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.215902 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888775-vcqdq" event={"ID":"ba0ac388-9127-47e0-8985-dd48570c8d27","Type":"ContainerDied","Data":"33f70d3bbb5dea11936ce04f726110c3bd79072ad1c1a2f9dc95c2cb2dbc81c9"} Oct 30 02:15:24.227184 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.225642 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888775-8tzkk" event={"ID":"bdf23f37-2870-4b9c-b558-9fea4452f4e5","Type":"ContainerDied","Data":"e9ef79542a91f5025ef25f8360f6842ede7843bb33c7c6bd526433c48c0a50dc"} Oct 30 02:15:24.227184 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.225824 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888775-8tzkk" Oct 30 02:15:24.230215 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.230189 6572 scope.go:122] "RemoveContainer" containerID="2c22c0e37f8ea7985e18a766f62ef4ab229a28b6e9b21a6096e443273e1989de" Oct 30 02:15:24.237012 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.236559 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888775-p4j7s" Oct 30 02:15:24.237012 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.236573 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888775-p4j7s" event={"ID":"8968e7e0-04c1-4469-9062-f7d31ee6dbf2","Type":"ContainerDied","Data":"09bd292cbadc3f620fca3ab083ecdd96e764b1430a3aa7c16e6774be0915aacf"} Oct 30 02:15:24.247767 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.247732 6572 scope.go:122] "RemoveContainer" containerID="39a696c948ceba59a65a157bd3afe843261f3a9b0bd4b5d07353300afaaa7292" Oct 30 02:15:24.260927 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.260894 6572 scope.go:122] "RemoveContainer" containerID="4ca16cea826c1086899ce766a8c4a731d60267e36fccede601a0fb7c3dcd019a" Oct 30 02:15:24.277938 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.277894 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888775-brm4t"] Oct 30 02:15:24.282303 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.282265 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-396f8e-29888775-brm4t"] Oct 30 02:15:24.311382 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.311344 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888775-vcqdq"] Oct 30 02:15:24.315916 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.315885 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-093d23-29888775-vcqdq"] Oct 30 02:15:24.359630 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.359591 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888775-8tzkk"] Oct 30 02:15:24.365137 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.365100 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5fb950-29888775-8tzkk"] Oct 30 02:15:24.402443 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.402401 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888775-p4j7s"] Oct 30 02:15:24.412024 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.411973 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-abb30f-29888775-p4j7s"] Oct 30 02:15:24.489384 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.489272 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-340ae6-29888775-8dfms"] Oct 30 02:15:24.547837 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.547801 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c7f35b4-a13a-4be3-be70-135dcc096f49" path="/var/lib/kubelet/pods/1c7f35b4-a13a-4be3-be70-135dcc096f49/volumes" Oct 30 02:15:24.548408 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.548384 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f68c408-6bab-42c7-b574-35d5d61c7544" path="/var/lib/kubelet/pods/1f68c408-6bab-42c7-b574-35d5d61c7544/volumes" Oct 30 02:15:24.548933 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.548914 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ed6af0f-eb0e-41f8-8b21-654babbb98b5" path="/var/lib/kubelet/pods/4ed6af0f-eb0e-41f8-8b21-654babbb98b5/volumes" Oct 30 02:15:24.549741 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.549719 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4" path="/var/lib/kubelet/pods/63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4/volumes" Oct 30 02:15:24.550208 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.550188 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6edd5565-a917-481f-9302-95011124898b" path="/var/lib/kubelet/pods/6edd5565-a917-481f-9302-95011124898b/volumes" Oct 30 02:15:24.550634 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.550617 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75a1834c-2859-4e9c-9d32-80a6f7247460" path="/var/lib/kubelet/pods/75a1834c-2859-4e9c-9d32-80a6f7247460/volumes" Oct 30 02:15:24.551364 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.551344 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8968e7e0-04c1-4469-9062-f7d31ee6dbf2" path="/var/lib/kubelet/pods/8968e7e0-04c1-4469-9062-f7d31ee6dbf2/volumes" Oct 30 02:15:24.551813 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.551795 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba0ac388-9127-47e0-8985-dd48570c8d27" path="/var/lib/kubelet/pods/ba0ac388-9127-47e0-8985-dd48570c8d27/volumes" Oct 30 02:15:24.552240 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.552222 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcf5437e-0075-467b-a4fd-82b5dc563489" path="/var/lib/kubelet/pods/bcf5437e-0075-467b-a4fd-82b5dc563489/volumes" Oct 30 02:15:24.552995 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.552974 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdf23f37-2870-4b9c-b558-9fea4452f4e5" path="/var/lib/kubelet/pods/bdf23f37-2870-4b9c-b558-9fea4452f4e5/volumes" Oct 30 02:15:24.553641 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.553613 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2bd8bc3-6252-42fc-9055-ca393b6f45d9" path="/var/lib/kubelet/pods/c2bd8bc3-6252-42fc-9055-ca393b6f45d9/volumes" Oct 30 02:15:24.554249 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.554229 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fec1f4d7-877e-4d7c-88d2-f2365469869a" path="/var/lib/kubelet/pods/fec1f4d7-877e-4d7c-88d2-f2365469869a/volumes" Oct 30 02:15:24.681714 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.681683 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-340ae6-29888775-8dfms" Oct 30 02:15:24.745590 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.745469 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39c584cd-b462-4d52-aa9d-786e9a0b8e94-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c584cd-b462-4d52-aa9d-786e9a0b8e94-maas-api-service-ca\") pod \"39c584cd-b462-4d52-aa9d-786e9a0b8e94\" (UID: \"39c584cd-b462-4d52-aa9d-786e9a0b8e94\") " Oct 30 02:15:24.745811 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.745629 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39c584cd-b462-4d52-aa9d-786e9a0b8e94-kube-api-access-5gpqh\" (UniqueName: \"kubernetes.io/projected/39c584cd-b462-4d52-aa9d-786e9a0b8e94-kube-api-access-5gpqh\") pod \"39c584cd-b462-4d52-aa9d-786e9a0b8e94\" (UID: \"39c584cd-b462-4d52-aa9d-786e9a0b8e94\") " Oct 30 02:15:24.745997 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.745967 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39c584cd-b462-4d52-aa9d-786e9a0b8e94-maas-api-service-ca" pod "39c584cd-b462-4d52-aa9d-786e9a0b8e94" (UID: "39c584cd-b462-4d52-aa9d-786e9a0b8e94"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:24.746279 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.746253 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c584cd-b462-4d52-aa9d-786e9a0b8e94-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:24.748381 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.748333 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39c584cd-b462-4d52-aa9d-786e9a0b8e94-kube-api-access-5gpqh" pod "39c584cd-b462-4d52-aa9d-786e9a0b8e94" (UID: "39c584cd-b462-4d52-aa9d-786e9a0b8e94"). InnerVolumeSpecName "kube-api-access-5gpqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:24.848093 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:24.848049 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gpqh\" (UniqueName: \"kubernetes.io/projected/39c584cd-b462-4d52-aa9d-786e9a0b8e94-kube-api-access-5gpqh\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:25.263114 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:25.263081 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-340ae6-29888775-8dfms" Oct 30 02:15:25.263114 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:25.263100 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-340ae6-29888775-8dfms" event={"ID":"39c584cd-b462-4d52-aa9d-786e9a0b8e94","Type":"ContainerDied","Data":"99db59d61ccb475091549443699d5a892a46f82bef091b1f36b0dbedeb707422"} Oct 30 02:15:25.263379 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:25.263158 6572 scope.go:122] "RemoveContainer" containerID="0e88e2b8c8c7a93b4e5be8d6b7402d59ef69a675bb95d992335684840cac4ba9" Oct 30 02:15:25.315863 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:25.315821 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-340ae6-29888775-8dfms"] Oct 30 02:15:25.326976 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:25.326934 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-340ae6-29888775-8dfms"] Oct 30 02:15:25.476063 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:25.476023 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:25.788211 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:25.788166 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-cd8173-8445cbb698-7wtm5" podUID="909b1459-46ea-4c3c-bc2e-1fb29527200a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:15:26.544896 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:26.544863 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39c584cd-b462-4d52-aa9d-786e9a0b8e94" path="/var/lib/kubelet/pods/39c584cd-b462-4d52-aa9d-786e9a0b8e94/volumes" Oct 30 02:15:27.039582 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:27.039543 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:28.681808 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:28.680830 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:28.872802 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:28.872723 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:29.200771 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.197908 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:29.640475 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.640379 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-657fba-b29m9"] Oct 30 02:15:29.641186 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641165 6572 cpu_manager.go:414] "containerMap: removing container" podUID="4ed6af0f-eb0e-41f8-8b21-654babbb98b5" containerName="cleanup" Oct 30 02:15:29.641186 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641184 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ed6af0f-eb0e-41f8-8b21-654babbb98b5" containerName="cleanup" Oct 30 02:15:29.641353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641199 6572 cpu_manager.go:414] "containerMap: removing container" podUID="c2bd8bc3-6252-42fc-9055-ca393b6f45d9" containerName="cleanup" Oct 30 02:15:29.641353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641204 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2bd8bc3-6252-42fc-9055-ca393b6f45d9" containerName="cleanup" Oct 30 02:15:29.641353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641212 6572 cpu_manager.go:414] "containerMap: removing container" podUID="bdf23f37-2870-4b9c-b558-9fea4452f4e5" containerName="cleanup" Oct 30 02:15:29.641353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641217 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdf23f37-2870-4b9c-b558-9fea4452f4e5" containerName="cleanup" Oct 30 02:15:29.641353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641223 6572 cpu_manager.go:414] "containerMap: removing container" podUID="3f9f8331-2090-4262-96a0-c3ea360e4375" containerName="cleanup" Oct 30 02:15:29.641353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641228 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f9f8331-2090-4262-96a0-c3ea360e4375" containerName="cleanup" Oct 30 02:15:29.641353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641237 6572 cpu_manager.go:414] "containerMap: removing container" podUID="bcf5437e-0075-467b-a4fd-82b5dc563489" containerName="cleanup" Oct 30 02:15:29.641353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641241 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcf5437e-0075-467b-a4fd-82b5dc563489" containerName="cleanup" Oct 30 02:15:29.641353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641246 6572 cpu_manager.go:414] "containerMap: removing container" podUID="da67c0ab-cf74-4396-9987-b1bb46eb802b" containerName="cleanup" Oct 30 02:15:29.641353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641251 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da67c0ab-cf74-4396-9987-b1bb46eb802b" containerName="cleanup" Oct 30 02:15:29.641353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641258 6572 cpu_manager.go:414] "containerMap: removing container" podUID="f641ac83-f4e6-4003-ae9c-be79e4ca0337" containerName="cleanup" Oct 30 02:15:29.641353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641262 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f641ac83-f4e6-4003-ae9c-be79e4ca0337" containerName="cleanup" Oct 30 02:15:29.641353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641269 6572 cpu_manager.go:414] "containerMap: removing container" podUID="99cb9b6a-51ec-41be-a85d-dd1de997fceb" containerName="cleanup" Oct 30 02:15:29.641353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641274 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99cb9b6a-51ec-41be-a85d-dd1de997fceb" containerName="cleanup" Oct 30 02:15:29.641353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641279 6572 cpu_manager.go:414] "containerMap: removing container" podUID="d0a98e88-a93c-462b-b2f5-68c61b386789" containerName="payload-pre-processing" Oct 30 02:15:29.641353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641284 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0a98e88-a93c-462b-b2f5-68c61b386789" containerName="payload-pre-processing" Oct 30 02:15:29.641353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641292 6572 cpu_manager.go:414] "containerMap: removing container" podUID="4ed6af0f-eb0e-41f8-8b21-654babbb98b5" containerName="cleanup" Oct 30 02:15:29.641353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641296 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ed6af0f-eb0e-41f8-8b21-654babbb98b5" containerName="cleanup" Oct 30 02:15:29.641353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641304 6572 cpu_manager.go:414] "containerMap: removing container" podUID="1f68c408-6bab-42c7-b574-35d5d61c7544" containerName="cleanup" Oct 30 02:15:29.641353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641308 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f68c408-6bab-42c7-b574-35d5d61c7544" containerName="cleanup" Oct 30 02:15:29.641353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641315 6572 cpu_manager.go:414] "containerMap: removing container" podUID="3f9f8331-2090-4262-96a0-c3ea360e4375" containerName="cleanup" Oct 30 02:15:29.641353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641319 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f9f8331-2090-4262-96a0-c3ea360e4375" containerName="cleanup" Oct 30 02:15:29.641353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641325 6572 cpu_manager.go:414] "containerMap: removing container" podUID="1f68c408-6bab-42c7-b574-35d5d61c7544" containerName="cleanup" Oct 30 02:15:29.641353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641328 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f68c408-6bab-42c7-b574-35d5d61c7544" containerName="cleanup" Oct 30 02:15:29.641353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641336 6572 cpu_manager.go:414] "containerMap: removing container" podUID="0dc3a619-596a-4e28-9845-9d1e2fc7bb06" containerName="cleanup" Oct 30 02:15:29.641353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641340 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dc3a619-596a-4e28-9845-9d1e2fc7bb06" containerName="cleanup" Oct 30 02:15:29.641353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641349 6572 cpu_manager.go:414] "containerMap: removing container" podUID="da67c0ab-cf74-4396-9987-b1bb46eb802b" containerName="cleanup" Oct 30 02:15:29.641353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641353 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da67c0ab-cf74-4396-9987-b1bb46eb802b" containerName="cleanup" Oct 30 02:15:29.641353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641362 6572 cpu_manager.go:414] "containerMap: removing container" podUID="75a1834c-2859-4e9c-9d32-80a6f7247460" containerName="cleanup" Oct 30 02:15:29.641353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641367 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a1834c-2859-4e9c-9d32-80a6f7247460" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641374 6572 cpu_manager.go:414] "containerMap: removing container" podUID="6edd5565-a917-481f-9302-95011124898b" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641378 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6edd5565-a917-481f-9302-95011124898b" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641383 6572 cpu_manager.go:414] "containerMap: removing container" podUID="bcf5437e-0075-467b-a4fd-82b5dc563489" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641387 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcf5437e-0075-467b-a4fd-82b5dc563489" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641392 6572 cpu_manager.go:414] "containerMap: removing container" podUID="bdf23f37-2870-4b9c-b558-9fea4452f4e5" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641396 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdf23f37-2870-4b9c-b558-9fea4452f4e5" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641403 6572 cpu_manager.go:414] "containerMap: removing container" podUID="3f9f8331-2090-4262-96a0-c3ea360e4375" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641407 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f9f8331-2090-4262-96a0-c3ea360e4375" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641413 6572 cpu_manager.go:414] "containerMap: removing container" podUID="fec1f4d7-877e-4d7c-88d2-f2365469869a" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641417 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fec1f4d7-877e-4d7c-88d2-f2365469869a" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641423 6572 cpu_manager.go:414] "containerMap: removing container" podUID="6edd5565-a917-481f-9302-95011124898b" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641427 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6edd5565-a917-481f-9302-95011124898b" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641434 6572 cpu_manager.go:414] "containerMap: removing container" podUID="39c584cd-b462-4d52-aa9d-786e9a0b8e94" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641439 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39c584cd-b462-4d52-aa9d-786e9a0b8e94" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641446 6572 cpu_manager.go:414] "containerMap: removing container" podUID="d66a87ee-dc89-4aba-845b-17b58bc6f4a6" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641454 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d66a87ee-dc89-4aba-845b-17b58bc6f4a6" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641460 6572 cpu_manager.go:414] "containerMap: removing container" podUID="2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641464 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641473 6572 cpu_manager.go:414] "containerMap: removing container" podUID="c2bd8bc3-6252-42fc-9055-ca393b6f45d9" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641478 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2bd8bc3-6252-42fc-9055-ca393b6f45d9" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641483 6572 cpu_manager.go:414] "containerMap: removing container" podUID="bdf23f37-2870-4b9c-b558-9fea4452f4e5" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641487 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdf23f37-2870-4b9c-b558-9fea4452f4e5" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641494 6572 cpu_manager.go:414] "containerMap: removing container" podUID="2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641498 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641504 6572 cpu_manager.go:414] "containerMap: removing container" podUID="f641ac83-f4e6-4003-ae9c-be79e4ca0337" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641508 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f641ac83-f4e6-4003-ae9c-be79e4ca0337" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641514 6572 cpu_manager.go:414] "containerMap: removing container" podUID="bcf5437e-0075-467b-a4fd-82b5dc563489" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641519 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcf5437e-0075-467b-a4fd-82b5dc563489" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641525 6572 cpu_manager.go:414] "containerMap: removing container" podUID="0dc3a619-596a-4e28-9845-9d1e2fc7bb06" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641529 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dc3a619-596a-4e28-9845-9d1e2fc7bb06" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641540 6572 cpu_manager.go:414] "containerMap: removing container" podUID="ba0ac388-9127-47e0-8985-dd48570c8d27" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641545 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba0ac388-9127-47e0-8985-dd48570c8d27" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641552 6572 cpu_manager.go:414] "containerMap: removing container" podUID="ba0ac388-9127-47e0-8985-dd48570c8d27" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641557 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba0ac388-9127-47e0-8985-dd48570c8d27" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641563 6572 cpu_manager.go:414] "containerMap: removing container" podUID="75a1834c-2859-4e9c-9d32-80a6f7247460" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641567 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a1834c-2859-4e9c-9d32-80a6f7247460" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641575 6572 cpu_manager.go:414] "containerMap: removing container" podUID="8968e7e0-04c1-4469-9062-f7d31ee6dbf2" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641578 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8968e7e0-04c1-4469-9062-f7d31ee6dbf2" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641584 6572 cpu_manager.go:414] "containerMap: removing container" podUID="99cb9b6a-51ec-41be-a85d-dd1de997fceb" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641588 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99cb9b6a-51ec-41be-a85d-dd1de997fceb" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641595 6572 cpu_manager.go:414] "containerMap: removing container" podUID="1c7f35b4-a13a-4be3-be70-135dcc096f49" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641599 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c7f35b4-a13a-4be3-be70-135dcc096f49" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641613 6572 cpu_manager.go:414] "containerMap: removing container" podUID="8968e7e0-04c1-4469-9062-f7d31ee6dbf2" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641618 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8968e7e0-04c1-4469-9062-f7d31ee6dbf2" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641625 6572 cpu_manager.go:414] "containerMap: removing container" podUID="63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641630 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641644 6572 cpu_manager.go:414] "containerMap: removing container" podUID="d66a87ee-dc89-4aba-845b-17b58bc6f4a6" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641648 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d66a87ee-dc89-4aba-845b-17b58bc6f4a6" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641654 6572 cpu_manager.go:414] "containerMap: removing container" podUID="c2bd8bc3-6252-42fc-9055-ca393b6f45d9" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641659 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2bd8bc3-6252-42fc-9055-ca393b6f45d9" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641666 6572 cpu_manager.go:414] "containerMap: removing container" podUID="39c584cd-b462-4d52-aa9d-786e9a0b8e94" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641670 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39c584cd-b462-4d52-aa9d-786e9a0b8e94" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641678 6572 cpu_manager.go:414] "containerMap: removing container" podUID="63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641682 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641690 6572 cpu_manager.go:414] "containerMap: removing container" podUID="d66a87ee-dc89-4aba-845b-17b58bc6f4a6" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641695 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d66a87ee-dc89-4aba-845b-17b58bc6f4a6" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641700 6572 cpu_manager.go:414] "containerMap: removing container" podUID="63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641705 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641717 6572 cpu_manager.go:414] "containerMap: removing container" podUID="cd10ed97-4e22-4efe-a0c6-7cff1ca2f157" containerName="payload-processing" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641722 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd10ed97-4e22-4efe-a0c6-7cff1ca2f157" containerName="payload-processing" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641730 6572 cpu_manager.go:414] "containerMap: removing container" podUID="fec1f4d7-877e-4d7c-88d2-f2365469869a" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641734 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fec1f4d7-877e-4d7c-88d2-f2365469869a" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641742 6572 cpu_manager.go:414] "containerMap: removing container" podUID="6edd5565-a917-481f-9302-95011124898b" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641761 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6edd5565-a917-481f-9302-95011124898b" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641771 6572 cpu_manager.go:414] "containerMap: removing container" podUID="39c584cd-b462-4d52-aa9d-786e9a0b8e94" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641776 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39c584cd-b462-4d52-aa9d-786e9a0b8e94" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641783 6572 cpu_manager.go:414] "containerMap: removing container" podUID="1c7f35b4-a13a-4be3-be70-135dcc096f49" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641787 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c7f35b4-a13a-4be3-be70-135dcc096f49" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641894 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ed6af0f-eb0e-41f8-8b21-654babbb98b5" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641902 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c7f35b4-a13a-4be3-be70-135dcc096f49" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641908 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="fec1f4d7-877e-4d7c-88d2-f2365469869a" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641914 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="6edd5565-a917-481f-9302-95011124898b" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641920 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="39c584cd-b462-4d52-aa9d-786e9a0b8e94" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641926 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641931 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c7f35b4-a13a-4be3-be70-135dcc096f49" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641937 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dc3a619-596a-4e28-9845-9d1e2fc7bb06" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641941 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba0ac388-9127-47e0-8985-dd48570c8d27" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641947 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ed6af0f-eb0e-41f8-8b21-654babbb98b5" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641956 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2bd8bc3-6252-42fc-9055-ca393b6f45d9" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641964 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdf23f37-2870-4b9c-b558-9fea4452f4e5" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641972 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f9f8331-2090-4262-96a0-c3ea360e4375" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641981 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcf5437e-0075-467b-a4fd-82b5dc563489" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641989 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="da67c0ab-cf74-4396-9987-b1bb46eb802b" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641994 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="f641ac83-f4e6-4003-ae9c-be79e4ca0337" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.641998 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="99cb9b6a-51ec-41be-a85d-dd1de997fceb" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.642002 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0a98e88-a93c-462b-b2f5-68c61b386789" containerName="payload-pre-processing" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.642009 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ed6af0f-eb0e-41f8-8b21-654babbb98b5" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.642015 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f68c408-6bab-42c7-b574-35d5d61c7544" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.642019 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f9f8331-2090-4262-96a0-c3ea360e4375" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.642024 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f68c408-6bab-42c7-b574-35d5d61c7544" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.642031 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="da67c0ab-cf74-4396-9987-b1bb46eb802b" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.642037 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="75a1834c-2859-4e9c-9d32-80a6f7247460" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.642043 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="6edd5565-a917-481f-9302-95011124898b" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.642048 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcf5437e-0075-467b-a4fd-82b5dc563489" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.642052 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdf23f37-2870-4b9c-b558-9fea4452f4e5" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.642057 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f9f8331-2090-4262-96a0-c3ea360e4375" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.642064 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="fec1f4d7-877e-4d7c-88d2-f2365469869a" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.642070 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="6edd5565-a917-481f-9302-95011124898b" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.642076 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="d66a87ee-dc89-4aba-845b-17b58bc6f4a6" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.642081 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2bd8bc3-6252-42fc-9055-ca393b6f45d9" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.642087 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdf23f37-2870-4b9c-b558-9fea4452f4e5" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.642093 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.642098 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcf5437e-0075-467b-a4fd-82b5dc563489" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.642103 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dc3a619-596a-4e28-9845-9d1e2fc7bb06" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.642109 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="99cb9b6a-51ec-41be-a85d-dd1de997fceb" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.642116 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba0ac388-9127-47e0-8985-dd48570c8d27" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.642121 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="75a1834c-2859-4e9c-9d32-80a6f7247460" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.642127 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="8968e7e0-04c1-4469-9062-f7d31ee6dbf2" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.642132 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="99cb9b6a-51ec-41be-a85d-dd1de997fceb" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.642140 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="75a1834c-2859-4e9c-9d32-80a6f7247460" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.642145 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="8968e7e0-04c1-4469-9062-f7d31ee6dbf2" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.642150 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.642157 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="d66a87ee-dc89-4aba-845b-17b58bc6f4a6" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.642162 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="39c584cd-b462-4d52-aa9d-786e9a0b8e94" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.642170 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.642176 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="63e4b7e1-1c7c-45f4-8de3-f23bc23ad2a4" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.642182 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="f641ac83-f4e6-4003-ae9c-be79e4ca0337" containerName="cleanup" Oct 30 02:15:29.642348 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.642188 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd10ed97-4e22-4efe-a0c6-7cff1ca2f157" containerName="payload-processing" Oct 30 02:15:29.647428 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.647406 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-657fba-b29m9" Oct 30 02:15:29.663827 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.661759 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:15:29.669507 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.669470 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:15:29.674773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.671025 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:15:29.674773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.672709 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:15:29.680581 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.680546 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-657fba-b29m9"] Oct 30 02:15:29.755386 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.755340 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhwds\" (UniqueName: \"kubernetes.io/projected/3fb220e0-3a46-4b35-9707-aca38ac5f288-kube-api-access-xhwds\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-657fba-b29m9\" (UID: \"3fb220e0-3a46-4b35-9707-aca38ac5f288\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-657fba-b29m9" Oct 30 02:15:29.755911 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.755528 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fb220e0-3a46-4b35-9707-aca38ac5f288-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-657fba-b29m9\" (UID: \"3fb220e0-3a46-4b35-9707-aca38ac5f288\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-657fba-b29m9" Oct 30 02:15:29.755911 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.755570 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fb220e0-3a46-4b35-9707-aca38ac5f288-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-657fba-b29m9\" (UID: \"3fb220e0-3a46-4b35-9707-aca38ac5f288\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-657fba-b29m9" Oct 30 02:15:29.857299 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.857255 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhwds\" (UniqueName: \"kubernetes.io/projected/3fb220e0-3a46-4b35-9707-aca38ac5f288-kube-api-access-xhwds\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-657fba-b29m9\" (UID: \"3fb220e0-3a46-4b35-9707-aca38ac5f288\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-657fba-b29m9" Oct 30 02:15:29.857495 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.857385 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fb220e0-3a46-4b35-9707-aca38ac5f288-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-657fba-b29m9\" (UID: \"3fb220e0-3a46-4b35-9707-aca38ac5f288\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-657fba-b29m9" Oct 30 02:15:29.857495 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.857410 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fb220e0-3a46-4b35-9707-aca38ac5f288-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-657fba-b29m9\" (UID: \"3fb220e0-3a46-4b35-9707-aca38ac5f288\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-657fba-b29m9" Oct 30 02:15:29.858109 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.858085 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fb220e0-3a46-4b35-9707-aca38ac5f288-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-657fba-b29m9\" (UID: \"3fb220e0-3a46-4b35-9707-aca38ac5f288\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-657fba-b29m9" Oct 30 02:15:29.859988 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.859960 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fb220e0-3a46-4b35-9707-aca38ac5f288-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-657fba-b29m9\" (UID: \"3fb220e0-3a46-4b35-9707-aca38ac5f288\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-657fba-b29m9" Oct 30 02:15:29.876153 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.876122 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhwds\" (UniqueName: \"kubernetes.io/projected/3fb220e0-3a46-4b35-9707-aca38ac5f288-kube-api-access-xhwds\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-657fba-b29m9\" (UID: \"3fb220e0-3a46-4b35-9707-aca38ac5f288\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-657fba-b29m9" Oct 30 02:15:29.965339 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:29.965292 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-657fba-b29m9" Oct 30 02:15:30.277048 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:30.277012 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-657fba-b29m9"] Oct 30 02:15:30.277827 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:15:30.277801 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fb220e0_3a46_4b35_9707_aca38ac5f288.slice/crio-d246e7567e89ec5517a0457fa75ee78d9c2861431e1be3715718d7ae2f94699c WatchSource:0}: Error finding container d246e7567e89ec5517a0457fa75ee78d9c2861431e1be3715718d7ae2f94699c: Status 404 returned error can't find the container with id d246e7567e89ec5517a0457fa75ee78d9c2861431e1be3715718d7ae2f94699c Oct 30 02:15:30.405774 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:30.399202 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-657fba-b29m9" event={"ID":"3fb220e0-3a46-4b35-9707-aca38ac5f288","Type":"ContainerStarted","Data":"d246e7567e89ec5517a0457fa75ee78d9c2861431e1be3715718d7ae2f94699c"} Oct 30 02:15:30.413778 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:30.411247 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:31.410860 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:31.410821 6572 generic.go:356] "Generic (PLEG): container finished" podID="3fb220e0-3a46-4b35-9707-aca38ac5f288" containerID="9bf888c0d9d426697c33561d1471dc1dc92c01bb32f0ce0f429e466f88b6061a" exitCode=0 Oct 30 02:15:31.411273 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:31.410903 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-657fba-b29m9" event={"ID":"3fb220e0-3a46-4b35-9707-aca38ac5f288","Type":"ContainerDied","Data":"9bf888c0d9d426697c33561d1471dc1dc92c01bb32f0ce0f429e466f88b6061a"} Oct 30 02:15:32.097674 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:32.096272 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:32.599095 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:32.599065 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-657fba-b29m9" Oct 30 02:15:32.615027 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:32.614994 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fb220e0-3a46-4b35-9707-aca38ac5f288-kube-api-access-xhwds\" (UniqueName: \"kubernetes.io/projected/3fb220e0-3a46-4b35-9707-aca38ac5f288-kube-api-access-xhwds\") pod \"3fb220e0-3a46-4b35-9707-aca38ac5f288\" (UID: \"3fb220e0-3a46-4b35-9707-aca38ac5f288\") " Oct 30 02:15:32.615242 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:32.615054 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fb220e0-3a46-4b35-9707-aca38ac5f288-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fb220e0-3a46-4b35-9707-aca38ac5f288-maas-api-openshift-ca\") pod \"3fb220e0-3a46-4b35-9707-aca38ac5f288\" (UID: \"3fb220e0-3a46-4b35-9707-aca38ac5f288\") " Oct 30 02:15:32.615242 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:32.615194 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fb220e0-3a46-4b35-9707-aca38ac5f288-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fb220e0-3a46-4b35-9707-aca38ac5f288-maas-api-xks-ca\") pod \"3fb220e0-3a46-4b35-9707-aca38ac5f288\" (UID: \"3fb220e0-3a46-4b35-9707-aca38ac5f288\") " Oct 30 02:15:32.615428 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:32.615402 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fb220e0-3a46-4b35-9707-aca38ac5f288-maas-api-openshift-ca" pod "3fb220e0-3a46-4b35-9707-aca38ac5f288" (UID: "3fb220e0-3a46-4b35-9707-aca38ac5f288"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:32.615899 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:32.615875 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fb220e0-3a46-4b35-9707-aca38ac5f288-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:32.617371 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:32.617349 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fb220e0-3a46-4b35-9707-aca38ac5f288-maas-api-xks-ca" pod "3fb220e0-3a46-4b35-9707-aca38ac5f288" (UID: "3fb220e0-3a46-4b35-9707-aca38ac5f288"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:15:32.617562 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:32.617542 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fb220e0-3a46-4b35-9707-aca38ac5f288-kube-api-access-xhwds" pod "3fb220e0-3a46-4b35-9707-aca38ac5f288" (UID: "3fb220e0-3a46-4b35-9707-aca38ac5f288"). InnerVolumeSpecName "kube-api-access-xhwds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:32.717077 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:32.716986 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fb220e0-3a46-4b35-9707-aca38ac5f288-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:32.717077 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:32.717017 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhwds\" (UniqueName: \"kubernetes.io/projected/3fb220e0-3a46-4b35-9707-aca38ac5f288-kube-api-access-xhwds\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:33.363663 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.363624 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j"] Oct 30 02:15:33.364468 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.364446 6572 cpu_manager.go:414] "containerMap: removing container" podUID="4ed6af0f-eb0e-41f8-8b21-654babbb98b5" containerName="cleanup" Oct 30 02:15:33.364468 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.364468 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ed6af0f-eb0e-41f8-8b21-654babbb98b5" containerName="cleanup" Oct 30 02:15:33.364689 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.364483 6572 cpu_manager.go:414] "containerMap: removing container" podUID="1f68c408-6bab-42c7-b574-35d5d61c7544" containerName="cleanup" Oct 30 02:15:33.364689 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.364488 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f68c408-6bab-42c7-b574-35d5d61c7544" containerName="cleanup" Oct 30 02:15:33.364689 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.364498 6572 cpu_manager.go:414] "containerMap: removing container" podUID="da67c0ab-cf74-4396-9987-b1bb46eb802b" containerName="cleanup" Oct 30 02:15:33.364689 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.364504 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da67c0ab-cf74-4396-9987-b1bb46eb802b" containerName="cleanup" Oct 30 02:15:33.364689 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.364514 6572 cpu_manager.go:414] "containerMap: removing container" podUID="fec1f4d7-877e-4d7c-88d2-f2365469869a" containerName="cleanup" Oct 30 02:15:33.364689 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.364519 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fec1f4d7-877e-4d7c-88d2-f2365469869a" containerName="cleanup" Oct 30 02:15:33.364689 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.364532 6572 cpu_manager.go:414] "containerMap: removing container" podUID="2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90" containerName="cleanup" Oct 30 02:15:33.364689 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.364538 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90" containerName="cleanup" Oct 30 02:15:33.364689 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.364553 6572 cpu_manager.go:414] "containerMap: removing container" podUID="3fb220e0-3a46-4b35-9707-aca38ac5f288" containerName="revoke-keys" Oct 30 02:15:33.364689 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.364559 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fb220e0-3a46-4b35-9707-aca38ac5f288" containerName="revoke-keys" Oct 30 02:15:33.364689 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.364565 6572 cpu_manager.go:414] "containerMap: removing container" podUID="f641ac83-f4e6-4003-ae9c-be79e4ca0337" containerName="cleanup" Oct 30 02:15:33.364689 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.364569 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f641ac83-f4e6-4003-ae9c-be79e4ca0337" containerName="cleanup" Oct 30 02:15:33.364689 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.364575 6572 cpu_manager.go:414] "containerMap: removing container" podUID="0dc3a619-596a-4e28-9845-9d1e2fc7bb06" containerName="cleanup" Oct 30 02:15:33.364689 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.364580 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dc3a619-596a-4e28-9845-9d1e2fc7bb06" containerName="cleanup" Oct 30 02:15:33.364689 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.364591 6572 cpu_manager.go:414] "containerMap: removing container" podUID="ba0ac388-9127-47e0-8985-dd48570c8d27" containerName="cleanup" Oct 30 02:15:33.364689 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.364599 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba0ac388-9127-47e0-8985-dd48570c8d27" containerName="cleanup" Oct 30 02:15:33.364689 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.364609 6572 cpu_manager.go:414] "containerMap: removing container" podUID="75a1834c-2859-4e9c-9d32-80a6f7247460" containerName="cleanup" Oct 30 02:15:33.364689 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.364615 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a1834c-2859-4e9c-9d32-80a6f7247460" containerName="cleanup" Oct 30 02:15:33.364689 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.364629 6572 cpu_manager.go:414] "containerMap: removing container" podUID="99cb9b6a-51ec-41be-a85d-dd1de997fceb" containerName="cleanup" Oct 30 02:15:33.364689 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.364635 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99cb9b6a-51ec-41be-a85d-dd1de997fceb" containerName="cleanup" Oct 30 02:15:33.364689 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.364646 6572 cpu_manager.go:414] "containerMap: removing container" podUID="1c7f35b4-a13a-4be3-be70-135dcc096f49" containerName="cleanup" Oct 30 02:15:33.364689 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.364652 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c7f35b4-a13a-4be3-be70-135dcc096f49" containerName="cleanup" Oct 30 02:15:33.364689 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.364673 6572 cpu_manager.go:414] "containerMap: removing container" podUID="8968e7e0-04c1-4469-9062-f7d31ee6dbf2" containerName="cleanup" Oct 30 02:15:33.364689 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.364680 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8968e7e0-04c1-4469-9062-f7d31ee6dbf2" containerName="cleanup" Oct 30 02:15:33.366120 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.364912 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="39c584cd-b462-4d52-aa9d-786e9a0b8e94" containerName="cleanup" Oct 30 02:15:33.366120 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.364930 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c7f35b4-a13a-4be3-be70-135dcc096f49" containerName="cleanup" Oct 30 02:15:33.366120 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.364944 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba0ac388-9127-47e0-8985-dd48570c8d27" containerName="cleanup" Oct 30 02:15:33.366120 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.364957 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="f641ac83-f4e6-4003-ae9c-be79e4ca0337" containerName="cleanup" Oct 30 02:15:33.366120 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.364967 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f68c408-6bab-42c7-b574-35d5d61c7544" containerName="cleanup" Oct 30 02:15:33.366120 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.364980 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="da67c0ab-cf74-4396-9987-b1bb46eb802b" containerName="cleanup" Oct 30 02:15:33.366120 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.364991 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2bd8bc3-6252-42fc-9055-ca393b6f45d9" containerName="cleanup" Oct 30 02:15:33.366120 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.365001 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fb220e0-3a46-4b35-9707-aca38ac5f288" containerName="revoke-keys" Oct 30 02:15:33.366120 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.365011 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c2dd0b9-9708-4fc8-91d2-c924b4ba9e90" containerName="cleanup" Oct 30 02:15:33.366120 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.365019 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dc3a619-596a-4e28-9845-9d1e2fc7bb06" containerName="cleanup" Oct 30 02:15:33.366120 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.365031 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="8968e7e0-04c1-4469-9062-f7d31ee6dbf2" containerName="cleanup" Oct 30 02:15:33.366120 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.365041 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="d66a87ee-dc89-4aba-845b-17b58bc6f4a6" containerName="cleanup" Oct 30 02:15:33.366120 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.365052 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="fec1f4d7-877e-4d7c-88d2-f2365469869a" containerName="cleanup" Oct 30 02:15:33.369486 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.369462 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j" Oct 30 02:15:33.390972 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.389616 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j"] Oct 30 02:15:33.394964 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.394929 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2d5aa9\"/\"test-model-2d5aa9-kserve-self-signed-certs\"" Oct 30 02:15:33.395162 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.395029 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2d5aa9\"/\"kube-root-ca.crt\"" Oct 30 02:15:33.395162 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.395126 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2d5aa9\"/\"openshift-service-ca.crt\"" Oct 30 02:15:33.395287 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.395234 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2d5aa9\"/\"default-dockercfg-kjg7j\"" Oct 30 02:15:33.428641 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.428604 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-dshm\") pod \"test-model-2d5aa9-kserve-648bdb9d6c-9nf5j\" (UID: \"d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9\") " pod="ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j" Oct 30 02:15:33.428905 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.428651 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-model-cache\") pod \"test-model-2d5aa9-kserve-648bdb9d6c-9nf5j\" (UID: \"d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9\") " pod="ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j" Oct 30 02:15:33.428905 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.428731 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn8zr\" (UniqueName: \"kubernetes.io/projected/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-kube-api-access-rn8zr\") pod \"test-model-2d5aa9-kserve-648bdb9d6c-9nf5j\" (UID: \"d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9\") " pod="ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j" Oct 30 02:15:33.429049 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.428904 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-tls-certs\") pod \"test-model-2d5aa9-kserve-648bdb9d6c-9nf5j\" (UID: \"d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9\") " pod="ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j" Oct 30 02:15:33.429114 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.429072 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-tmp-dir\") pod \"test-model-2d5aa9-kserve-648bdb9d6c-9nf5j\" (UID: \"d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9\") " pod="ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j" Oct 30 02:15:33.429170 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.429142 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-home\") pod \"test-model-2d5aa9-kserve-648bdb9d6c-9nf5j\" (UID: \"d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9\") " pod="ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j" Oct 30 02:15:33.432574 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.432543 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-657fba-b29m9" Oct 30 02:15:33.432793 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.432569 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-657fba-b29m9" event={"ID":"3fb220e0-3a46-4b35-9707-aca38ac5f288","Type":"ContainerDied","Data":"d246e7567e89ec5517a0457fa75ee78d9c2861431e1be3715718d7ae2f94699c"} Oct 30 02:15:33.432793 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.432624 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d246e7567e89ec5517a0457fa75ee78d9c2861431e1be3715718d7ae2f94699c" Oct 30 02:15:33.524682 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.524649 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:33.530495 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.530464 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-tls-certs\") pod \"test-model-2d5aa9-kserve-648bdb9d6c-9nf5j\" (UID: \"d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9\") " pod="ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j" Oct 30 02:15:33.530649 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.530575 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-tmp-dir\") pod \"test-model-2d5aa9-kserve-648bdb9d6c-9nf5j\" (UID: \"d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9\") " pod="ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j" Oct 30 02:15:33.530649 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.530643 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-home\") pod \"test-model-2d5aa9-kserve-648bdb9d6c-9nf5j\" (UID: \"d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9\") " pod="ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j" Oct 30 02:15:33.530803 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.530692 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-dshm\") pod \"test-model-2d5aa9-kserve-648bdb9d6c-9nf5j\" (UID: \"d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9\") " pod="ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j" Oct 30 02:15:33.530803 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.530743 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-model-cache\") pod \"test-model-2d5aa9-kserve-648bdb9d6c-9nf5j\" (UID: \"d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9\") " pod="ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j" Oct 30 02:15:33.530922 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.530812 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn8zr\" (UniqueName: \"kubernetes.io/projected/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-kube-api-access-rn8zr\") pod \"test-model-2d5aa9-kserve-648bdb9d6c-9nf5j\" (UID: \"d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9\") " pod="ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j" Oct 30 02:15:33.531129 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.531103 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-home\") pod \"test-model-2d5aa9-kserve-648bdb9d6c-9nf5j\" (UID: \"d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9\") " pod="ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j" Oct 30 02:15:33.531275 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.531149 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-tmp-dir\") pod \"test-model-2d5aa9-kserve-648bdb9d6c-9nf5j\" (UID: \"d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9\") " pod="ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j" Oct 30 02:15:33.531423 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.531402 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-model-cache\") pod \"test-model-2d5aa9-kserve-648bdb9d6c-9nf5j\" (UID: \"d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9\") " pod="ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j" Oct 30 02:15:33.533475 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.533446 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-tls-certs\") pod \"test-model-2d5aa9-kserve-648bdb9d6c-9nf5j\" (UID: \"d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9\") " pod="ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j" Oct 30 02:15:33.533560 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.533496 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-dshm\") pod \"test-model-2d5aa9-kserve-648bdb9d6c-9nf5j\" (UID: \"d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9\") " pod="ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j" Oct 30 02:15:33.549583 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.549551 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn8zr\" (UniqueName: \"kubernetes.io/projected/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-kube-api-access-rn8zr\") pod \"test-model-2d5aa9-kserve-648bdb9d6c-9nf5j\" (UID: \"d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9\") " pod="ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j" Oct 30 02:15:33.702783 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.702276 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j" Oct 30 02:15:33.822539 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:33.821789 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:34.090472 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:34.090441 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j"] Oct 30 02:15:34.090813 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:15:34.090784 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1ab3c1f_ad36_4a68_b0d0_66e95e88ecb9.slice/crio-ef0f8782ee3cd5e27b71d68af888ce952feca689d9ab1ea760a26bf13f9c6375 WatchSource:0}: Error finding container ef0f8782ee3cd5e27b71d68af888ce952feca689d9ab1ea760a26bf13f9c6375: Status 404 returned error can't find the container with id ef0f8782ee3cd5e27b71d68af888ce952feca689d9ab1ea760a26bf13f9c6375 Oct 30 02:15:34.447002 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:34.446947 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j" event={"ID":"d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9","Type":"ContainerStarted","Data":"f1c5c2b48433dee0ca2668ff481054d06c63273b57d6ed9605c888149da3cd2e"} Oct 30 02:15:34.447002 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:34.447006 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j" event={"ID":"d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9","Type":"ContainerStarted","Data":"ef0f8782ee3cd5e27b71d68af888ce952feca689d9ab1ea760a26bf13f9c6375"} Oct 30 02:15:34.488228 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:34.488169 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j" podStartSLOduration=1.488153839 podStartE2EDuration="1.488153839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:34.485066122 +0000 UTC m=+3460.061491523" watchObservedRunningTime="2026-10-30 02:15:34.488153839 +0000 UTC m=+3460.064579240" Oct 30 02:15:34.703324 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:34.703225 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j" Oct 30 02:15:35.391140 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:35.391105 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:35.787592 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:35.787548 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-cd8173-8445cbb698-7wtm5" podUID="909b1459-46ea-4c3c-bc2e-1fb29527200a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:15:37.116104 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:37.116060 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:38.768823 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:38.768776 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:40.047373 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:40.047333 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-657fba-8485db5fdd-5spds"] Oct 30 02:15:40.078240 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:40.078198 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-657fba-6d7784b879-n5xjk"] Oct 30 02:15:40.220682 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:40.220633 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:40.406147 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:40.406052 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:41.790562 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:41.790453 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:41.956597 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:41.956562 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-cd8173-8445cbb698-7wtm5" Oct 30 02:15:42.110043 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:42.110010 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/909b1459-46ea-4c3c-bc2e-1fb29527200a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/909b1459-46ea-4c3c-bc2e-1fb29527200a-plugins-config-volume\") pod \"909b1459-46ea-4c3c-bc2e-1fb29527200a\" (UID: \"909b1459-46ea-4c3c-bc2e-1fb29527200a\") " Oct 30 02:15:42.110247 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:42.110072 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/909b1459-46ea-4c3c-bc2e-1fb29527200a-kube-api-access-6hdpg\" (UniqueName: \"kubernetes.io/projected/909b1459-46ea-4c3c-bc2e-1fb29527200a-kube-api-access-6hdpg\") pod \"909b1459-46ea-4c3c-bc2e-1fb29527200a\" (UID: \"909b1459-46ea-4c3c-bc2e-1fb29527200a\") " Oct 30 02:15:42.110456 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:42.110434 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/909b1459-46ea-4c3c-bc2e-1fb29527200a-plugins-config-volume" pod "909b1459-46ea-4c3c-bc2e-1fb29527200a" (UID: "909b1459-46ea-4c3c-bc2e-1fb29527200a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:42.110633 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:42.110618 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/909b1459-46ea-4c3c-bc2e-1fb29527200a-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:42.112394 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:42.112371 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/909b1459-46ea-4c3c-bc2e-1fb29527200a-kube-api-access-6hdpg" pod "909b1459-46ea-4c3c-bc2e-1fb29527200a" (UID: "909b1459-46ea-4c3c-bc2e-1fb29527200a"). InnerVolumeSpecName "kube-api-access-6hdpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:42.212303 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:42.212216 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hdpg\" (UniqueName: \"kubernetes.io/projected/909b1459-46ea-4c3c-bc2e-1fb29527200a-kube-api-access-6hdpg\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:42.531920 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:42.531834 6572 generic.go:356] "Generic (PLEG): container finished" podID="909b1459-46ea-4c3c-bc2e-1fb29527200a" containerID="62ea3c1253f7e97159102d95e5784501cc79b28778bed95989c3ae698979a14a" exitCode=0 Oct 30 02:15:42.531920 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:42.531888 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-cd8173-8445cbb698-7wtm5" Oct 30 02:15:42.532126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:42.531918 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-cd8173-8445cbb698-7wtm5" event={"ID":"909b1459-46ea-4c3c-bc2e-1fb29527200a","Type":"ContainerDied","Data":"62ea3c1253f7e97159102d95e5784501cc79b28778bed95989c3ae698979a14a"} Oct 30 02:15:42.532126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:42.531983 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-cd8173-8445cbb698-7wtm5" event={"ID":"909b1459-46ea-4c3c-bc2e-1fb29527200a","Type":"ContainerDied","Data":"1ff3c9b50c63b15341ecb78f3c27a1db8bd9fbaf866b9b1318d1e477c1d9b5ff"} Oct 30 02:15:42.532126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:42.532012 6572 scope.go:122] "RemoveContainer" containerID="62ea3c1253f7e97159102d95e5784501cc79b28778bed95989c3ae698979a14a" Oct 30 02:15:42.551808 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:42.551781 6572 scope.go:122] "RemoveContainer" containerID="62ea3c1253f7e97159102d95e5784501cc79b28778bed95989c3ae698979a14a" Oct 30 02:15:42.554086 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:42.553962 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62ea3c1253f7e97159102d95e5784501cc79b28778bed95989c3ae698979a14a\": container with ID starting with 62ea3c1253f7e97159102d95e5784501cc79b28778bed95989c3ae698979a14a not found: ID does not exist" containerID="62ea3c1253f7e97159102d95e5784501cc79b28778bed95989c3ae698979a14a" Oct 30 02:15:42.554310 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:42.554280 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62ea3c1253f7e97159102d95e5784501cc79b28778bed95989c3ae698979a14a"} err="failed to get container status \"62ea3c1253f7e97159102d95e5784501cc79b28778bed95989c3ae698979a14a\": rpc error: code = NotFound desc = could not find container \"62ea3c1253f7e97159102d95e5784501cc79b28778bed95989c3ae698979a14a\": container with ID starting with 62ea3c1253f7e97159102d95e5784501cc79b28778bed95989c3ae698979a14a not found: ID does not exist" Oct 30 02:15:42.585895 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:42.585855 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-cd8173-8445cbb698-7wtm5"] Oct 30 02:15:42.595561 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:42.595532 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-cd8173-8445cbb698-7wtm5"] Oct 30 02:15:43.273256 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:43.273219 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:43.702708 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:43.702674 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j" Oct 30 02:15:43.715463 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:43.715430 6572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j" Oct 30 02:15:44.294284 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:44.294246 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95ced13a-prqjw"] Oct 30 02:15:44.295002 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:44.294984 6572 cpu_manager.go:414] "containerMap: removing container" podUID="909b1459-46ea-4c3c-bc2e-1fb29527200a" containerName="payload-processing" Oct 30 02:15:44.295002 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:44.295002 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="909b1459-46ea-4c3c-bc2e-1fb29527200a" containerName="payload-processing" Oct 30 02:15:44.295154 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:44.295145 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="909b1459-46ea-4c3c-bc2e-1fb29527200a" containerName="payload-processing" Oct 30 02:15:44.300397 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:44.300367 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95ced13a-prqjw" Oct 30 02:15:44.314632 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:44.314609 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:15:44.317607 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:44.317578 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95ced13a-prqjw"] Oct 30 02:15:44.322570 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:44.322548 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:15:44.322704 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:44.322587 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:15:44.322704 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:44.322591 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:15:44.446191 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:44.446156 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d91b3759-1119-4059-922c-aa54b8d2922c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95ced13a-prqjw\" (UID: \"d91b3759-1119-4059-922c-aa54b8d2922c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95ced13a-prqjw" Oct 30 02:15:44.446387 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:44.446201 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d91b3759-1119-4059-922c-aa54b8d2922c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95ced13a-prqjw\" (UID: \"d91b3759-1119-4059-922c-aa54b8d2922c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95ced13a-prqjw" Oct 30 02:15:44.446387 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:44.446357 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6qt9\" (UniqueName: \"kubernetes.io/projected/d91b3759-1119-4059-922c-aa54b8d2922c-kube-api-access-q6qt9\") pod \"maas-api-revoke-sub-95ced13a-prqjw\" (UID: \"d91b3759-1119-4059-922c-aa54b8d2922c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95ced13a-prqjw" Oct 30 02:15:44.546571 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:44.546478 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="909b1459-46ea-4c3c-bc2e-1fb29527200a" path="/var/lib/kubelet/pods/909b1459-46ea-4c3c-bc2e-1fb29527200a/volumes" Oct 30 02:15:44.547975 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:44.547946 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d91b3759-1119-4059-922c-aa54b8d2922c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95ced13a-prqjw\" (UID: \"d91b3759-1119-4059-922c-aa54b8d2922c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95ced13a-prqjw" Oct 30 02:15:44.548119 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:44.547993 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d91b3759-1119-4059-922c-aa54b8d2922c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95ced13a-prqjw\" (UID: \"d91b3759-1119-4059-922c-aa54b8d2922c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95ced13a-prqjw" Oct 30 02:15:44.548119 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:44.548070 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6qt9\" (UniqueName: \"kubernetes.io/projected/d91b3759-1119-4059-922c-aa54b8d2922c-kube-api-access-q6qt9\") pod \"maas-api-revoke-sub-95ced13a-prqjw\" (UID: \"d91b3759-1119-4059-922c-aa54b8d2922c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95ced13a-prqjw" Oct 30 02:15:44.548718 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:44.548685 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d91b3759-1119-4059-922c-aa54b8d2922c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95ced13a-prqjw\" (UID: \"d91b3759-1119-4059-922c-aa54b8d2922c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95ced13a-prqjw" Oct 30 02:15:44.550463 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:44.550436 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d91b3759-1119-4059-922c-aa54b8d2922c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95ced13a-prqjw\" (UID: \"d91b3759-1119-4059-922c-aa54b8d2922c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95ced13a-prqjw" Oct 30 02:15:44.566544 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:44.566515 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j" Oct 30 02:15:44.566680 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:44.566621 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6qt9\" (UniqueName: \"kubernetes.io/projected/d91b3759-1119-4059-922c-aa54b8d2922c-kube-api-access-q6qt9\") pod \"maas-api-revoke-sub-95ced13a-prqjw\" (UID: \"d91b3759-1119-4059-922c-aa54b8d2922c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95ced13a-prqjw" Oct 30 02:15:44.619514 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:44.619474 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95ced13a-prqjw" Oct 30 02:15:44.731437 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:15:44.731265 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:15:44.791060 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:44.790939 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:44.942624 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:44.942591 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95ced13a-prqjw"] Oct 30 02:15:44.943303 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:15:44.943269 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd91b3759_1119_4059_922c_aa54b8d2922c.slice/crio-b1c6ba3a33d8de005fe95659a2963a9acadc035b3e865a71af6a91e598c8f47f WatchSource:0}: Error finding container b1c6ba3a33d8de005fe95659a2963a9acadc035b3e865a71af6a91e598c8f47f: Status 404 returned error can't find the container with id b1c6ba3a33d8de005fe95659a2963a9acadc035b3e865a71af6a91e598c8f47f Oct 30 02:15:45.047719 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:45.047659 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-657fba-8485db5fdd-5spds" podUID="8ba16392-25f9-4163-9003-941b83f70c6d" containerName="payload-processing" containerID="cri-o://f5bc9ef526b52f21aa6d007258077fe79157f1221d8ffa7084277b6c029150ce" gracePeriod=25 Oct 30 02:15:45.079402 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:45.079354 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-657fba-6d7784b879-n5xjk" podUID="d336f442-61fa-4ba4-81a6-08eb9f852bad" containerName="payload-pre-processing" containerID="cri-o://5b40fad244cd547e94cb3fe03bcbd20baa635a43d5cbfa8d052f70865f2ce164" gracePeriod=25 Oct 30 02:15:45.565432 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:45.565318 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95ced13a-prqjw" event={"ID":"d91b3759-1119-4059-922c-aa54b8d2922c","Type":"ContainerStarted","Data":"a073480a2f43747cda58293cfe3a706c208665652f4c1cf6ff871e4d40dba75e"} Oct 30 02:15:45.565432 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:45.565390 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95ced13a-prqjw" event={"ID":"d91b3759-1119-4059-922c-aa54b8d2922c","Type":"ContainerStarted","Data":"b1c6ba3a33d8de005fe95659a2963a9acadc035b3e865a71af6a91e598c8f47f"} Oct 30 02:15:45.606221 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:45.606159 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95ced13a-prqjw" podStartSLOduration=1.606144478 podStartE2EDuration="1.606144478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:45.601169183 +0000 UTC m=+3471.177594586" watchObservedRunningTime="2026-10-30 02:15:45.606144478 +0000 UTC m=+3471.182569880" Oct 30 02:15:46.312442 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:46.312403 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:46.474564 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:46.474505 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-657fba-8485db5fdd-5spds" podUID="8ba16392-25f9-4163-9003-941b83f70c6d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:15:46.576932 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:46.576840 6572 generic.go:356] "Generic (PLEG): container finished" podID="d91b3759-1119-4059-922c-aa54b8d2922c" containerID="a073480a2f43747cda58293cfe3a706c208665652f4c1cf6ff871e4d40dba75e" exitCode=0 Oct 30 02:15:46.576932 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:46.576899 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95ced13a-prqjw" event={"ID":"d91b3759-1119-4059-922c-aa54b8d2922c","Type":"ContainerDied","Data":"a073480a2f43747cda58293cfe3a706c208665652f4c1cf6ff871e4d40dba75e"} Oct 30 02:15:47.763221 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:47.763192 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95ced13a-prqjw" Oct 30 02:15:47.858305 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:47.858221 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d91b3759-1119-4059-922c-aa54b8d2922c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d91b3759-1119-4059-922c-aa54b8d2922c-maas-api-xks-ca\") pod \"d91b3759-1119-4059-922c-aa54b8d2922c\" (UID: \"d91b3759-1119-4059-922c-aa54b8d2922c\") " Oct 30 02:15:47.858707 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:47.858602 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d91b3759-1119-4059-922c-aa54b8d2922c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d91b3759-1119-4059-922c-aa54b8d2922c-maas-api-openshift-ca\") pod \"d91b3759-1119-4059-922c-aa54b8d2922c\" (UID: \"d91b3759-1119-4059-922c-aa54b8d2922c\") " Oct 30 02:15:47.858707 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:47.858664 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d91b3759-1119-4059-922c-aa54b8d2922c-kube-api-access-q6qt9\" (UniqueName: \"kubernetes.io/projected/d91b3759-1119-4059-922c-aa54b8d2922c-kube-api-access-q6qt9\") pod \"d91b3759-1119-4059-922c-aa54b8d2922c\" (UID: \"d91b3759-1119-4059-922c-aa54b8d2922c\") " Oct 30 02:15:47.874398 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:47.874314 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d91b3759-1119-4059-922c-aa54b8d2922c-maas-api-openshift-ca" pod "d91b3759-1119-4059-922c-aa54b8d2922c" (UID: "d91b3759-1119-4059-922c-aa54b8d2922c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:47.880480 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:47.880245 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:47.898161 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:47.898082 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d91b3759-1119-4059-922c-aa54b8d2922c-maas-api-xks-ca" pod "d91b3759-1119-4059-922c-aa54b8d2922c" (UID: "d91b3759-1119-4059-922c-aa54b8d2922c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:15:47.902396 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:47.900124 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91b3759-1119-4059-922c-aa54b8d2922c-kube-api-access-q6qt9" pod "d91b3759-1119-4059-922c-aa54b8d2922c" (UID: "d91b3759-1119-4059-922c-aa54b8d2922c"). InnerVolumeSpecName "kube-api-access-q6qt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:47.960600 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:47.960562 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d91b3759-1119-4059-922c-aa54b8d2922c-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:47.960600 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:47.960595 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d91b3759-1119-4059-922c-aa54b8d2922c-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:47.960600 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:47.960609 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6qt9\" (UniqueName: \"kubernetes.io/projected/d91b3759-1119-4059-922c-aa54b8d2922c-kube-api-access-q6qt9\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:48.597620 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:48.597589 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95ced13a-prqjw" Oct 30 02:15:48.597838 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:48.597623 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95ced13a-prqjw" event={"ID":"d91b3759-1119-4059-922c-aa54b8d2922c","Type":"ContainerDied","Data":"b1c6ba3a33d8de005fe95659a2963a9acadc035b3e865a71af6a91e598c8f47f"} Oct 30 02:15:48.597838 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:48.597679 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1c6ba3a33d8de005fe95659a2963a9acadc035b3e865a71af6a91e598c8f47f" Oct 30 02:15:49.573589 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:49.573552 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:51.179805 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:51.179723 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:52.442664 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:52.442613 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:53.038189 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:53.038142 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:53.071275 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:53.071238 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-cd8173-zd4ct"] Oct 30 02:15:53.081982 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:53.081949 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-cd8173-zd4ct"] Oct 30 02:15:53.237050 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:53.237019 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:53.663811 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:53.663733 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:54.398101 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:54.398062 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-657fba-6d7784b879-n5xjk" podUID="d336f442-61fa-4ba4-81a6-08eb9f852bad" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:15:54.546422 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:54.546385 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8e0de10-d72a-4a21-9f72-4d27d10c1623" path="/var/lib/kubelet/pods/c8e0de10-d72a-4a21-9f72-4d27d10c1623/volumes" Oct 30 02:15:54.872004 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:54.871947 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:55.288061 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:55.287933 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:55.627829 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:55.627720 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b1ffb61-dsvfq"] Oct 30 02:15:55.628493 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:55.628476 6572 cpu_manager.go:414] "containerMap: removing container" podUID="d91b3759-1119-4059-922c-aa54b8d2922c" containerName="revoke-keys" Oct 30 02:15:55.628541 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:55.628494 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d91b3759-1119-4059-922c-aa54b8d2922c" containerName="revoke-keys" Oct 30 02:15:55.628635 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:55.628625 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="d91b3759-1119-4059-922c-aa54b8d2922c" containerName="revoke-keys" Oct 30 02:15:55.633814 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:55.633735 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1ffb61-dsvfq" Oct 30 02:15:55.650398 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:55.650365 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b1ffb61-dsvfq"] Oct 30 02:15:55.656571 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:55.656278 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:15:55.656571 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:55.656278 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:15:55.656869 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:55.656650 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:15:55.658706 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:55.658679 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:15:55.695363 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:55.695325 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b1ffb61-dsvfq\" (UID: \"c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1ffb61-dsvfq" Oct 30 02:15:55.695697 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:55.695676 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b1ffb61-dsvfq\" (UID: \"c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1ffb61-dsvfq" Oct 30 02:15:55.695861 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:55.695845 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbtgl\" (UniqueName: \"kubernetes.io/projected/c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475-kube-api-access-lbtgl\") pod \"maas-api-revoke-sub-6b1ffb61-dsvfq\" (UID: \"c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1ffb61-dsvfq" Oct 30 02:15:55.797441 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:55.797398 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b1ffb61-dsvfq\" (UID: \"c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1ffb61-dsvfq" Oct 30 02:15:55.797673 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:55.797558 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbtgl\" (UniqueName: \"kubernetes.io/projected/c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475-kube-api-access-lbtgl\") pod \"maas-api-revoke-sub-6b1ffb61-dsvfq\" (UID: \"c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1ffb61-dsvfq" Oct 30 02:15:55.797673 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:55.797589 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b1ffb61-dsvfq\" (UID: \"c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1ffb61-dsvfq" Oct 30 02:15:55.798289 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:55.798265 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b1ffb61-dsvfq\" (UID: \"c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1ffb61-dsvfq" Oct 30 02:15:55.799906 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:55.799883 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b1ffb61-dsvfq\" (UID: \"c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1ffb61-dsvfq" Oct 30 02:15:55.816330 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:55.816293 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbtgl\" (UniqueName: \"kubernetes.io/projected/c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475-kube-api-access-lbtgl\") pod \"maas-api-revoke-sub-6b1ffb61-dsvfq\" (UID: \"c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1ffb61-dsvfq" Oct 30 02:15:55.953877 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:55.953834 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1ffb61-dsvfq" Oct 30 02:15:56.261307 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:56.261273 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b1ffb61-dsvfq"] Oct 30 02:15:56.261456 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:15:56.261333 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3bd2ccc_4ec5_4e6b_ba5b_5df3dc003475.slice/crio-eb6263e440e8bd491e6f2ebd241e9ff95d003fc4906035d8bcca6227b8e27f87 WatchSource:0}: Error finding container eb6263e440e8bd491e6f2ebd241e9ff95d003fc4906035d8bcca6227b8e27f87: Status 404 returned error can't find the container with id eb6263e440e8bd491e6f2ebd241e9ff95d003fc4906035d8bcca6227b8e27f87 Oct 30 02:15:56.455286 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:56.455246 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:56.474649 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:56.474611 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-657fba-8485db5fdd-5spds" podUID="8ba16392-25f9-4163-9003-941b83f70c6d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:15:56.698376 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:56.698338 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1ffb61-dsvfq" event={"ID":"c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475","Type":"ContainerStarted","Data":"eb6263e440e8bd491e6f2ebd241e9ff95d003fc4906035d8bcca6227b8e27f87"} Oct 30 02:15:57.710611 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:57.710575 6572 generic.go:356] "Generic (PLEG): container finished" podID="c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475" containerID="bf1b10e17cf04829cad9aa8c34891b8498af834494e4ee1a7fece3361dcd645d" exitCode=0 Oct 30 02:15:57.711151 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:57.710656 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1ffb61-dsvfq" event={"ID":"c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475","Type":"ContainerDied","Data":"bf1b10e17cf04829cad9aa8c34891b8498af834494e4ee1a7fece3361dcd645d"} Oct 30 02:15:57.940940 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:57.940899 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:58.899205 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:58.899176 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1ffb61-dsvfq" Oct 30 02:15:59.061475 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:59.061377 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475-maas-api-xks-ca\") pod \"c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475\" (UID: \"c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475\") " Oct 30 02:15:59.061475 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:59.061462 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475-kube-api-access-lbtgl\" (UniqueName: \"kubernetes.io/projected/c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475-kube-api-access-lbtgl\") pod \"c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475\" (UID: \"c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475\") " Oct 30 02:15:59.061702 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:59.061603 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475-maas-api-openshift-ca\") pod \"c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475\" (UID: \"c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475\") " Oct 30 02:15:59.062014 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:59.061988 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475-maas-api-openshift-ca" pod "c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475" (UID: "c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:59.062318 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:59.062300 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:59.063838 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:59.063809 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475-kube-api-access-lbtgl" pod "c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475" (UID: "c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475"). InnerVolumeSpecName "kube-api-access-lbtgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:59.063940 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:59.063857 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475-maas-api-xks-ca" pod "c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475" (UID: "c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:15:59.163713 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:59.163676 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:59.163713 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:59.163709 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbtgl\" (UniqueName: \"kubernetes.io/projected/c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475-kube-api-access-lbtgl\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:15:59.436823 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:59.436787 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:59.635264 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:59.635154 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:59.730299 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:59.730262 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1ffb61-dsvfq" Oct 30 02:15:59.730299 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:59.730292 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b1ffb61-dsvfq" event={"ID":"c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475","Type":"ContainerDied","Data":"eb6263e440e8bd491e6f2ebd241e9ff95d003fc4906035d8bcca6227b8e27f87"} Oct 30 02:15:59.730568 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:59.730340 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb6263e440e8bd491e6f2ebd241e9ff95d003fc4906035d8bcca6227b8e27f87" Oct 30 02:15:59.875767 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:15:59.869725 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:00.276771 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:00.276711 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b6b7fc-mfnf6"] Oct 30 02:16:00.277512 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:00.277493 6572 cpu_manager.go:414] "containerMap: removing container" podUID="c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475" containerName="revoke-keys" Oct 30 02:16:00.277584 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:00.277513 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475" containerName="revoke-keys" Oct 30 02:16:00.277678 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:00.277668 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475" containerName="revoke-keys" Oct 30 02:16:00.283460 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:00.283438 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b6b7fc-mfnf6" Oct 30 02:16:00.299922 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:00.299891 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b6b7fc-mfnf6"] Oct 30 02:16:00.305232 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:00.305194 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:16:00.305422 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:00.305231 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:16:00.305422 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:00.305263 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:16:00.310619 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:00.310586 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:16:00.417019 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:00.416977 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4fe447d-cdf6-43c6-b785-7a57ad00eba2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b6b7fc-mfnf6\" (UID: \"b4fe447d-cdf6-43c6-b785-7a57ad00eba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b6b7fc-mfnf6" Oct 30 02:16:00.417212 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:00.417173 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4fe447d-cdf6-43c6-b785-7a57ad00eba2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b6b7fc-mfnf6\" (UID: \"b4fe447d-cdf6-43c6-b785-7a57ad00eba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b6b7fc-mfnf6" Oct 30 02:16:00.417301 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:00.417212 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r5jg\" (UniqueName: \"kubernetes.io/projected/b4fe447d-cdf6-43c6-b785-7a57ad00eba2-kube-api-access-6r5jg\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b6b7fc-mfnf6\" (UID: \"b4fe447d-cdf6-43c6-b785-7a57ad00eba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b6b7fc-mfnf6" Oct 30 02:16:00.518425 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:00.518382 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4fe447d-cdf6-43c6-b785-7a57ad00eba2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b6b7fc-mfnf6\" (UID: \"b4fe447d-cdf6-43c6-b785-7a57ad00eba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b6b7fc-mfnf6" Oct 30 02:16:00.518425 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:00.518428 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r5jg\" (UniqueName: \"kubernetes.io/projected/b4fe447d-cdf6-43c6-b785-7a57ad00eba2-kube-api-access-6r5jg\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b6b7fc-mfnf6\" (UID: \"b4fe447d-cdf6-43c6-b785-7a57ad00eba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b6b7fc-mfnf6" Oct 30 02:16:00.518706 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:00.518575 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4fe447d-cdf6-43c6-b785-7a57ad00eba2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b6b7fc-mfnf6\" (UID: \"b4fe447d-cdf6-43c6-b785-7a57ad00eba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b6b7fc-mfnf6" Oct 30 02:16:00.519183 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:00.519160 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4fe447d-cdf6-43c6-b785-7a57ad00eba2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b6b7fc-mfnf6\" (UID: \"b4fe447d-cdf6-43c6-b785-7a57ad00eba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b6b7fc-mfnf6" Oct 30 02:16:00.521076 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:00.521049 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4fe447d-cdf6-43c6-b785-7a57ad00eba2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b6b7fc-mfnf6\" (UID: \"b4fe447d-cdf6-43c6-b785-7a57ad00eba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b6b7fc-mfnf6" Oct 30 02:16:00.537544 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:00.537451 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r5jg\" (UniqueName: \"kubernetes.io/projected/b4fe447d-cdf6-43c6-b785-7a57ad00eba2-kube-api-access-6r5jg\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b6b7fc-mfnf6\" (UID: \"b4fe447d-cdf6-43c6-b785-7a57ad00eba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b6b7fc-mfnf6" Oct 30 02:16:00.603421 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:00.603380 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b6b7fc-mfnf6" Oct 30 02:16:00.905961 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:00.905928 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b6b7fc-mfnf6"] Oct 30 02:16:00.906843 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:16:00.906816 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4fe447d_cdf6_43c6_b785_7a57ad00eba2.slice/crio-b4f0d7b8650a3080e86e087fc6f82275c12701373c81e0e18cdd8098143f767f WatchSource:0}: Error finding container b4f0d7b8650a3080e86e087fc6f82275c12701373c81e0e18cdd8098143f767f: Status 404 returned error can't find the container with id b4f0d7b8650a3080e86e087fc6f82275c12701373c81e0e18cdd8098143f767f Oct 30 02:16:01.028879 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:01.028836 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bd81604-p5fjs"] Oct 30 02:16:01.033080 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:01.033048 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bd81604-p5fjs"] Oct 30 02:16:01.283285 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:01.282841 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:01.751619 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:01.751578 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b6b7fc-mfnf6" event={"ID":"b4fe447d-cdf6-43c6-b785-7a57ad00eba2","Type":"ContainerStarted","Data":"e0f16c01b5af3a9e9be408f1d4fe642888b5909bb7f24715cdf4e486dbfd27e1"} Oct 30 02:16:01.751852 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:01.751641 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b6b7fc-mfnf6" event={"ID":"b4fe447d-cdf6-43c6-b785-7a57ad00eba2","Type":"ContainerStarted","Data":"b4f0d7b8650a3080e86e087fc6f82275c12701373c81e0e18cdd8098143f767f"} Oct 30 02:16:01.792497 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:01.792430 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b6b7fc-mfnf6" podStartSLOduration=1.7924145820000001 podStartE2EDuration="1.792414582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:16:01.788484311 +0000 UTC m=+3487.364909714" watchObservedRunningTime="2026-10-30 02:16:01.792414582 +0000 UTC m=+3487.368839977" Oct 30 02:16:02.544899 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:02.544861 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e537af8-9a0b-4ff3-bb95-00cd183fa206" path="/var/lib/kubelet/pods/5e537af8-9a0b-4ff3-bb95-00cd183fa206/volumes" Oct 30 02:16:02.806971 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:02.806877 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:02.807178 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:02.806981 6572 generic.go:356] "Generic (PLEG): container finished" podID="b4fe447d-cdf6-43c6-b785-7a57ad00eba2" containerID="e0f16c01b5af3a9e9be408f1d4fe642888b5909bb7f24715cdf4e486dbfd27e1" exitCode=0 Oct 30 02:16:02.807178 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:02.807015 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b6b7fc-mfnf6" event={"ID":"b4fe447d-cdf6-43c6-b785-7a57ad00eba2","Type":"ContainerDied","Data":"e0f16c01b5af3a9e9be408f1d4fe642888b5909bb7f24715cdf4e486dbfd27e1"} Oct 30 02:16:04.002810 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:04.002784 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b6b7fc-mfnf6" Oct 30 02:16:04.108732 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:04.108619 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4fe447d-cdf6-43c6-b785-7a57ad00eba2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4fe447d-cdf6-43c6-b785-7a57ad00eba2-maas-api-openshift-ca\") pod \"b4fe447d-cdf6-43c6-b785-7a57ad00eba2\" (UID: \"b4fe447d-cdf6-43c6-b785-7a57ad00eba2\") " Oct 30 02:16:04.108732 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:04.108716 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4fe447d-cdf6-43c6-b785-7a57ad00eba2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4fe447d-cdf6-43c6-b785-7a57ad00eba2-maas-api-xks-ca\") pod \"b4fe447d-cdf6-43c6-b785-7a57ad00eba2\" (UID: \"b4fe447d-cdf6-43c6-b785-7a57ad00eba2\") " Oct 30 02:16:04.109026 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:04.108820 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4fe447d-cdf6-43c6-b785-7a57ad00eba2-kube-api-access-6r5jg\" (UniqueName: \"kubernetes.io/projected/b4fe447d-cdf6-43c6-b785-7a57ad00eba2-kube-api-access-6r5jg\") pod \"b4fe447d-cdf6-43c6-b785-7a57ad00eba2\" (UID: \"b4fe447d-cdf6-43c6-b785-7a57ad00eba2\") " Oct 30 02:16:04.109178 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:04.109146 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4fe447d-cdf6-43c6-b785-7a57ad00eba2-maas-api-openshift-ca" pod "b4fe447d-cdf6-43c6-b785-7a57ad00eba2" (UID: "b4fe447d-cdf6-43c6-b785-7a57ad00eba2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:16:04.109582 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:04.109556 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4fe447d-cdf6-43c6-b785-7a57ad00eba2-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:04.111402 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:04.111371 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4fe447d-cdf6-43c6-b785-7a57ad00eba2-maas-api-xks-ca" pod "b4fe447d-cdf6-43c6-b785-7a57ad00eba2" (UID: "b4fe447d-cdf6-43c6-b785-7a57ad00eba2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:16:04.111883 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:04.111860 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4fe447d-cdf6-43c6-b785-7a57ad00eba2-kube-api-access-6r5jg" pod "b4fe447d-cdf6-43c6-b785-7a57ad00eba2" (UID: "b4fe447d-cdf6-43c6-b785-7a57ad00eba2"). InnerVolumeSpecName "kube-api-access-6r5jg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:16:04.210639 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:04.210598 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4fe447d-cdf6-43c6-b785-7a57ad00eba2-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:04.210639 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:04.210627 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r5jg\" (UniqueName: \"kubernetes.io/projected/b4fe447d-cdf6-43c6-b785-7a57ad00eba2-kube-api-access-6r5jg\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:04.397602 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:04.397508 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-657fba-6d7784b879-n5xjk" podUID="d336f442-61fa-4ba4-81a6-08eb9f852bad" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:16:04.586212 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:04.585325 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:04.832516 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:04.832484 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b6b7fc-mfnf6" Oct 30 02:16:04.832780 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:04.832534 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b6b7fc-mfnf6" event={"ID":"b4fe447d-cdf6-43c6-b785-7a57ad00eba2","Type":"ContainerDied","Data":"b4f0d7b8650a3080e86e087fc6f82275c12701373c81e0e18cdd8098143f767f"} Oct 30 02:16:04.832780 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:04.832629 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4f0d7b8650a3080e86e087fc6f82275c12701373c81e0e18cdd8098143f767f" Oct 30 02:16:05.388008 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:05.387944 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-657fba-8485db5fdd-5spds" Oct 30 02:16:05.422273 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:05.422248 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-657fba-6d7784b879-n5xjk" Oct 30 02:16:05.434295 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:05.434263 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ba16392-25f9-4163-9003-941b83f70c6d-kube-api-access-9wmv9\" (UniqueName: \"kubernetes.io/projected/8ba16392-25f9-4163-9003-941b83f70c6d-kube-api-access-9wmv9\") pod \"8ba16392-25f9-4163-9003-941b83f70c6d\" (UID: \"8ba16392-25f9-4163-9003-941b83f70c6d\") " Oct 30 02:16:05.434295 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:05.434297 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ba16392-25f9-4163-9003-941b83f70c6d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba16392-25f9-4163-9003-941b83f70c6d-plugins-config-volume\") pod \"8ba16392-25f9-4163-9003-941b83f70c6d\" (UID: \"8ba16392-25f9-4163-9003-941b83f70c6d\") " Oct 30 02:16:05.434792 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:05.434767 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba16392-25f9-4163-9003-941b83f70c6d-plugins-config-volume" pod "8ba16392-25f9-4163-9003-941b83f70c6d" (UID: "8ba16392-25f9-4163-9003-941b83f70c6d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:16:05.435134 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:05.435087 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba16392-25f9-4163-9003-941b83f70c6d-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:05.436856 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:05.436825 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba16392-25f9-4163-9003-941b83f70c6d-kube-api-access-9wmv9" pod "8ba16392-25f9-4163-9003-941b83f70c6d" (UID: "8ba16392-25f9-4163-9003-941b83f70c6d"). InnerVolumeSpecName "kube-api-access-9wmv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:16:05.536312 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:05.536280 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d336f442-61fa-4ba4-81a6-08eb9f852bad-kube-api-access-z6d5t\" (UniqueName: \"kubernetes.io/projected/d336f442-61fa-4ba4-81a6-08eb9f852bad-kube-api-access-z6d5t\") pod \"d336f442-61fa-4ba4-81a6-08eb9f852bad\" (UID: \"d336f442-61fa-4ba4-81a6-08eb9f852bad\") " Oct 30 02:16:05.536528 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:05.536328 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d336f442-61fa-4ba4-81a6-08eb9f852bad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d336f442-61fa-4ba4-81a6-08eb9f852bad-plugins-config-volume\") pod \"d336f442-61fa-4ba4-81a6-08eb9f852bad\" (UID: \"d336f442-61fa-4ba4-81a6-08eb9f852bad\") " Oct 30 02:16:05.536833 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:05.536804 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d336f442-61fa-4ba4-81a6-08eb9f852bad-plugins-config-volume" pod "d336f442-61fa-4ba4-81a6-08eb9f852bad" (UID: "d336f442-61fa-4ba4-81a6-08eb9f852bad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:16:05.536944 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:05.536864 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wmv9\" (UniqueName: \"kubernetes.io/projected/8ba16392-25f9-4163-9003-941b83f70c6d-kube-api-access-9wmv9\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:05.538653 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:05.538632 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d336f442-61fa-4ba4-81a6-08eb9f852bad-kube-api-access-z6d5t" pod "d336f442-61fa-4ba4-81a6-08eb9f852bad" (UID: "d336f442-61fa-4ba4-81a6-08eb9f852bad"). InnerVolumeSpecName "kube-api-access-z6d5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:16:05.599722 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:05.599685 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:05.637971 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:05.637934 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6d5t\" (UniqueName: \"kubernetes.io/projected/d336f442-61fa-4ba4-81a6-08eb9f852bad-kube-api-access-z6d5t\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:05.637971 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:05.637966 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d336f442-61fa-4ba4-81a6-08eb9f852bad-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:05.842943 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:05.842906 6572 generic.go:356] "Generic (PLEG): container finished" podID="d336f442-61fa-4ba4-81a6-08eb9f852bad" containerID="5b40fad244cd547e94cb3fe03bcbd20baa635a43d5cbfa8d052f70865f2ce164" exitCode=0 Oct 30 02:16:05.843170 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:05.842978 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-657fba-6d7784b879-n5xjk" Oct 30 02:16:05.843170 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:05.842992 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-657fba-6d7784b879-n5xjk" event={"ID":"d336f442-61fa-4ba4-81a6-08eb9f852bad","Type":"ContainerDied","Data":"5b40fad244cd547e94cb3fe03bcbd20baa635a43d5cbfa8d052f70865f2ce164"} Oct 30 02:16:05.843170 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:05.843058 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-657fba-6d7784b879-n5xjk" event={"ID":"d336f442-61fa-4ba4-81a6-08eb9f852bad","Type":"ContainerDied","Data":"c34efd72763ba771eb282c87b45fd7a38dbd17148e954a9eb191bb92d43d5150"} Oct 30 02:16:05.843170 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:05.843092 6572 scope.go:122] "RemoveContainer" containerID="5b40fad244cd547e94cb3fe03bcbd20baa635a43d5cbfa8d052f70865f2ce164" Oct 30 02:16:05.844592 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:05.844571 6572 generic.go:356] "Generic (PLEG): container finished" podID="8ba16392-25f9-4163-9003-941b83f70c6d" containerID="f5bc9ef526b52f21aa6d007258077fe79157f1221d8ffa7084277b6c029150ce" exitCode=0 Oct 30 02:16:05.844703 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:05.844628 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-657fba-8485db5fdd-5spds" Oct 30 02:16:05.844703 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:05.844658 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-657fba-8485db5fdd-5spds" event={"ID":"8ba16392-25f9-4163-9003-941b83f70c6d","Type":"ContainerDied","Data":"f5bc9ef526b52f21aa6d007258077fe79157f1221d8ffa7084277b6c029150ce"} Oct 30 02:16:05.844871 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:05.844720 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-657fba-8485db5fdd-5spds" event={"ID":"8ba16392-25f9-4163-9003-941b83f70c6d","Type":"ContainerDied","Data":"7b4192c9353441d583e655e72fb78cc0ce0ab6ee3c358c26a900f2b44031816d"} Oct 30 02:16:05.856730 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:05.856675 6572 scope.go:122] "RemoveContainer" containerID="5b40fad244cd547e94cb3fe03bcbd20baa635a43d5cbfa8d052f70865f2ce164" Oct 30 02:16:05.857236 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:16:05.857207 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b40fad244cd547e94cb3fe03bcbd20baa635a43d5cbfa8d052f70865f2ce164\": container with ID starting with 5b40fad244cd547e94cb3fe03bcbd20baa635a43d5cbfa8d052f70865f2ce164 not found: ID does not exist" containerID="5b40fad244cd547e94cb3fe03bcbd20baa635a43d5cbfa8d052f70865f2ce164" Oct 30 02:16:05.857363 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:05.857244 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b40fad244cd547e94cb3fe03bcbd20baa635a43d5cbfa8d052f70865f2ce164"} err="failed to get container status \"5b40fad244cd547e94cb3fe03bcbd20baa635a43d5cbfa8d052f70865f2ce164\": rpc error: code = NotFound desc = could not find container \"5b40fad244cd547e94cb3fe03bcbd20baa635a43d5cbfa8d052f70865f2ce164\": container with ID starting with 5b40fad244cd547e94cb3fe03bcbd20baa635a43d5cbfa8d052f70865f2ce164 not found: ID does not exist" Oct 30 02:16:05.857363 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:05.857263 6572 scope.go:122] "RemoveContainer" containerID="f5bc9ef526b52f21aa6d007258077fe79157f1221d8ffa7084277b6c029150ce" Oct 30 02:16:05.869152 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:05.869133 6572 scope.go:122] "RemoveContainer" containerID="f5bc9ef526b52f21aa6d007258077fe79157f1221d8ffa7084277b6c029150ce" Oct 30 02:16:05.869440 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:16:05.869422 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5bc9ef526b52f21aa6d007258077fe79157f1221d8ffa7084277b6c029150ce\": container with ID starting with f5bc9ef526b52f21aa6d007258077fe79157f1221d8ffa7084277b6c029150ce not found: ID does not exist" containerID="f5bc9ef526b52f21aa6d007258077fe79157f1221d8ffa7084277b6c029150ce" Oct 30 02:16:05.869514 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:05.869451 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5bc9ef526b52f21aa6d007258077fe79157f1221d8ffa7084277b6c029150ce"} err="failed to get container status \"f5bc9ef526b52f21aa6d007258077fe79157f1221d8ffa7084277b6c029150ce\": rpc error: code = NotFound desc = could not find container \"f5bc9ef526b52f21aa6d007258077fe79157f1221d8ffa7084277b6c029150ce\": container with ID starting with f5bc9ef526b52f21aa6d007258077fe79157f1221d8ffa7084277b6c029150ce not found: ID does not exist" Oct 30 02:16:05.895509 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:05.895476 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-657fba-8485db5fdd-5spds"] Oct 30 02:16:05.907463 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:05.907429 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-657fba-8485db5fdd-5spds"] Oct 30 02:16:05.939708 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:05.939674 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-657fba-6d7784b879-n5xjk"] Oct 30 02:16:05.946191 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:05.946156 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-657fba-6d7784b879-n5xjk"] Oct 30 02:16:06.049901 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:06.049868 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:06.544397 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:06.544361 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ba16392-25f9-4163-9003-941b83f70c6d" path="/var/lib/kubelet/pods/8ba16392-25f9-4163-9003-941b83f70c6d/volumes" Oct 30 02:16:06.544908 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:06.544889 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d336f442-61fa-4ba4-81a6-08eb9f852bad" path="/var/lib/kubelet/pods/d336f442-61fa-4ba4-81a6-08eb9f852bad/volumes" Oct 30 02:16:07.352716 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:07.352629 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:08.722311 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:08.720875 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:08.979596 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:08.979481 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:10.079112 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:10.079073 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:10.917269 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:10.917229 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:11.031417 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:11.031379 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c951947-rhrl4"] Oct 30 02:16:11.041816 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:11.041786 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c951947-rhrl4"] Oct 30 02:16:11.099863 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:11.099816 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:11.305450 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:11.305355 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:12.037993 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:12.034008 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:12.543965 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:12.543929 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9a81242-e317-4648-ab26-831696c7e08e" path="/var/lib/kubelet/pods/f9a81242-e317-4648-ab26-831696c7e08e/volumes" Oct 30 02:16:13.402237 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:13.402161 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:14.690253 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:16:14.690211 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:16:14.784346 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:14.784299 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:16.138525 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:16.138490 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:16.915377 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:16.915340 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb"] Oct 30 02:16:17.062206 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:17.062168 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:17.209363 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:17.209266 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aaf045-ttm2c"] Oct 30 02:16:17.210037 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:17.210020 6572 cpu_manager.go:414] "containerMap: removing container" podUID="d336f442-61fa-4ba4-81a6-08eb9f852bad" containerName="payload-pre-processing" Oct 30 02:16:17.210097 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:17.210039 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d336f442-61fa-4ba4-81a6-08eb9f852bad" containerName="payload-pre-processing" Oct 30 02:16:17.210097 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:17.210054 6572 cpu_manager.go:414] "containerMap: removing container" podUID="b4fe447d-cdf6-43c6-b785-7a57ad00eba2" containerName="revoke-keys" Oct 30 02:16:17.210097 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:17.210060 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4fe447d-cdf6-43c6-b785-7a57ad00eba2" containerName="revoke-keys" Oct 30 02:16:17.210208 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:17.210127 6572 cpu_manager.go:414] "containerMap: removing container" podUID="8ba16392-25f9-4163-9003-941b83f70c6d" containerName="payload-processing" Oct 30 02:16:17.210208 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:17.210133 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ba16392-25f9-4163-9003-941b83f70c6d" containerName="payload-processing" Oct 30 02:16:17.210302 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:17.210240 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ba16392-25f9-4163-9003-941b83f70c6d" containerName="payload-processing" Oct 30 02:16:17.210302 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:17.210251 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="d336f442-61fa-4ba4-81a6-08eb9f852bad" containerName="payload-pre-processing" Oct 30 02:16:17.210302 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:17.210259 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4fe447d-cdf6-43c6-b785-7a57ad00eba2" containerName="revoke-keys" Oct 30 02:16:17.216092 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:17.216065 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aaf045-ttm2c" Oct 30 02:16:17.232906 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:17.232730 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:16:17.234652 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:17.234081 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aaf045-ttm2c"] Oct 30 02:16:17.238283 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:17.238258 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:16:17.265742 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:17.265704 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:16:17.265969 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:17.265842 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:16:17.375721 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:17.375677 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0255eea-e6a3-4288-92c9-6a5c4900d961-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-aaf045-ttm2c\" (UID: \"a0255eea-e6a3-4288-92c9-6a5c4900d961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aaf045-ttm2c" Oct 30 02:16:17.375943 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:17.375730 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7k2l\" (UniqueName: \"kubernetes.io/projected/a0255eea-e6a3-4288-92c9-6a5c4900d961-kube-api-access-x7k2l\") pod \"maas-api-revoke-keys-e2e-inf-a-aaf045-ttm2c\" (UID: \"a0255eea-e6a3-4288-92c9-6a5c4900d961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aaf045-ttm2c" Oct 30 02:16:17.375943 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:17.375834 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0255eea-e6a3-4288-92c9-6a5c4900d961-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-aaf045-ttm2c\" (UID: \"a0255eea-e6a3-4288-92c9-6a5c4900d961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aaf045-ttm2c" Oct 30 02:16:17.477489 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:17.477381 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0255eea-e6a3-4288-92c9-6a5c4900d961-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-aaf045-ttm2c\" (UID: \"a0255eea-e6a3-4288-92c9-6a5c4900d961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aaf045-ttm2c" Oct 30 02:16:17.477489 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:17.477449 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7k2l\" (UniqueName: \"kubernetes.io/projected/a0255eea-e6a3-4288-92c9-6a5c4900d961-kube-api-access-x7k2l\") pod \"maas-api-revoke-keys-e2e-inf-a-aaf045-ttm2c\" (UID: \"a0255eea-e6a3-4288-92c9-6a5c4900d961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aaf045-ttm2c" Oct 30 02:16:17.477489 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:17.477489 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0255eea-e6a3-4288-92c9-6a5c4900d961-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-aaf045-ttm2c\" (UID: \"a0255eea-e6a3-4288-92c9-6a5c4900d961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aaf045-ttm2c" Oct 30 02:16:17.478089 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:17.478067 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0255eea-e6a3-4288-92c9-6a5c4900d961-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-aaf045-ttm2c\" (UID: \"a0255eea-e6a3-4288-92c9-6a5c4900d961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aaf045-ttm2c" Oct 30 02:16:17.480003 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:17.479973 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0255eea-e6a3-4288-92c9-6a5c4900d961-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-aaf045-ttm2c\" (UID: \"a0255eea-e6a3-4288-92c9-6a5c4900d961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aaf045-ttm2c" Oct 30 02:16:17.495910 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:17.495878 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7k2l\" (UniqueName: \"kubernetes.io/projected/a0255eea-e6a3-4288-92c9-6a5c4900d961-kube-api-access-x7k2l\") pod \"maas-api-revoke-keys-e2e-inf-a-aaf045-ttm2c\" (UID: \"a0255eea-e6a3-4288-92c9-6a5c4900d961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aaf045-ttm2c" Oct 30 02:16:17.537874 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:17.537834 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aaf045-ttm2c" Oct 30 02:16:17.779836 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:17.758868 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:17.912997 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:17.912966 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aaf045-ttm2c"] Oct 30 02:16:17.913980 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:16:17.913957 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0255eea_e6a3_4288_92c9_6a5c4900d961.slice/crio-2a102a6b77cbaeee426eb1e9cb52b2fcb3240cd5b5a40d40f0061c17b5221091 WatchSource:0}: Error finding container 2a102a6b77cbaeee426eb1e9cb52b2fcb3240cd5b5a40d40f0061c17b5221091: Status 404 returned error can't find the container with id 2a102a6b77cbaeee426eb1e9cb52b2fcb3240cd5b5a40d40f0061c17b5221091 Oct 30 02:16:17.953662 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:17.953628 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aaf045-ttm2c" event={"ID":"a0255eea-e6a3-4288-92c9-6a5c4900d961","Type":"ContainerStarted","Data":"2a102a6b77cbaeee426eb1e9cb52b2fcb3240cd5b5a40d40f0061c17b5221091"} Oct 30 02:16:18.967840 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:18.967398 6572 generic.go:356] "Generic (PLEG): container finished" podID="a0255eea-e6a3-4288-92c9-6a5c4900d961" containerID="dfc84eef8dfada8b8f69b0ac6e7233031feda3010b1f0a6875574f446d9089de" exitCode=0 Oct 30 02:16:18.967840 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:18.967463 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aaf045-ttm2c" event={"ID":"a0255eea-e6a3-4288-92c9-6a5c4900d961","Type":"ContainerDied","Data":"dfc84eef8dfada8b8f69b0ac6e7233031feda3010b1f0a6875574f446d9089de"} Oct 30 02:16:19.053920 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:19.053880 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:19.545842 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:19.545722 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:19.683223 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:19.683186 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-657fba-b29m9"] Oct 30 02:16:19.713617 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:19.713574 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-657fba-b29m9"] Oct 30 02:16:19.739344 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:19.739307 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:19.926281 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:19.926245 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:20.160055 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:20.160028 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aaf045-ttm2c" Oct 30 02:16:20.225682 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:20.225586 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0255eea-e6a3-4288-92c9-6a5c4900d961-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0255eea-e6a3-4288-92c9-6a5c4900d961-maas-api-openshift-ca\") pod \"a0255eea-e6a3-4288-92c9-6a5c4900d961\" (UID: \"a0255eea-e6a3-4288-92c9-6a5c4900d961\") " Oct 30 02:16:20.225904 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:20.225738 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0255eea-e6a3-4288-92c9-6a5c4900d961-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0255eea-e6a3-4288-92c9-6a5c4900d961-maas-api-xks-ca\") pod \"a0255eea-e6a3-4288-92c9-6a5c4900d961\" (UID: \"a0255eea-e6a3-4288-92c9-6a5c4900d961\") " Oct 30 02:16:20.225904 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:20.225873 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0255eea-e6a3-4288-92c9-6a5c4900d961-kube-api-access-x7k2l\" (UniqueName: \"kubernetes.io/projected/a0255eea-e6a3-4288-92c9-6a5c4900d961-kube-api-access-x7k2l\") pod \"a0255eea-e6a3-4288-92c9-6a5c4900d961\" (UID: \"a0255eea-e6a3-4288-92c9-6a5c4900d961\") " Oct 30 02:16:20.226094 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:20.226064 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0255eea-e6a3-4288-92c9-6a5c4900d961-maas-api-openshift-ca" pod "a0255eea-e6a3-4288-92c9-6a5c4900d961" (UID: "a0255eea-e6a3-4288-92c9-6a5c4900d961"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:16:20.226697 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:20.226668 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0255eea-e6a3-4288-92c9-6a5c4900d961-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:20.228207 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:20.228174 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0255eea-e6a3-4288-92c9-6a5c4900d961-kube-api-access-x7k2l" pod "a0255eea-e6a3-4288-92c9-6a5c4900d961" (UID: "a0255eea-e6a3-4288-92c9-6a5c4900d961"). InnerVolumeSpecName "kube-api-access-x7k2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:16:20.228482 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:20.228446 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0255eea-e6a3-4288-92c9-6a5c4900d961-maas-api-xks-ca" pod "a0255eea-e6a3-4288-92c9-6a5c4900d961" (UID: "a0255eea-e6a3-4288-92c9-6a5c4900d961"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:16:20.306940 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:20.306904 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:20.328138 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:20.328096 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0255eea-e6a3-4288-92c9-6a5c4900d961-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:20.328138 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:20.328131 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7k2l\" (UniqueName: \"kubernetes.io/projected/a0255eea-e6a3-4288-92c9-6a5c4900d961-kube-api-access-x7k2l\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:20.545303 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:20.545219 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fb220e0-3a46-4b35-9707-aca38ac5f288" path="/var/lib/kubelet/pods/3fb220e0-3a46-4b35-9707-aca38ac5f288/volumes" Oct 30 02:16:20.988147 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:20.988114 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aaf045-ttm2c" Oct 30 02:16:20.988363 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:20.988146 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aaf045-ttm2c" event={"ID":"a0255eea-e6a3-4288-92c9-6a5c4900d961","Type":"ContainerDied","Data":"2a102a6b77cbaeee426eb1e9cb52b2fcb3240cd5b5a40d40f0061c17b5221091"} Oct 30 02:16:20.988363 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:20.988197 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a102a6b77cbaeee426eb1e9cb52b2fcb3240cd5b5a40d40f0061c17b5221091" Oct 30 02:16:21.487235 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:21.487200 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:22.839793 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:22.839738 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:24.102540 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:24.102503 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:25.402838 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:25.401125 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:26.397812 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:26.397741 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:26.591616 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:26.591468 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l"] Oct 30 02:16:26.592881 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:26.591894 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" podUID="c2afdd4b-323c-41fc-9919-ed0d7cbce70f" containerName="istio-proxy" containerID="cri-o://ac429be00f0ab50134a85f48b699af3457cd0985722309a01200c59b8adf90fd" gracePeriod=30 Oct 30 02:16:26.594515 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:26.594483 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:26.797451 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:26.797416 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:27.181258 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:27.181225 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:27.381803 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:27.381766 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:27.435307 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:27.435217 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-803472-rwvsc"] Oct 30 02:16:27.435984 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:27.435967 6572 cpu_manager.go:414] "containerMap: removing container" podUID="a0255eea-e6a3-4288-92c9-6a5c4900d961" containerName="revoke-keys" Oct 30 02:16:27.436063 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:27.435985 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0255eea-e6a3-4288-92c9-6a5c4900d961" containerName="revoke-keys" Oct 30 02:16:27.436106 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:27.436089 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0255eea-e6a3-4288-92c9-6a5c4900d961" containerName="revoke-keys" Oct 30 02:16:27.440015 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:27.439994 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-803472-rwvsc" Oct 30 02:16:27.460164 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:27.460132 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-803472-rwvsc"] Oct 30 02:16:27.466907 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:27.466874 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:16:27.466907 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:27.466900 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:16:27.467144 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:27.466880 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:16:27.467144 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:27.466985 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:16:27.563468 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:27.563427 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53436d95-4e4a-4d93-b47b-629c4861dac5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-803472-rwvsc\" (UID: \"53436d95-4e4a-4d93-b47b-629c4861dac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-803472-rwvsc" Oct 30 02:16:27.563696 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:27.563670 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbdlt\" (UniqueName: \"kubernetes.io/projected/53436d95-4e4a-4d93-b47b-629c4861dac5-kube-api-access-sbdlt\") pod \"maas-api-revoke-keys-e2e-worker-w4-803472-rwvsc\" (UID: \"53436d95-4e4a-4d93-b47b-629c4861dac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-803472-rwvsc" Oct 30 02:16:27.563837 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:27.563817 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53436d95-4e4a-4d93-b47b-629c4861dac5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-803472-rwvsc\" (UID: \"53436d95-4e4a-4d93-b47b-629c4861dac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-803472-rwvsc" Oct 30 02:16:27.664998 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:27.664959 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbdlt\" (UniqueName: \"kubernetes.io/projected/53436d95-4e4a-4d93-b47b-629c4861dac5-kube-api-access-sbdlt\") pod \"maas-api-revoke-keys-e2e-worker-w4-803472-rwvsc\" (UID: \"53436d95-4e4a-4d93-b47b-629c4861dac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-803472-rwvsc" Oct 30 02:16:27.665495 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:27.665058 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53436d95-4e4a-4d93-b47b-629c4861dac5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-803472-rwvsc\" (UID: \"53436d95-4e4a-4d93-b47b-629c4861dac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-803472-rwvsc" Oct 30 02:16:27.665495 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:27.665186 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53436d95-4e4a-4d93-b47b-629c4861dac5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-803472-rwvsc\" (UID: \"53436d95-4e4a-4d93-b47b-629c4861dac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-803472-rwvsc" Oct 30 02:16:27.666077 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:27.666049 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53436d95-4e4a-4d93-b47b-629c4861dac5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-803472-rwvsc\" (UID: \"53436d95-4e4a-4d93-b47b-629c4861dac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-803472-rwvsc" Oct 30 02:16:27.667709 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:27.667686 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53436d95-4e4a-4d93-b47b-629c4861dac5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-803472-rwvsc\" (UID: \"53436d95-4e4a-4d93-b47b-629c4861dac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-803472-rwvsc" Oct 30 02:16:27.684021 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:27.683990 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbdlt\" (UniqueName: \"kubernetes.io/projected/53436d95-4e4a-4d93-b47b-629c4861dac5-kube-api-access-sbdlt\") pod \"maas-api-revoke-keys-e2e-worker-w4-803472-rwvsc\" (UID: \"53436d95-4e4a-4d93-b47b-629c4861dac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-803472-rwvsc" Oct 30 02:16:27.756512 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:27.756425 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-803472-rwvsc" Oct 30 02:16:28.058977 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:28.058939 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-803472-rwvsc"] Oct 30 02:16:28.059817 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:16:28.059721 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53436d95_4e4a_4d93_b47b_629c4861dac5.slice/crio-054d11347e23dcae8c97fb56afd24043481b896d5d2685700b81b8c274dd6d8c WatchSource:0}: Error finding container 054d11347e23dcae8c97fb56afd24043481b896d5d2685700b81b8c274dd6d8c: Status 404 returned error can't find the container with id 054d11347e23dcae8c97fb56afd24043481b896d5d2685700b81b8c274dd6d8c Oct 30 02:16:28.206169 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:28.206118 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:28.401313 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:28.401275 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:28.608831 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:28.607736 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:29.064955 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:29.064922 6572 generic.go:356] "Generic (PLEG): container finished" podID="53436d95-4e4a-4d93-b47b-629c4861dac5" containerID="eccde603d684e44a03331f5fc3ca7e67e6ff59ab48c7550dbaf31e9535197719" exitCode=0 Oct 30 02:16:29.065356 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:29.065004 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-803472-rwvsc" event={"ID":"53436d95-4e4a-4d93-b47b-629c4861dac5","Type":"ContainerDied","Data":"eccde603d684e44a03331f5fc3ca7e67e6ff59ab48c7550dbaf31e9535197719"} Oct 30 02:16:29.065356 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:29.065058 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-803472-rwvsc" event={"ID":"53436d95-4e4a-4d93-b47b-629c4861dac5","Type":"ContainerStarted","Data":"054d11347e23dcae8c97fb56afd24043481b896d5d2685700b81b8c274dd6d8c"} Oct 30 02:16:29.851706 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:29.851664 6572 patch_prober.go:29] interesting pod/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:16:29.852081 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:29.852046 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" podUID="c2afdd4b-323c-41fc-9919-ed0d7cbce70f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:16:29.892782 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:29.884433 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:30.258162 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:30.258129 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-803472-rwvsc" Oct 30 02:16:30.362709 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:30.362606 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53436d95-4e4a-4d93-b47b-629c4861dac5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53436d95-4e4a-4d93-b47b-629c4861dac5-maas-api-openshift-ca\") pod \"53436d95-4e4a-4d93-b47b-629c4861dac5\" (UID: \"53436d95-4e4a-4d93-b47b-629c4861dac5\") " Oct 30 02:16:30.362933 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:30.362837 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53436d95-4e4a-4d93-b47b-629c4861dac5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53436d95-4e4a-4d93-b47b-629c4861dac5-maas-api-xks-ca\") pod \"53436d95-4e4a-4d93-b47b-629c4861dac5\" (UID: \"53436d95-4e4a-4d93-b47b-629c4861dac5\") " Oct 30 02:16:30.362985 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:30.362976 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53436d95-4e4a-4d93-b47b-629c4861dac5-kube-api-access-sbdlt\" (UniqueName: \"kubernetes.io/projected/53436d95-4e4a-4d93-b47b-629c4861dac5-kube-api-access-sbdlt\") pod \"53436d95-4e4a-4d93-b47b-629c4861dac5\" (UID: \"53436d95-4e4a-4d93-b47b-629c4861dac5\") " Oct 30 02:16:30.363090 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:30.363062 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53436d95-4e4a-4d93-b47b-629c4861dac5-maas-api-openshift-ca" pod "53436d95-4e4a-4d93-b47b-629c4861dac5" (UID: "53436d95-4e4a-4d93-b47b-629c4861dac5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:16:30.363463 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:30.363431 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53436d95-4e4a-4d93-b47b-629c4861dac5-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:30.365216 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:30.365184 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53436d95-4e4a-4d93-b47b-629c4861dac5-maas-api-xks-ca" pod "53436d95-4e4a-4d93-b47b-629c4861dac5" (UID: "53436d95-4e4a-4d93-b47b-629c4861dac5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:16:30.365360 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:30.365285 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53436d95-4e4a-4d93-b47b-629c4861dac5-kube-api-access-sbdlt" pod "53436d95-4e4a-4d93-b47b-629c4861dac5" (UID: "53436d95-4e4a-4d93-b47b-629c4861dac5"). InnerVolumeSpecName "kube-api-access-sbdlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:16:30.464654 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:30.464616 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53436d95-4e4a-4d93-b47b-629c4861dac5-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:30.464654 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:30.464651 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbdlt\" (UniqueName: \"kubernetes.io/projected/53436d95-4e4a-4d93-b47b-629c4861dac5-kube-api-access-sbdlt\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:31.090504 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:31.090472 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-803472-rwvsc" Oct 30 02:16:31.090693 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:31.090471 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-803472-rwvsc" event={"ID":"53436d95-4e4a-4d93-b47b-629c4861dac5","Type":"ContainerDied","Data":"054d11347e23dcae8c97fb56afd24043481b896d5d2685700b81b8c274dd6d8c"} Oct 30 02:16:31.090693 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:31.090616 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="054d11347e23dcae8c97fb56afd24043481b896d5d2685700b81b8c274dd6d8c" Oct 30 02:16:31.283975 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:31.283940 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:31.875246 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:31.875184 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" Oct 30 02:16:31.884958 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:31.884931 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-istio-data\") pod \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\" (UID: \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\") " Oct 30 02:16:31.885131 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:31.885067 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-istio-token\" (UniqueName: \"kubernetes.io/projected/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-istio-token\") pod \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\" (UID: \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\") " Oct 30 02:16:31.885205 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:31.885157 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-workload-socket\") pod \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\" (UID: \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\") " Oct 30 02:16:31.885205 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:31.885191 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-istio-data" pod "c2afdd4b-323c-41fc-9919-ed0d7cbce70f" (UID: "c2afdd4b-323c-41fc-9919-ed0d7cbce70f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:31.885314 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:31.885198 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-workload-certs\") pod \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\" (UID: \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\") " Oct 30 02:16:31.885314 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:31.885242 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-istiod-ca-cert\") pod \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\" (UID: \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\") " Oct 30 02:16:31.885314 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:31.885278 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-istio-envoy\") pod \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\" (UID: \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\") " Oct 30 02:16:31.885314 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:31.885296 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-kube-api-access-8chm8\" (UniqueName: \"kubernetes.io/projected/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-kube-api-access-8chm8\") pod \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\" (UID: \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\") " Oct 30 02:16:31.885537 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:31.885344 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-credential-socket\") pod \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\" (UID: \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\") " Oct 30 02:16:31.885537 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:31.885371 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-istio-podinfo\") pod \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\" (UID: \"c2afdd4b-323c-41fc-9919-ed0d7cbce70f\") " Oct 30 02:16:31.885537 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:31.885475 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-workload-socket" pod "c2afdd4b-323c-41fc-9919-ed0d7cbce70f" (UID: "c2afdd4b-323c-41fc-9919-ed0d7cbce70f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:31.885537 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:31.885490 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-workload-certs" pod "c2afdd4b-323c-41fc-9919-ed0d7cbce70f" (UID: "c2afdd4b-323c-41fc-9919-ed0d7cbce70f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:31.885792 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:31.885714 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-istiod-ca-cert" pod "c2afdd4b-323c-41fc-9919-ed0d7cbce70f" (UID: "c2afdd4b-323c-41fc-9919-ed0d7cbce70f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:16:31.885792 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:31.885727 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-credential-socket" pod "c2afdd4b-323c-41fc-9919-ed0d7cbce70f" (UID: "c2afdd4b-323c-41fc-9919-ed0d7cbce70f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:31.886216 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:31.886193 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:31.886335 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:31.886219 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:31.886335 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:31.886236 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:31.886335 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:31.886250 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:31.886335 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:31.886265 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:31.887592 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:31.887567 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-istio-token" pod "c2afdd4b-323c-41fc-9919-ed0d7cbce70f" (UID: "c2afdd4b-323c-41fc-9919-ed0d7cbce70f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:16:31.887807 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:31.887776 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-kube-api-access-8chm8" pod "c2afdd4b-323c-41fc-9919-ed0d7cbce70f" (UID: "c2afdd4b-323c-41fc-9919-ed0d7cbce70f"). InnerVolumeSpecName "kube-api-access-8chm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:16:31.887961 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:31.887944 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-istio-podinfo" pod "c2afdd4b-323c-41fc-9919-ed0d7cbce70f" (UID: "c2afdd4b-323c-41fc-9919-ed0d7cbce70f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:16:31.888030 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:31.888015 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-istio-envoy" pod "c2afdd4b-323c-41fc-9919-ed0d7cbce70f" (UID: "c2afdd4b-323c-41fc-9919-ed0d7cbce70f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:31.988279 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:31.988246 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:31.988279 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:31.988274 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:31.988279 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:31.988286 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8chm8\" (UniqueName: \"kubernetes.io/projected/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-kube-api-access-8chm8\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:31.988563 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:31.988296 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2afdd4b-323c-41fc-9919-ed0d7cbce70f-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:32.074728 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:32.074692 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86"] Oct 30 02:16:32.081095 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:32.081068 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn"] Oct 30 02:16:32.104241 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:32.104210 6572 generic.go:356] "Generic (PLEG): container finished" podID="c2afdd4b-323c-41fc-9919-ed0d7cbce70f" containerID="ac429be00f0ab50134a85f48b699af3457cd0985722309a01200c59b8adf90fd" exitCode=0 Oct 30 02:16:32.104445 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:32.104258 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" Oct 30 02:16:32.104445 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:32.104289 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" event={"ID":"c2afdd4b-323c-41fc-9919-ed0d7cbce70f","Type":"ContainerDied","Data":"ac429be00f0ab50134a85f48b699af3457cd0985722309a01200c59b8adf90fd"} Oct 30 02:16:32.104445 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:32.104348 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l" event={"ID":"c2afdd4b-323c-41fc-9919-ed0d7cbce70f","Type":"ContainerDied","Data":"bfe82c925666a9c977d4f6afed4b8efed79f79a2bcffb84cc98055c9f084f0f2"} Oct 30 02:16:32.104445 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:32.104382 6572 scope.go:122] "RemoveContainer" containerID="ac429be00f0ab50134a85f48b699af3457cd0985722309a01200c59b8adf90fd" Oct 30 02:16:32.117899 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:32.117876 6572 scope.go:122] "RemoveContainer" containerID="ac429be00f0ab50134a85f48b699af3457cd0985722309a01200c59b8adf90fd" Oct 30 02:16:32.118213 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:16:32.118193 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac429be00f0ab50134a85f48b699af3457cd0985722309a01200c59b8adf90fd\": container with ID starting with ac429be00f0ab50134a85f48b699af3457cd0985722309a01200c59b8adf90fd not found: ID does not exist" containerID="ac429be00f0ab50134a85f48b699af3457cd0985722309a01200c59b8adf90fd" Oct 30 02:16:32.118274 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:32.118223 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac429be00f0ab50134a85f48b699af3457cd0985722309a01200c59b8adf90fd"} err="failed to get container status \"ac429be00f0ab50134a85f48b699af3457cd0985722309a01200c59b8adf90fd\": rpc error: code = NotFound desc = could not find container \"ac429be00f0ab50134a85f48b699af3457cd0985722309a01200c59b8adf90fd\": container with ID starting with ac429be00f0ab50134a85f48b699af3457cd0985722309a01200c59b8adf90fd not found: ID does not exist" Oct 30 02:16:32.202162 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:32.202118 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l"] Oct 30 02:16:32.206300 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:32.206273 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-76k5l"] Oct 30 02:16:32.241583 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:32.241543 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:32.286484 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:32.286439 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb" podUID="3636031c-7bdd-4706-ab91-dfcf7bdb21be" containerName="main" containerID="cri-o://bcb2293f39beb059f491b70c11f115a6bf691a4390bbe5c5f64fef9d62fff6cd" gracePeriod=45 Oct 30 02:16:32.311855 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:16:32.311825 6572 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3636031c_7bdd_4706_ab91_dfcf7bdb21be.slice/crio-conmon-bcb2293f39beb059f491b70c11f115a6bf691a4390bbe5c5f64fef9d62fff6cd.scope\": RecentStats: unable to find data in memory cache]" Oct 30 02:16:32.312036 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:16:32.311917 6572 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3636031c_7bdd_4706_ab91_dfcf7bdb21be.slice/crio-bcb2293f39beb059f491b70c11f115a6bf691a4390bbe5c5f64fef9d62fff6cd.scope\": RecentStats: unable to find data in memory cache]" Oct 30 02:16:32.545550 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:32.545505 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2afdd4b-323c-41fc-9919-ed0d7cbce70f" path="/var/lib/kubelet/pods/c2afdd4b-323c-41fc-9919-ed0d7cbce70f/volumes" Oct 30 02:16:32.553730 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:32.553699 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb" Oct 30 02:16:32.602828 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:32.602797 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3636031c-7bdd-4706-ab91-dfcf7bdb21be-kube-api-access-w24v5\" (UniqueName: \"kubernetes.io/projected/3636031c-7bdd-4706-ab91-dfcf7bdb21be-kube-api-access-w24v5\") pod \"3636031c-7bdd-4706-ab91-dfcf7bdb21be\" (UID: \"3636031c-7bdd-4706-ab91-dfcf7bdb21be\") " Oct 30 02:16:32.603040 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:32.602867 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3636031c-7bdd-4706-ab91-dfcf7bdb21be-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3636031c-7bdd-4706-ab91-dfcf7bdb21be-model-cache\") pod \"3636031c-7bdd-4706-ab91-dfcf7bdb21be\" (UID: \"3636031c-7bdd-4706-ab91-dfcf7bdb21be\") " Oct 30 02:16:32.603040 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:32.602916 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3636031c-7bdd-4706-ab91-dfcf7bdb21be-tls-certs\" (UniqueName: \"kubernetes.io/secret/3636031c-7bdd-4706-ab91-dfcf7bdb21be-tls-certs\") pod \"3636031c-7bdd-4706-ab91-dfcf7bdb21be\" (UID: \"3636031c-7bdd-4706-ab91-dfcf7bdb21be\") " Oct 30 02:16:32.603040 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:32.602946 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3636031c-7bdd-4706-ab91-dfcf7bdb21be-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3636031c-7bdd-4706-ab91-dfcf7bdb21be-tmp-dir\") pod \"3636031c-7bdd-4706-ab91-dfcf7bdb21be\" (UID: \"3636031c-7bdd-4706-ab91-dfcf7bdb21be\") " Oct 30 02:16:32.603215 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:32.603122 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3636031c-7bdd-4706-ab91-dfcf7bdb21be-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3636031c-7bdd-4706-ab91-dfcf7bdb21be-dshm\") pod \"3636031c-7bdd-4706-ab91-dfcf7bdb21be\" (UID: \"3636031c-7bdd-4706-ab91-dfcf7bdb21be\") " Oct 30 02:16:32.603215 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:32.603129 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3636031c-7bdd-4706-ab91-dfcf7bdb21be-model-cache" pod "3636031c-7bdd-4706-ab91-dfcf7bdb21be" (UID: "3636031c-7bdd-4706-ab91-dfcf7bdb21be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:32.603339 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:32.603240 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3636031c-7bdd-4706-ab91-dfcf7bdb21be-home\" (UniqueName: \"kubernetes.io/empty-dir/3636031c-7bdd-4706-ab91-dfcf7bdb21be-home\") pod \"3636031c-7bdd-4706-ab91-dfcf7bdb21be\" (UID: \"3636031c-7bdd-4706-ab91-dfcf7bdb21be\") " Oct 30 02:16:32.603404 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:32.603351 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3636031c-7bdd-4706-ab91-dfcf7bdb21be-tmp-dir" pod "3636031c-7bdd-4706-ab91-dfcf7bdb21be" (UID: "3636031c-7bdd-4706-ab91-dfcf7bdb21be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:32.603608 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:32.603587 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3636031c-7bdd-4706-ab91-dfcf7bdb21be-home" pod "3636031c-7bdd-4706-ab91-dfcf7bdb21be" (UID: "3636031c-7bdd-4706-ab91-dfcf7bdb21be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:32.604120 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:32.604093 6572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3636031c-7bdd-4706-ab91-dfcf7bdb21be-home\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:32.604229 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:32.604125 6572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3636031c-7bdd-4706-ab91-dfcf7bdb21be-model-cache\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:32.604229 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:32.604140 6572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3636031c-7bdd-4706-ab91-dfcf7bdb21be-tmp-dir\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:32.605613 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:32.605586 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3636031c-7bdd-4706-ab91-dfcf7bdb21be-kube-api-access-w24v5" pod "3636031c-7bdd-4706-ab91-dfcf7bdb21be" (UID: "3636031c-7bdd-4706-ab91-dfcf7bdb21be"). InnerVolumeSpecName "kube-api-access-w24v5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:16:32.605720 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:32.605669 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3636031c-7bdd-4706-ab91-dfcf7bdb21be-tls-certs" pod "3636031c-7bdd-4706-ab91-dfcf7bdb21be" (UID: "3636031c-7bdd-4706-ab91-dfcf7bdb21be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:16:32.605848 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:32.605795 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3636031c-7bdd-4706-ab91-dfcf7bdb21be-dshm" pod "3636031c-7bdd-4706-ab91-dfcf7bdb21be" (UID: "3636031c-7bdd-4706-ab91-dfcf7bdb21be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:32.705249 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:32.705210 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w24v5\" (UniqueName: \"kubernetes.io/projected/3636031c-7bdd-4706-ab91-dfcf7bdb21be-kube-api-access-w24v5\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:32.705249 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:32.705244 6572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3636031c-7bdd-4706-ab91-dfcf7bdb21be-tls-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:32.705249 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:32.705254 6572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3636031c-7bdd-4706-ab91-dfcf7bdb21be-dshm\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:32.766857 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:32.766727 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:33.115036 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:33.114936 6572 generic.go:356] "Generic (PLEG): container finished" podID="3636031c-7bdd-4706-ab91-dfcf7bdb21be" containerID="bcb2293f39beb059f491b70c11f115a6bf691a4390bbe5c5f64fef9d62fff6cd" exitCode=0 Oct 30 02:16:33.115036 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:33.114993 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb" Oct 30 02:16:33.115036 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:33.115024 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb" event={"ID":"3636031c-7bdd-4706-ab91-dfcf7bdb21be","Type":"ContainerDied","Data":"bcb2293f39beb059f491b70c11f115a6bf691a4390bbe5c5f64fef9d62fff6cd"} Oct 30 02:16:33.115337 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:33.115064 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb" event={"ID":"3636031c-7bdd-4706-ab91-dfcf7bdb21be","Type":"ContainerDied","Data":"f5214bc28b2bea305d1159954934a889cbc0deb16246c4a34fb3e944ceefdd9f"} Oct 30 02:16:33.115337 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:33.115082 6572 scope.go:122] "RemoveContainer" containerID="bcb2293f39beb059f491b70c11f115a6bf691a4390bbe5c5f64fef9d62fff6cd" Oct 30 02:16:33.127549 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:33.127529 6572 scope.go:122] "RemoveContainer" containerID="bcb2293f39beb059f491b70c11f115a6bf691a4390bbe5c5f64fef9d62fff6cd" Oct 30 02:16:33.127869 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:16:33.127848 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcb2293f39beb059f491b70c11f115a6bf691a4390bbe5c5f64fef9d62fff6cd\": container with ID starting with bcb2293f39beb059f491b70c11f115a6bf691a4390bbe5c5f64fef9d62fff6cd not found: ID does not exist" containerID="bcb2293f39beb059f491b70c11f115a6bf691a4390bbe5c5f64fef9d62fff6cd" Oct 30 02:16:33.127970 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:33.127876 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcb2293f39beb059f491b70c11f115a6bf691a4390bbe5c5f64fef9d62fff6cd"} err="failed to get container status \"bcb2293f39beb059f491b70c11f115a6bf691a4390bbe5c5f64fef9d62fff6cd\": rpc error: code = NotFound desc = could not find container \"bcb2293f39beb059f491b70c11f115a6bf691a4390bbe5c5f64fef9d62fff6cd\": container with ID starting with bcb2293f39beb059f491b70c11f115a6bf691a4390bbe5c5f64fef9d62fff6cd not found: ID does not exist" Oct 30 02:16:33.165417 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:33.165387 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb"] Oct 30 02:16:33.177196 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:33.177162 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-22e99d/test-model-22e99d-kserve-5cc5465c4f-ctcfb"] Oct 30 02:16:34.177596 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:34.177497 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:34.543555 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:34.543468 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3636031c-7bdd-4706-ab91-dfcf7bdb21be" path="/var/lib/kubelet/pods/3636031c-7bdd-4706-ab91-dfcf7bdb21be/volumes" Oct 30 02:16:35.297142 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:35.297107 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-679e07df-98pc7"] Oct 30 02:16:35.297863 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:35.297843 6572 cpu_manager.go:414] "containerMap: removing container" podUID="3636031c-7bdd-4706-ab91-dfcf7bdb21be" containerName="main" Oct 30 02:16:35.297863 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:35.297860 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3636031c-7bdd-4706-ab91-dfcf7bdb21be" containerName="main" Oct 30 02:16:35.298053 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:35.297870 6572 cpu_manager.go:414] "containerMap: removing container" podUID="c2afdd4b-323c-41fc-9919-ed0d7cbce70f" containerName="istio-proxy" Oct 30 02:16:35.298053 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:35.297877 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2afdd4b-323c-41fc-9919-ed0d7cbce70f" containerName="istio-proxy" Oct 30 02:16:35.298053 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:35.297895 6572 cpu_manager.go:414] "containerMap: removing container" podUID="53436d95-4e4a-4d93-b47b-629c4861dac5" containerName="revoke-keys" Oct 30 02:16:35.298053 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:35.297901 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53436d95-4e4a-4d93-b47b-629c4861dac5" containerName="revoke-keys" Oct 30 02:16:35.298053 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:35.298030 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="3636031c-7bdd-4706-ab91-dfcf7bdb21be" containerName="main" Oct 30 02:16:35.298053 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:35.298036 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2afdd4b-323c-41fc-9919-ed0d7cbce70f" containerName="istio-proxy" Oct 30 02:16:35.298053 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:35.298046 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="53436d95-4e4a-4d93-b47b-629c4861dac5" containerName="revoke-keys" Oct 30 02:16:35.303239 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:35.303219 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-679e07df-98pc7" Oct 30 02:16:35.319529 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:35.319502 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-679e07df-98pc7"] Oct 30 02:16:35.324814 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:35.324743 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:16:35.324971 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:35.324826 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:16:35.324971 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:35.324833 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:16:35.328531 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:35.328508 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:16:35.343219 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:35.343183 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-679e07df-98pc7\" (UID: \"e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-679e07df-98pc7" Oct 30 02:16:35.343438 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:35.343418 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c86jr\" (UniqueName: \"kubernetes.io/projected/e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd-kube-api-access-c86jr\") pod \"maas-api-revoke-sub-679e07df-98pc7\" (UID: \"e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-679e07df-98pc7" Oct 30 02:16:35.343524 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:35.343450 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-679e07df-98pc7\" (UID: \"e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-679e07df-98pc7" Oct 30 02:16:35.453095 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:35.453034 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-679e07df-98pc7\" (UID: \"e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-679e07df-98pc7" Oct 30 02:16:35.453803 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:35.453743 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c86jr\" (UniqueName: \"kubernetes.io/projected/e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd-kube-api-access-c86jr\") pod \"maas-api-revoke-sub-679e07df-98pc7\" (UID: \"e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-679e07df-98pc7" Oct 30 02:16:35.454067 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:35.454047 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-679e07df-98pc7\" (UID: \"e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-679e07df-98pc7" Oct 30 02:16:35.455614 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:35.455593 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-679e07df-98pc7\" (UID: \"e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-679e07df-98pc7" Oct 30 02:16:35.474597 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:35.474281 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-679e07df-98pc7\" (UID: \"e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-679e07df-98pc7" Oct 30 02:16:35.484193 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:35.484155 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c86jr\" (UniqueName: \"kubernetes.io/projected/e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd-kube-api-access-c86jr\") pod \"maas-api-revoke-sub-679e07df-98pc7\" (UID: \"e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-679e07df-98pc7" Oct 30 02:16:35.487915 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:35.485368 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:35.621919 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:35.621818 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-679e07df-98pc7" Oct 30 02:16:35.925935 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:35.925900 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-679e07df-98pc7"] Oct 30 02:16:35.927103 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:16:35.927073 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode19c3c2f_04ff_47c0_a75e_5db37c9c0cdd.slice/crio-dc74fa05cee918834ebbb55360cdeab7ca9e2703e3cbc209048ef94bfa5e2838 WatchSource:0}: Error finding container dc74fa05cee918834ebbb55360cdeab7ca9e2703e3cbc209048ef94bfa5e2838: Status 404 returned error can't find the container with id dc74fa05cee918834ebbb55360cdeab7ca9e2703e3cbc209048ef94bfa5e2838 Oct 30 02:16:36.146054 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:36.146007 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-679e07df-98pc7" event={"ID":"e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd","Type":"ContainerStarted","Data":"dc74fa05cee918834ebbb55360cdeab7ca9e2703e3cbc209048ef94bfa5e2838"} Oct 30 02:16:36.787317 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:36.787271 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:37.158390 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:37.158289 6572 generic.go:356] "Generic (PLEG): container finished" podID="e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd" containerID="94e93b07e2ed8f31a0b8e46a268a51fbeef3ad56c350248cd7f3f83d850dfc82" exitCode=0 Oct 30 02:16:37.158390 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:37.158361 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-679e07df-98pc7" event={"ID":"e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd","Type":"ContainerDied","Data":"94e93b07e2ed8f31a0b8e46a268a51fbeef3ad56c350248cd7f3f83d850dfc82"} Oct 30 02:16:37.944699 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:37.944658 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-803472-d7dc9c774-4cr8j"] Oct 30 02:16:37.978119 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:37.978081 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-803472-5ffb57c959-qll7m"] Oct 30 02:16:38.169162 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:38.169131 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:38.357855 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:38.357794 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:38.394641 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:38.394604 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-679e07df-98pc7" Oct 30 02:16:38.452049 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:38.452012 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd-maas-api-openshift-ca\") pod \"e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd\" (UID: \"e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd\") " Oct 30 02:16:38.452230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:38.452169 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd-kube-api-access-c86jr\" (UniqueName: \"kubernetes.io/projected/e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd-kube-api-access-c86jr\") pod \"e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd\" (UID: \"e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd\") " Oct 30 02:16:38.452328 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:38.452225 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd-maas-api-xks-ca\") pod \"e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd\" (UID: \"e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd\") " Oct 30 02:16:38.452510 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:38.452480 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd-maas-api-openshift-ca" pod "e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd" (UID: "e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:16:38.452944 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:38.452917 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:38.455216 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:38.455181 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd-kube-api-access-c86jr" pod "e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd" (UID: "e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd"). InnerVolumeSpecName "kube-api-access-c86jr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:16:38.456898 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:38.456869 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd-maas-api-xks-ca" pod "e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd" (UID: "e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:16:38.606876 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:38.604990 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c86jr\" (UniqueName: \"kubernetes.io/projected/e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd-kube-api-access-c86jr\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:38.606876 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:38.605028 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:39.176572 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:39.176538 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-679e07df-98pc7" Oct 30 02:16:39.177033 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:39.176568 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-679e07df-98pc7" event={"ID":"e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd","Type":"ContainerDied","Data":"dc74fa05cee918834ebbb55360cdeab7ca9e2703e3cbc209048ef94bfa5e2838"} Oct 30 02:16:39.177033 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:39.176623 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc74fa05cee918834ebbb55360cdeab7ca9e2703e3cbc209048ef94bfa5e2838" Oct 30 02:16:39.697718 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:39.697682 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:41.165192 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:41.165122 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:41.368370 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:41.368335 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:41.564426 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:41.564318 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:42.577965 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:42.577911 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:42.945056 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:42.944992 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-803472-d7dc9c774-4cr8j" podUID="c3884225-2803-4ea6-8f09-96e0696ac6d3" containerName="payload-processing" containerID="cri-o://163c7455042f57e14b1363a3d980771807048ab9a07ea33452bc91145117d08a" gracePeriod=25 Oct 30 02:16:42.978538 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:42.978499 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-803472-5ffb57c959-qll7m" podUID="d888789f-78c5-4606-94e1-7fcc77a6c16e" containerName="payload-pre-processing" containerID="cri-o://0ef5b3485d8e8aadbeec2fb15675471d134d8610db80cf70182b00a3d54e661d" gracePeriod=25 Oct 30 02:16:43.981235 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:43.981191 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:44.688867 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:16:44.688821 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:16:44.981845 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:44.981737 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad9c077d-vmd8b"] Oct 30 02:16:44.982491 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:44.982473 6572 cpu_manager.go:414] "containerMap: removing container" podUID="e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd" containerName="revoke-keys" Oct 30 02:16:44.982491 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:44.982490 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd" containerName="revoke-keys" Oct 30 02:16:44.982662 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:44.982620 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd" containerName="revoke-keys" Oct 30 02:16:44.986645 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:44.986618 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad9c077d-vmd8b" Oct 30 02:16:44.990058 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:44.990028 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-deb53ecd-gfhc8"] Oct 30 02:16:44.994733 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:44.994705 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb53ecd-gfhc8" Oct 30 02:16:45.000556 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:45.000445 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:16:45.000892 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:45.000873 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:16:45.006949 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:45.006915 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad9c077d-vmd8b"] Oct 30 02:16:45.010192 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:45.010162 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:16:45.010377 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:45.010201 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:16:45.022077 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:45.022038 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-deb53ecd-gfhc8"] Oct 30 02:16:45.113731 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:45.113696 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba3e43fb-87be-445f-875a-280d6787c4c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad9c077d-vmd8b\" (UID: \"ba3e43fb-87be-445f-875a-280d6787c4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad9c077d-vmd8b" Oct 30 02:16:45.113731 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:45.113737 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hrc4\" (UniqueName: \"kubernetes.io/projected/ba3e43fb-87be-445f-875a-280d6787c4c7-kube-api-access-7hrc4\") pod \"maas-api-revoke-sub-ad9c077d-vmd8b\" (UID: \"ba3e43fb-87be-445f-875a-280d6787c4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad9c077d-vmd8b" Oct 30 02:16:45.114004 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:45.113987 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlr5l\" (UniqueName: \"kubernetes.io/projected/dfa07f94-1c0a-4855-8849-5d3e753ab795-kube-api-access-dlr5l\") pod \"maas-api-revoke-sub-deb53ecd-gfhc8\" (UID: \"dfa07f94-1c0a-4855-8849-5d3e753ab795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb53ecd-gfhc8" Oct 30 02:16:45.114047 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:45.114017 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfa07f94-1c0a-4855-8849-5d3e753ab795-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-deb53ecd-gfhc8\" (UID: \"dfa07f94-1c0a-4855-8849-5d3e753ab795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb53ecd-gfhc8" Oct 30 02:16:45.114119 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:45.114107 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba3e43fb-87be-445f-875a-280d6787c4c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad9c077d-vmd8b\" (UID: \"ba3e43fb-87be-445f-875a-280d6787c4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad9c077d-vmd8b" Oct 30 02:16:45.114172 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:45.114135 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfa07f94-1c0a-4855-8849-5d3e753ab795-maas-api-xks-ca\") pod \"maas-api-revoke-sub-deb53ecd-gfhc8\" (UID: \"dfa07f94-1c0a-4855-8849-5d3e753ab795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb53ecd-gfhc8" Oct 30 02:16:45.151139 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:45.151107 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:45.215996 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:45.215960 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlr5l\" (UniqueName: \"kubernetes.io/projected/dfa07f94-1c0a-4855-8849-5d3e753ab795-kube-api-access-dlr5l\") pod \"maas-api-revoke-sub-deb53ecd-gfhc8\" (UID: \"dfa07f94-1c0a-4855-8849-5d3e753ab795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb53ecd-gfhc8" Oct 30 02:16:45.215996 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:45.216008 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfa07f94-1c0a-4855-8849-5d3e753ab795-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-deb53ecd-gfhc8\" (UID: \"dfa07f94-1c0a-4855-8849-5d3e753ab795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb53ecd-gfhc8" Oct 30 02:16:45.216259 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:45.216066 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba3e43fb-87be-445f-875a-280d6787c4c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad9c077d-vmd8b\" (UID: \"ba3e43fb-87be-445f-875a-280d6787c4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad9c077d-vmd8b" Oct 30 02:16:45.216259 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:45.216094 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfa07f94-1c0a-4855-8849-5d3e753ab795-maas-api-xks-ca\") pod \"maas-api-revoke-sub-deb53ecd-gfhc8\" (UID: \"dfa07f94-1c0a-4855-8849-5d3e753ab795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb53ecd-gfhc8" Oct 30 02:16:45.216259 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:45.216137 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba3e43fb-87be-445f-875a-280d6787c4c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad9c077d-vmd8b\" (UID: \"ba3e43fb-87be-445f-875a-280d6787c4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad9c077d-vmd8b" Oct 30 02:16:45.216259 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:45.216158 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hrc4\" (UniqueName: \"kubernetes.io/projected/ba3e43fb-87be-445f-875a-280d6787c4c7-kube-api-access-7hrc4\") pod \"maas-api-revoke-sub-ad9c077d-vmd8b\" (UID: \"ba3e43fb-87be-445f-875a-280d6787c4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad9c077d-vmd8b" Oct 30 02:16:45.216938 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:45.216895 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfa07f94-1c0a-4855-8849-5d3e753ab795-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-deb53ecd-gfhc8\" (UID: \"dfa07f94-1c0a-4855-8849-5d3e753ab795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb53ecd-gfhc8" Oct 30 02:16:45.217064 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:45.216962 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba3e43fb-87be-445f-875a-280d6787c4c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad9c077d-vmd8b\" (UID: \"ba3e43fb-87be-445f-875a-280d6787c4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad9c077d-vmd8b" Oct 30 02:16:45.218787 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:45.218725 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba3e43fb-87be-445f-875a-280d6787c4c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad9c077d-vmd8b\" (UID: \"ba3e43fb-87be-445f-875a-280d6787c4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad9c077d-vmd8b" Oct 30 02:16:45.219075 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:45.219058 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfa07f94-1c0a-4855-8849-5d3e753ab795-maas-api-xks-ca\") pod \"maas-api-revoke-sub-deb53ecd-gfhc8\" (UID: \"dfa07f94-1c0a-4855-8849-5d3e753ab795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb53ecd-gfhc8" Oct 30 02:16:45.253839 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:45.245321 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hrc4\" (UniqueName: \"kubernetes.io/projected/ba3e43fb-87be-445f-875a-280d6787c4c7-kube-api-access-7hrc4\") pod \"maas-api-revoke-sub-ad9c077d-vmd8b\" (UID: \"ba3e43fb-87be-445f-875a-280d6787c4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad9c077d-vmd8b" Oct 30 02:16:45.277771 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:45.258857 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlr5l\" (UniqueName: \"kubernetes.io/projected/dfa07f94-1c0a-4855-8849-5d3e753ab795-kube-api-access-dlr5l\") pod \"maas-api-revoke-sub-deb53ecd-gfhc8\" (UID: \"dfa07f94-1c0a-4855-8849-5d3e753ab795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb53ecd-gfhc8" Oct 30 02:16:45.318217 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:45.314768 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad9c077d-vmd8b" Oct 30 02:16:45.362526 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:45.362170 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb53ecd-gfhc8" Oct 30 02:16:45.732474 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:45.732432 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:45.746079 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:45.746035 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aaf045-ttm2c"] Oct 30 02:16:45.754975 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:45.754939 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aaf045-ttm2c"] Oct 30 02:16:45.815115 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:45.815083 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-deb53ecd-gfhc8"] Oct 30 02:16:45.817185 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:16:45.817156 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfa07f94_1c0a_4855_8849_5d3e753ab795.slice/crio-d5f35f61c651c8b23c0382a1502024e5c04740f95e7f18875d1e9d24def016d8 WatchSource:0}: Error finding container d5f35f61c651c8b23c0382a1502024e5c04740f95e7f18875d1e9d24def016d8: Status 404 returned error can't find the container with id d5f35f61c651c8b23c0382a1502024e5c04740f95e7f18875d1e9d24def016d8 Oct 30 02:16:45.867553 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:45.867527 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad9c077d-vmd8b"] Oct 30 02:16:45.869462 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:16:45.869434 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba3e43fb_87be_445f_875a_280d6787c4c7.slice/crio-26a49cb2fdec1a45a88d5ed1fc1efc7a135f5cd39b056e86c794fd86930157d6 WatchSource:0}: Error finding container 26a49cb2fdec1a45a88d5ed1fc1efc7a135f5cd39b056e86c794fd86930157d6: Status 404 returned error can't find the container with id 26a49cb2fdec1a45a88d5ed1fc1efc7a135f5cd39b056e86c794fd86930157d6 Oct 30 02:16:45.976795 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:45.976742 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:46.155905 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:46.155870 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:46.170462 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:46.170426 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b6b7fc-mfnf6"] Oct 30 02:16:46.175668 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:46.175626 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b6b7fc-mfnf6"] Oct 30 02:16:46.268853 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:46.268816 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad9c077d-vmd8b" event={"ID":"ba3e43fb-87be-445f-875a-280d6787c4c7","Type":"ContainerStarted","Data":"26a49cb2fdec1a45a88d5ed1fc1efc7a135f5cd39b056e86c794fd86930157d6"} Oct 30 02:16:46.270635 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:46.270609 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb53ecd-gfhc8" event={"ID":"dfa07f94-1c0a-4855-8849-5d3e753ab795","Type":"ContainerStarted","Data":"d5f35f61c651c8b23c0382a1502024e5c04740f95e7f18875d1e9d24def016d8"} Oct 30 02:16:46.348280 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:46.348236 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:46.543845 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:46.543812 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0255eea-e6a3-4288-92c9-6a5c4900d961" path="/var/lib/kubelet/pods/a0255eea-e6a3-4288-92c9-6a5c4900d961/volumes" Oct 30 02:16:46.544330 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:46.544316 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4fe447d-cdf6-43c6-b785-7a57ad00eba2" path="/var/lib/kubelet/pods/b4fe447d-cdf6-43c6-b785-7a57ad00eba2/volumes" Oct 30 02:16:46.552999 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:46.552972 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:46.760502 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:46.760419 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:46.943990 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:46.943957 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-561b8748-8lbtl"] Oct 30 02:16:46.948645 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:46.948622 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-561b8748-8lbtl" Oct 30 02:16:46.967729 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:46.967694 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-561b8748-8lbtl"] Oct 30 02:16:47.054840 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.054814 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdrj4\" (UniqueName: \"kubernetes.io/projected/5211f68e-c27b-4d50-a49d-b47ea4adf58a-kube-api-access-hdrj4\") pod \"maas-api-revoke-sub-561b8748-8lbtl\" (UID: \"5211f68e-c27b-4d50-a49d-b47ea4adf58a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-561b8748-8lbtl" Oct 30 02:16:47.054953 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.054870 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5211f68e-c27b-4d50-a49d-b47ea4adf58a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-561b8748-8lbtl\" (UID: \"5211f68e-c27b-4d50-a49d-b47ea4adf58a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-561b8748-8lbtl" Oct 30 02:16:47.055086 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.055069 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5211f68e-c27b-4d50-a49d-b47ea4adf58a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-561b8748-8lbtl\" (UID: \"5211f68e-c27b-4d50-a49d-b47ea4adf58a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-561b8748-8lbtl" Oct 30 02:16:47.098455 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.098423 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" podUID="0ee3f8e3-9b55-4ce5-afaf-86780a26391e" containerName="main" containerID="cri-o://4b95215344c7160a1f57af88d29d3c69196f43edc38c6deb6ff4684dc8f7de69" gracePeriod=45 Oct 30 02:16:47.104287 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.104254 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" podUID="fac3ca55-64af-42d9-90b1-0e821b893282" containerName="main" containerID="cri-o://a27836cd757f1537db62bc82910b80f5d5cbf7494695ef25a9f49c9f95e2b840" gracePeriod=45 Oct 30 02:16:47.147108 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.147077 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" podUID="0ee3f8e3-9b55-4ce5-afaf-86780a26391e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": dial tcp 10.133.0.65:8000: connect: connection refused" Oct 30 02:16:47.156046 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.156024 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5211f68e-c27b-4d50-a49d-b47ea4adf58a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-561b8748-8lbtl\" (UID: \"5211f68e-c27b-4d50-a49d-b47ea4adf58a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-561b8748-8lbtl" Oct 30 02:16:47.156637 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.156609 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdrj4\" (UniqueName: \"kubernetes.io/projected/5211f68e-c27b-4d50-a49d-b47ea4adf58a-kube-api-access-hdrj4\") pod \"maas-api-revoke-sub-561b8748-8lbtl\" (UID: \"5211f68e-c27b-4d50-a49d-b47ea4adf58a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-561b8748-8lbtl" Oct 30 02:16:47.156736 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.156678 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5211f68e-c27b-4d50-a49d-b47ea4adf58a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-561b8748-8lbtl\" (UID: \"5211f68e-c27b-4d50-a49d-b47ea4adf58a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-561b8748-8lbtl" Oct 30 02:16:47.157338 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.157313 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5211f68e-c27b-4d50-a49d-b47ea4adf58a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-561b8748-8lbtl\" (UID: \"5211f68e-c27b-4d50-a49d-b47ea4adf58a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-561b8748-8lbtl" Oct 30 02:16:47.158712 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.158686 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5211f68e-c27b-4d50-a49d-b47ea4adf58a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-561b8748-8lbtl\" (UID: \"5211f68e-c27b-4d50-a49d-b47ea4adf58a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-561b8748-8lbtl" Oct 30 02:16:47.175195 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.175173 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdrj4\" (UniqueName: \"kubernetes.io/projected/5211f68e-c27b-4d50-a49d-b47ea4adf58a-kube-api-access-hdrj4\") pod \"maas-api-revoke-sub-561b8748-8lbtl\" (UID: \"5211f68e-c27b-4d50-a49d-b47ea4adf58a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-561b8748-8lbtl" Oct 30 02:16:47.284117 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.284078 6572 generic.go:356] "Generic (PLEG): container finished" podID="ba3e43fb-87be-445f-875a-280d6787c4c7" containerID="e41838dbfa6605dd6be4f541609fa8feb160c7a8cac51522af133d6fc6dc3f66" exitCode=0 Oct 30 02:16:47.284353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.284301 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad9c077d-vmd8b" event={"ID":"ba3e43fb-87be-445f-875a-280d6787c4c7","Type":"ContainerDied","Data":"e41838dbfa6605dd6be4f541609fa8feb160c7a8cac51522af133d6fc6dc3f66"} Oct 30 02:16:47.298341 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.297146 6572 generic.go:356] "Generic (PLEG): container finished" podID="0ee3f8e3-9b55-4ce5-afaf-86780a26391e" containerID="4b95215344c7160a1f57af88d29d3c69196f43edc38c6deb6ff4684dc8f7de69" exitCode=0 Oct 30 02:16:47.298341 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.297267 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" event={"ID":"0ee3f8e3-9b55-4ce5-afaf-86780a26391e","Type":"ContainerDied","Data":"4b95215344c7160a1f57af88d29d3c69196f43edc38c6deb6ff4684dc8f7de69"} Oct 30 02:16:47.302790 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.301664 6572 generic.go:356] "Generic (PLEG): container finished" podID="dfa07f94-1c0a-4855-8849-5d3e753ab795" containerID="a9cc05bdd2d284c300796157eea3e56887ec86bbc2645fb920a5888304af6570" exitCode=0 Oct 30 02:16:47.302790 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.301771 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb53ecd-gfhc8" event={"ID":"dfa07f94-1c0a-4855-8849-5d3e753ab795","Type":"ContainerDied","Data":"a9cc05bdd2d284c300796157eea3e56887ec86bbc2645fb920a5888304af6570"} Oct 30 02:16:47.304038 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.304008 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-561b8748-8lbtl" Oct 30 02:16:47.304388 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.304360 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" podUID="fac3ca55-64af-42d9-90b1-0e821b893282" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": dial tcp 10.133.0.63:8000: connect: connection refused" Oct 30 02:16:47.304557 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.304532 6572 generic.go:356] "Generic (PLEG): container finished" podID="fac3ca55-64af-42d9-90b1-0e821b893282" containerID="a27836cd757f1537db62bc82910b80f5d5cbf7494695ef25a9f49c9f95e2b840" exitCode=0 Oct 30 02:16:47.304654 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.304634 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" event={"ID":"fac3ca55-64af-42d9-90b1-0e821b893282","Type":"ContainerDied","Data":"a27836cd757f1537db62bc82910b80f5d5cbf7494695ef25a9f49c9f95e2b840"} Oct 30 02:16:47.454490 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.454125 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" Oct 30 02:16:47.459184 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.459156 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" Oct 30 02:16:47.494908 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.494659 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-803472-5ffb57c959-qll7m" podUID="d888789f-78c5-4606-94e1-7fcc77a6c16e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:16:47.495781 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.495735 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-803472-d7dc9c774-4cr8j" podUID="c3884225-2803-4ea6-8f09-96e0696ac6d3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:16:47.567340 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.567248 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fac3ca55-64af-42d9-90b1-0e821b893282-tls-certs\" (UniqueName: \"kubernetes.io/secret/fac3ca55-64af-42d9-90b1-0e821b893282-tls-certs\") pod \"fac3ca55-64af-42d9-90b1-0e821b893282\" (UID: \"fac3ca55-64af-42d9-90b1-0e821b893282\") " Oct 30 02:16:47.567517 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.567354 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-tmp-dir\") pod \"0ee3f8e3-9b55-4ce5-afaf-86780a26391e\" (UID: \"0ee3f8e3-9b55-4ce5-afaf-86780a26391e\") " Oct 30 02:16:47.567517 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.567405 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-dshm\") pod \"0ee3f8e3-9b55-4ce5-afaf-86780a26391e\" (UID: \"0ee3f8e3-9b55-4ce5-afaf-86780a26391e\") " Oct 30 02:16:47.567517 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.567425 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-tls-certs\" (UniqueName: \"kubernetes.io/secret/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-tls-certs\") pod \"0ee3f8e3-9b55-4ce5-afaf-86780a26391e\" (UID: \"0ee3f8e3-9b55-4ce5-afaf-86780a26391e\") " Oct 30 02:16:47.567517 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.567451 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-model-cache\") pod \"0ee3f8e3-9b55-4ce5-afaf-86780a26391e\" (UID: \"0ee3f8e3-9b55-4ce5-afaf-86780a26391e\") " Oct 30 02:16:47.567517 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.567485 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-home\" (UniqueName: \"kubernetes.io/empty-dir/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-home\") pod \"0ee3f8e3-9b55-4ce5-afaf-86780a26391e\" (UID: \"0ee3f8e3-9b55-4ce5-afaf-86780a26391e\") " Oct 30 02:16:47.567804 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.567522 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac3ca55-64af-42d9-90b1-0e821b893282-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fac3ca55-64af-42d9-90b1-0e821b893282-tmp-dir\") pod \"fac3ca55-64af-42d9-90b1-0e821b893282\" (UID: \"fac3ca55-64af-42d9-90b1-0e821b893282\") " Oct 30 02:16:47.567804 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.567563 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac3ca55-64af-42d9-90b1-0e821b893282-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fac3ca55-64af-42d9-90b1-0e821b893282-model-cache\") pod \"fac3ca55-64af-42d9-90b1-0e821b893282\" (UID: \"fac3ca55-64af-42d9-90b1-0e821b893282\") " Oct 30 02:16:47.567804 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.567582 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac3ca55-64af-42d9-90b1-0e821b893282-home\" (UniqueName: \"kubernetes.io/empty-dir/fac3ca55-64af-42d9-90b1-0e821b893282-home\") pod \"fac3ca55-64af-42d9-90b1-0e821b893282\" (UID: \"fac3ca55-64af-42d9-90b1-0e821b893282\") " Oct 30 02:16:47.567804 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.567649 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fac3ca55-64af-42d9-90b1-0e821b893282-kube-api-access-8rlnn\" (UniqueName: \"kubernetes.io/projected/fac3ca55-64af-42d9-90b1-0e821b893282-kube-api-access-8rlnn\") pod \"fac3ca55-64af-42d9-90b1-0e821b893282\" (UID: \"fac3ca55-64af-42d9-90b1-0e821b893282\") " Oct 30 02:16:47.567804 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.567668 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac3ca55-64af-42d9-90b1-0e821b893282-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fac3ca55-64af-42d9-90b1-0e821b893282-dshm\") pod \"fac3ca55-64af-42d9-90b1-0e821b893282\" (UID: \"fac3ca55-64af-42d9-90b1-0e821b893282\") " Oct 30 02:16:47.567804 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.567695 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-kube-api-access-m8rdj\" (UniqueName: \"kubernetes.io/projected/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-kube-api-access-m8rdj\") pod \"0ee3f8e3-9b55-4ce5-afaf-86780a26391e\" (UID: \"0ee3f8e3-9b55-4ce5-afaf-86780a26391e\") " Oct 30 02:16:47.569051 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.569021 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac3ca55-64af-42d9-90b1-0e821b893282-home" pod "fac3ca55-64af-42d9-90b1-0e821b893282" (UID: "fac3ca55-64af-42d9-90b1-0e821b893282"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:47.569283 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.569258 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac3ca55-64af-42d9-90b1-0e821b893282-tmp-dir" pod "fac3ca55-64af-42d9-90b1-0e821b893282" (UID: "fac3ca55-64af-42d9-90b1-0e821b893282"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:47.569464 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.569445 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac3ca55-64af-42d9-90b1-0e821b893282-model-cache" pod "fac3ca55-64af-42d9-90b1-0e821b893282" (UID: "fac3ca55-64af-42d9-90b1-0e821b893282"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:47.569867 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.569827 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-home" pod "0ee3f8e3-9b55-4ce5-afaf-86780a26391e" (UID: "0ee3f8e3-9b55-4ce5-afaf-86780a26391e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:47.570058 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.570034 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-model-cache" pod "0ee3f8e3-9b55-4ce5-afaf-86780a26391e" (UID: "0ee3f8e3-9b55-4ce5-afaf-86780a26391e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:47.570296 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.570271 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-tmp-dir" pod "0ee3f8e3-9b55-4ce5-afaf-86780a26391e" (UID: "0ee3f8e3-9b55-4ce5-afaf-86780a26391e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:47.571119 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.571090 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-kube-api-access-m8rdj" pod "0ee3f8e3-9b55-4ce5-afaf-86780a26391e" (UID: "0ee3f8e3-9b55-4ce5-afaf-86780a26391e"). InnerVolumeSpecName "kube-api-access-m8rdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:16:47.571978 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.571957 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac3ca55-64af-42d9-90b1-0e821b893282-dshm" pod "fac3ca55-64af-42d9-90b1-0e821b893282" (UID: "fac3ca55-64af-42d9-90b1-0e821b893282"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:47.572494 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.572466 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fac3ca55-64af-42d9-90b1-0e821b893282-kube-api-access-8rlnn" pod "fac3ca55-64af-42d9-90b1-0e821b893282" (UID: "fac3ca55-64af-42d9-90b1-0e821b893282"). InnerVolumeSpecName "kube-api-access-8rlnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:16:47.573162 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.573137 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fac3ca55-64af-42d9-90b1-0e821b893282-tls-certs" pod "fac3ca55-64af-42d9-90b1-0e821b893282" (UID: "fac3ca55-64af-42d9-90b1-0e821b893282"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:16:47.575080 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.575061 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-tls-certs" pod "0ee3f8e3-9b55-4ce5-afaf-86780a26391e" (UID: "0ee3f8e3-9b55-4ce5-afaf-86780a26391e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:16:47.575374 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.575353 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-dshm" pod "0ee3f8e3-9b55-4ce5-afaf-86780a26391e" (UID: "0ee3f8e3-9b55-4ce5-afaf-86780a26391e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:47.668825 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.668794 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-561b8748-8lbtl"] Oct 30 02:16:47.668945 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:16:47.668830 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5211f68e_c27b_4d50_a49d_b47ea4adf58a.slice/crio-a498cdec5ba1055f0ae38b69394e3ac271f086aabacdc29a4d5aab3a259232b8 WatchSource:0}: Error finding container a498cdec5ba1055f0ae38b69394e3ac271f086aabacdc29a4d5aab3a259232b8: Status 404 returned error can't find the container with id a498cdec5ba1055f0ae38b69394e3ac271f086aabacdc29a4d5aab3a259232b8 Oct 30 02:16:47.669841 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.669819 6572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-dshm\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:47.669909 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.669847 6572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-tls-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:47.669909 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.669862 6572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-model-cache\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:47.669909 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.669875 6572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-home\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:47.669909 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.669887 6572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fac3ca55-64af-42d9-90b1-0e821b893282-tmp-dir\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:47.669909 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.669900 6572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fac3ca55-64af-42d9-90b1-0e821b893282-model-cache\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:47.670294 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.669912 6572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fac3ca55-64af-42d9-90b1-0e821b893282-home\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:47.670294 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.669927 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rlnn\" (UniqueName: \"kubernetes.io/projected/fac3ca55-64af-42d9-90b1-0e821b893282-kube-api-access-8rlnn\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:47.670294 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.669941 6572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fac3ca55-64af-42d9-90b1-0e821b893282-dshm\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:47.670294 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.669956 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8rdj\" (UniqueName: \"kubernetes.io/projected/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-kube-api-access-m8rdj\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:47.670294 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.669969 6572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fac3ca55-64af-42d9-90b1-0e821b893282-tls-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:47.670294 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.669984 6572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ee3f8e3-9b55-4ce5-afaf-86780a26391e-tmp-dir\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:47.981514 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:47.979273 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:48.316632 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:48.316601 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" Oct 30 02:16:48.317176 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:48.316606 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86" event={"ID":"0ee3f8e3-9b55-4ce5-afaf-86780a26391e","Type":"ContainerDied","Data":"8fd1d60406b0c9a2863c6fb3a360379efefaaedf3e2d0e7f7be65febd3481888"} Oct 30 02:16:48.317176 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:48.316737 6572 scope.go:122] "RemoveContainer" containerID="4b95215344c7160a1f57af88d29d3c69196f43edc38c6deb6ff4684dc8f7de69" Oct 30 02:16:48.318449 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:48.318428 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" Oct 30 02:16:48.318573 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:48.318479 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn" event={"ID":"fac3ca55-64af-42d9-90b1-0e821b893282","Type":"ContainerDied","Data":"531ae5a539bc2904251050dbadbb9b0a3a598eafc68163c44b51308c545f1ec0"} Oct 30 02:16:48.321296 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:48.320742 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-561b8748-8lbtl" event={"ID":"5211f68e-c27b-4d50-a49d-b47ea4adf58a","Type":"ContainerStarted","Data":"2e07265b9705ec06899bf35e7c895d3e43ce66b81f008d8ba64fcf9b860b1bd0"} Oct 30 02:16:48.321296 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:48.320837 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-561b8748-8lbtl" event={"ID":"5211f68e-c27b-4d50-a49d-b47ea4adf58a","Type":"ContainerStarted","Data":"a498cdec5ba1055f0ae38b69394e3ac271f086aabacdc29a4d5aab3a259232b8"} Oct 30 02:16:48.340996 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:48.340925 6572 scope.go:122] "RemoveContainer" containerID="a27836cd757f1537db62bc82910b80f5d5cbf7494695ef25a9f49c9f95e2b840" Oct 30 02:16:48.372400 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:48.372336 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-561b8748-8lbtl" podStartSLOduration=2.372316613 podStartE2EDuration="2.372316613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:16:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:16:48.367573828 +0000 UTC m=+3533.943999241" watchObservedRunningTime="2026-10-30 02:16:48.372316613 +0000 UTC m=+3533.948742017" Oct 30 02:16:48.423105 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:48.423044 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn"] Oct 30 02:16:48.429785 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:48.428962 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-803472/simulator-w4-803472-kserve-69d5d5f45c-n75kn"] Oct 30 02:16:48.461322 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:48.461284 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86"] Oct 30 02:16:48.465376 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:48.465339 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-803472/premium-w4-803472-kserve-68694cd7b8-b5j86"] Oct 30 02:16:48.499908 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:48.499842 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb53ecd-gfhc8" Oct 30 02:16:48.564784 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:48.564720 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ee3f8e3-9b55-4ce5-afaf-86780a26391e" path="/var/lib/kubelet/pods/0ee3f8e3-9b55-4ce5-afaf-86780a26391e/volumes" Oct 30 02:16:48.565568 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:48.565542 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fac3ca55-64af-42d9-90b1-0e821b893282" path="/var/lib/kubelet/pods/fac3ca55-64af-42d9-90b1-0e821b893282/volumes" Oct 30 02:16:48.568145 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:48.568122 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad9c077d-vmd8b" Oct 30 02:16:48.588486 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:48.588457 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfa07f94-1c0a-4855-8849-5d3e753ab795-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfa07f94-1c0a-4855-8849-5d3e753ab795-maas-api-openshift-ca\") pod \"dfa07f94-1c0a-4855-8849-5d3e753ab795\" (UID: \"dfa07f94-1c0a-4855-8849-5d3e753ab795\") " Oct 30 02:16:48.588623 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:48.588503 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfa07f94-1c0a-4855-8849-5d3e753ab795-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfa07f94-1c0a-4855-8849-5d3e753ab795-maas-api-xks-ca\") pod \"dfa07f94-1c0a-4855-8849-5d3e753ab795\" (UID: \"dfa07f94-1c0a-4855-8849-5d3e753ab795\") " Oct 30 02:16:48.588623 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:48.588573 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfa07f94-1c0a-4855-8849-5d3e753ab795-kube-api-access-dlr5l\" (UniqueName: \"kubernetes.io/projected/dfa07f94-1c0a-4855-8849-5d3e753ab795-kube-api-access-dlr5l\") pod \"dfa07f94-1c0a-4855-8849-5d3e753ab795\" (UID: \"dfa07f94-1c0a-4855-8849-5d3e753ab795\") " Oct 30 02:16:48.588973 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:48.588946 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfa07f94-1c0a-4855-8849-5d3e753ab795-maas-api-openshift-ca" pod "dfa07f94-1c0a-4855-8849-5d3e753ab795" (UID: "dfa07f94-1c0a-4855-8849-5d3e753ab795"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:16:48.589246 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:48.589223 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfa07f94-1c0a-4855-8849-5d3e753ab795-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:48.591220 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:48.591194 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfa07f94-1c0a-4855-8849-5d3e753ab795-kube-api-access-dlr5l" pod "dfa07f94-1c0a-4855-8849-5d3e753ab795" (UID: "dfa07f94-1c0a-4855-8849-5d3e753ab795"). InnerVolumeSpecName "kube-api-access-dlr5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:16:48.591429 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:48.591406 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfa07f94-1c0a-4855-8849-5d3e753ab795-maas-api-xks-ca" pod "dfa07f94-1c0a-4855-8849-5d3e753ab795" (UID: "dfa07f94-1c0a-4855-8849-5d3e753ab795"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:16:48.689941 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:48.689911 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba3e43fb-87be-445f-875a-280d6787c4c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba3e43fb-87be-445f-875a-280d6787c4c7-maas-api-openshift-ca\") pod \"ba3e43fb-87be-445f-875a-280d6787c4c7\" (UID: \"ba3e43fb-87be-445f-875a-280d6787c4c7\") " Oct 30 02:16:48.690118 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:48.690108 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba3e43fb-87be-445f-875a-280d6787c4c7-kube-api-access-7hrc4\" (UniqueName: \"kubernetes.io/projected/ba3e43fb-87be-445f-875a-280d6787c4c7-kube-api-access-7hrc4\") pod \"ba3e43fb-87be-445f-875a-280d6787c4c7\" (UID: \"ba3e43fb-87be-445f-875a-280d6787c4c7\") " Oct 30 02:16:48.690173 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:48.690157 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba3e43fb-87be-445f-875a-280d6787c4c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba3e43fb-87be-445f-875a-280d6787c4c7-maas-api-xks-ca\") pod \"ba3e43fb-87be-445f-875a-280d6787c4c7\" (UID: \"ba3e43fb-87be-445f-875a-280d6787c4c7\") " Oct 30 02:16:48.690316 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:48.690292 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba3e43fb-87be-445f-875a-280d6787c4c7-maas-api-openshift-ca" pod "ba3e43fb-87be-445f-875a-280d6787c4c7" (UID: "ba3e43fb-87be-445f-875a-280d6787c4c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:16:48.691033 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:48.691002 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba3e43fb-87be-445f-875a-280d6787c4c7-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:48.691177 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:48.691037 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfa07f94-1c0a-4855-8849-5d3e753ab795-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:48.691177 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:48.691067 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlr5l\" (UniqueName: \"kubernetes.io/projected/dfa07f94-1c0a-4855-8849-5d3e753ab795-kube-api-access-dlr5l\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:48.692361 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:48.692338 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba3e43fb-87be-445f-875a-280d6787c4c7-kube-api-access-7hrc4" pod "ba3e43fb-87be-445f-875a-280d6787c4c7" (UID: "ba3e43fb-87be-445f-875a-280d6787c4c7"). InnerVolumeSpecName "kube-api-access-7hrc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:16:48.692455 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:48.692365 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba3e43fb-87be-445f-875a-280d6787c4c7-maas-api-xks-ca" pod "ba3e43fb-87be-445f-875a-280d6787c4c7" (UID: "ba3e43fb-87be-445f-875a-280d6787c4c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:16:48.792424 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:48.792347 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hrc4\" (UniqueName: \"kubernetes.io/projected/ba3e43fb-87be-445f-875a-280d6787c4c7-kube-api-access-7hrc4\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:48.792424 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:48.792377 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba3e43fb-87be-445f-875a-280d6787c4c7-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:49.294192 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:49.294157 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:49.330579 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:49.330545 6572 generic.go:356] "Generic (PLEG): container finished" podID="5211f68e-c27b-4d50-a49d-b47ea4adf58a" containerID="2e07265b9705ec06899bf35e7c895d3e43ce66b81f008d8ba64fcf9b860b1bd0" exitCode=0 Oct 30 02:16:49.331045 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:49.330629 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-561b8748-8lbtl" event={"ID":"5211f68e-c27b-4d50-a49d-b47ea4adf58a","Type":"ContainerDied","Data":"2e07265b9705ec06899bf35e7c895d3e43ce66b81f008d8ba64fcf9b860b1bd0"} Oct 30 02:16:49.332237 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:49.332217 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad9c077d-vmd8b" Oct 30 02:16:49.332365 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:49.332253 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad9c077d-vmd8b" event={"ID":"ba3e43fb-87be-445f-875a-280d6787c4c7","Type":"ContainerDied","Data":"26a49cb2fdec1a45a88d5ed1fc1efc7a135f5cd39b056e86c794fd86930157d6"} Oct 30 02:16:49.332365 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:49.332295 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26a49cb2fdec1a45a88d5ed1fc1efc7a135f5cd39b056e86c794fd86930157d6" Oct 30 02:16:49.335014 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:49.334995 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb53ecd-gfhc8" Oct 30 02:16:49.335014 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:49.335008 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-deb53ecd-gfhc8" event={"ID":"dfa07f94-1c0a-4855-8849-5d3e753ab795","Type":"ContainerDied","Data":"d5f35f61c651c8b23c0382a1502024e5c04740f95e7f18875d1e9d24def016d8"} Oct 30 02:16:49.335168 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:49.335042 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5f35f61c651c8b23c0382a1502024e5c04740f95e7f18875d1e9d24def016d8" Oct 30 02:16:50.501359 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:50.501336 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-561b8748-8lbtl" Oct 30 02:16:50.571712 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:50.570773 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:50.616964 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:50.616883 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5211f68e-c27b-4d50-a49d-b47ea4adf58a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5211f68e-c27b-4d50-a49d-b47ea4adf58a-maas-api-openshift-ca\") pod \"5211f68e-c27b-4d50-a49d-b47ea4adf58a\" (UID: \"5211f68e-c27b-4d50-a49d-b47ea4adf58a\") " Oct 30 02:16:50.617104 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:50.616979 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5211f68e-c27b-4d50-a49d-b47ea4adf58a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5211f68e-c27b-4d50-a49d-b47ea4adf58a-maas-api-xks-ca\") pod \"5211f68e-c27b-4d50-a49d-b47ea4adf58a\" (UID: \"5211f68e-c27b-4d50-a49d-b47ea4adf58a\") " Oct 30 02:16:50.617222 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:50.617195 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5211f68e-c27b-4d50-a49d-b47ea4adf58a-kube-api-access-hdrj4\" (UniqueName: \"kubernetes.io/projected/5211f68e-c27b-4d50-a49d-b47ea4adf58a-kube-api-access-hdrj4\") pod \"5211f68e-c27b-4d50-a49d-b47ea4adf58a\" (UID: \"5211f68e-c27b-4d50-a49d-b47ea4adf58a\") " Oct 30 02:16:50.617267 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:50.617227 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5211f68e-c27b-4d50-a49d-b47ea4adf58a-maas-api-openshift-ca" pod "5211f68e-c27b-4d50-a49d-b47ea4adf58a" (UID: "5211f68e-c27b-4d50-a49d-b47ea4adf58a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:16:50.617985 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:50.617961 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5211f68e-c27b-4d50-a49d-b47ea4adf58a-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:50.619190 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:50.619167 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5211f68e-c27b-4d50-a49d-b47ea4adf58a-maas-api-xks-ca" pod "5211f68e-c27b-4d50-a49d-b47ea4adf58a" (UID: "5211f68e-c27b-4d50-a49d-b47ea4adf58a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:16:50.619372 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:50.619352 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5211f68e-c27b-4d50-a49d-b47ea4adf58a-kube-api-access-hdrj4" pod "5211f68e-c27b-4d50-a49d-b47ea4adf58a" (UID: "5211f68e-c27b-4d50-a49d-b47ea4adf58a"). InnerVolumeSpecName "kube-api-access-hdrj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:16:50.719878 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:50.719839 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5211f68e-c27b-4d50-a49d-b47ea4adf58a-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:50.719878 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:50.719871 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdrj4\" (UniqueName: \"kubernetes.io/projected/5211f68e-c27b-4d50-a49d-b47ea4adf58a-kube-api-access-hdrj4\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:51.355041 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:51.355010 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-561b8748-8lbtl" Oct 30 02:16:51.355279 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:51.355050 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-561b8748-8lbtl" event={"ID":"5211f68e-c27b-4d50-a49d-b47ea4adf58a","Type":"ContainerDied","Data":"a498cdec5ba1055f0ae38b69394e3ac271f086aabacdc29a4d5aab3a259232b8"} Oct 30 02:16:51.355279 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:51.355112 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a498cdec5ba1055f0ae38b69394e3ac271f086aabacdc29a4d5aab3a259232b8" Oct 30 02:16:51.921773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:51.920771 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:52.997642 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:52.997599 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:53.190411 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:53.190375 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:53.397859 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:53.395287 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:53.756775 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:53.748693 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:53.893535 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:53.893505 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22e99d-vjz8p"] Oct 30 02:16:53.894211 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:53.894195 6572 cpu_manager.go:414] "containerMap: removing container" podUID="dfa07f94-1c0a-4855-8849-5d3e753ab795" containerName="revoke-keys" Oct 30 02:16:53.894278 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:53.894212 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfa07f94-1c0a-4855-8849-5d3e753ab795" containerName="revoke-keys" Oct 30 02:16:53.894278 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:53.894229 6572 cpu_manager.go:414] "containerMap: removing container" podUID="ba3e43fb-87be-445f-875a-280d6787c4c7" containerName="revoke-keys" Oct 30 02:16:53.894278 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:53.894234 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba3e43fb-87be-445f-875a-280d6787c4c7" containerName="revoke-keys" Oct 30 02:16:53.894278 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:53.894243 6572 cpu_manager.go:414] "containerMap: removing container" podUID="fac3ca55-64af-42d9-90b1-0e821b893282" containerName="main" Oct 30 02:16:53.894278 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:53.894249 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fac3ca55-64af-42d9-90b1-0e821b893282" containerName="main" Oct 30 02:16:53.894278 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:53.894256 6572 cpu_manager.go:414] "containerMap: removing container" podUID="5211f68e-c27b-4d50-a49d-b47ea4adf58a" containerName="revoke-keys" Oct 30 02:16:53.894278 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:53.894261 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5211f68e-c27b-4d50-a49d-b47ea4adf58a" containerName="revoke-keys" Oct 30 02:16:53.894278 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:53.894268 6572 cpu_manager.go:414] "containerMap: removing container" podUID="0ee3f8e3-9b55-4ce5-afaf-86780a26391e" containerName="main" Oct 30 02:16:53.894278 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:53.894272 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ee3f8e3-9b55-4ce5-afaf-86780a26391e" containerName="main" Oct 30 02:16:53.894559 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:53.894382 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfa07f94-1c0a-4855-8849-5d3e753ab795" containerName="revoke-keys" Oct 30 02:16:53.894559 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:53.894392 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba3e43fb-87be-445f-875a-280d6787c4c7" containerName="revoke-keys" Oct 30 02:16:53.894559 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:53.894398 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="fac3ca55-64af-42d9-90b1-0e821b893282" containerName="main" Oct 30 02:16:53.894559 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:53.894405 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="5211f68e-c27b-4d50-a49d-b47ea4adf58a" containerName="revoke-keys" Oct 30 02:16:53.894559 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:53.894411 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ee3f8e3-9b55-4ce5-afaf-86780a26391e" containerName="main" Oct 30 02:16:53.899521 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:53.899503 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22e99d-vjz8p" Oct 30 02:16:53.921889 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:53.921854 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:16:53.922051 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:53.922000 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:16:53.923133 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:53.923103 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22e99d-vjz8p"] Oct 30 02:16:53.929505 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:53.929480 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:16:53.929914 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:53.929895 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:16:54.002135 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:54.002097 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f84693b2-6f55-4fae-a301-2cd285e7001b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-22e99d-vjz8p\" (UID: \"f84693b2-6f55-4fae-a301-2cd285e7001b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22e99d-vjz8p" Oct 30 02:16:54.002622 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:54.002216 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbtdm\" (UniqueName: \"kubernetes.io/projected/f84693b2-6f55-4fae-a301-2cd285e7001b-kube-api-access-rbtdm\") pod \"maas-api-revoke-keys-e2e-inf-b-22e99d-vjz8p\" (UID: \"f84693b2-6f55-4fae-a301-2cd285e7001b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22e99d-vjz8p" Oct 30 02:16:54.002622 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:54.002441 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f84693b2-6f55-4fae-a301-2cd285e7001b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-22e99d-vjz8p\" (UID: \"f84693b2-6f55-4fae-a301-2cd285e7001b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22e99d-vjz8p" Oct 30 02:16:54.103959 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:54.103874 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f84693b2-6f55-4fae-a301-2cd285e7001b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-22e99d-vjz8p\" (UID: \"f84693b2-6f55-4fae-a301-2cd285e7001b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22e99d-vjz8p" Oct 30 02:16:54.104116 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:54.104010 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f84693b2-6f55-4fae-a301-2cd285e7001b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-22e99d-vjz8p\" (UID: \"f84693b2-6f55-4fae-a301-2cd285e7001b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22e99d-vjz8p" Oct 30 02:16:54.104116 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:54.104047 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbtdm\" (UniqueName: \"kubernetes.io/projected/f84693b2-6f55-4fae-a301-2cd285e7001b-kube-api-access-rbtdm\") pod \"maas-api-revoke-keys-e2e-inf-b-22e99d-vjz8p\" (UID: \"f84693b2-6f55-4fae-a301-2cd285e7001b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22e99d-vjz8p" Oct 30 02:16:54.104584 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:54.104559 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f84693b2-6f55-4fae-a301-2cd285e7001b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-22e99d-vjz8p\" (UID: \"f84693b2-6f55-4fae-a301-2cd285e7001b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22e99d-vjz8p" Oct 30 02:16:54.106440 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:54.106416 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f84693b2-6f55-4fae-a301-2cd285e7001b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-22e99d-vjz8p\" (UID: \"f84693b2-6f55-4fae-a301-2cd285e7001b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22e99d-vjz8p" Oct 30 02:16:54.128456 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:54.128428 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-264bdd-zqlhm"] Oct 30 02:16:54.132967 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:54.132944 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbtdm\" (UniqueName: \"kubernetes.io/projected/f84693b2-6f55-4fae-a301-2cd285e7001b-kube-api-access-rbtdm\") pod \"maas-api-revoke-keys-e2e-inf-b-22e99d-vjz8p\" (UID: \"f84693b2-6f55-4fae-a301-2cd285e7001b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22e99d-vjz8p" Oct 30 02:16:54.133562 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:54.133544 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-264bdd-zqlhm" Oct 30 02:16:54.151380 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:54.151358 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-264bdd-zqlhm"] Oct 30 02:16:54.225937 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:54.225903 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22e99d-vjz8p" Oct 30 02:16:54.306645 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:54.306608 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb304a68-5357-4ad1-9cf4-94b3796da9c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-264bdd-zqlhm\" (UID: \"bb304a68-5357-4ad1-9cf4-94b3796da9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-264bdd-zqlhm" Oct 30 02:16:54.306829 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:54.306669 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t2f6\" (UniqueName: \"kubernetes.io/projected/bb304a68-5357-4ad1-9cf4-94b3796da9c4-kube-api-access-2t2f6\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-264bdd-zqlhm\" (UID: \"bb304a68-5357-4ad1-9cf4-94b3796da9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-264bdd-zqlhm" Oct 30 02:16:54.307140 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:54.307101 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb304a68-5357-4ad1-9cf4-94b3796da9c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-264bdd-zqlhm\" (UID: \"bb304a68-5357-4ad1-9cf4-94b3796da9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-264bdd-zqlhm" Oct 30 02:16:54.409008 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:54.408928 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb304a68-5357-4ad1-9cf4-94b3796da9c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-264bdd-zqlhm\" (UID: \"bb304a68-5357-4ad1-9cf4-94b3796da9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-264bdd-zqlhm" Oct 30 02:16:54.409008 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:54.408979 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb304a68-5357-4ad1-9cf4-94b3796da9c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-264bdd-zqlhm\" (UID: \"bb304a68-5357-4ad1-9cf4-94b3796da9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-264bdd-zqlhm" Oct 30 02:16:54.409008 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:54.409009 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t2f6\" (UniqueName: \"kubernetes.io/projected/bb304a68-5357-4ad1-9cf4-94b3796da9c4-kube-api-access-2t2f6\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-264bdd-zqlhm\" (UID: \"bb304a68-5357-4ad1-9cf4-94b3796da9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-264bdd-zqlhm" Oct 30 02:16:54.409584 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:54.409556 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb304a68-5357-4ad1-9cf4-94b3796da9c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-264bdd-zqlhm\" (UID: \"bb304a68-5357-4ad1-9cf4-94b3796da9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-264bdd-zqlhm" Oct 30 02:16:54.411344 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:54.411319 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb304a68-5357-4ad1-9cf4-94b3796da9c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-264bdd-zqlhm\" (UID: \"bb304a68-5357-4ad1-9cf4-94b3796da9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-264bdd-zqlhm" Oct 30 02:16:54.427596 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:54.427573 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t2f6\" (UniqueName: \"kubernetes.io/projected/bb304a68-5357-4ad1-9cf4-94b3796da9c4-kube-api-access-2t2f6\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-264bdd-zqlhm\" (UID: \"bb304a68-5357-4ad1-9cf4-94b3796da9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-264bdd-zqlhm" Oct 30 02:16:54.461510 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:54.461483 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-264bdd-zqlhm" Oct 30 02:16:54.530450 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:54.530422 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22e99d-vjz8p"] Oct 30 02:16:54.764639 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:16:54.764610 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb304a68_5357_4ad1_9cf4_94b3796da9c4.slice/crio-658d96a833bef8af39fc64c86a43487875afe4acdd97b23da0061355f9ce041f WatchSource:0}: Error finding container 658d96a833bef8af39fc64c86a43487875afe4acdd97b23da0061355f9ce041f: Status 404 returned error can't find the container with id 658d96a833bef8af39fc64c86a43487875afe4acdd97b23da0061355f9ce041f Oct 30 02:16:54.764774 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:54.764732 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-264bdd-zqlhm"] Oct 30 02:16:55.055918 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:55.055879 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:55.436581 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:55.436544 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22e99d-vjz8p" event={"ID":"f84693b2-6f55-4fae-a301-2cd285e7001b","Type":"ContainerStarted","Data":"21c40c702c16de5c187a7111fdd4a51f38b49be37abc9b4269ee7904add9fd10"} Oct 30 02:16:55.436800 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:55.436607 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22e99d-vjz8p" event={"ID":"f84693b2-6f55-4fae-a301-2cd285e7001b","Type":"ContainerStarted","Data":"0ff9c3e70b0d1571dfeeb025b4b1090ceb626b18eea5c21c7412c2a6b27ddf9e"} Oct 30 02:16:55.438306 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:55.438279 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-264bdd-zqlhm" event={"ID":"bb304a68-5357-4ad1-9cf4-94b3796da9c4","Type":"ContainerStarted","Data":"8277a0e52c0b0f27058b15b8f32c99426659d55b9c2389c64ba9e1b21b8ad422"} Oct 30 02:16:55.438441 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:55.438324 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-264bdd-zqlhm" event={"ID":"bb304a68-5357-4ad1-9cf4-94b3796da9c4","Type":"ContainerStarted","Data":"658d96a833bef8af39fc64c86a43487875afe4acdd97b23da0061355f9ce041f"} Oct 30 02:16:55.477016 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:55.476966 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22e99d-vjz8p" podStartSLOduration=2.476954513 podStartE2EDuration="2.476954513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:16:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:16:55.472424498 +0000 UTC m=+3541.048849901" watchObservedRunningTime="2026-10-30 02:16:55.476954513 +0000 UTC m=+3541.053379951" Oct 30 02:16:55.512134 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:55.512036 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-264bdd-zqlhm" podStartSLOduration=1.512021962 podStartE2EDuration="1.512021962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:16:55.508523319 +0000 UTC m=+3541.084948723" watchObservedRunningTime="2026-10-30 02:16:55.512021962 +0000 UTC m=+3541.088447364" Oct 30 02:16:56.449873 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:56.449834 6572 generic.go:356] "Generic (PLEG): container finished" podID="f84693b2-6f55-4fae-a301-2cd285e7001b" containerID="21c40c702c16de5c187a7111fdd4a51f38b49be37abc9b4269ee7904add9fd10" exitCode=0 Oct 30 02:16:56.450338 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:56.449876 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22e99d-vjz8p" event={"ID":"f84693b2-6f55-4fae-a301-2cd285e7001b","Type":"ContainerDied","Data":"21c40c702c16de5c187a7111fdd4a51f38b49be37abc9b4269ee7904add9fd10"} Oct 30 02:16:56.451818 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:56.451795 6572 generic.go:356] "Generic (PLEG): container finished" podID="bb304a68-5357-4ad1-9cf4-94b3796da9c4" containerID="8277a0e52c0b0f27058b15b8f32c99426659d55b9c2389c64ba9e1b21b8ad422" exitCode=0 Oct 30 02:16:56.451962 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:56.451823 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-264bdd-zqlhm" event={"ID":"bb304a68-5357-4ad1-9cf4-94b3796da9c4","Type":"ContainerDied","Data":"8277a0e52c0b0f27058b15b8f32c99426659d55b9c2389c64ba9e1b21b8ad422"} Oct 30 02:16:56.550206 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:56.550179 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:57.495531 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:57.495146 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-803472-d7dc9c774-4cr8j" podUID="c3884225-2803-4ea6-8f09-96e0696ac6d3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:16:57.496428 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:57.496391 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-803472-5ffb57c959-qll7m" podUID="d888789f-78c5-4606-94e1-7fcc77a6c16e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:16:57.661580 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:57.661548 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-264bdd-zqlhm" Oct 30 02:16:57.699317 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:57.699276 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22e99d-vjz8p" Oct 30 02:16:57.767654 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:57.767550 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb304a68-5357-4ad1-9cf4-94b3796da9c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb304a68-5357-4ad1-9cf4-94b3796da9c4-maas-api-openshift-ca\") pod \"bb304a68-5357-4ad1-9cf4-94b3796da9c4\" (UID: \"bb304a68-5357-4ad1-9cf4-94b3796da9c4\") " Oct 30 02:16:57.767836 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:57.767656 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb304a68-5357-4ad1-9cf4-94b3796da9c4-kube-api-access-2t2f6\" (UniqueName: \"kubernetes.io/projected/bb304a68-5357-4ad1-9cf4-94b3796da9c4-kube-api-access-2t2f6\") pod \"bb304a68-5357-4ad1-9cf4-94b3796da9c4\" (UID: \"bb304a68-5357-4ad1-9cf4-94b3796da9c4\") " Oct 30 02:16:57.767836 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:57.767717 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f84693b2-6f55-4fae-a301-2cd285e7001b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f84693b2-6f55-4fae-a301-2cd285e7001b-maas-api-xks-ca\") pod \"f84693b2-6f55-4fae-a301-2cd285e7001b\" (UID: \"f84693b2-6f55-4fae-a301-2cd285e7001b\") " Oct 30 02:16:57.767836 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:57.767830 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb304a68-5357-4ad1-9cf4-94b3796da9c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb304a68-5357-4ad1-9cf4-94b3796da9c4-maas-api-xks-ca\") pod \"bb304a68-5357-4ad1-9cf4-94b3796da9c4\" (UID: \"bb304a68-5357-4ad1-9cf4-94b3796da9c4\") " Oct 30 02:16:57.768096 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:57.767867 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f84693b2-6f55-4fae-a301-2cd285e7001b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f84693b2-6f55-4fae-a301-2cd285e7001b-maas-api-openshift-ca\") pod \"f84693b2-6f55-4fae-a301-2cd285e7001b\" (UID: \"f84693b2-6f55-4fae-a301-2cd285e7001b\") " Oct 30 02:16:57.768096 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:57.767924 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f84693b2-6f55-4fae-a301-2cd285e7001b-kube-api-access-rbtdm\" (UniqueName: \"kubernetes.io/projected/f84693b2-6f55-4fae-a301-2cd285e7001b-kube-api-access-rbtdm\") pod \"f84693b2-6f55-4fae-a301-2cd285e7001b\" (UID: \"f84693b2-6f55-4fae-a301-2cd285e7001b\") " Oct 30 02:16:57.768096 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:57.767989 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb304a68-5357-4ad1-9cf4-94b3796da9c4-maas-api-openshift-ca" pod "bb304a68-5357-4ad1-9cf4-94b3796da9c4" (UID: "bb304a68-5357-4ad1-9cf4-94b3796da9c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:16:57.768344 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:57.768321 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f84693b2-6f55-4fae-a301-2cd285e7001b-maas-api-openshift-ca" pod "f84693b2-6f55-4fae-a301-2cd285e7001b" (UID: "f84693b2-6f55-4fae-a301-2cd285e7001b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:16:57.768681 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:57.768657 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f84693b2-6f55-4fae-a301-2cd285e7001b-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:57.768815 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:57.768685 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb304a68-5357-4ad1-9cf4-94b3796da9c4-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:57.770483 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:57.770455 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb304a68-5357-4ad1-9cf4-94b3796da9c4-kube-api-access-2t2f6" pod "bb304a68-5357-4ad1-9cf4-94b3796da9c4" (UID: "bb304a68-5357-4ad1-9cf4-94b3796da9c4"). InnerVolumeSpecName "kube-api-access-2t2f6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:16:57.770585 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:57.770502 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb304a68-5357-4ad1-9cf4-94b3796da9c4-maas-api-xks-ca" pod "bb304a68-5357-4ad1-9cf4-94b3796da9c4" (UID: "bb304a68-5357-4ad1-9cf4-94b3796da9c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:16:57.770668 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:57.770619 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f84693b2-6f55-4fae-a301-2cd285e7001b-maas-api-xks-ca" pod "f84693b2-6f55-4fae-a301-2cd285e7001b" (UID: "f84693b2-6f55-4fae-a301-2cd285e7001b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:16:57.772259 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:57.772234 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f84693b2-6f55-4fae-a301-2cd285e7001b-kube-api-access-rbtdm" pod "f84693b2-6f55-4fae-a301-2cd285e7001b" (UID: "f84693b2-6f55-4fae-a301-2cd285e7001b"). InnerVolumeSpecName "kube-api-access-rbtdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:16:57.870122 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:57.870085 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t2f6\" (UniqueName: \"kubernetes.io/projected/bb304a68-5357-4ad1-9cf4-94b3796da9c4-kube-api-access-2t2f6\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:57.870122 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:57.870120 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f84693b2-6f55-4fae-a301-2cd285e7001b-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:57.870334 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:57.870133 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb304a68-5357-4ad1-9cf4-94b3796da9c4-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:57.870334 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:57.870146 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbtdm\" (UniqueName: \"kubernetes.io/projected/f84693b2-6f55-4fae-a301-2cd285e7001b-kube-api-access-rbtdm\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:16:57.956617 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:57.956579 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:58.472140 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:58.472106 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22e99d-vjz8p" Oct 30 02:16:58.472360 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:58.472140 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22e99d-vjz8p" event={"ID":"f84693b2-6f55-4fae-a301-2cd285e7001b","Type":"ContainerDied","Data":"0ff9c3e70b0d1571dfeeb025b4b1090ceb626b18eea5c21c7412c2a6b27ddf9e"} Oct 30 02:16:58.472360 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:58.472191 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ff9c3e70b0d1571dfeeb025b4b1090ceb626b18eea5c21c7412c2a6b27ddf9e" Oct 30 02:16:58.473816 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:58.473789 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-264bdd-zqlhm" Oct 30 02:16:58.473816 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:58.473803 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-264bdd-zqlhm" event={"ID":"bb304a68-5357-4ad1-9cf4-94b3796da9c4","Type":"ContainerDied","Data":"658d96a833bef8af39fc64c86a43487875afe4acdd97b23da0061355f9ce041f"} Oct 30 02:16:58.474024 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:58.473842 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="658d96a833bef8af39fc64c86a43487875afe4acdd97b23da0061355f9ce041f" Oct 30 02:16:59.347680 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:59.347636 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:59.412506 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:16:59.412476 6572 scope.go:122] "RemoveContainer" containerID="99ebf356dbc2d2e637ef5e8a9aca934caf6338ac02c72957dbc862f30a7e4b15" Oct 30 02:17:00.778196 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:00.778159 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:01.556779 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:01.555939 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f"] Oct 30 02:17:01.556779 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:01.556275 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" podUID="364b9686-9847-4ef4-a5b1-7f611a248b64" containerName="istio-proxy" containerID="cri-o://90ea182817356e05150d77f344aaa6ec82d66a25f3c17c8ed6afec9995a0d2e1" gracePeriod=30 Oct 30 02:17:01.579255 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:01.579206 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf"] Oct 30 02:17:01.579586 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:01.579557 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" podUID="1147e056-c134-4bd0-8c0f-19de2e33eead" containerName="istio-proxy" containerID="cri-o://1f0b6965d1cee50ed6490374ca948a7d8b74666b0e6d63385c139e3fd39b2261" gracePeriod=30 Oct 30 02:17:01.732778 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:01.726688 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:01.859878 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:01.858764 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f89697-8lcv9"] Oct 30 02:17:01.859878 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:01.859858 6572 cpu_manager.go:414] "containerMap: removing container" podUID="bb304a68-5357-4ad1-9cf4-94b3796da9c4" containerName="revoke-keys" Oct 30 02:17:01.860551 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:01.859883 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb304a68-5357-4ad1-9cf4-94b3796da9c4" containerName="revoke-keys" Oct 30 02:17:01.860551 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:01.859915 6572 cpu_manager.go:414] "containerMap: removing container" podUID="f84693b2-6f55-4fae-a301-2cd285e7001b" containerName="revoke-keys" Oct 30 02:17:01.860551 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:01.859925 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f84693b2-6f55-4fae-a301-2cd285e7001b" containerName="revoke-keys" Oct 30 02:17:01.860551 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:01.860150 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="f84693b2-6f55-4fae-a301-2cd285e7001b" containerName="revoke-keys" Oct 30 02:17:01.860551 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:01.860184 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb304a68-5357-4ad1-9cf4-94b3796da9c4" containerName="revoke-keys" Oct 30 02:17:01.867473 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:01.867443 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f89697-8lcv9" Oct 30 02:17:01.884699 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:01.884529 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:17:01.884699 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:01.884664 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:17:01.897025 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:01.896986 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f89697-8lcv9"] Oct 30 02:17:01.959806 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:01.959766 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:17:01.959973 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:01.959818 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:17:02.052890 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:02.052849 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vwjc\" (UniqueName: \"kubernetes.io/projected/0ac6e981-474c-4232-afcc-2b72ad4835f7-kube-api-access-9vwjc\") pod \"maas-api-revoke-keys-e2e-inf-a-f89697-8lcv9\" (UID: \"0ac6e981-474c-4232-afcc-2b72ad4835f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f89697-8lcv9" Oct 30 02:17:02.053093 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:02.052984 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ac6e981-474c-4232-afcc-2b72ad4835f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f89697-8lcv9\" (UID: \"0ac6e981-474c-4232-afcc-2b72ad4835f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f89697-8lcv9" Oct 30 02:17:02.053197 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:02.053179 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ac6e981-474c-4232-afcc-2b72ad4835f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f89697-8lcv9\" (UID: \"0ac6e981-474c-4232-afcc-2b72ad4835f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f89697-8lcv9" Oct 30 02:17:02.173844 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:02.158786 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ac6e981-474c-4232-afcc-2b72ad4835f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f89697-8lcv9\" (UID: \"0ac6e981-474c-4232-afcc-2b72ad4835f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f89697-8lcv9" Oct 30 02:17:02.173844 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:02.158884 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vwjc\" (UniqueName: \"kubernetes.io/projected/0ac6e981-474c-4232-afcc-2b72ad4835f7-kube-api-access-9vwjc\") pod \"maas-api-revoke-keys-e2e-inf-a-f89697-8lcv9\" (UID: \"0ac6e981-474c-4232-afcc-2b72ad4835f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f89697-8lcv9" Oct 30 02:17:02.173844 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:02.158993 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ac6e981-474c-4232-afcc-2b72ad4835f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f89697-8lcv9\" (UID: \"0ac6e981-474c-4232-afcc-2b72ad4835f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f89697-8lcv9" Oct 30 02:17:02.173844 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:02.164768 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ac6e981-474c-4232-afcc-2b72ad4835f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f89697-8lcv9\" (UID: \"0ac6e981-474c-4232-afcc-2b72ad4835f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f89697-8lcv9" Oct 30 02:17:02.193888 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:02.193852 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ac6e981-474c-4232-afcc-2b72ad4835f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f89697-8lcv9\" (UID: \"0ac6e981-474c-4232-afcc-2b72ad4835f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f89697-8lcv9" Oct 30 02:17:02.217769 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:02.194773 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vwjc\" (UniqueName: \"kubernetes.io/projected/0ac6e981-474c-4232-afcc-2b72ad4835f7-kube-api-access-9vwjc\") pod \"maas-api-revoke-keys-e2e-inf-a-f89697-8lcv9\" (UID: \"0ac6e981-474c-4232-afcc-2b72ad4835f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f89697-8lcv9" Oct 30 02:17:02.492433 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:02.492327 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f89697-8lcv9" Oct 30 02:17:02.574454 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:02.573100 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:02.923325 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:17:02.923292 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ac6e981_474c_4232_afcc_2b72ad4835f7.slice/crio-cfd9d1970a299e7fbb73888b7a57ae48522d1d577cc70e53b67dcb4d7f1a2aca WatchSource:0}: Error finding container cfd9d1970a299e7fbb73888b7a57ae48522d1d577cc70e53b67dcb4d7f1a2aca: Status 404 returned error can't find the container with id cfd9d1970a299e7fbb73888b7a57ae48522d1d577cc70e53b67dcb4d7f1a2aca Oct 30 02:17:02.924600 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:02.924571 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f89697-8lcv9"] Oct 30 02:17:03.347608 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:03.347580 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-803472-d7dc9c774-4cr8j" Oct 30 02:17:03.351494 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:03.351471 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-803472-5ffb57c959-qll7m" Oct 30 02:17:03.391759 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:03.391709 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3884225-2803-4ea6-8f09-96e0696ac6d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3884225-2803-4ea6-8f09-96e0696ac6d3-plugins-config-volume\") pod \"c3884225-2803-4ea6-8f09-96e0696ac6d3\" (UID: \"c3884225-2803-4ea6-8f09-96e0696ac6d3\") " Oct 30 02:17:03.391930 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:03.391837 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d888789f-78c5-4606-94e1-7fcc77a6c16e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d888789f-78c5-4606-94e1-7fcc77a6c16e-plugins-config-volume\") pod \"d888789f-78c5-4606-94e1-7fcc77a6c16e\" (UID: \"d888789f-78c5-4606-94e1-7fcc77a6c16e\") " Oct 30 02:17:03.391930 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:03.391877 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3884225-2803-4ea6-8f09-96e0696ac6d3-kube-api-access-rgd56\" (UniqueName: \"kubernetes.io/projected/c3884225-2803-4ea6-8f09-96e0696ac6d3-kube-api-access-rgd56\") pod \"c3884225-2803-4ea6-8f09-96e0696ac6d3\" (UID: \"c3884225-2803-4ea6-8f09-96e0696ac6d3\") " Oct 30 02:17:03.392134 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:03.392077 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d888789f-78c5-4606-94e1-7fcc77a6c16e-kube-api-access-tfzw7\" (UniqueName: \"kubernetes.io/projected/d888789f-78c5-4606-94e1-7fcc77a6c16e-kube-api-access-tfzw7\") pod \"d888789f-78c5-4606-94e1-7fcc77a6c16e\" (UID: \"d888789f-78c5-4606-94e1-7fcc77a6c16e\") " Oct 30 02:17:03.398792 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:03.392233 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3884225-2803-4ea6-8f09-96e0696ac6d3-plugins-config-volume" pod "c3884225-2803-4ea6-8f09-96e0696ac6d3" (UID: "c3884225-2803-4ea6-8f09-96e0696ac6d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:17:03.398792 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:03.392653 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3884225-2803-4ea6-8f09-96e0696ac6d3-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:03.398792 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:03.392707 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d888789f-78c5-4606-94e1-7fcc77a6c16e-plugins-config-volume" pod "d888789f-78c5-4606-94e1-7fcc77a6c16e" (UID: "d888789f-78c5-4606-94e1-7fcc77a6c16e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:17:03.398792 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:03.394875 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3884225-2803-4ea6-8f09-96e0696ac6d3-kube-api-access-rgd56" pod "c3884225-2803-4ea6-8f09-96e0696ac6d3" (UID: "c3884225-2803-4ea6-8f09-96e0696ac6d3"). InnerVolumeSpecName "kube-api-access-rgd56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:03.398792 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:03.395453 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d888789f-78c5-4606-94e1-7fcc77a6c16e-kube-api-access-tfzw7" pod "d888789f-78c5-4606-94e1-7fcc77a6c16e" (UID: "d888789f-78c5-4606-94e1-7fcc77a6c16e"). InnerVolumeSpecName "kube-api-access-tfzw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:03.494649 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:03.494567 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfzw7\" (UniqueName: \"kubernetes.io/projected/d888789f-78c5-4606-94e1-7fcc77a6c16e-kube-api-access-tfzw7\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:03.494649 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:03.494609 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d888789f-78c5-4606-94e1-7fcc77a6c16e-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:03.494649 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:03.494627 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgd56\" (UniqueName: \"kubernetes.io/projected/c3884225-2803-4ea6-8f09-96e0696ac6d3-kube-api-access-rgd56\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:03.595839 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:03.595804 6572 generic.go:356] "Generic (PLEG): container finished" podID="d888789f-78c5-4606-94e1-7fcc77a6c16e" containerID="0ef5b3485d8e8aadbeec2fb15675471d134d8610db80cf70182b00a3d54e661d" exitCode=0 Oct 30 02:17:03.595839 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:03.595841 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-803472-5ffb57c959-qll7m" Oct 30 02:17:03.596100 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:03.595883 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-803472-5ffb57c959-qll7m" event={"ID":"d888789f-78c5-4606-94e1-7fcc77a6c16e","Type":"ContainerDied","Data":"0ef5b3485d8e8aadbeec2fb15675471d134d8610db80cf70182b00a3d54e661d"} Oct 30 02:17:03.596100 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:03.595938 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-803472-5ffb57c959-qll7m" event={"ID":"d888789f-78c5-4606-94e1-7fcc77a6c16e","Type":"ContainerDied","Data":"3b3235c7d646fca355e22ba8156fc7f8b33224829c739f052a5341c91021fddd"} Oct 30 02:17:03.596100 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:03.595957 6572 scope.go:122] "RemoveContainer" containerID="0ef5b3485d8e8aadbeec2fb15675471d134d8610db80cf70182b00a3d54e661d" Oct 30 02:17:03.598116 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:03.598085 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f89697-8lcv9" event={"ID":"0ac6e981-474c-4232-afcc-2b72ad4835f7","Type":"ContainerStarted","Data":"ecacd790474ae75b0b020d56c888b5ff60330a19707faf85921190f71733bb6e"} Oct 30 02:17:03.598259 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:03.598136 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f89697-8lcv9" event={"ID":"0ac6e981-474c-4232-afcc-2b72ad4835f7","Type":"ContainerStarted","Data":"cfd9d1970a299e7fbb73888b7a57ae48522d1d577cc70e53b67dcb4d7f1a2aca"} Oct 30 02:17:03.599865 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:03.599838 6572 generic.go:356] "Generic (PLEG): container finished" podID="c3884225-2803-4ea6-8f09-96e0696ac6d3" containerID="163c7455042f57e14b1363a3d980771807048ab9a07ea33452bc91145117d08a" exitCode=0 Oct 30 02:17:03.600004 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:03.599875 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-803472-d7dc9c774-4cr8j" Oct 30 02:17:03.600004 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:03.599911 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-803472-d7dc9c774-4cr8j" event={"ID":"c3884225-2803-4ea6-8f09-96e0696ac6d3","Type":"ContainerDied","Data":"163c7455042f57e14b1363a3d980771807048ab9a07ea33452bc91145117d08a"} Oct 30 02:17:03.600004 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:03.599954 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-803472-d7dc9c774-4cr8j" event={"ID":"c3884225-2803-4ea6-8f09-96e0696ac6d3","Type":"ContainerDied","Data":"340f4879b9233e22d067db22a77f30c140361c3fc9175140e78ce6adc3a298ad"} Oct 30 02:17:03.610651 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:03.610629 6572 scope.go:122] "RemoveContainer" containerID="0ef5b3485d8e8aadbeec2fb15675471d134d8610db80cf70182b00a3d54e661d" Oct 30 02:17:03.610993 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:17:03.610971 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ef5b3485d8e8aadbeec2fb15675471d134d8610db80cf70182b00a3d54e661d\": container with ID starting with 0ef5b3485d8e8aadbeec2fb15675471d134d8610db80cf70182b00a3d54e661d not found: ID does not exist" containerID="0ef5b3485d8e8aadbeec2fb15675471d134d8610db80cf70182b00a3d54e661d" Oct 30 02:17:03.611066 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:03.611001 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ef5b3485d8e8aadbeec2fb15675471d134d8610db80cf70182b00a3d54e661d"} err="failed to get container status \"0ef5b3485d8e8aadbeec2fb15675471d134d8610db80cf70182b00a3d54e661d\": rpc error: code = NotFound desc = could not find container \"0ef5b3485d8e8aadbeec2fb15675471d134d8610db80cf70182b00a3d54e661d\": container with ID starting with 0ef5b3485d8e8aadbeec2fb15675471d134d8610db80cf70182b00a3d54e661d not found: ID does not exist" Oct 30 02:17:03.611066 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:03.611018 6572 scope.go:122] "RemoveContainer" containerID="163c7455042f57e14b1363a3d980771807048ab9a07ea33452bc91145117d08a" Oct 30 02:17:03.625193 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:03.625170 6572 scope.go:122] "RemoveContainer" containerID="163c7455042f57e14b1363a3d980771807048ab9a07ea33452bc91145117d08a" Oct 30 02:17:03.625470 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:17:03.625452 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"163c7455042f57e14b1363a3d980771807048ab9a07ea33452bc91145117d08a\": container with ID starting with 163c7455042f57e14b1363a3d980771807048ab9a07ea33452bc91145117d08a not found: ID does not exist" containerID="163c7455042f57e14b1363a3d980771807048ab9a07ea33452bc91145117d08a" Oct 30 02:17:03.625519 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:03.625479 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"163c7455042f57e14b1363a3d980771807048ab9a07ea33452bc91145117d08a"} err="failed to get container status \"163c7455042f57e14b1363a3d980771807048ab9a07ea33452bc91145117d08a\": rpc error: code = NotFound desc = could not find container \"163c7455042f57e14b1363a3d980771807048ab9a07ea33452bc91145117d08a\": container with ID starting with 163c7455042f57e14b1363a3d980771807048ab9a07ea33452bc91145117d08a not found: ID does not exist" Oct 30 02:17:03.648135 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:03.648084 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f89697-8lcv9" podStartSLOduration=2.648070002 podStartE2EDuration="2.648070002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:17:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:17:03.646315675 +0000 UTC m=+3549.222741078" watchObservedRunningTime="2026-10-30 02:17:03.648070002 +0000 UTC m=+3549.224495397" Oct 30 02:17:03.685689 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:03.685653 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-803472-5ffb57c959-qll7m"] Oct 30 02:17:03.689612 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:03.689583 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-803472-5ffb57c959-qll7m"] Oct 30 02:17:03.735187 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:03.735136 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-803472-d7dc9c774-4cr8j"] Oct 30 02:17:03.739291 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:03.739261 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-803472-d7dc9c774-4cr8j"] Oct 30 02:17:04.073584 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:04.073549 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:04.307663 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:04.307624 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-264bdd-65d6c7bf46-cq56s"] Oct 30 02:17:04.528727 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:04.528690 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:04.573185 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:04.573151 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3884225-2803-4ea6-8f09-96e0696ac6d3" path="/var/lib/kubelet/pods/c3884225-2803-4ea6-8f09-96e0696ac6d3/volumes" Oct 30 02:17:04.574093 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:04.574066 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d888789f-78c5-4606-94e1-7fcc77a6c16e" path="/var/lib/kubelet/pods/d888789f-78c5-4606-94e1-7fcc77a6c16e/volumes" Oct 30 02:17:04.615162 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:04.615131 6572 generic.go:356] "Generic (PLEG): container finished" podID="0ac6e981-474c-4232-afcc-2b72ad4835f7" containerID="ecacd790474ae75b0b020d56c888b5ff60330a19707faf85921190f71733bb6e" exitCode=0 Oct 30 02:17:04.615341 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:04.615221 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f89697-8lcv9" event={"ID":"0ac6e981-474c-4232-afcc-2b72ad4835f7","Type":"ContainerDied","Data":"ecacd790474ae75b0b020d56c888b5ff60330a19707faf85921190f71733bb6e"} Oct 30 02:17:04.711333 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:04.711295 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:05.052567 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:05.052529 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-22e99d-6c8f7f8c7f-pbfcx"] Oct 30 02:17:05.233120 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:05.233083 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:05.417420 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:05.417385 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:05.668734 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:05.659037 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:05.923541 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:05.923472 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f89697-8lcv9" Oct 30 02:17:06.061324 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.061291 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ac6e981-474c-4232-afcc-2b72ad4835f7-kube-api-access-9vwjc\" (UniqueName: \"kubernetes.io/projected/0ac6e981-474c-4232-afcc-2b72ad4835f7-kube-api-access-9vwjc\") pod \"0ac6e981-474c-4232-afcc-2b72ad4835f7\" (UID: \"0ac6e981-474c-4232-afcc-2b72ad4835f7\") " Oct 30 02:17:06.061515 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.061404 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ac6e981-474c-4232-afcc-2b72ad4835f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ac6e981-474c-4232-afcc-2b72ad4835f7-maas-api-xks-ca\") pod \"0ac6e981-474c-4232-afcc-2b72ad4835f7\" (UID: \"0ac6e981-474c-4232-afcc-2b72ad4835f7\") " Oct 30 02:17:06.061600 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.061586 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ac6e981-474c-4232-afcc-2b72ad4835f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ac6e981-474c-4232-afcc-2b72ad4835f7-maas-api-openshift-ca\") pod \"0ac6e981-474c-4232-afcc-2b72ad4835f7\" (UID: \"0ac6e981-474c-4232-afcc-2b72ad4835f7\") " Oct 30 02:17:06.061991 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.061959 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ac6e981-474c-4232-afcc-2b72ad4835f7-maas-api-openshift-ca" pod "0ac6e981-474c-4232-afcc-2b72ad4835f7" (UID: "0ac6e981-474c-4232-afcc-2b72ad4835f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:17:06.062262 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.062235 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ac6e981-474c-4232-afcc-2b72ad4835f7-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:06.063543 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.063518 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ac6e981-474c-4232-afcc-2b72ad4835f7-maas-api-xks-ca" pod "0ac6e981-474c-4232-afcc-2b72ad4835f7" (UID: "0ac6e981-474c-4232-afcc-2b72ad4835f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:17:06.063656 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.063639 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ac6e981-474c-4232-afcc-2b72ad4835f7-kube-api-access-9vwjc" pod "0ac6e981-474c-4232-afcc-2b72ad4835f7" (UID: "0ac6e981-474c-4232-afcc-2b72ad4835f7"). InnerVolumeSpecName "kube-api-access-9vwjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:06.163507 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.163471 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vwjc\" (UniqueName: \"kubernetes.io/projected/0ac6e981-474c-4232-afcc-2b72ad4835f7-kube-api-access-9vwjc\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:06.163507 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.163502 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ac6e981-474c-4232-afcc-2b72ad4835f7-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:06.720624 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.720564 6572 generic.go:356] "Generic (PLEG): container finished" podID="1147e056-c134-4bd0-8c0f-19de2e33eead" containerID="1f0b6965d1cee50ed6490374ca948a7d8b74666b0e6d63385c139e3fd39b2261" exitCode=0 Oct 30 02:17:06.721214 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.720646 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" event={"ID":"1147e056-c134-4bd0-8c0f-19de2e33eead","Type":"ContainerDied","Data":"1f0b6965d1cee50ed6490374ca948a7d8b74666b0e6d63385c139e3fd39b2261"} Oct 30 02:17:06.721214 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.720715 6572 scope.go:122] "RemoveContainer" containerID="c2eb3bb8f543d76951b0ba49b3b61bfbbd9ddb686d3e939d0a1a72b452844742" Oct 30 02:17:06.723085 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.723060 6572 generic.go:356] "Generic (PLEG): container finished" podID="364b9686-9847-4ef4-a5b1-7f611a248b64" containerID="90ea182817356e05150d77f344aaa6ec82d66a25f3c17c8ed6afec9995a0d2e1" exitCode=0 Oct 30 02:17:06.723219 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.723135 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" event={"ID":"364b9686-9847-4ef4-a5b1-7f611a248b64","Type":"ContainerDied","Data":"90ea182817356e05150d77f344aaa6ec82d66a25f3c17c8ed6afec9995a0d2e1"} Oct 30 02:17:06.725054 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.725033 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f89697-8lcv9" Oct 30 02:17:06.728932 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.725065 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f89697-8lcv9" event={"ID":"0ac6e981-474c-4232-afcc-2b72ad4835f7","Type":"ContainerDied","Data":"cfd9d1970a299e7fbb73888b7a57ae48522d1d577cc70e53b67dcb4d7f1a2aca"} Oct 30 02:17:06.728932 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.725116 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfd9d1970a299e7fbb73888b7a57ae48522d1d577cc70e53b67dcb4d7f1a2aca" Oct 30 02:17:06.856440 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.856404 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" Oct 30 02:17:06.950691 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.950658 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:06.966075 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.966041 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:17:06.975718 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.975687 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/364b9686-9847-4ef4-a5b1-7f611a248b64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/364b9686-9847-4ef4-a5b1-7f611a248b64-workload-socket\") pod \"364b9686-9847-4ef4-a5b1-7f611a248b64\" (UID: \"364b9686-9847-4ef4-a5b1-7f611a248b64\") " Oct 30 02:17:06.975902 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.975759 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/364b9686-9847-4ef4-a5b1-7f611a248b64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/364b9686-9847-4ef4-a5b1-7f611a248b64-istio-data\") pod \"364b9686-9847-4ef4-a5b1-7f611a248b64\" (UID: \"364b9686-9847-4ef4-a5b1-7f611a248b64\") " Oct 30 02:17:06.975902 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.975796 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/364b9686-9847-4ef4-a5b1-7f611a248b64-kube-api-access-wzd9b\" (UniqueName: \"kubernetes.io/projected/364b9686-9847-4ef4-a5b1-7f611a248b64-kube-api-access-wzd9b\") pod \"364b9686-9847-4ef4-a5b1-7f611a248b64\" (UID: \"364b9686-9847-4ef4-a5b1-7f611a248b64\") " Oct 30 02:17:06.975902 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.975825 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/364b9686-9847-4ef4-a5b1-7f611a248b64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/364b9686-9847-4ef4-a5b1-7f611a248b64-credential-socket\") pod \"364b9686-9847-4ef4-a5b1-7f611a248b64\" (UID: \"364b9686-9847-4ef4-a5b1-7f611a248b64\") " Oct 30 02:17:06.976079 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.976025 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/364b9686-9847-4ef4-a5b1-7f611a248b64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/364b9686-9847-4ef4-a5b1-7f611a248b64-workload-certs\") pod \"364b9686-9847-4ef4-a5b1-7f611a248b64\" (UID: \"364b9686-9847-4ef4-a5b1-7f611a248b64\") " Oct 30 02:17:06.976079 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.976016 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/364b9686-9847-4ef4-a5b1-7f611a248b64-workload-socket" pod "364b9686-9847-4ef4-a5b1-7f611a248b64" (UID: "364b9686-9847-4ef4-a5b1-7f611a248b64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:06.976199 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.976070 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/364b9686-9847-4ef4-a5b1-7f611a248b64-istio-data" pod "364b9686-9847-4ef4-a5b1-7f611a248b64" (UID: "364b9686-9847-4ef4-a5b1-7f611a248b64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:06.976199 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.976136 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/364b9686-9847-4ef4-a5b1-7f611a248b64-istio-token\" (UniqueName: \"kubernetes.io/projected/364b9686-9847-4ef4-a5b1-7f611a248b64-istio-token\") pod \"364b9686-9847-4ef4-a5b1-7f611a248b64\" (UID: \"364b9686-9847-4ef4-a5b1-7f611a248b64\") " Oct 30 02:17:06.976199 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.976152 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/364b9686-9847-4ef4-a5b1-7f611a248b64-credential-socket" pod "364b9686-9847-4ef4-a5b1-7f611a248b64" (UID: "364b9686-9847-4ef4-a5b1-7f611a248b64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:06.976356 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.976257 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/364b9686-9847-4ef4-a5b1-7f611a248b64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/364b9686-9847-4ef4-a5b1-7f611a248b64-istio-envoy\") pod \"364b9686-9847-4ef4-a5b1-7f611a248b64\" (UID: \"364b9686-9847-4ef4-a5b1-7f611a248b64\") " Oct 30 02:17:06.976356 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.976296 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/364b9686-9847-4ef4-a5b1-7f611a248b64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/364b9686-9847-4ef4-a5b1-7f611a248b64-istiod-ca-cert\") pod \"364b9686-9847-4ef4-a5b1-7f611a248b64\" (UID: \"364b9686-9847-4ef4-a5b1-7f611a248b64\") " Oct 30 02:17:06.976356 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.976259 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/364b9686-9847-4ef4-a5b1-7f611a248b64-workload-certs" pod "364b9686-9847-4ef4-a5b1-7f611a248b64" (UID: "364b9686-9847-4ef4-a5b1-7f611a248b64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:06.976512 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.976425 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/364b9686-9847-4ef4-a5b1-7f611a248b64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/364b9686-9847-4ef4-a5b1-7f611a248b64-istio-podinfo\") pod \"364b9686-9847-4ef4-a5b1-7f611a248b64\" (UID: \"364b9686-9847-4ef4-a5b1-7f611a248b64\") " Oct 30 02:17:06.976719 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.976698 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/364b9686-9847-4ef4-a5b1-7f611a248b64-istiod-ca-cert" pod "364b9686-9847-4ef4-a5b1-7f611a248b64" (UID: "364b9686-9847-4ef4-a5b1-7f611a248b64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:17:06.977099 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.977081 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/364b9686-9847-4ef4-a5b1-7f611a248b64-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:06.977189 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.977101 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/364b9686-9847-4ef4-a5b1-7f611a248b64-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:06.977189 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.977110 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/364b9686-9847-4ef4-a5b1-7f611a248b64-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:06.977189 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.977118 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/364b9686-9847-4ef4-a5b1-7f611a248b64-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:06.977189 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.977126 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/364b9686-9847-4ef4-a5b1-7f611a248b64-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:06.978687 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.978660 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/364b9686-9847-4ef4-a5b1-7f611a248b64-istio-envoy" pod "364b9686-9847-4ef4-a5b1-7f611a248b64" (UID: "364b9686-9847-4ef4-a5b1-7f611a248b64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:06.978822 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.978717 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/364b9686-9847-4ef4-a5b1-7f611a248b64-kube-api-access-wzd9b" pod "364b9686-9847-4ef4-a5b1-7f611a248b64" (UID: "364b9686-9847-4ef4-a5b1-7f611a248b64"). InnerVolumeSpecName "kube-api-access-wzd9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:06.978822 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.978741 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/364b9686-9847-4ef4-a5b1-7f611a248b64-istio-token" pod "364b9686-9847-4ef4-a5b1-7f611a248b64" (UID: "364b9686-9847-4ef4-a5b1-7f611a248b64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:06.978822 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:06.978783 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/364b9686-9847-4ef4-a5b1-7f611a248b64-istio-podinfo" pod "364b9686-9847-4ef4-a5b1-7f611a248b64" (UID: "364b9686-9847-4ef4-a5b1-7f611a248b64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:17:07.033388 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.033343 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76881e36-25w2k"] Oct 30 02:17:07.037621 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.037583 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76881e36-25w2k"] Oct 30 02:17:07.078083 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.078043 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1147e056-c134-4bd0-8c0f-19de2e33eead-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1147e056-c134-4bd0-8c0f-19de2e33eead-credential-socket\") pod \"1147e056-c134-4bd0-8c0f-19de2e33eead\" (UID: \"1147e056-c134-4bd0-8c0f-19de2e33eead\") " Oct 30 02:17:07.078298 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.078097 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1147e056-c134-4bd0-8c0f-19de2e33eead-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1147e056-c134-4bd0-8c0f-19de2e33eead-istio-data\") pod \"1147e056-c134-4bd0-8c0f-19de2e33eead\" (UID: \"1147e056-c134-4bd0-8c0f-19de2e33eead\") " Oct 30 02:17:07.078298 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.078179 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1147e056-c134-4bd0-8c0f-19de2e33eead-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1147e056-c134-4bd0-8c0f-19de2e33eead-workload-socket\") pod \"1147e056-c134-4bd0-8c0f-19de2e33eead\" (UID: \"1147e056-c134-4bd0-8c0f-19de2e33eead\") " Oct 30 02:17:07.078298 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.078233 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1147e056-c134-4bd0-8c0f-19de2e33eead-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1147e056-c134-4bd0-8c0f-19de2e33eead-istiod-ca-cert\") pod \"1147e056-c134-4bd0-8c0f-19de2e33eead\" (UID: \"1147e056-c134-4bd0-8c0f-19de2e33eead\") " Oct 30 02:17:07.078298 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.078267 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1147e056-c134-4bd0-8c0f-19de2e33eead-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1147e056-c134-4bd0-8c0f-19de2e33eead-istio-envoy\") pod \"1147e056-c134-4bd0-8c0f-19de2e33eead\" (UID: \"1147e056-c134-4bd0-8c0f-19de2e33eead\") " Oct 30 02:17:07.078590 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.078350 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1147e056-c134-4bd0-8c0f-19de2e33eead-istio-token\" (UniqueName: \"kubernetes.io/projected/1147e056-c134-4bd0-8c0f-19de2e33eead-istio-token\") pod \"1147e056-c134-4bd0-8c0f-19de2e33eead\" (UID: \"1147e056-c134-4bd0-8c0f-19de2e33eead\") " Oct 30 02:17:07.078590 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.078382 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1147e056-c134-4bd0-8c0f-19de2e33eead-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1147e056-c134-4bd0-8c0f-19de2e33eead-workload-certs\") pod \"1147e056-c134-4bd0-8c0f-19de2e33eead\" (UID: \"1147e056-c134-4bd0-8c0f-19de2e33eead\") " Oct 30 02:17:07.078590 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.078386 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1147e056-c134-4bd0-8c0f-19de2e33eead-credential-socket" pod "1147e056-c134-4bd0-8c0f-19de2e33eead" (UID: "1147e056-c134-4bd0-8c0f-19de2e33eead"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:07.078590 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.078445 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1147e056-c134-4bd0-8c0f-19de2e33eead-kube-api-access-dvk9g\" (UniqueName: \"kubernetes.io/projected/1147e056-c134-4bd0-8c0f-19de2e33eead-kube-api-access-dvk9g\") pod \"1147e056-c134-4bd0-8c0f-19de2e33eead\" (UID: \"1147e056-c134-4bd0-8c0f-19de2e33eead\") " Oct 30 02:17:07.078590 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.078564 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1147e056-c134-4bd0-8c0f-19de2e33eead-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1147e056-c134-4bd0-8c0f-19de2e33eead-istio-podinfo\") pod \"1147e056-c134-4bd0-8c0f-19de2e33eead\" (UID: \"1147e056-c134-4bd0-8c0f-19de2e33eead\") " Oct 30 02:17:07.078875 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.078449 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1147e056-c134-4bd0-8c0f-19de2e33eead-istio-data" pod "1147e056-c134-4bd0-8c0f-19de2e33eead" (UID: "1147e056-c134-4bd0-8c0f-19de2e33eead"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:07.078875 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.078584 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1147e056-c134-4bd0-8c0f-19de2e33eead-workload-socket" pod "1147e056-c134-4bd0-8c0f-19de2e33eead" (UID: "1147e056-c134-4bd0-8c0f-19de2e33eead"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:07.078875 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.078676 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1147e056-c134-4bd0-8c0f-19de2e33eead-istiod-ca-cert" pod "1147e056-c134-4bd0-8c0f-19de2e33eead" (UID: "1147e056-c134-4bd0-8c0f-19de2e33eead"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:17:07.079034 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.078882 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1147e056-c134-4bd0-8c0f-19de2e33eead-workload-certs" pod "1147e056-c134-4bd0-8c0f-19de2e33eead" (UID: "1147e056-c134-4bd0-8c0f-19de2e33eead"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:07.079261 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.079236 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzd9b\" (UniqueName: \"kubernetes.io/projected/364b9686-9847-4ef4-a5b1-7f611a248b64-kube-api-access-wzd9b\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:07.079372 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.079266 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1147e056-c134-4bd0-8c0f-19de2e33eead-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:07.079372 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.079283 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1147e056-c134-4bd0-8c0f-19de2e33eead-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:07.079372 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.079298 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/364b9686-9847-4ef4-a5b1-7f611a248b64-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:07.079372 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.079313 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1147e056-c134-4bd0-8c0f-19de2e33eead-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:07.079372 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.079326 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/364b9686-9847-4ef4-a5b1-7f611a248b64-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:07.079372 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.079339 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1147e056-c134-4bd0-8c0f-19de2e33eead-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:07.079372 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.079351 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1147e056-c134-4bd0-8c0f-19de2e33eead-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:07.079372 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.079364 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/364b9686-9847-4ef4-a5b1-7f611a248b64-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:07.080735 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.080710 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1147e056-c134-4bd0-8c0f-19de2e33eead-istio-envoy" pod "1147e056-c134-4bd0-8c0f-19de2e33eead" (UID: "1147e056-c134-4bd0-8c0f-19de2e33eead"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:07.080879 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.080712 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1147e056-c134-4bd0-8c0f-19de2e33eead-istio-token" pod "1147e056-c134-4bd0-8c0f-19de2e33eead" (UID: "1147e056-c134-4bd0-8c0f-19de2e33eead"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:07.080879 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.080794 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1147e056-c134-4bd0-8c0f-19de2e33eead-kube-api-access-dvk9g" pod "1147e056-c134-4bd0-8c0f-19de2e33eead" (UID: "1147e056-c134-4bd0-8c0f-19de2e33eead"). InnerVolumeSpecName "kube-api-access-dvk9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:07.080879 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.080814 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1147e056-c134-4bd0-8c0f-19de2e33eead-istio-podinfo" pod "1147e056-c134-4bd0-8c0f-19de2e33eead" (UID: "1147e056-c134-4bd0-8c0f-19de2e33eead"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:17:07.189846 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.183458 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1147e056-c134-4bd0-8c0f-19de2e33eead-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:07.189846 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.183500 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvk9g\" (UniqueName: \"kubernetes.io/projected/1147e056-c134-4bd0-8c0f-19de2e33eead-kube-api-access-dvk9g\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:07.189846 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.183516 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1147e056-c134-4bd0-8c0f-19de2e33eead-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:07.189846 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.183531 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1147e056-c134-4bd0-8c0f-19de2e33eead-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:07.414484 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.414451 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:07.455837 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.455731 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-264bdd-zqlhm"] Oct 30 02:17:07.459982 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.459953 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-264bdd-zqlhm"] Oct 30 02:17:07.607730 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.607696 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:07.736030 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.735933 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" Oct 30 02:17:07.736030 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.735967 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f" event={"ID":"364b9686-9847-4ef4-a5b1-7f611a248b64","Type":"ContainerDied","Data":"65792ac4d49d8f61f59f4fd740c9d681f0d7ba20936b5de2f3ed6d0bd125366d"} Oct 30 02:17:07.736030 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.736032 6572 scope.go:122] "RemoveContainer" containerID="90ea182817356e05150d77f344aaa6ec82d66a25f3c17c8ed6afec9995a0d2e1" Oct 30 02:17:07.738325 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.738287 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" event={"ID":"1147e056-c134-4bd0-8c0f-19de2e33eead","Type":"ContainerDied","Data":"53f5437e0c0bd5ab34a4b6dec802fb47130d1d363eb1a94aca70eebec279ab63"} Oct 30 02:17:07.738472 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.738301 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf" Oct 30 02:17:07.749120 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.748986 6572 scope.go:122] "RemoveContainer" containerID="1f0b6965d1cee50ed6490374ca948a7d8b74666b0e6d63385c139e3fd39b2261" Oct 30 02:17:07.787880 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.787841 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf"] Oct 30 02:17:07.792315 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.792279 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:07.799473 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.799437 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-bshgf"] Oct 30 02:17:07.823220 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.823185 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f"] Oct 30 02:17:07.829862 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:07.829827 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-6jg5f"] Oct 30 02:17:08.036686 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:08.036585 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:08.548460 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:08.548409 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1147e056-c134-4bd0-8c0f-19de2e33eead" path="/var/lib/kubelet/pods/1147e056-c134-4bd0-8c0f-19de2e33eead/volumes" Oct 30 02:17:08.549385 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:08.549353 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="364b9686-9847-4ef4-a5b1-7f611a248b64" path="/var/lib/kubelet/pods/364b9686-9847-4ef4-a5b1-7f611a248b64/volumes" Oct 30 02:17:08.550202 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:08.550178 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59ee674a-29e5-4865-a2db-15aa20fee9d7" path="/var/lib/kubelet/pods/59ee674a-29e5-4865-a2db-15aa20fee9d7/volumes" Oct 30 02:17:08.551394 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:08.551366 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb304a68-5357-4ad1-9cf4-94b3796da9c4" path="/var/lib/kubelet/pods/bb304a68-5357-4ad1-9cf4-94b3796da9c4/volumes" Oct 30 02:17:08.868901 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:08.868179 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:09.047585 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:09.047550 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:09.241199 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:09.241164 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:09.308932 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:09.308881 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-264bdd-65d6c7bf46-cq56s" podUID="6dbd7a2c-c776-496e-ac5a-6a97046ab3f7" containerName="payload-processing" containerID="cri-o://1aea154994541a3bfe5be6dda7f2b5680d88761896e8d4b8361a9b694fa7d032" gracePeriod=25 Oct 30 02:17:09.447149 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:09.447112 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:09.719876 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:09.719849 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-264bdd-65d6c7bf46-cq56s" Oct 30 02:17:09.770359 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:09.770265 6572 generic.go:356] "Generic (PLEG): container finished" podID="6dbd7a2c-c776-496e-ac5a-6a97046ab3f7" containerID="1aea154994541a3bfe5be6dda7f2b5680d88761896e8d4b8361a9b694fa7d032" exitCode=0 Oct 30 02:17:09.770556 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:09.770529 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-264bdd-65d6c7bf46-cq56s" Oct 30 02:17:09.770629 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:09.770557 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-264bdd-65d6c7bf46-cq56s" event={"ID":"6dbd7a2c-c776-496e-ac5a-6a97046ab3f7","Type":"ContainerDied","Data":"1aea154994541a3bfe5be6dda7f2b5680d88761896e8d4b8361a9b694fa7d032"} Oct 30 02:17:09.770629 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:09.770612 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-264bdd-65d6c7bf46-cq56s" event={"ID":"6dbd7a2c-c776-496e-ac5a-6a97046ab3f7","Type":"ContainerDied","Data":"c63946815fef4f4082a770fac17c9ee0dbbcdc5ef634206e0dfcbd2ab4128554"} Oct 30 02:17:09.770783 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:09.770641 6572 scope.go:122] "RemoveContainer" containerID="1aea154994541a3bfe5be6dda7f2b5680d88761896e8d4b8361a9b694fa7d032" Oct 30 02:17:09.794833 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:09.794804 6572 scope.go:122] "RemoveContainer" containerID="1aea154994541a3bfe5be6dda7f2b5680d88761896e8d4b8361a9b694fa7d032" Oct 30 02:17:09.795171 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:17:09.795147 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aea154994541a3bfe5be6dda7f2b5680d88761896e8d4b8361a9b694fa7d032\": container with ID starting with 1aea154994541a3bfe5be6dda7f2b5680d88761896e8d4b8361a9b694fa7d032 not found: ID does not exist" containerID="1aea154994541a3bfe5be6dda7f2b5680d88761896e8d4b8361a9b694fa7d032" Oct 30 02:17:09.795273 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:09.795179 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aea154994541a3bfe5be6dda7f2b5680d88761896e8d4b8361a9b694fa7d032"} err="failed to get container status \"1aea154994541a3bfe5be6dda7f2b5680d88761896e8d4b8361a9b694fa7d032\": rpc error: code = NotFound desc = could not find container \"1aea154994541a3bfe5be6dda7f2b5680d88761896e8d4b8361a9b694fa7d032\": container with ID starting with 1aea154994541a3bfe5be6dda7f2b5680d88761896e8d4b8361a9b694fa7d032 not found: ID does not exist" Oct 30 02:17:09.844998 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:09.844967 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dbd7a2c-c776-496e-ac5a-6a97046ab3f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dbd7a2c-c776-496e-ac5a-6a97046ab3f7-plugins-config-volume\") pod \"6dbd7a2c-c776-496e-ac5a-6a97046ab3f7\" (UID: \"6dbd7a2c-c776-496e-ac5a-6a97046ab3f7\") " Oct 30 02:17:09.845238 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:09.845222 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dbd7a2c-c776-496e-ac5a-6a97046ab3f7-kube-api-access-fsbxl\" (UniqueName: \"kubernetes.io/projected/6dbd7a2c-c776-496e-ac5a-6a97046ab3f7-kube-api-access-fsbxl\") pod \"6dbd7a2c-c776-496e-ac5a-6a97046ab3f7\" (UID: \"6dbd7a2c-c776-496e-ac5a-6a97046ab3f7\") " Oct 30 02:17:09.845376 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:09.845350 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dbd7a2c-c776-496e-ac5a-6a97046ab3f7-plugins-config-volume" pod "6dbd7a2c-c776-496e-ac5a-6a97046ab3f7" (UID: "6dbd7a2c-c776-496e-ac5a-6a97046ab3f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:17:09.845890 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:09.845870 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dbd7a2c-c776-496e-ac5a-6a97046ab3f7-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:09.847657 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:09.847628 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dbd7a2c-c776-496e-ac5a-6a97046ab3f7-kube-api-access-fsbxl" pod "6dbd7a2c-c776-496e-ac5a-6a97046ab3f7" (UID: "6dbd7a2c-c776-496e-ac5a-6a97046ab3f7"). InnerVolumeSpecName "kube-api-access-fsbxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:09.947641 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:09.947598 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsbxl\" (UniqueName: \"kubernetes.io/projected/6dbd7a2c-c776-496e-ac5a-6a97046ab3f7-kube-api-access-fsbxl\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:10.053507 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:10.053408 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22e99d-6c8f7f8c7f-pbfcx" podUID="1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a" containerName="payload-pre-processing" containerID="cri-o://89958f4d886480638e260c469a27e7f3101c6984c5b79052808339a90db17663" gracePeriod=25 Oct 30 02:17:10.116908 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:10.116868 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-264bdd-65d6c7bf46-cq56s"] Oct 30 02:17:10.128093 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:10.128057 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-264bdd-65d6c7bf46-cq56s"] Oct 30 02:17:10.163434 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:10.163404 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:10.545126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:10.545094 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dbd7a2c-c776-496e-ac5a-6a97046ab3f7" path="/var/lib/kubelet/pods/6dbd7a2c-c776-496e-ac5a-6a97046ab3f7/volumes" Oct 30 02:17:12.346477 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:12.346438 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:12.543982 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:12.543947 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:13.019603 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:13.019560 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22e99d-6c8f7f8c7f-pbfcx" podUID="1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:17:13.672999 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:13.672962 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:14.413772 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:14.413691 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:14.617454 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:14.617414 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:14.688153 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:17:14.688064 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:17:14.842628 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:14.836501 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:15.160370 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.160329 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:15.343536 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.343498 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:15.369648 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.369616 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eda881-qqklx"] Oct 30 02:17:15.370286 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.370268 6572 cpu_manager.go:414] "containerMap: removing container" podUID="6dbd7a2c-c776-496e-ac5a-6a97046ab3f7" containerName="payload-processing" Oct 30 02:17:15.370386 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.370288 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dbd7a2c-c776-496e-ac5a-6a97046ab3f7" containerName="payload-processing" Oct 30 02:17:15.370386 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.370302 6572 cpu_manager.go:414] "containerMap: removing container" podUID="364b9686-9847-4ef4-a5b1-7f611a248b64" containerName="istio-proxy" Oct 30 02:17:15.370386 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.370308 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="364b9686-9847-4ef4-a5b1-7f611a248b64" containerName="istio-proxy" Oct 30 02:17:15.370386 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.370319 6572 cpu_manager.go:414] "containerMap: removing container" podUID="0ac6e981-474c-4232-afcc-2b72ad4835f7" containerName="revoke-keys" Oct 30 02:17:15.370386 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.370325 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ac6e981-474c-4232-afcc-2b72ad4835f7" containerName="revoke-keys" Oct 30 02:17:15.370386 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.370335 6572 cpu_manager.go:414] "containerMap: removing container" podUID="1147e056-c134-4bd0-8c0f-19de2e33eead" containerName="istio-proxy" Oct 30 02:17:15.370386 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.370339 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1147e056-c134-4bd0-8c0f-19de2e33eead" containerName="istio-proxy" Oct 30 02:17:15.370386 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.370367 6572 cpu_manager.go:414] "containerMap: removing container" podUID="d888789f-78c5-4606-94e1-7fcc77a6c16e" containerName="payload-pre-processing" Oct 30 02:17:15.370386 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.370372 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d888789f-78c5-4606-94e1-7fcc77a6c16e" containerName="payload-pre-processing" Oct 30 02:17:15.370386 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.370387 6572 cpu_manager.go:414] "containerMap: removing container" podUID="c3884225-2803-4ea6-8f09-96e0696ac6d3" containerName="payload-processing" Oct 30 02:17:15.370386 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.370392 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3884225-2803-4ea6-8f09-96e0696ac6d3" containerName="payload-processing" Oct 30 02:17:15.370986 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.370400 6572 cpu_manager.go:414] "containerMap: removing container" podUID="1147e056-c134-4bd0-8c0f-19de2e33eead" containerName="istio-proxy" Oct 30 02:17:15.370986 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.370405 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1147e056-c134-4bd0-8c0f-19de2e33eead" containerName="istio-proxy" Oct 30 02:17:15.370986 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.370501 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dbd7a2c-c776-496e-ac5a-6a97046ab3f7" containerName="payload-processing" Oct 30 02:17:15.370986 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.370507 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="364b9686-9847-4ef4-a5b1-7f611a248b64" containerName="istio-proxy" Oct 30 02:17:15.370986 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.370515 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ac6e981-474c-4232-afcc-2b72ad4835f7" containerName="revoke-keys" Oct 30 02:17:15.370986 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.370522 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="1147e056-c134-4bd0-8c0f-19de2e33eead" containerName="istio-proxy" Oct 30 02:17:15.370986 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.370532 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="d888789f-78c5-4606-94e1-7fcc77a6c16e" containerName="payload-pre-processing" Oct 30 02:17:15.370986 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.370540 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3884225-2803-4ea6-8f09-96e0696ac6d3" containerName="payload-processing" Oct 30 02:17:15.375504 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.375482 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eda881-qqklx" Oct 30 02:17:15.391996 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.391958 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eda881-qqklx"] Oct 30 02:17:15.397240 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.397221 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:17:15.397471 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.397437 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:17:15.397609 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.397222 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:17:15.402182 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.402162 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:17:15.465681 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.465593 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/685f53b7-90dd-46cd-8d51-abc73d483531-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-eda881-qqklx\" (UID: \"685f53b7-90dd-46cd-8d51-abc73d483531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eda881-qqklx" Oct 30 02:17:15.465681 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.465633 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjn6g\" (UniqueName: \"kubernetes.io/projected/685f53b7-90dd-46cd-8d51-abc73d483531-kube-api-access-tjn6g\") pod \"maas-api-revoke-keys-e2e-worker-w2-eda881-qqklx\" (UID: \"685f53b7-90dd-46cd-8d51-abc73d483531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eda881-qqklx" Oct 30 02:17:15.465926 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.465788 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/685f53b7-90dd-46cd-8d51-abc73d483531-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-eda881-qqklx\" (UID: \"685f53b7-90dd-46cd-8d51-abc73d483531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eda881-qqklx" Oct 30 02:17:15.567223 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.567188 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/685f53b7-90dd-46cd-8d51-abc73d483531-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-eda881-qqklx\" (UID: \"685f53b7-90dd-46cd-8d51-abc73d483531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eda881-qqklx" Oct 30 02:17:15.567474 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.567444 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/685f53b7-90dd-46cd-8d51-abc73d483531-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-eda881-qqklx\" (UID: \"685f53b7-90dd-46cd-8d51-abc73d483531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eda881-qqklx" Oct 30 02:17:15.567561 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.567489 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjn6g\" (UniqueName: \"kubernetes.io/projected/685f53b7-90dd-46cd-8d51-abc73d483531-kube-api-access-tjn6g\") pod \"maas-api-revoke-keys-e2e-worker-w2-eda881-qqklx\" (UID: \"685f53b7-90dd-46cd-8d51-abc73d483531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eda881-qqklx" Oct 30 02:17:15.568004 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.567980 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/685f53b7-90dd-46cd-8d51-abc73d483531-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-eda881-qqklx\" (UID: \"685f53b7-90dd-46cd-8d51-abc73d483531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eda881-qqklx" Oct 30 02:17:15.569886 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.569867 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/685f53b7-90dd-46cd-8d51-abc73d483531-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-eda881-qqklx\" (UID: \"685f53b7-90dd-46cd-8d51-abc73d483531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eda881-qqklx" Oct 30 02:17:15.585192 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.585170 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjn6g\" (UniqueName: \"kubernetes.io/projected/685f53b7-90dd-46cd-8d51-abc73d483531-kube-api-access-tjn6g\") pod \"maas-api-revoke-keys-e2e-worker-w2-eda881-qqklx\" (UID: \"685f53b7-90dd-46cd-8d51-abc73d483531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eda881-qqklx" Oct 30 02:17:15.691000 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.690958 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eda881-qqklx" Oct 30 02:17:15.989941 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:15.989904 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eda881-qqklx"] Oct 30 02:17:15.990268 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:17:15.990245 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod685f53b7_90dd_46cd_8d51_abc73d483531.slice/crio-c03f981825f392eab8610b5c477e5e80f8387bc9df7d46bfb7426809cf0956fc WatchSource:0}: Error finding container c03f981825f392eab8610b5c477e5e80f8387bc9df7d46bfb7426809cf0956fc: Status 404 returned error can't find the container with id c03f981825f392eab8610b5c477e5e80f8387bc9df7d46bfb7426809cf0956fc Oct 30 02:17:16.371428 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:16.371333 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:16.565408 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:16.565383 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:16.892563 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:16.892530 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eda881-qqklx" event={"ID":"685f53b7-90dd-46cd-8d51-abc73d483531","Type":"ContainerStarted","Data":"7ab1be6e736f865fe10a51d74846f202d7db143c8387d31f0e3d5cc9711ed397"} Oct 30 02:17:16.893000 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:16.892590 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eda881-qqklx" event={"ID":"685f53b7-90dd-46cd-8d51-abc73d483531","Type":"ContainerStarted","Data":"c03f981825f392eab8610b5c477e5e80f8387bc9df7d46bfb7426809cf0956fc"} Oct 30 02:17:16.931705 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:16.931656 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eda881-qqklx" podStartSLOduration=1.931644854 podStartE2EDuration="1.931644854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:17:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:17:16.931336564 +0000 UTC m=+3562.507761965" watchObservedRunningTime="2026-10-30 02:17:16.931644854 +0000 UTC m=+3562.508070255" Oct 30 02:17:16.992567 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:16.992531 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:17.032548 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:17.032514 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f89697-8lcv9"] Oct 30 02:17:17.039185 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:17.039149 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f89697-8lcv9"] Oct 30 02:17:17.201309 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:17.201271 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:17.584971 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:17.584875 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:17.902342 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:17.902256 6572 generic.go:356] "Generic (PLEG): container finished" podID="685f53b7-90dd-46cd-8d51-abc73d483531" containerID="7ab1be6e736f865fe10a51d74846f202d7db143c8387d31f0e3d5cc9711ed397" exitCode=0 Oct 30 02:17:17.902719 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:17.902341 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eda881-qqklx" event={"ID":"685f53b7-90dd-46cd-8d51-abc73d483531","Type":"ContainerDied","Data":"7ab1be6e736f865fe10a51d74846f202d7db143c8387d31f0e3d5cc9711ed397"} Oct 30 02:17:18.545882 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:18.545842 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ac6e981-474c-4232-afcc-2b72ad4835f7" path="/var/lib/kubelet/pods/0ac6e981-474c-4232-afcc-2b72ad4835f7/volumes" Oct 30 02:17:18.581944 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:18.581911 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:19.030851 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:19.030818 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9f4cb55-hj454"] Oct 30 02:17:19.038848 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:19.038823 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9f4cb55-hj454"] Oct 30 02:17:19.102610 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:19.102580 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eda881-qqklx" Oct 30 02:17:19.118147 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:19.118116 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:19.129997 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:19.129971 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-803472-rwvsc"] Oct 30 02:17:19.133582 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:19.133555 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-803472-rwvsc"] Oct 30 02:17:19.225813 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:19.221222 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/685f53b7-90dd-46cd-8d51-abc73d483531-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/685f53b7-90dd-46cd-8d51-abc73d483531-maas-api-openshift-ca\") pod \"685f53b7-90dd-46cd-8d51-abc73d483531\" (UID: \"685f53b7-90dd-46cd-8d51-abc73d483531\") " Oct 30 02:17:19.225813 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:19.221392 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/685f53b7-90dd-46cd-8d51-abc73d483531-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/685f53b7-90dd-46cd-8d51-abc73d483531-maas-api-xks-ca\") pod \"685f53b7-90dd-46cd-8d51-abc73d483531\" (UID: \"685f53b7-90dd-46cd-8d51-abc73d483531\") " Oct 30 02:17:19.225813 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:19.221441 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/685f53b7-90dd-46cd-8d51-abc73d483531-kube-api-access-tjn6g\" (UniqueName: \"kubernetes.io/projected/685f53b7-90dd-46cd-8d51-abc73d483531-kube-api-access-tjn6g\") pod \"685f53b7-90dd-46cd-8d51-abc73d483531\" (UID: \"685f53b7-90dd-46cd-8d51-abc73d483531\") " Oct 30 02:17:19.225813 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:19.225197 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/685f53b7-90dd-46cd-8d51-abc73d483531-maas-api-openshift-ca" pod "685f53b7-90dd-46cd-8d51-abc73d483531" (UID: "685f53b7-90dd-46cd-8d51-abc73d483531"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:17:19.243291 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:19.242348 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/685f53b7-90dd-46cd-8d51-abc73d483531-maas-api-xks-ca" pod "685f53b7-90dd-46cd-8d51-abc73d483531" (UID: "685f53b7-90dd-46cd-8d51-abc73d483531"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:17:19.247954 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:19.247844 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/685f53b7-90dd-46cd-8d51-abc73d483531-kube-api-access-tjn6g" pod "685f53b7-90dd-46cd-8d51-abc73d483531" (UID: "685f53b7-90dd-46cd-8d51-abc73d483531"). InnerVolumeSpecName "kube-api-access-tjn6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:19.305322 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:19.305289 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:19.323699 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:19.323671 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/685f53b7-90dd-46cd-8d51-abc73d483531-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:19.323699 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:19.323697 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/685f53b7-90dd-46cd-8d51-abc73d483531-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:19.323891 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:19.323707 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjn6g\" (UniqueName: \"kubernetes.io/projected/685f53b7-90dd-46cd-8d51-abc73d483531-kube-api-access-tjn6g\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:19.694420 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:19.694389 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:19.945449 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:19.945368 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eda881-qqklx" Oct 30 02:17:19.945449 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:19.945400 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eda881-qqklx" event={"ID":"685f53b7-90dd-46cd-8d51-abc73d483531","Type":"ContainerDied","Data":"c03f981825f392eab8610b5c477e5e80f8387bc9df7d46bfb7426809cf0956fc"} Oct 30 02:17:19.945449 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:19.945444 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c03f981825f392eab8610b5c477e5e80f8387bc9df7d46bfb7426809cf0956fc" Oct 30 02:17:20.165774 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:20.160857 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n"] Oct 30 02:17:20.191908 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:20.191873 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd"] Oct 30 02:17:20.373138 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:20.373037 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:20.544040 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:20.544006 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ad57d82-5011-4542-9301-3f042e876dde" path="/var/lib/kubelet/pods/1ad57d82-5011-4542-9301-3f042e876dde/volumes" Oct 30 02:17:20.544522 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:20.544497 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53436d95-4e4a-4d93-b47b-629c4861dac5" path="/var/lib/kubelet/pods/53436d95-4e4a-4d93-b47b-629c4861dac5/volumes" Oct 30 02:17:20.940275 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:20.931895 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:22.164157 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:22.164122 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:22.586308 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:22.586214 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:22.858609 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:22.858511 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:22.881235 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:22.881200 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22e99d-vjz8p"] Oct 30 02:17:22.892911 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:22.892887 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22e99d-vjz8p"] Oct 30 02:17:23.020515 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:23.020473 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22e99d-6c8f7f8c7f-pbfcx" podUID="1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:17:23.074894 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:23.074854 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:23.388838 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:23.388805 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:24.324366 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:24.324328 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:24.523412 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:24.523382 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:24.548493 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:24.548457 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f84693b2-6f55-4fae-a301-2cd285e7001b" path="/var/lib/kubelet/pods/f84693b2-6f55-4fae-a301-2cd285e7001b/volumes" Oct 30 02:17:24.548893 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:24.548872 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5"] Oct 30 02:17:24.549310 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:24.549281 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" podUID="166123b7-cdef-49d7-8e6c-2a2a993b9bb8" containerName="istio-proxy" containerID="cri-o://1edaf94c752d86ca6cb8f395b33fcd7f8ba41d4f91cb5bc654b4cbc9c6d1fd4e" gracePeriod=30 Oct 30 02:17:24.718457 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:24.718422 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:25.120543 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:25.120452 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-601d4b-4vzjt"] Oct 30 02:17:25.121292 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:25.121266 6572 cpu_manager.go:414] "containerMap: removing container" podUID="685f53b7-90dd-46cd-8d51-abc73d483531" containerName="revoke-keys" Oct 30 02:17:25.121292 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:25.121290 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="685f53b7-90dd-46cd-8d51-abc73d483531" containerName="revoke-keys" Oct 30 02:17:25.121553 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:25.121535 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="685f53b7-90dd-46cd-8d51-abc73d483531" containerName="revoke-keys" Oct 30 02:17:25.121553 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:25.121553 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="1147e056-c134-4bd0-8c0f-19de2e33eead" containerName="istio-proxy" Oct 30 02:17:25.126867 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:25.126844 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-601d4b-4vzjt" Oct 30 02:17:25.141416 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:25.141394 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:17:25.141581 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:25.141557 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:17:25.141797 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:25.141640 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:17:25.141797 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:25.141483 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:17:25.145052 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:25.145029 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-601d4b-4vzjt"] Oct 30 02:17:25.232533 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:25.232502 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82bd130d-77a4-414d-8fac-57e607a0d1b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-601d4b-4vzjt\" (UID: \"82bd130d-77a4-414d-8fac-57e607a0d1b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-601d4b-4vzjt" Oct 30 02:17:25.232704 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:25.232542 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82bd130d-77a4-414d-8fac-57e607a0d1b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-601d4b-4vzjt\" (UID: \"82bd130d-77a4-414d-8fac-57e607a0d1b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-601d4b-4vzjt" Oct 30 02:17:25.232704 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:25.232645 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbz9q\" (UniqueName: \"kubernetes.io/projected/82bd130d-77a4-414d-8fac-57e607a0d1b6-kube-api-access-jbz9q\") pod \"maas-api-revoke-keys-e2e-inf-b-601d4b-4vzjt\" (UID: \"82bd130d-77a4-414d-8fac-57e607a0d1b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-601d4b-4vzjt" Oct 30 02:17:25.334111 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:25.334081 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82bd130d-77a4-414d-8fac-57e607a0d1b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-601d4b-4vzjt\" (UID: \"82bd130d-77a4-414d-8fac-57e607a0d1b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-601d4b-4vzjt" Oct 30 02:17:25.334270 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:25.334120 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82bd130d-77a4-414d-8fac-57e607a0d1b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-601d4b-4vzjt\" (UID: \"82bd130d-77a4-414d-8fac-57e607a0d1b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-601d4b-4vzjt" Oct 30 02:17:25.334270 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:25.334229 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbz9q\" (UniqueName: \"kubernetes.io/projected/82bd130d-77a4-414d-8fac-57e607a0d1b6-kube-api-access-jbz9q\") pod \"maas-api-revoke-keys-e2e-inf-b-601d4b-4vzjt\" (UID: \"82bd130d-77a4-414d-8fac-57e607a0d1b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-601d4b-4vzjt" Oct 30 02:17:25.334831 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:25.334810 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82bd130d-77a4-414d-8fac-57e607a0d1b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-601d4b-4vzjt\" (UID: \"82bd130d-77a4-414d-8fac-57e607a0d1b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-601d4b-4vzjt" Oct 30 02:17:25.336350 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:25.336328 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82bd130d-77a4-414d-8fac-57e607a0d1b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-601d4b-4vzjt\" (UID: \"82bd130d-77a4-414d-8fac-57e607a0d1b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-601d4b-4vzjt" Oct 30 02:17:25.353574 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:25.353544 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbz9q\" (UniqueName: \"kubernetes.io/projected/82bd130d-77a4-414d-8fac-57e607a0d1b6-kube-api-access-jbz9q\") pod \"maas-api-revoke-keys-e2e-inf-b-601d4b-4vzjt\" (UID: \"82bd130d-77a4-414d-8fac-57e607a0d1b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-601d4b-4vzjt" Oct 30 02:17:25.440927 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:25.440901 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-601d4b-4vzjt" Oct 30 02:17:25.745506 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:17:25.745476 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82bd130d_77a4_414d_8fac_57e607a0d1b6.slice/crio-ba10488d4e620063096c05f6946d534e35da43874cffc6d9ef11e2c455397c7c WatchSource:0}: Error finding container ba10488d4e620063096c05f6946d534e35da43874cffc6d9ef11e2c455397c7c: Status 404 returned error can't find the container with id ba10488d4e620063096c05f6946d534e35da43874cffc6d9ef11e2c455397c7c Oct 30 02:17:25.746137 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:25.746038 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-601d4b-4vzjt"] Oct 30 02:17:25.794723 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:25.794695 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-eda881-5c7cd4d9b-hqnjh"] Oct 30 02:17:25.830056 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:25.830031 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-eda881-67d94b78bc-6n9dq"] Oct 30 02:17:25.920929 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:25.920895 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:26.003939 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:26.003853 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-601d4b-4vzjt" event={"ID":"82bd130d-77a4-414d-8fac-57e607a0d1b6","Type":"ContainerStarted","Data":"ba10488d4e620063096c05f6946d534e35da43874cffc6d9ef11e2c455397c7c"} Oct 30 02:17:26.140768 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:26.140409 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:26.142362 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:26.142334 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk"] Oct 30 02:17:26.142543 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:26.142521 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" podUID="f4f3636d-3177-48b9-88a1-14c250c18bba" containerName="istio-proxy" containerID="cri-o://dbb780afbb7c78259bc02247888634c5e49991148e4f036b8be5f7af7003edac" gracePeriod=30 Oct 30 02:17:26.343864 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:26.343775 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:27.014223 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:27.014123 6572 generic.go:356] "Generic (PLEG): container finished" podID="82bd130d-77a4-414d-8fac-57e607a0d1b6" containerID="beff4ab4e00f4555bcdb265d0f6e0206d77bddd2fa444e6d6c18522b5a04a45b" exitCode=0 Oct 30 02:17:27.014223 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:27.014165 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-601d4b-4vzjt" event={"ID":"82bd130d-77a4-414d-8fac-57e607a0d1b6","Type":"ContainerDied","Data":"beff4ab4e00f4555bcdb265d0f6e0206d77bddd2fa444e6d6c18522b5a04a45b"} Oct 30 02:17:27.203714 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:27.203673 6572 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:17:27.203940 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:27.203772 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" podUID="f4f3636d-3177-48b9-88a1-14c250c18bba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:17:27.654353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:27.654314 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:28.182838 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:28.182817 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-601d4b-4vzjt" Oct 30 02:17:28.280861 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:28.280788 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82bd130d-77a4-414d-8fac-57e607a0d1b6-kube-api-access-jbz9q\" (UniqueName: \"kubernetes.io/projected/82bd130d-77a4-414d-8fac-57e607a0d1b6-kube-api-access-jbz9q\") pod \"82bd130d-77a4-414d-8fac-57e607a0d1b6\" (UID: \"82bd130d-77a4-414d-8fac-57e607a0d1b6\") " Oct 30 02:17:28.281003 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:28.280978 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82bd130d-77a4-414d-8fac-57e607a0d1b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82bd130d-77a4-414d-8fac-57e607a0d1b6-maas-api-openshift-ca\") pod \"82bd130d-77a4-414d-8fac-57e607a0d1b6\" (UID: \"82bd130d-77a4-414d-8fac-57e607a0d1b6\") " Oct 30 02:17:28.281079 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:28.281063 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82bd130d-77a4-414d-8fac-57e607a0d1b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82bd130d-77a4-414d-8fac-57e607a0d1b6-maas-api-xks-ca\") pod \"82bd130d-77a4-414d-8fac-57e607a0d1b6\" (UID: \"82bd130d-77a4-414d-8fac-57e607a0d1b6\") " Oct 30 02:17:28.281309 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:28.281286 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82bd130d-77a4-414d-8fac-57e607a0d1b6-maas-api-openshift-ca" pod "82bd130d-77a4-414d-8fac-57e607a0d1b6" (UID: "82bd130d-77a4-414d-8fac-57e607a0d1b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:17:28.281572 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:28.281558 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82bd130d-77a4-414d-8fac-57e607a0d1b6-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:28.283022 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:28.282996 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82bd130d-77a4-414d-8fac-57e607a0d1b6-maas-api-xks-ca" pod "82bd130d-77a4-414d-8fac-57e607a0d1b6" (UID: "82bd130d-77a4-414d-8fac-57e607a0d1b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:17:28.283135 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:28.283068 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82bd130d-77a4-414d-8fac-57e607a0d1b6-kube-api-access-jbz9q" pod "82bd130d-77a4-414d-8fac-57e607a0d1b6" (UID: "82bd130d-77a4-414d-8fac-57e607a0d1b6"). InnerVolumeSpecName "kube-api-access-jbz9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:28.382352 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:28.382320 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbz9q\" (UniqueName: \"kubernetes.io/projected/82bd130d-77a4-414d-8fac-57e607a0d1b6-kube-api-access-jbz9q\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:28.382352 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:28.382350 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82bd130d-77a4-414d-8fac-57e607a0d1b6-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:29.034253 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:29.034223 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-601d4b-4vzjt" Oct 30 02:17:29.034462 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:29.034259 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-601d4b-4vzjt" event={"ID":"82bd130d-77a4-414d-8fac-57e607a0d1b6","Type":"ContainerDied","Data":"ba10488d4e620063096c05f6946d534e35da43874cffc6d9ef11e2c455397c7c"} Oct 30 02:17:29.034462 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:29.034319 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba10488d4e620063096c05f6946d534e35da43874cffc6d9ef11e2c455397c7c" Oct 30 02:17:29.088676 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:29.088640 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:29.606010 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:29.605978 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:29.840591 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:29.840500 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:29.882496 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:29.882468 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:17:30.078479 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.067848 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:30.087350 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.087244 6572 generic.go:356] "Generic (PLEG): container finished" podID="166123b7-cdef-49d7-8e6c-2a2a993b9bb8" containerID="1edaf94c752d86ca6cb8f395b33fcd7f8ba41d4f91cb5bc654b4cbc9c6d1fd4e" exitCode=0 Oct 30 02:17:30.087909 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.087851 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" Oct 30 02:17:30.088294 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.088258 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" event={"ID":"166123b7-cdef-49d7-8e6c-2a2a993b9bb8","Type":"ContainerDied","Data":"1edaf94c752d86ca6cb8f395b33fcd7f8ba41d4f91cb5bc654b4cbc9c6d1fd4e"} Oct 30 02:17:30.088570 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.088510 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5" event={"ID":"166123b7-cdef-49d7-8e6c-2a2a993b9bb8","Type":"ContainerDied","Data":"1aa9835bd0feaf2a602684ab6b4cee6a47897e6a79212c74abf012018bbb1722"} Oct 30 02:17:30.088729 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.088710 6572 scope.go:122] "RemoveContainer" containerID="1edaf94c752d86ca6cb8f395b33fcd7f8ba41d4f91cb5bc654b4cbc9c6d1fd4e" Oct 30 02:17:30.105875 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.105841 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-istiod-ca-cert\") pod \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\" (UID: \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\") " Oct 30 02:17:30.106088 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.106005 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-istio-envoy\") pod \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\" (UID: \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\") " Oct 30 02:17:30.106088 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.106046 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-workload-certs\") pod \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\" (UID: \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\") " Oct 30 02:17:30.106088 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.106064 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-istio-token\" (UniqueName: \"kubernetes.io/projected/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-istio-token\") pod \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\" (UID: \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\") " Oct 30 02:17:30.106278 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.106107 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-istio-data\") pod \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\" (UID: \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\") " Oct 30 02:17:30.106278 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.106156 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-istio-podinfo\") pod \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\" (UID: \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\") " Oct 30 02:17:30.106278 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.106187 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-kube-api-access-fn9pf\" (UniqueName: \"kubernetes.io/projected/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-kube-api-access-fn9pf\") pod \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\" (UID: \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\") " Oct 30 02:17:30.106278 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.106218 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-credential-socket\") pod \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\" (UID: \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\") " Oct 30 02:17:30.106278 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.106237 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-workload-socket\") pod \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\" (UID: \"166123b7-cdef-49d7-8e6c-2a2a993b9bb8\") " Oct 30 02:17:30.106875 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.106846 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-workload-socket" pod "166123b7-cdef-49d7-8e6c-2a2a993b9bb8" (UID: "166123b7-cdef-49d7-8e6c-2a2a993b9bb8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:30.107187 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.107161 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-istiod-ca-cert" pod "166123b7-cdef-49d7-8e6c-2a2a993b9bb8" (UID: "166123b7-cdef-49d7-8e6c-2a2a993b9bb8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:17:30.107285 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.107258 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-workload-certs" pod "166123b7-cdef-49d7-8e6c-2a2a993b9bb8" (UID: "166123b7-cdef-49d7-8e6c-2a2a993b9bb8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:30.107381 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.107363 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-istio-data" pod "166123b7-cdef-49d7-8e6c-2a2a993b9bb8" (UID: "166123b7-cdef-49d7-8e6c-2a2a993b9bb8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:30.107430 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.107389 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-credential-socket" pod "166123b7-cdef-49d7-8e6c-2a2a993b9bb8" (UID: "166123b7-cdef-49d7-8e6c-2a2a993b9bb8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:30.109678 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.109637 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-kube-api-access-fn9pf" pod "166123b7-cdef-49d7-8e6c-2a2a993b9bb8" (UID: "166123b7-cdef-49d7-8e6c-2a2a993b9bb8"). InnerVolumeSpecName "kube-api-access-fn9pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:30.109874 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.109841 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-istio-podinfo" pod "166123b7-cdef-49d7-8e6c-2a2a993b9bb8" (UID: "166123b7-cdef-49d7-8e6c-2a2a993b9bb8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:17:30.109963 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.109904 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-istio-envoy" pod "166123b7-cdef-49d7-8e6c-2a2a993b9bb8" (UID: "166123b7-cdef-49d7-8e6c-2a2a993b9bb8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:30.110050 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.110033 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-istio-token" pod "166123b7-cdef-49d7-8e6c-2a2a993b9bb8" (UID: "166123b7-cdef-49d7-8e6c-2a2a993b9bb8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:30.171531 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.171507 6572 scope.go:122] "RemoveContainer" containerID="1b1062c93a39d9f873c2611743f1ba3423beb5cc68ddf14d4b57baa66761cd63" Oct 30 02:17:30.184729 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.184684 6572 scope.go:122] "RemoveContainer" containerID="1edaf94c752d86ca6cb8f395b33fcd7f8ba41d4f91cb5bc654b4cbc9c6d1fd4e" Oct 30 02:17:30.185192 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:17:30.185157 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1edaf94c752d86ca6cb8f395b33fcd7f8ba41d4f91cb5bc654b4cbc9c6d1fd4e\": container with ID starting with 1edaf94c752d86ca6cb8f395b33fcd7f8ba41d4f91cb5bc654b4cbc9c6d1fd4e not found: ID does not exist" containerID="1edaf94c752d86ca6cb8f395b33fcd7f8ba41d4f91cb5bc654b4cbc9c6d1fd4e" Oct 30 02:17:30.185309 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.185192 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1edaf94c752d86ca6cb8f395b33fcd7f8ba41d4f91cb5bc654b4cbc9c6d1fd4e"} err="failed to get container status \"1edaf94c752d86ca6cb8f395b33fcd7f8ba41d4f91cb5bc654b4cbc9c6d1fd4e\": rpc error: code = NotFound desc = could not find container \"1edaf94c752d86ca6cb8f395b33fcd7f8ba41d4f91cb5bc654b4cbc9c6d1fd4e\": container with ID starting with 1edaf94c752d86ca6cb8f395b33fcd7f8ba41d4f91cb5bc654b4cbc9c6d1fd4e not found: ID does not exist" Oct 30 02:17:30.185309 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.185213 6572 scope.go:122] "RemoveContainer" containerID="1b1062c93a39d9f873c2611743f1ba3423beb5cc68ddf14d4b57baa66761cd63" Oct 30 02:17:30.185534 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:17:30.185516 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b1062c93a39d9f873c2611743f1ba3423beb5cc68ddf14d4b57baa66761cd63\": container with ID starting with 1b1062c93a39d9f873c2611743f1ba3423beb5cc68ddf14d4b57baa66761cd63 not found: ID does not exist" containerID="1b1062c93a39d9f873c2611743f1ba3423beb5cc68ddf14d4b57baa66761cd63" Oct 30 02:17:30.185598 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.185539 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b1062c93a39d9f873c2611743f1ba3423beb5cc68ddf14d4b57baa66761cd63"} err="failed to get container status \"1b1062c93a39d9f873c2611743f1ba3423beb5cc68ddf14d4b57baa66761cd63\": rpc error: code = NotFound desc = could not find container \"1b1062c93a39d9f873c2611743f1ba3423beb5cc68ddf14d4b57baa66761cd63\": container with ID starting with 1b1062c93a39d9f873c2611743f1ba3423beb5cc68ddf14d4b57baa66761cd63 not found: ID does not exist" Oct 30 02:17:30.208510 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.208461 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:30.208510 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.208505 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:30.208510 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.208520 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn9pf\" (UniqueName: \"kubernetes.io/projected/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-kube-api-access-fn9pf\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:30.208913 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.208539 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:30.208913 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.208553 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:30.208913 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.208569 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:30.208913 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.208583 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:30.208913 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.208596 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:30.208913 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.208609 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/166123b7-cdef-49d7-8e6c-2a2a993b9bb8-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:30.322819 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.322782 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22e99d-6c8f7f8c7f-pbfcx" Oct 30 02:17:30.395142 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.395043 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d44413-vtt7z"] Oct 30 02:17:30.396077 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.396051 6572 cpu_manager.go:414] "containerMap: removing container" podUID="82bd130d-77a4-414d-8fac-57e607a0d1b6" containerName="revoke-keys" Oct 30 02:17:30.396167 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.396081 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82bd130d-77a4-414d-8fac-57e607a0d1b6" containerName="revoke-keys" Oct 30 02:17:30.396167 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.396109 6572 cpu_manager.go:414] "containerMap: removing container" podUID="166123b7-cdef-49d7-8e6c-2a2a993b9bb8" containerName="istio-proxy" Oct 30 02:17:30.396167 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.396118 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="166123b7-cdef-49d7-8e6c-2a2a993b9bb8" containerName="istio-proxy" Oct 30 02:17:30.396167 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.396151 6572 cpu_manager.go:414] "containerMap: removing container" podUID="1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a" containerName="payload-pre-processing" Oct 30 02:17:30.396167 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.396160 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a" containerName="payload-pre-processing" Oct 30 02:17:30.396437 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.396210 6572 cpu_manager.go:414] "containerMap: removing container" podUID="166123b7-cdef-49d7-8e6c-2a2a993b9bb8" containerName="istio-proxy" Oct 30 02:17:30.396437 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.396219 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="166123b7-cdef-49d7-8e6c-2a2a993b9bb8" containerName="istio-proxy" Oct 30 02:17:30.396437 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.396376 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a" containerName="payload-pre-processing" Oct 30 02:17:30.396437 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.396398 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="166123b7-cdef-49d7-8e6c-2a2a993b9bb8" containerName="istio-proxy" Oct 30 02:17:30.396437 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.396413 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="82bd130d-77a4-414d-8fac-57e607a0d1b6" containerName="revoke-keys" Oct 30 02:17:30.396437 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.396425 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="166123b7-cdef-49d7-8e6c-2a2a993b9bb8" containerName="istio-proxy" Oct 30 02:17:30.400485 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.400460 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d44413-vtt7z" Oct 30 02:17:30.419500 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.419462 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d44413-vtt7z"] Oct 30 02:17:30.419673 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.419618 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:17:30.419779 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.419712 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:17:30.421144 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.421114 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a-kube-api-access-7hnh6\" (UniqueName: \"kubernetes.io/projected/1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a-kube-api-access-7hnh6\") pod \"1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a\" (UID: \"1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a\") " Oct 30 02:17:30.421297 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.421158 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a-plugins-config-volume\") pod \"1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a\" (UID: \"1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a\") " Oct 30 02:17:30.422219 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.422196 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a-plugins-config-volume" pod "1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a" (UID: "1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:17:30.424098 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.424072 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a-kube-api-access-7hnh6" pod "1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a" (UID: "1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a"). InnerVolumeSpecName "kube-api-access-7hnh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:30.438740 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.438701 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:17:30.439106 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.438798 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:17:30.489359 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.489314 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5"] Oct 30 02:17:30.495778 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.495726 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-7s8n5"] Oct 30 02:17:30.523448 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.523408 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e3680ff-1ee0-4f4d-ada9-325766ae0a97-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d44413-vtt7z\" (UID: \"9e3680ff-1ee0-4f4d-ada9-325766ae0a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d44413-vtt7z" Oct 30 02:17:30.523681 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.523554 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztf64\" (UniqueName: \"kubernetes.io/projected/9e3680ff-1ee0-4f4d-ada9-325766ae0a97-kube-api-access-ztf64\") pod \"maas-api-revoke-keys-e2e-extmod-a-d44413-vtt7z\" (UID: \"9e3680ff-1ee0-4f4d-ada9-325766ae0a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d44413-vtt7z" Oct 30 02:17:30.523681 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.523633 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e3680ff-1ee0-4f4d-ada9-325766ae0a97-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d44413-vtt7z\" (UID: \"9e3680ff-1ee0-4f4d-ada9-325766ae0a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d44413-vtt7z" Oct 30 02:17:30.523948 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.523922 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hnh6\" (UniqueName: \"kubernetes.io/projected/1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a-kube-api-access-7hnh6\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:30.523993 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.523955 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:30.556809 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.552414 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="166123b7-cdef-49d7-8e6c-2a2a993b9bb8" path="/var/lib/kubelet/pods/166123b7-cdef-49d7-8e6c-2a2a993b9bb8/volumes" Oct 30 02:17:30.600142 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.599922 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:30.625738 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.625691 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e3680ff-1ee0-4f4d-ada9-325766ae0a97-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d44413-vtt7z\" (UID: \"9e3680ff-1ee0-4f4d-ada9-325766ae0a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d44413-vtt7z" Oct 30 02:17:30.626284 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.625832 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztf64\" (UniqueName: \"kubernetes.io/projected/9e3680ff-1ee0-4f4d-ada9-325766ae0a97-kube-api-access-ztf64\") pod \"maas-api-revoke-keys-e2e-extmod-a-d44413-vtt7z\" (UID: \"9e3680ff-1ee0-4f4d-ada9-325766ae0a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d44413-vtt7z" Oct 30 02:17:30.626284 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.625882 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e3680ff-1ee0-4f4d-ada9-325766ae0a97-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d44413-vtt7z\" (UID: \"9e3680ff-1ee0-4f4d-ada9-325766ae0a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d44413-vtt7z" Oct 30 02:17:30.626764 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.626723 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e3680ff-1ee0-4f4d-ada9-325766ae0a97-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d44413-vtt7z\" (UID: \"9e3680ff-1ee0-4f4d-ada9-325766ae0a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d44413-vtt7z" Oct 30 02:17:30.628412 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.628387 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e3680ff-1ee0-4f4d-ada9-325766ae0a97-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d44413-vtt7z\" (UID: \"9e3680ff-1ee0-4f4d-ada9-325766ae0a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d44413-vtt7z" Oct 30 02:17:30.648684 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.648586 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztf64\" (UniqueName: \"kubernetes.io/projected/9e3680ff-1ee0-4f4d-ada9-325766ae0a97-kube-api-access-ztf64\") pod \"maas-api-revoke-keys-e2e-extmod-a-d44413-vtt7z\" (UID: \"9e3680ff-1ee0-4f4d-ada9-325766ae0a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d44413-vtt7z" Oct 30 02:17:30.724585 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.724542 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d44413-vtt7z" Oct 30 02:17:30.795732 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.795550 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-eda881-5c7cd4d9b-hqnjh" podUID="2de76834-8ea2-4e53-8e91-ebd04ae69886" containerName="payload-processing" containerID="cri-o://ef418e27639b8c25bc243d572d9415a6b4434f39f26edd2ce9aabbf9af948479" gracePeriod=25 Oct 30 02:17:30.838150 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:30.836689 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eda881-67d94b78bc-6n9dq" podUID="6d6c050f-1131-4874-b064-3ef68d3b990b" containerName="payload-pre-processing" containerID="cri-o://6737b3a464cecce2a8a42b64ad22a0549c0a53b74b0a127f19e40fb215c6e941" gracePeriod=25 Oct 30 02:17:31.027942 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.027905 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df3d431b-hflx7"] Oct 30 02:17:31.031352 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.031328 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df3d431b-hflx7"] Oct 30 02:17:31.040813 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:17:31.040783 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e3680ff_1ee0_4f4d_ada9_325766ae0a97.slice/crio-af1aa4684d055e7e1a5825f18dd29d164a0a7adc0f3a2e88507782ab1541dfda WatchSource:0}: Error finding container af1aa4684d055e7e1a5825f18dd29d164a0a7adc0f3a2e88507782ab1541dfda: Status 404 returned error can't find the container with id af1aa4684d055e7e1a5825f18dd29d164a0a7adc0f3a2e88507782ab1541dfda Oct 30 02:17:31.041674 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.041646 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d44413-vtt7z"] Oct 30 02:17:31.097813 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.097773 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d44413-vtt7z" event={"ID":"9e3680ff-1ee0-4f4d-ada9-325766ae0a97","Type":"ContainerStarted","Data":"af1aa4684d055e7e1a5825f18dd29d164a0a7adc0f3a2e88507782ab1541dfda"} Oct 30 02:17:31.100449 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.100421 6572 generic.go:356] "Generic (PLEG): container finished" podID="1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a" containerID="89958f4d886480638e260c469a27e7f3101c6984c5b79052808339a90db17663" exitCode=0 Oct 30 02:17:31.100626 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.100479 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22e99d-6c8f7f8c7f-pbfcx" event={"ID":"1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a","Type":"ContainerDied","Data":"89958f4d886480638e260c469a27e7f3101c6984c5b79052808339a90db17663"} Oct 30 02:17:31.100626 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.100500 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22e99d-6c8f7f8c7f-pbfcx" Oct 30 02:17:31.100626 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.100516 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22e99d-6c8f7f8c7f-pbfcx" event={"ID":"1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a","Type":"ContainerDied","Data":"f053131fdfb324c855bbac137e18dee00e2f7886aa6bccb14ff2d957d0a0ccdd"} Oct 30 02:17:31.100626 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.100537 6572 scope.go:122] "RemoveContainer" containerID="89958f4d886480638e260c469a27e7f3101c6984c5b79052808339a90db17663" Oct 30 02:17:31.154176 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.154145 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-22e99d-6c8f7f8c7f-pbfcx"] Oct 30 02:17:31.159468 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.159440 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-22e99d-6c8f7f8c7f-pbfcx"] Oct 30 02:17:31.179913 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.179889 6572 scope.go:122] "RemoveContainer" containerID="89958f4d886480638e260c469a27e7f3101c6984c5b79052808339a90db17663" Oct 30 02:17:31.180281 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:17:31.180257 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89958f4d886480638e260c469a27e7f3101c6984c5b79052808339a90db17663\": container with ID starting with 89958f4d886480638e260c469a27e7f3101c6984c5b79052808339a90db17663 not found: ID does not exist" containerID="89958f4d886480638e260c469a27e7f3101c6984c5b79052808339a90db17663" Oct 30 02:17:31.180361 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.180294 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89958f4d886480638e260c469a27e7f3101c6984c5b79052808339a90db17663"} err="failed to get container status \"89958f4d886480638e260c469a27e7f3101c6984c5b79052808339a90db17663\": rpc error: code = NotFound desc = could not find container \"89958f4d886480638e260c469a27e7f3101c6984c5b79052808339a90db17663\": container with ID starting with 89958f4d886480638e260c469a27e7f3101c6984c5b79052808339a90db17663 not found: ID does not exist" Oct 30 02:17:31.399109 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.399080 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 02:17:31.541239 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.541195 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f3636d-3177-48b9-88a1-14c250c18bba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f3636d-3177-48b9-88a1-14c250c18bba-istio-envoy\") pod \"f4f3636d-3177-48b9-88a1-14c250c18bba\" (UID: \"f4f3636d-3177-48b9-88a1-14c250c18bba\") " Oct 30 02:17:31.541239 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.541242 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4f3636d-3177-48b9-88a1-14c250c18bba-kube-api-access-9z4hf\" (UniqueName: \"kubernetes.io/projected/f4f3636d-3177-48b9-88a1-14c250c18bba-kube-api-access-9z4hf\") pod \"f4f3636d-3177-48b9-88a1-14c250c18bba\" (UID: \"f4f3636d-3177-48b9-88a1-14c250c18bba\") " Oct 30 02:17:31.541505 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.541287 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4f3636d-3177-48b9-88a1-14c250c18bba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f3636d-3177-48b9-88a1-14c250c18bba-istiod-ca-cert\") pod \"f4f3636d-3177-48b9-88a1-14c250c18bba\" (UID: \"f4f3636d-3177-48b9-88a1-14c250c18bba\") " Oct 30 02:17:31.541505 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.541321 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4f3636d-3177-48b9-88a1-14c250c18bba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f3636d-3177-48b9-88a1-14c250c18bba-istio-podinfo\") pod \"f4f3636d-3177-48b9-88a1-14c250c18bba\" (UID: \"f4f3636d-3177-48b9-88a1-14c250c18bba\") " Oct 30 02:17:31.541505 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.541358 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f3636d-3177-48b9-88a1-14c250c18bba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f3636d-3177-48b9-88a1-14c250c18bba-workload-certs\") pod \"f4f3636d-3177-48b9-88a1-14c250c18bba\" (UID: \"f4f3636d-3177-48b9-88a1-14c250c18bba\") " Oct 30 02:17:31.541505 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.541407 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f3636d-3177-48b9-88a1-14c250c18bba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f3636d-3177-48b9-88a1-14c250c18bba-credential-socket\") pod \"f4f3636d-3177-48b9-88a1-14c250c18bba\" (UID: \"f4f3636d-3177-48b9-88a1-14c250c18bba\") " Oct 30 02:17:31.541505 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.541471 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4f3636d-3177-48b9-88a1-14c250c18bba-istio-token\" (UniqueName: \"kubernetes.io/projected/f4f3636d-3177-48b9-88a1-14c250c18bba-istio-token\") pod \"f4f3636d-3177-48b9-88a1-14c250c18bba\" (UID: \"f4f3636d-3177-48b9-88a1-14c250c18bba\") " Oct 30 02:17:31.541781 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.541510 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f3636d-3177-48b9-88a1-14c250c18bba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f3636d-3177-48b9-88a1-14c250c18bba-istio-data\") pod \"f4f3636d-3177-48b9-88a1-14c250c18bba\" (UID: \"f4f3636d-3177-48b9-88a1-14c250c18bba\") " Oct 30 02:17:31.541781 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.541582 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f3636d-3177-48b9-88a1-14c250c18bba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f3636d-3177-48b9-88a1-14c250c18bba-workload-socket\") pod \"f4f3636d-3177-48b9-88a1-14c250c18bba\" (UID: \"f4f3636d-3177-48b9-88a1-14c250c18bba\") " Oct 30 02:17:31.541896 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.541792 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4f3636d-3177-48b9-88a1-14c250c18bba-istiod-ca-cert" pod "f4f3636d-3177-48b9-88a1-14c250c18bba" (UID: "f4f3636d-3177-48b9-88a1-14c250c18bba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:17:31.541989 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.541961 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f3636d-3177-48b9-88a1-14c250c18bba-workload-certs" pod "f4f3636d-3177-48b9-88a1-14c250c18bba" (UID: "f4f3636d-3177-48b9-88a1-14c250c18bba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:31.542055 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.541985 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f3636d-3177-48b9-88a1-14c250c18bba-credential-socket" pod "f4f3636d-3177-48b9-88a1-14c250c18bba" (UID: "f4f3636d-3177-48b9-88a1-14c250c18bba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:31.542170 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.542147 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f3636d-3177-48b9-88a1-14c250c18bba-istio-data" pod "f4f3636d-3177-48b9-88a1-14c250c18bba" (UID: "f4f3636d-3177-48b9-88a1-14c250c18bba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:31.542242 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.542200 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f3636d-3177-48b9-88a1-14c250c18bba-workload-socket" pod "f4f3636d-3177-48b9-88a1-14c250c18bba" (UID: "f4f3636d-3177-48b9-88a1-14c250c18bba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:31.542393 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.542365 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f3636d-3177-48b9-88a1-14c250c18bba-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:31.542393 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.542389 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f3636d-3177-48b9-88a1-14c250c18bba-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:31.542535 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.542403 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f3636d-3177-48b9-88a1-14c250c18bba-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:31.542535 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.542418 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f3636d-3177-48b9-88a1-14c250c18bba-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:31.542535 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.542433 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f3636d-3177-48b9-88a1-14c250c18bba-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:31.543970 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.543916 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f3636d-3177-48b9-88a1-14c250c18bba-kube-api-access-9z4hf" pod "f4f3636d-3177-48b9-88a1-14c250c18bba" (UID: "f4f3636d-3177-48b9-88a1-14c250c18bba"). InnerVolumeSpecName "kube-api-access-9z4hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:31.543970 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.543961 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f3636d-3177-48b9-88a1-14c250c18bba-istio-token" pod "f4f3636d-3177-48b9-88a1-14c250c18bba" (UID: "f4f3636d-3177-48b9-88a1-14c250c18bba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:31.544148 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.544059 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4f3636d-3177-48b9-88a1-14c250c18bba-istio-podinfo" pod "f4f3636d-3177-48b9-88a1-14c250c18bba" (UID: "f4f3636d-3177-48b9-88a1-14c250c18bba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:17:31.544250 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.544234 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f3636d-3177-48b9-88a1-14c250c18bba-istio-envoy" pod "f4f3636d-3177-48b9-88a1-14c250c18bba" (UID: "f4f3636d-3177-48b9-88a1-14c250c18bba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:31.644349 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.644312 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f3636d-3177-48b9-88a1-14c250c18bba-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:31.644349 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.644344 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z4hf\" (UniqueName: \"kubernetes.io/projected/f4f3636d-3177-48b9-88a1-14c250c18bba-kube-api-access-9z4hf\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:31.644349 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.644354 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f3636d-3177-48b9-88a1-14c250c18bba-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:31.644844 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:31.644363 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f3636d-3177-48b9-88a1-14c250c18bba-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:32.024731 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:32.024697 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:32.111480 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:32.111446 6572 generic.go:356] "Generic (PLEG): container finished" podID="9e3680ff-1ee0-4f4d-ada9-325766ae0a97" containerID="09a0ebab352001c669eeb09ee68595167b520e5488bad754d7380cbcab88e0d8" exitCode=0 Oct 30 02:17:32.111666 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:32.111541 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d44413-vtt7z" event={"ID":"9e3680ff-1ee0-4f4d-ada9-325766ae0a97","Type":"ContainerDied","Data":"09a0ebab352001c669eeb09ee68595167b520e5488bad754d7380cbcab88e0d8"} Oct 30 02:17:32.113511 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:32.113483 6572 generic.go:356] "Generic (PLEG): container finished" podID="f4f3636d-3177-48b9-88a1-14c250c18bba" containerID="dbb780afbb7c78259bc02247888634c5e49991148e4f036b8be5f7af7003edac" exitCode=0 Oct 30 02:17:32.113610 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:32.113525 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" Oct 30 02:17:32.113610 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:32.113545 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" event={"ID":"f4f3636d-3177-48b9-88a1-14c250c18bba","Type":"ContainerDied","Data":"dbb780afbb7c78259bc02247888634c5e49991148e4f036b8be5f7af7003edac"} Oct 30 02:17:32.113610 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:32.113597 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk" event={"ID":"f4f3636d-3177-48b9-88a1-14c250c18bba","Type":"ContainerDied","Data":"4f8feef36bed139b6dda96638e6a57f41930d774e807cdb3be71d5ac01e5a33f"} Oct 30 02:17:32.113781 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:32.113620 6572 scope.go:122] "RemoveContainer" containerID="dbb780afbb7c78259bc02247888634c5e49991148e4f036b8be5f7af7003edac" Oct 30 02:17:32.125339 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:32.125317 6572 scope.go:122] "RemoveContainer" containerID="a58560fdefc97caffea04f97a78f5e744d0da3554019f5f85c813ab4a27ff796" Oct 30 02:17:32.135663 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:32.135646 6572 scope.go:122] "RemoveContainer" containerID="dbb780afbb7c78259bc02247888634c5e49991148e4f036b8be5f7af7003edac" Oct 30 02:17:32.135948 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:17:32.135929 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbb780afbb7c78259bc02247888634c5e49991148e4f036b8be5f7af7003edac\": container with ID starting with dbb780afbb7c78259bc02247888634c5e49991148e4f036b8be5f7af7003edac not found: ID does not exist" containerID="dbb780afbb7c78259bc02247888634c5e49991148e4f036b8be5f7af7003edac" Oct 30 02:17:32.136009 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:32.135957 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbb780afbb7c78259bc02247888634c5e49991148e4f036b8be5f7af7003edac"} err="failed to get container status \"dbb780afbb7c78259bc02247888634c5e49991148e4f036b8be5f7af7003edac\": rpc error: code = NotFound desc = could not find container \"dbb780afbb7c78259bc02247888634c5e49991148e4f036b8be5f7af7003edac\": container with ID starting with dbb780afbb7c78259bc02247888634c5e49991148e4f036b8be5f7af7003edac not found: ID does not exist" Oct 30 02:17:32.136009 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:32.135974 6572 scope.go:122] "RemoveContainer" containerID="a58560fdefc97caffea04f97a78f5e744d0da3554019f5f85c813ab4a27ff796" Oct 30 02:17:32.136186 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:17:32.136171 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a58560fdefc97caffea04f97a78f5e744d0da3554019f5f85c813ab4a27ff796\": container with ID starting with a58560fdefc97caffea04f97a78f5e744d0da3554019f5f85c813ab4a27ff796 not found: ID does not exist" containerID="a58560fdefc97caffea04f97a78f5e744d0da3554019f5f85c813ab4a27ff796" Oct 30 02:17:32.136232 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:32.136190 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a58560fdefc97caffea04f97a78f5e744d0da3554019f5f85c813ab4a27ff796"} err="failed to get container status \"a58560fdefc97caffea04f97a78f5e744d0da3554019f5f85c813ab4a27ff796\": rpc error: code = NotFound desc = could not find container \"a58560fdefc97caffea04f97a78f5e744d0da3554019f5f85c813ab4a27ff796\": container with ID starting with a58560fdefc97caffea04f97a78f5e744d0da3554019f5f85c813ab4a27ff796 not found: ID does not exist" Oct 30 02:17:32.185029 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:32.184971 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk"] Oct 30 02:17:32.196463 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:32.196436 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-4l7bk"] Oct 30 02:17:32.212679 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:32.212643 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:32.543272 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:32.543190 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0143bfe1-c549-4cac-baa1-e207c659e9f2" path="/var/lib/kubelet/pods/0143bfe1-c549-4cac-baa1-e207c659e9f2/volumes" Oct 30 02:17:32.543636 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:32.543622 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a" path="/var/lib/kubelet/pods/1a0fc9d8-ce02-4e21-80cb-4a13e8a2063a/volumes" Oct 30 02:17:32.544092 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:32.544078 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4f3636d-3177-48b9-88a1-14c250c18bba" path="/var/lib/kubelet/pods/f4f3636d-3177-48b9-88a1-14c250c18bba/volumes" Oct 30 02:17:33.286097 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:33.286072 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d44413-vtt7z" Oct 30 02:17:33.364886 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:33.364853 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e3680ff-1ee0-4f4d-ada9-325766ae0a97-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e3680ff-1ee0-4f4d-ada9-325766ae0a97-maas-api-openshift-ca\") pod \"9e3680ff-1ee0-4f4d-ada9-325766ae0a97\" (UID: \"9e3680ff-1ee0-4f4d-ada9-325766ae0a97\") " Oct 30 02:17:33.365053 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:33.364962 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e3680ff-1ee0-4f4d-ada9-325766ae0a97-kube-api-access-ztf64\" (UniqueName: \"kubernetes.io/projected/9e3680ff-1ee0-4f4d-ada9-325766ae0a97-kube-api-access-ztf64\") pod \"9e3680ff-1ee0-4f4d-ada9-325766ae0a97\" (UID: \"9e3680ff-1ee0-4f4d-ada9-325766ae0a97\") " Oct 30 02:17:33.365249 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:33.365210 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e3680ff-1ee0-4f4d-ada9-325766ae0a97-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e3680ff-1ee0-4f4d-ada9-325766ae0a97-maas-api-xks-ca\") pod \"9e3680ff-1ee0-4f4d-ada9-325766ae0a97\" (UID: \"9e3680ff-1ee0-4f4d-ada9-325766ae0a97\") " Oct 30 02:17:33.365362 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:33.365259 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e3680ff-1ee0-4f4d-ada9-325766ae0a97-maas-api-openshift-ca" pod "9e3680ff-1ee0-4f4d-ada9-325766ae0a97" (UID: "9e3680ff-1ee0-4f4d-ada9-325766ae0a97"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:17:33.365779 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:33.365726 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e3680ff-1ee0-4f4d-ada9-325766ae0a97-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:33.367575 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:33.367553 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e3680ff-1ee0-4f4d-ada9-325766ae0a97-kube-api-access-ztf64" pod "9e3680ff-1ee0-4f4d-ada9-325766ae0a97" (UID: "9e3680ff-1ee0-4f4d-ada9-325766ae0a97"). InnerVolumeSpecName "kube-api-access-ztf64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:33.367699 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:33.367638 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e3680ff-1ee0-4f4d-ada9-325766ae0a97-maas-api-xks-ca" pod "9e3680ff-1ee0-4f4d-ada9-325766ae0a97" (UID: "9e3680ff-1ee0-4f4d-ada9-325766ae0a97"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:17:33.467467 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:33.467434 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztf64\" (UniqueName: \"kubernetes.io/projected/9e3680ff-1ee0-4f4d-ada9-325766ae0a97-kube-api-access-ztf64\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:33.467467 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:33.467468 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e3680ff-1ee0-4f4d-ada9-325766ae0a97-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:33.476856 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:33.476822 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:34.140468 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:34.140436 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d44413-vtt7z" Oct 30 02:17:34.140661 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:34.140468 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d44413-vtt7z" event={"ID":"9e3680ff-1ee0-4f4d-ada9-325766ae0a97","Type":"ContainerDied","Data":"af1aa4684d055e7e1a5825f18dd29d164a0a7adc0f3a2e88507782ab1541dfda"} Oct 30 02:17:34.140661 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:34.140553 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af1aa4684d055e7e1a5825f18dd29d164a0a7adc0f3a2e88507782ab1541dfda" Oct 30 02:17:34.867781 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:34.867728 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:35.241672 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.241629 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" podUID="9a56da7a-9b92-40e2-b08b-43cf591b5221" containerName="main" containerID="cri-o://a183fcef174a0070df08e7ee8b84a662ee426e569957c4b47e109619135c666e" gracePeriod=45 Oct 30 02:17:35.242531 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.242469 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" podUID="e76339df-9659-4bea-80a4-5f18c4a73fd6" containerName="main" containerID="cri-o://b6aff951c15f7b34689a6c804f655051fb9bd1254c1a35c8bba7cfd5b379701a" gracePeriod=45 Oct 30 02:17:35.538634 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.538606 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" Oct 30 02:17:35.542002 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.541975 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-601d4b-b48cfb7b-zk29s"] Oct 30 02:17:35.542531 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.542515 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" Oct 30 02:17:35.573067 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.573042 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-601d4b-565cc7b475-mhc5s"] Oct 30 02:17:35.593700 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.593671 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a56da7a-9b92-40e2-b08b-43cf591b5221-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a56da7a-9b92-40e2-b08b-43cf591b5221-dshm\") pod \"9a56da7a-9b92-40e2-b08b-43cf591b5221\" (UID: \"9a56da7a-9b92-40e2-b08b-43cf591b5221\") " Oct 30 02:17:35.594435 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.594406 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a56da7a-9b92-40e2-b08b-43cf591b5221-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a56da7a-9b92-40e2-b08b-43cf591b5221-model-cache\") pod \"9a56da7a-9b92-40e2-b08b-43cf591b5221\" (UID: \"9a56da7a-9b92-40e2-b08b-43cf591b5221\") " Oct 30 02:17:35.594535 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.594479 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a56da7a-9b92-40e2-b08b-43cf591b5221-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a56da7a-9b92-40e2-b08b-43cf591b5221-tls-certs\") pod \"9a56da7a-9b92-40e2-b08b-43cf591b5221\" (UID: \"9a56da7a-9b92-40e2-b08b-43cf591b5221\") " Oct 30 02:17:35.594535 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.594512 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e76339df-9659-4bea-80a4-5f18c4a73fd6-tls-certs\" (UniqueName: \"kubernetes.io/secret/e76339df-9659-4bea-80a4-5f18c4a73fd6-tls-certs\") pod \"e76339df-9659-4bea-80a4-5f18c4a73fd6\" (UID: \"e76339df-9659-4bea-80a4-5f18c4a73fd6\") " Oct 30 02:17:35.594671 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.594547 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e76339df-9659-4bea-80a4-5f18c4a73fd6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e76339df-9659-4bea-80a4-5f18c4a73fd6-dshm\") pod \"e76339df-9659-4bea-80a4-5f18c4a73fd6\" (UID: \"e76339df-9659-4bea-80a4-5f18c4a73fd6\") " Oct 30 02:17:35.594671 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.594590 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a56da7a-9b92-40e2-b08b-43cf591b5221-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a56da7a-9b92-40e2-b08b-43cf591b5221-tmp-dir\") pod \"9a56da7a-9b92-40e2-b08b-43cf591b5221\" (UID: \"9a56da7a-9b92-40e2-b08b-43cf591b5221\") " Oct 30 02:17:35.594808 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.594682 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a56da7a-9b92-40e2-b08b-43cf591b5221-kube-api-access-zfmhf\" (UniqueName: \"kubernetes.io/projected/9a56da7a-9b92-40e2-b08b-43cf591b5221-kube-api-access-zfmhf\") pod \"9a56da7a-9b92-40e2-b08b-43cf591b5221\" (UID: \"9a56da7a-9b92-40e2-b08b-43cf591b5221\") " Oct 30 02:17:35.595494 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.595356 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e76339df-9659-4bea-80a4-5f18c4a73fd6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e76339df-9659-4bea-80a4-5f18c4a73fd6-tmp-dir\") pod \"e76339df-9659-4bea-80a4-5f18c4a73fd6\" (UID: \"e76339df-9659-4bea-80a4-5f18c4a73fd6\") " Oct 30 02:17:35.595494 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.595452 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e76339df-9659-4bea-80a4-5f18c4a73fd6-home\" (UniqueName: \"kubernetes.io/empty-dir/e76339df-9659-4bea-80a4-5f18c4a73fd6-home\") pod \"e76339df-9659-4bea-80a4-5f18c4a73fd6\" (UID: \"e76339df-9659-4bea-80a4-5f18c4a73fd6\") " Oct 30 02:17:35.597620 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.596476 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e76339df-9659-4bea-80a4-5f18c4a73fd6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e76339df-9659-4bea-80a4-5f18c4a73fd6-model-cache\") pod \"e76339df-9659-4bea-80a4-5f18c4a73fd6\" (UID: \"e76339df-9659-4bea-80a4-5f18c4a73fd6\") " Oct 30 02:17:35.597620 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.596539 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e76339df-9659-4bea-80a4-5f18c4a73fd6-kube-api-access-v529k\" (UniqueName: \"kubernetes.io/projected/e76339df-9659-4bea-80a4-5f18c4a73fd6-kube-api-access-v529k\") pod \"e76339df-9659-4bea-80a4-5f18c4a73fd6\" (UID: \"e76339df-9659-4bea-80a4-5f18c4a73fd6\") " Oct 30 02:17:35.597620 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.596605 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a56da7a-9b92-40e2-b08b-43cf591b5221-home\" (UniqueName: \"kubernetes.io/empty-dir/9a56da7a-9b92-40e2-b08b-43cf591b5221-home\") pod \"9a56da7a-9b92-40e2-b08b-43cf591b5221\" (UID: \"9a56da7a-9b92-40e2-b08b-43cf591b5221\") " Oct 30 02:17:35.597620 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.595871 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a56da7a-9b92-40e2-b08b-43cf591b5221-tmp-dir" pod "9a56da7a-9b92-40e2-b08b-43cf591b5221" (UID: "9a56da7a-9b92-40e2-b08b-43cf591b5221"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:35.597620 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.595952 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a56da7a-9b92-40e2-b08b-43cf591b5221-model-cache" pod "9a56da7a-9b92-40e2-b08b-43cf591b5221" (UID: "9a56da7a-9b92-40e2-b08b-43cf591b5221"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:35.597620 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.595971 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e76339df-9659-4bea-80a4-5f18c4a73fd6-home" pod "e76339df-9659-4bea-80a4-5f18c4a73fd6" (UID: "e76339df-9659-4bea-80a4-5f18c4a73fd6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:35.597620 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.596372 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e76339df-9659-4bea-80a4-5f18c4a73fd6-tmp-dir" pod "e76339df-9659-4bea-80a4-5f18c4a73fd6" (UID: "e76339df-9659-4bea-80a4-5f18c4a73fd6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:35.598136 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.598102 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a56da7a-9b92-40e2-b08b-43cf591b5221-home" pod "9a56da7a-9b92-40e2-b08b-43cf591b5221" (UID: "9a56da7a-9b92-40e2-b08b-43cf591b5221"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:35.598349 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.598318 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e76339df-9659-4bea-80a4-5f18c4a73fd6-model-cache" pod "e76339df-9659-4bea-80a4-5f18c4a73fd6" (UID: "e76339df-9659-4bea-80a4-5f18c4a73fd6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:35.598469 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.598348 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a56da7a-9b92-40e2-b08b-43cf591b5221-tls-certs" pod "9a56da7a-9b92-40e2-b08b-43cf591b5221" (UID: "9a56da7a-9b92-40e2-b08b-43cf591b5221"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:17:35.600172 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.600141 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e76339df-9659-4bea-80a4-5f18c4a73fd6-kube-api-access-v529k" pod "e76339df-9659-4bea-80a4-5f18c4a73fd6" (UID: "e76339df-9659-4bea-80a4-5f18c4a73fd6"). InnerVolumeSpecName "kube-api-access-v529k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:35.601163 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.601137 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e76339df-9659-4bea-80a4-5f18c4a73fd6-tls-certs" pod "e76339df-9659-4bea-80a4-5f18c4a73fd6" (UID: "e76339df-9659-4bea-80a4-5f18c4a73fd6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:17:35.601345 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.601141 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e76339df-9659-4bea-80a4-5f18c4a73fd6-dshm" pod "e76339df-9659-4bea-80a4-5f18c4a73fd6" (UID: "e76339df-9659-4bea-80a4-5f18c4a73fd6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:35.601571 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.601551 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a56da7a-9b92-40e2-b08b-43cf591b5221-dshm" pod "9a56da7a-9b92-40e2-b08b-43cf591b5221" (UID: "9a56da7a-9b92-40e2-b08b-43cf591b5221"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:35.603343 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.603316 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a56da7a-9b92-40e2-b08b-43cf591b5221-kube-api-access-zfmhf" pod "9a56da7a-9b92-40e2-b08b-43cf591b5221" (UID: "9a56da7a-9b92-40e2-b08b-43cf591b5221"). InnerVolumeSpecName "kube-api-access-zfmhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:35.698546 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.698446 6572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a56da7a-9b92-40e2-b08b-43cf591b5221-home\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:35.698546 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.698480 6572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a56da7a-9b92-40e2-b08b-43cf591b5221-dshm\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:35.698546 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.698492 6572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a56da7a-9b92-40e2-b08b-43cf591b5221-model-cache\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:35.698546 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.698506 6572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a56da7a-9b92-40e2-b08b-43cf591b5221-tls-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:35.698546 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.698517 6572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e76339df-9659-4bea-80a4-5f18c4a73fd6-tls-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:35.698546 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.698528 6572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e76339df-9659-4bea-80a4-5f18c4a73fd6-dshm\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:35.698546 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.698538 6572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a56da7a-9b92-40e2-b08b-43cf591b5221-tmp-dir\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:35.698546 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.698550 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfmhf\" (UniqueName: \"kubernetes.io/projected/9a56da7a-9b92-40e2-b08b-43cf591b5221-kube-api-access-zfmhf\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:35.698546 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.698561 6572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e76339df-9659-4bea-80a4-5f18c4a73fd6-tmp-dir\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:35.699245 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.698574 6572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e76339df-9659-4bea-80a4-5f18c4a73fd6-home\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:35.699245 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.698585 6572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e76339df-9659-4bea-80a4-5f18c4a73fd6-model-cache\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:35.699245 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.698597 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v529k\" (UniqueName: \"kubernetes.io/projected/e76339df-9659-4bea-80a4-5f18c4a73fd6-kube-api-access-v529k\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:35.775445 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.775356 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:35.972762 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:35.972710 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:36.197860 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:36.197822 6572 generic.go:356] "Generic (PLEG): container finished" podID="9a56da7a-9b92-40e2-b08b-43cf591b5221" containerID="a183fcef174a0070df08e7ee8b84a662ee426e569957c4b47e109619135c666e" exitCode=0 Oct 30 02:17:36.198822 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:36.198791 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" event={"ID":"9a56da7a-9b92-40e2-b08b-43cf591b5221","Type":"ContainerDied","Data":"a183fcef174a0070df08e7ee8b84a662ee426e569957c4b47e109619135c666e"} Oct 30 02:17:36.199023 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:36.199001 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" event={"ID":"9a56da7a-9b92-40e2-b08b-43cf591b5221","Type":"ContainerDied","Data":"d1f7ff677eb0162b34704c6b45b68d57cd5cd8bbb9fb2152cc7b9cd5c2f64824"} Oct 30 02:17:36.199269 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:36.199251 6572 scope.go:122] "RemoveContainer" containerID="a183fcef174a0070df08e7ee8b84a662ee426e569957c4b47e109619135c666e" Oct 30 02:17:36.199654 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:36.199638 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" Oct 30 02:17:36.211134 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:36.210709 6572 generic.go:356] "Generic (PLEG): container finished" podID="e76339df-9659-4bea-80a4-5f18c4a73fd6" containerID="b6aff951c15f7b34689a6c804f655051fb9bd1254c1a35c8bba7cfd5b379701a" exitCode=0 Oct 30 02:17:36.211134 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:36.210989 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" Oct 30 02:17:36.211333 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:36.211233 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" event={"ID":"e76339df-9659-4bea-80a4-5f18c4a73fd6","Type":"ContainerDied","Data":"b6aff951c15f7b34689a6c804f655051fb9bd1254c1a35c8bba7cfd5b379701a"} Oct 30 02:17:36.211333 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:36.211283 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n" event={"ID":"e76339df-9659-4bea-80a4-5f18c4a73fd6","Type":"ContainerDied","Data":"8f70f23b95f9231eeea315d81b9147d3ec3fb39683c87ce0b013502b6cd9ecc4"} Oct 30 02:17:36.228843 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:36.228733 6572 scope.go:122] "RemoveContainer" containerID="a183fcef174a0070df08e7ee8b84a662ee426e569957c4b47e109619135c666e" Oct 30 02:17:36.229291 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:17:36.229261 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a183fcef174a0070df08e7ee8b84a662ee426e569957c4b47e109619135c666e\": container with ID starting with a183fcef174a0070df08e7ee8b84a662ee426e569957c4b47e109619135c666e not found: ID does not exist" containerID="a183fcef174a0070df08e7ee8b84a662ee426e569957c4b47e109619135c666e" Oct 30 02:17:36.229378 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:36.229301 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a183fcef174a0070df08e7ee8b84a662ee426e569957c4b47e109619135c666e"} err="failed to get container status \"a183fcef174a0070df08e7ee8b84a662ee426e569957c4b47e109619135c666e\": rpc error: code = NotFound desc = could not find container \"a183fcef174a0070df08e7ee8b84a662ee426e569957c4b47e109619135c666e\": container with ID starting with a183fcef174a0070df08e7ee8b84a662ee426e569957c4b47e109619135c666e not found: ID does not exist" Oct 30 02:17:36.229378 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:36.229326 6572 scope.go:122] "RemoveContainer" containerID="b6aff951c15f7b34689a6c804f655051fb9bd1254c1a35c8bba7cfd5b379701a" Oct 30 02:17:36.247107 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:36.247075 6572 scope.go:122] "RemoveContainer" containerID="b6aff951c15f7b34689a6c804f655051fb9bd1254c1a35c8bba7cfd5b379701a" Oct 30 02:17:36.247732 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:17:36.247574 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6aff951c15f7b34689a6c804f655051fb9bd1254c1a35c8bba7cfd5b379701a\": container with ID starting with b6aff951c15f7b34689a6c804f655051fb9bd1254c1a35c8bba7cfd5b379701a not found: ID does not exist" containerID="b6aff951c15f7b34689a6c804f655051fb9bd1254c1a35c8bba7cfd5b379701a" Oct 30 02:17:36.247732 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:36.247620 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6aff951c15f7b34689a6c804f655051fb9bd1254c1a35c8bba7cfd5b379701a"} err="failed to get container status \"b6aff951c15f7b34689a6c804f655051fb9bd1254c1a35c8bba7cfd5b379701a\": rpc error: code = NotFound desc = could not find container \"b6aff951c15f7b34689a6c804f655051fb9bd1254c1a35c8bba7cfd5b379701a\": container with ID starting with b6aff951c15f7b34689a6c804f655051fb9bd1254c1a35c8bba7cfd5b379701a not found: ID does not exist" Oct 30 02:17:36.264341 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:36.264315 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd"] Oct 30 02:17:36.274801 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:36.274775 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd"] Oct 30 02:17:36.324721 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:36.324514 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n"] Oct 30 02:17:36.329682 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:36.329642 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-eda881/distinct-2-w2-eda881-kserve-6cc664b7f9-ttl6n"] Oct 30 02:17:36.348708 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:36.348669 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:36.528907 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:36.528815 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/distinct-w2-eda881-kserve-7d9d4c5588-dqwqd" podUID="9a56da7a-9b92-40e2-b08b-43cf591b5221" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:17:36.543425 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:36.543391 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a56da7a-9b92-40e2-b08b-43cf591b5221" path="/var/lib/kubelet/pods/9a56da7a-9b92-40e2-b08b-43cf591b5221/volumes" Oct 30 02:17:36.543903 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:36.543886 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e76339df-9659-4bea-80a4-5f18c4a73fd6" path="/var/lib/kubelet/pods/e76339df-9659-4bea-80a4-5f18c4a73fd6/volumes" Oct 30 02:17:37.749589 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:37.749557 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:39.069184 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:39.069139 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:39.339548 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:39.339448 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eda881-67d94b78bc-6n9dq" podUID="6d6c050f-1131-4874-b064-3ef68d3b990b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:17:39.340046 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:39.340022 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-eda881-5c7cd4d9b-hqnjh" podUID="2de76834-8ea2-4e53-8e91-ebd04ae69886" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:17:40.192072 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.192022 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5db98a24-7k4zk"] Oct 30 02:17:40.193238 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.193213 6572 cpu_manager.go:414] "containerMap: removing container" podUID="9a56da7a-9b92-40e2-b08b-43cf591b5221" containerName="main" Oct 30 02:17:40.193331 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.193240 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a56da7a-9b92-40e2-b08b-43cf591b5221" containerName="main" Oct 30 02:17:40.193331 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.193272 6572 cpu_manager.go:414] "containerMap: removing container" podUID="e76339df-9659-4bea-80a4-5f18c4a73fd6" containerName="main" Oct 30 02:17:40.193331 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.193280 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e76339df-9659-4bea-80a4-5f18c4a73fd6" containerName="main" Oct 30 02:17:40.193331 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.193297 6572 cpu_manager.go:414] "containerMap: removing container" podUID="f4f3636d-3177-48b9-88a1-14c250c18bba" containerName="istio-proxy" Oct 30 02:17:40.193331 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.193306 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4f3636d-3177-48b9-88a1-14c250c18bba" containerName="istio-proxy" Oct 30 02:17:40.193703 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.193339 6572 cpu_manager.go:414] "containerMap: removing container" podUID="f4f3636d-3177-48b9-88a1-14c250c18bba" containerName="istio-proxy" Oct 30 02:17:40.193703 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.193347 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4f3636d-3177-48b9-88a1-14c250c18bba" containerName="istio-proxy" Oct 30 02:17:40.193703 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.193378 6572 cpu_manager.go:414] "containerMap: removing container" podUID="9e3680ff-1ee0-4f4d-ada9-325766ae0a97" containerName="revoke-keys" Oct 30 02:17:40.193703 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.193385 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e3680ff-1ee0-4f4d-ada9-325766ae0a97" containerName="revoke-keys" Oct 30 02:17:40.193703 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.193530 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a56da7a-9b92-40e2-b08b-43cf591b5221" containerName="main" Oct 30 02:17:40.193703 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.193546 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="e76339df-9659-4bea-80a4-5f18c4a73fd6" containerName="main" Oct 30 02:17:40.193703 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.193556 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4f3636d-3177-48b9-88a1-14c250c18bba" containerName="istio-proxy" Oct 30 02:17:40.193703 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.193587 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e3680ff-1ee0-4f4d-ada9-325766ae0a97" containerName="revoke-keys" Oct 30 02:17:40.199526 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.199499 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5db98a24-7k4zk" Oct 30 02:17:40.204338 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.204307 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99fd05c4-wd9cz"] Oct 30 02:17:40.205069 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.205048 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4f3636d-3177-48b9-88a1-14c250c18bba" containerName="istio-proxy" Oct 30 02:17:40.208990 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.208964 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fd05c4-wd9cz" Oct 30 02:17:40.214305 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.214273 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:17:40.214443 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.214366 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:17:40.215461 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.215436 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5db98a24-7k4zk"] Oct 30 02:17:40.221169 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.221145 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:17:40.221845 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.221819 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:17:40.233951 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.233916 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99fd05c4-wd9cz"] Oct 30 02:17:40.260411 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.260368 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a36c6723-df85-48fa-9b09-e466c2c28500-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5db98a24-7k4zk\" (UID: \"a36c6723-df85-48fa-9b09-e466c2c28500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5db98a24-7k4zk" Oct 30 02:17:40.260644 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.260567 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh7sq\" (UniqueName: \"kubernetes.io/projected/a36c6723-df85-48fa-9b09-e466c2c28500-kube-api-access-vh7sq\") pod \"maas-api-revoke-sub-5db98a24-7k4zk\" (UID: \"a36c6723-df85-48fa-9b09-e466c2c28500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5db98a24-7k4zk" Oct 30 02:17:40.260644 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.260614 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ltsv\" (UniqueName: \"kubernetes.io/projected/af284c10-c406-439b-98ad-c0a5a35b9a8e-kube-api-access-6ltsv\") pod \"maas-api-revoke-sub-99fd05c4-wd9cz\" (UID: \"af284c10-c406-439b-98ad-c0a5a35b9a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fd05c4-wd9cz" Oct 30 02:17:40.260809 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.260781 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af284c10-c406-439b-98ad-c0a5a35b9a8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99fd05c4-wd9cz\" (UID: \"af284c10-c406-439b-98ad-c0a5a35b9a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fd05c4-wd9cz" Oct 30 02:17:40.260880 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.260833 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a36c6723-df85-48fa-9b09-e466c2c28500-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5db98a24-7k4zk\" (UID: \"a36c6723-df85-48fa-9b09-e466c2c28500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5db98a24-7k4zk" Oct 30 02:17:40.260929 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.260876 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af284c10-c406-439b-98ad-c0a5a35b9a8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99fd05c4-wd9cz\" (UID: \"af284c10-c406-439b-98ad-c0a5a35b9a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fd05c4-wd9cz" Oct 30 02:17:40.362439 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.362397 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh7sq\" (UniqueName: \"kubernetes.io/projected/a36c6723-df85-48fa-9b09-e466c2c28500-kube-api-access-vh7sq\") pod \"maas-api-revoke-sub-5db98a24-7k4zk\" (UID: \"a36c6723-df85-48fa-9b09-e466c2c28500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5db98a24-7k4zk" Oct 30 02:17:40.362635 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.362459 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ltsv\" (UniqueName: \"kubernetes.io/projected/af284c10-c406-439b-98ad-c0a5a35b9a8e-kube-api-access-6ltsv\") pod \"maas-api-revoke-sub-99fd05c4-wd9cz\" (UID: \"af284c10-c406-439b-98ad-c0a5a35b9a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fd05c4-wd9cz" Oct 30 02:17:40.362635 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.362560 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af284c10-c406-439b-98ad-c0a5a35b9a8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99fd05c4-wd9cz\" (UID: \"af284c10-c406-439b-98ad-c0a5a35b9a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fd05c4-wd9cz" Oct 30 02:17:40.362635 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.362596 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a36c6723-df85-48fa-9b09-e466c2c28500-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5db98a24-7k4zk\" (UID: \"a36c6723-df85-48fa-9b09-e466c2c28500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5db98a24-7k4zk" Oct 30 02:17:40.362635 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.362630 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af284c10-c406-439b-98ad-c0a5a35b9a8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99fd05c4-wd9cz\" (UID: \"af284c10-c406-439b-98ad-c0a5a35b9a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fd05c4-wd9cz" Oct 30 02:17:40.362966 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.362725 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a36c6723-df85-48fa-9b09-e466c2c28500-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5db98a24-7k4zk\" (UID: \"a36c6723-df85-48fa-9b09-e466c2c28500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5db98a24-7k4zk" Oct 30 02:17:40.363635 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.363610 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a36c6723-df85-48fa-9b09-e466c2c28500-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5db98a24-7k4zk\" (UID: \"a36c6723-df85-48fa-9b09-e466c2c28500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5db98a24-7k4zk" Oct 30 02:17:40.363873 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.363609 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af284c10-c406-439b-98ad-c0a5a35b9a8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99fd05c4-wd9cz\" (UID: \"af284c10-c406-439b-98ad-c0a5a35b9a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fd05c4-wd9cz" Oct 30 02:17:40.366078 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.366036 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af284c10-c406-439b-98ad-c0a5a35b9a8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99fd05c4-wd9cz\" (UID: \"af284c10-c406-439b-98ad-c0a5a35b9a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fd05c4-wd9cz" Oct 30 02:17:40.366406 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.366384 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a36c6723-df85-48fa-9b09-e466c2c28500-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5db98a24-7k4zk\" (UID: \"a36c6723-df85-48fa-9b09-e466c2c28500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5db98a24-7k4zk" Oct 30 02:17:40.383802 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.383764 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh7sq\" (UniqueName: \"kubernetes.io/projected/a36c6723-df85-48fa-9b09-e466c2c28500-kube-api-access-vh7sq\") pod \"maas-api-revoke-sub-5db98a24-7k4zk\" (UID: \"a36c6723-df85-48fa-9b09-e466c2c28500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5db98a24-7k4zk" Oct 30 02:17:40.383992 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.383944 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ltsv\" (UniqueName: \"kubernetes.io/projected/af284c10-c406-439b-98ad-c0a5a35b9a8e-kube-api-access-6ltsv\") pod \"maas-api-revoke-sub-99fd05c4-wd9cz\" (UID: \"af284c10-c406-439b-98ad-c0a5a35b9a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fd05c4-wd9cz" Oct 30 02:17:40.517086 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.516988 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5db98a24-7k4zk" Oct 30 02:17:40.546417 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.543405 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-601d4b-b48cfb7b-zk29s" podUID="7c580bbf-cee6-4937-8ea7-fd44837e076d" containerName="payload-processing" containerID="cri-o://7bcf40de3cb91740af4e29865456475c612e1a3ef6a87dc9fa9aa259f69d165e" gracePeriod=25 Oct 30 02:17:40.546417 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.543732 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fd05c4-wd9cz" Oct 30 02:17:40.577059 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.575828 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-601d4b-565cc7b475-mhc5s" podUID="f7f30150-5a1b-4c5b-8044-cfe1774eb76b" containerName="payload-pre-processing" containerID="cri-o://6433cfff0722adfd087c6fa1004d86ed4ebb409fe889485bccce6d7c835f9a05" gracePeriod=25 Oct 30 02:17:40.629774 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.618266 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:40.833985 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.833878 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d44413-86f78b95f6-v547l"] Oct 30 02:17:40.857699 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.857655 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d44413-c6c7b8c75-qqdbt"] Oct 30 02:17:40.942788 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.942721 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5db98a24-7k4zk"] Oct 30 02:17:40.967279 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:17:40.967246 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda36c6723_df85_48fa_9b09_e466c2c28500.slice/crio-6f3cd8db8f39ecb4049ca7349e10e1d8f1f14235d413e0071de48e447a3f6087 WatchSource:0}: Error finding container 6f3cd8db8f39ecb4049ca7349e10e1d8f1f14235d413e0071de48e447a3f6087: Status 404 returned error can't find the container with id 6f3cd8db8f39ecb4049ca7349e10e1d8f1f14235d413e0071de48e447a3f6087 Oct 30 02:17:40.981862 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.981822 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99fd05c4-wd9cz"] Oct 30 02:17:40.986215 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:17:40.986188 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf284c10_c406_439b_98ad_c0a5a35b9a8e.slice/crio-0494f3ac78e26f928579214cd3cc2bfe76434b11e8dd617790f16ed926f9f9fe WatchSource:0}: Error finding container 0494f3ac78e26f928579214cd3cc2bfe76434b11e8dd617790f16ed926f9f9fe: Status 404 returned error can't find the container with id 0494f3ac78e26f928579214cd3cc2bfe76434b11e8dd617790f16ed926f9f9fe Oct 30 02:17:40.998384 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.998080 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-601d4b-b48cfb7b-zk29s" podUID="7c580bbf-cee6-4937-8ea7-fd44837e076d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:17:40.998384 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:40.998169 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-601d4b-565cc7b475-mhc5s" podUID="f7f30150-5a1b-4c5b-8044-cfe1774eb76b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:17:41.028655 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:41.028572 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07155f44-bp5t5"] Oct 30 02:17:41.031984 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:41.031954 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07155f44-bp5t5"] Oct 30 02:17:41.059603 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:41.059249 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:41.249869 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:41.249822 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:41.266244 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:41.266208 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5db98a24-7k4zk" event={"ID":"a36c6723-df85-48fa-9b09-e466c2c28500","Type":"ContainerStarted","Data":"6f3cd8db8f39ecb4049ca7349e10e1d8f1f14235d413e0071de48e447a3f6087"} Oct 30 02:17:41.267827 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:41.267797 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fd05c4-wd9cz" event={"ID":"af284c10-c406-439b-98ad-c0a5a35b9a8e","Type":"ContainerStarted","Data":"0494f3ac78e26f928579214cd3cc2bfe76434b11e8dd617790f16ed926f9f9fe"} Oct 30 02:17:42.088803 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:42.088730 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:42.278481 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:42.278382 6572 generic.go:356] "Generic (PLEG): container finished" podID="af284c10-c406-439b-98ad-c0a5a35b9a8e" containerID="7d0e036012f02e4d54410f69c769ee7e33010a9b2edaf2dc71d8f58af3a88038" exitCode=0 Oct 30 02:17:42.278481 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:42.278458 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fd05c4-wd9cz" event={"ID":"af284c10-c406-439b-98ad-c0a5a35b9a8e","Type":"ContainerDied","Data":"7d0e036012f02e4d54410f69c769ee7e33010a9b2edaf2dc71d8f58af3a88038"} Oct 30 02:17:42.280506 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:42.280478 6572 generic.go:356] "Generic (PLEG): container finished" podID="a36c6723-df85-48fa-9b09-e466c2c28500" containerID="1456e34002e2c429a8aecd69470bdc96afd718ac5826d71f52def3eea2a6e451" exitCode=0 Oct 30 02:17:42.280624 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:42.280554 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5db98a24-7k4zk" event={"ID":"a36c6723-df85-48fa-9b09-e466c2c28500","Type":"ContainerDied","Data":"1456e34002e2c429a8aecd69470bdc96afd718ac5826d71f52def3eea2a6e451"} Oct 30 02:17:42.543557 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:42.543470 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8a485b3-17b6-4680-ba7b-02a2f7a795c4" path="/var/lib/kubelet/pods/d8a485b3-17b6-4680-ba7b-02a2f7a795c4/volumes" Oct 30 02:17:43.490777 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:43.490736 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5db98a24-7k4zk" Oct 30 02:17:43.493321 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:43.493298 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fd05c4-wd9cz" Oct 30 02:17:43.579082 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:43.578994 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:43.619138 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:43.619097 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a36c6723-df85-48fa-9b09-e466c2c28500-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a36c6723-df85-48fa-9b09-e466c2c28500-maas-api-xks-ca\") pod \"a36c6723-df85-48fa-9b09-e466c2c28500\" (UID: \"a36c6723-df85-48fa-9b09-e466c2c28500\") " Oct 30 02:17:43.619357 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:43.619294 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af284c10-c406-439b-98ad-c0a5a35b9a8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af284c10-c406-439b-98ad-c0a5a35b9a8e-maas-api-openshift-ca\") pod \"af284c10-c406-439b-98ad-c0a5a35b9a8e\" (UID: \"af284c10-c406-439b-98ad-c0a5a35b9a8e\") " Oct 30 02:17:43.619428 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:43.619403 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af284c10-c406-439b-98ad-c0a5a35b9a8e-kube-api-access-6ltsv\" (UniqueName: \"kubernetes.io/projected/af284c10-c406-439b-98ad-c0a5a35b9a8e-kube-api-access-6ltsv\") pod \"af284c10-c406-439b-98ad-c0a5a35b9a8e\" (UID: \"af284c10-c406-439b-98ad-c0a5a35b9a8e\") " Oct 30 02:17:43.619485 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:43.619442 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a36c6723-df85-48fa-9b09-e466c2c28500-kube-api-access-vh7sq\" (UniqueName: \"kubernetes.io/projected/a36c6723-df85-48fa-9b09-e466c2c28500-kube-api-access-vh7sq\") pod \"a36c6723-df85-48fa-9b09-e466c2c28500\" (UID: \"a36c6723-df85-48fa-9b09-e466c2c28500\") " Oct 30 02:17:43.619696 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:43.619650 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a36c6723-df85-48fa-9b09-e466c2c28500-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a36c6723-df85-48fa-9b09-e466c2c28500-maas-api-openshift-ca\") pod \"a36c6723-df85-48fa-9b09-e466c2c28500\" (UID: \"a36c6723-df85-48fa-9b09-e466c2c28500\") " Oct 30 02:17:43.619829 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:43.619698 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af284c10-c406-439b-98ad-c0a5a35b9a8e-maas-api-openshift-ca" pod "af284c10-c406-439b-98ad-c0a5a35b9a8e" (UID: "af284c10-c406-439b-98ad-c0a5a35b9a8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:17:43.619829 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:43.619818 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af284c10-c406-439b-98ad-c0a5a35b9a8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af284c10-c406-439b-98ad-c0a5a35b9a8e-maas-api-xks-ca\") pod \"af284c10-c406-439b-98ad-c0a5a35b9a8e\" (UID: \"af284c10-c406-439b-98ad-c0a5a35b9a8e\") " Oct 30 02:17:43.620193 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:43.620153 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a36c6723-df85-48fa-9b09-e466c2c28500-maas-api-openshift-ca" pod "a36c6723-df85-48fa-9b09-e466c2c28500" (UID: "a36c6723-df85-48fa-9b09-e466c2c28500"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:17:43.620608 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:43.620585 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a36c6723-df85-48fa-9b09-e466c2c28500-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:43.620688 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:43.620611 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af284c10-c406-439b-98ad-c0a5a35b9a8e-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:43.621707 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:43.621673 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a36c6723-df85-48fa-9b09-e466c2c28500-maas-api-xks-ca" pod "a36c6723-df85-48fa-9b09-e466c2c28500" (UID: "a36c6723-df85-48fa-9b09-e466c2c28500"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:17:43.622066 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:43.622039 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af284c10-c406-439b-98ad-c0a5a35b9a8e-kube-api-access-6ltsv" pod "af284c10-c406-439b-98ad-c0a5a35b9a8e" (UID: "af284c10-c406-439b-98ad-c0a5a35b9a8e"). InnerVolumeSpecName "kube-api-access-6ltsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:43.622231 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:43.622213 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af284c10-c406-439b-98ad-c0a5a35b9a8e-maas-api-xks-ca" pod "af284c10-c406-439b-98ad-c0a5a35b9a8e" (UID: "af284c10-c406-439b-98ad-c0a5a35b9a8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:17:43.622698 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:43.622673 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a36c6723-df85-48fa-9b09-e466c2c28500-kube-api-access-vh7sq" pod "a36c6723-df85-48fa-9b09-e466c2c28500" (UID: "a36c6723-df85-48fa-9b09-e466c2c28500"). InnerVolumeSpecName "kube-api-access-vh7sq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:43.721546 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:43.721504 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af284c10-c406-439b-98ad-c0a5a35b9a8e-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:43.721546 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:43.721541 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a36c6723-df85-48fa-9b09-e466c2c28500-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:43.721546 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:43.721554 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ltsv\" (UniqueName: \"kubernetes.io/projected/af284c10-c406-439b-98ad-c0a5a35b9a8e-kube-api-access-6ltsv\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:43.721841 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:43.721570 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh7sq\" (UniqueName: \"kubernetes.io/projected/a36c6723-df85-48fa-9b09-e466c2c28500-kube-api-access-vh7sq\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:44.300612 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:44.300583 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5db98a24-7k4zk" Oct 30 02:17:44.300892 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:44.300620 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5db98a24-7k4zk" event={"ID":"a36c6723-df85-48fa-9b09-e466c2c28500","Type":"ContainerDied","Data":"6f3cd8db8f39ecb4049ca7349e10e1d8f1f14235d413e0071de48e447a3f6087"} Oct 30 02:17:44.300892 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:44.300678 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f3cd8db8f39ecb4049ca7349e10e1d8f1f14235d413e0071de48e447a3f6087" Oct 30 02:17:44.302622 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:44.302591 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fd05c4-wd9cz" Oct 30 02:17:44.302787 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:44.302600 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fd05c4-wd9cz" event={"ID":"af284c10-c406-439b-98ad-c0a5a35b9a8e","Type":"ContainerDied","Data":"0494f3ac78e26f928579214cd3cc2bfe76434b11e8dd617790f16ed926f9f9fe"} Oct 30 02:17:44.302787 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:44.302714 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0494f3ac78e26f928579214cd3cc2bfe76434b11e8dd617790f16ed926f9f9fe" Oct 30 02:17:44.689716 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:17:44.689676 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:17:44.941942 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:44.941829 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:45.834659 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:45.834601 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-d44413-86f78b95f6-v547l" podUID="1e5005f7-fccd-4447-b09d-dd991c1c7733" containerName="payload-processing" containerID="cri-o://512472424c4eda1866a70f9230e096f63b90958173aa5510108308a64ef6f812" gracePeriod=25 Oct 30 02:17:45.858836 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:45.858799 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d44413-c6c7b8c75-qqdbt" podUID="96c5b66f-d98b-491a-93ae-4b2587cf9809" containerName="payload-pre-processing" containerID="cri-o://e8b9af652baa34b52f5e9747ad0e65dcdec8d32cab773c5e99812ce57d60bf6b" gracePeriod=25 Oct 30 02:17:46.150483 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:46.150452 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d44413-86f78b95f6-v547l" Oct 30 02:17:46.180805 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:46.180768 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d44413-c6c7b8c75-qqdbt" Oct 30 02:17:46.239642 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:46.239425 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:46.257457 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:46.257408 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e5005f7-fccd-4447-b09d-dd991c1c7733-kube-api-access-bdsqp\" (UniqueName: \"kubernetes.io/projected/1e5005f7-fccd-4447-b09d-dd991c1c7733-kube-api-access-bdsqp\") pod \"1e5005f7-fccd-4447-b09d-dd991c1c7733\" (UID: \"1e5005f7-fccd-4447-b09d-dd991c1c7733\") " Oct 30 02:17:46.257952 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:46.257918 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e5005f7-fccd-4447-b09d-dd991c1c7733-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e5005f7-fccd-4447-b09d-dd991c1c7733-plugins-config-volume\") pod \"1e5005f7-fccd-4447-b09d-dd991c1c7733\" (UID: \"1e5005f7-fccd-4447-b09d-dd991c1c7733\") " Oct 30 02:17:46.258201 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:46.258184 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96c5b66f-d98b-491a-93ae-4b2587cf9809-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96c5b66f-d98b-491a-93ae-4b2587cf9809-plugins-config-volume\") pod \"96c5b66f-d98b-491a-93ae-4b2587cf9809\" (UID: \"96c5b66f-d98b-491a-93ae-4b2587cf9809\") " Oct 30 02:17:46.261413 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:46.258366 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e5005f7-fccd-4447-b09d-dd991c1c7733-plugins-config-volume" pod "1e5005f7-fccd-4447-b09d-dd991c1c7733" (UID: "1e5005f7-fccd-4447-b09d-dd991c1c7733"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:17:46.261413 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:46.258595 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96c5b66f-d98b-491a-93ae-4b2587cf9809-plugins-config-volume" pod "96c5b66f-d98b-491a-93ae-4b2587cf9809" (UID: "96c5b66f-d98b-491a-93ae-4b2587cf9809"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:17:46.261628 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:46.261606 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96c5b66f-d98b-491a-93ae-4b2587cf9809-kube-api-access-pfztl\" (UniqueName: \"kubernetes.io/projected/96c5b66f-d98b-491a-93ae-4b2587cf9809-kube-api-access-pfztl\") pod \"96c5b66f-d98b-491a-93ae-4b2587cf9809\" (UID: \"96c5b66f-d98b-491a-93ae-4b2587cf9809\") " Oct 30 02:17:46.262290 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:46.262267 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e5005f7-fccd-4447-b09d-dd991c1c7733-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:46.262290 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:46.262289 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96c5b66f-d98b-491a-93ae-4b2587cf9809-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:46.264562 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:46.264381 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e5005f7-fccd-4447-b09d-dd991c1c7733-kube-api-access-bdsqp" pod "1e5005f7-fccd-4447-b09d-dd991c1c7733" (UID: "1e5005f7-fccd-4447-b09d-dd991c1c7733"). InnerVolumeSpecName "kube-api-access-bdsqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:46.264562 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:46.264528 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96c5b66f-d98b-491a-93ae-4b2587cf9809-kube-api-access-pfztl" pod "96c5b66f-d98b-491a-93ae-4b2587cf9809" (UID: "96c5b66f-d98b-491a-93ae-4b2587cf9809"). InnerVolumeSpecName "kube-api-access-pfztl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:46.324859 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:46.324823 6572 generic.go:356] "Generic (PLEG): container finished" podID="1e5005f7-fccd-4447-b09d-dd991c1c7733" containerID="512472424c4eda1866a70f9230e096f63b90958173aa5510108308a64ef6f812" exitCode=0 Oct 30 02:17:46.325071 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:46.324877 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d44413-86f78b95f6-v547l" Oct 30 02:17:46.325071 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:46.324916 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d44413-86f78b95f6-v547l" event={"ID":"1e5005f7-fccd-4447-b09d-dd991c1c7733","Type":"ContainerDied","Data":"512472424c4eda1866a70f9230e096f63b90958173aa5510108308a64ef6f812"} Oct 30 02:17:46.325071 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:46.324981 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d44413-86f78b95f6-v547l" event={"ID":"1e5005f7-fccd-4447-b09d-dd991c1c7733","Type":"ContainerDied","Data":"b71fc06e6f0cae5522805ea2c9a0217c7ba1248fdb2df7f33b907642790a7989"} Oct 30 02:17:46.325071 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:46.325016 6572 scope.go:122] "RemoveContainer" containerID="512472424c4eda1866a70f9230e096f63b90958173aa5510108308a64ef6f812" Oct 30 02:17:46.336500 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:46.336270 6572 generic.go:356] "Generic (PLEG): container finished" podID="96c5b66f-d98b-491a-93ae-4b2587cf9809" containerID="e8b9af652baa34b52f5e9747ad0e65dcdec8d32cab773c5e99812ce57d60bf6b" exitCode=0 Oct 30 02:17:46.336500 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:46.336377 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d44413-c6c7b8c75-qqdbt" Oct 30 02:17:46.336500 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:46.336409 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d44413-c6c7b8c75-qqdbt" event={"ID":"96c5b66f-d98b-491a-93ae-4b2587cf9809","Type":"ContainerDied","Data":"e8b9af652baa34b52f5e9747ad0e65dcdec8d32cab773c5e99812ce57d60bf6b"} Oct 30 02:17:46.336858 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:46.336512 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d44413-c6c7b8c75-qqdbt" event={"ID":"96c5b66f-d98b-491a-93ae-4b2587cf9809","Type":"ContainerDied","Data":"2315d26c5b358f32204ad93a3118e1c58e260c5d5eee8cb025202a4c3eee43ed"} Oct 30 02:17:46.357318 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:46.357283 6572 scope.go:122] "RemoveContainer" containerID="512472424c4eda1866a70f9230e096f63b90958173aa5510108308a64ef6f812" Oct 30 02:17:46.357669 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:17:46.357642 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"512472424c4eda1866a70f9230e096f63b90958173aa5510108308a64ef6f812\": container with ID starting with 512472424c4eda1866a70f9230e096f63b90958173aa5510108308a64ef6f812 not found: ID does not exist" containerID="512472424c4eda1866a70f9230e096f63b90958173aa5510108308a64ef6f812" Oct 30 02:17:46.357791 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:46.357679 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"512472424c4eda1866a70f9230e096f63b90958173aa5510108308a64ef6f812"} err="failed to get container status \"512472424c4eda1866a70f9230e096f63b90958173aa5510108308a64ef6f812\": rpc error: code = NotFound desc = could not find container \"512472424c4eda1866a70f9230e096f63b90958173aa5510108308a64ef6f812\": container with ID starting with 512472424c4eda1866a70f9230e096f63b90958173aa5510108308a64ef6f812 not found: ID does not exist" Oct 30 02:17:46.357791 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:46.357698 6572 scope.go:122] "RemoveContainer" containerID="e8b9af652baa34b52f5e9747ad0e65dcdec8d32cab773c5e99812ce57d60bf6b" Oct 30 02:17:46.363472 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:46.363445 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdsqp\" (UniqueName: \"kubernetes.io/projected/1e5005f7-fccd-4447-b09d-dd991c1c7733-kube-api-access-bdsqp\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:46.363560 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:46.363475 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfztl\" (UniqueName: \"kubernetes.io/projected/96c5b66f-d98b-491a-93ae-4b2587cf9809-kube-api-access-pfztl\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:46.374603 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:46.374581 6572 scope.go:122] "RemoveContainer" containerID="e8b9af652baa34b52f5e9747ad0e65dcdec8d32cab773c5e99812ce57d60bf6b" Oct 30 02:17:46.374993 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:17:46.374970 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8b9af652baa34b52f5e9747ad0e65dcdec8d32cab773c5e99812ce57d60bf6b\": container with ID starting with e8b9af652baa34b52f5e9747ad0e65dcdec8d32cab773c5e99812ce57d60bf6b not found: ID does not exist" containerID="e8b9af652baa34b52f5e9747ad0e65dcdec8d32cab773c5e99812ce57d60bf6b" Oct 30 02:17:46.375056 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:46.375003 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8b9af652baa34b52f5e9747ad0e65dcdec8d32cab773c5e99812ce57d60bf6b"} err="failed to get container status \"e8b9af652baa34b52f5e9747ad0e65dcdec8d32cab773c5e99812ce57d60bf6b\": rpc error: code = NotFound desc = could not find container \"e8b9af652baa34b52f5e9747ad0e65dcdec8d32cab773c5e99812ce57d60bf6b\": container with ID starting with e8b9af652baa34b52f5e9747ad0e65dcdec8d32cab773c5e99812ce57d60bf6b not found: ID does not exist" Oct 30 02:17:46.399731 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:46.399697 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d44413-86f78b95f6-v547l"] Oct 30 02:17:46.403838 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:46.403801 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d44413-86f78b95f6-v547l"] Oct 30 02:17:46.439803 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:46.438277 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d44413-c6c7b8c75-qqdbt"] Oct 30 02:17:46.442813 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:46.442779 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d44413-c6c7b8c75-qqdbt"] Oct 30 02:17:46.543384 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:46.543349 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e5005f7-fccd-4447-b09d-dd991c1c7733" path="/var/lib/kubelet/pods/1e5005f7-fccd-4447-b09d-dd991c1c7733/volumes" Oct 30 02:17:46.543835 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:46.543821 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96c5b66f-d98b-491a-93ae-4b2587cf9809" path="/var/lib/kubelet/pods/96c5b66f-d98b-491a-93ae-4b2587cf9809/volumes" Oct 30 02:17:47.392732 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:47.392699 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:47.734738 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:47.734702 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:47.931993 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:47.931960 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:48.026586 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:48.026481 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-601d4b-4vzjt"] Oct 30 02:17:48.036422 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:48.036389 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-601d4b-4vzjt"] Oct 30 02:17:48.117864 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:48.117829 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:48.543152 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:48.543115 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82bd130d-77a4-414d-8fac-57e607a0d1b6" path="/var/lib/kubelet/pods/82bd130d-77a4-414d-8fac-57e607a0d1b6/volumes" Oct 30 02:17:48.581767 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:48.581722 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:48.606924 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:48.606886 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d44413-vtt7z"] Oct 30 02:17:48.618086 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:48.618043 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d44413-vtt7z"] Oct 30 02:17:48.777202 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:48.777163 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:49.125405 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:49.125370 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:49.340056 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:49.340012 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eda881-67d94b78bc-6n9dq" podUID="6d6c050f-1131-4874-b064-3ef68d3b990b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:17:49.340308 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:49.340285 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-eda881-5c7cd4d9b-hqnjh" podUID="2de76834-8ea2-4e53-8e91-ebd04ae69886" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:17:50.260823 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:50.260790 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:50.543143 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:50.543049 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e3680ff-1ee0-4f4d-ada9-325766ae0a97" path="/var/lib/kubelet/pods/9e3680ff-1ee0-4f4d-ada9-325766ae0a97/volumes" Oct 30 02:17:50.997949 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:50.997902 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-601d4b-565cc7b475-mhc5s" podUID="f7f30150-5a1b-4c5b-8044-cfe1774eb76b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:17:50.998144 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:50.997970 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-601d4b-b48cfb7b-zk29s" podUID="7c580bbf-cee6-4937-8ea7-fd44837e076d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:17:51.124420 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:51.124374 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-eda881-5c7cd4d9b-hqnjh" Oct 30 02:17:51.181516 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:51.181479 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2de76834-8ea2-4e53-8e91-ebd04ae69886-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de76834-8ea2-4e53-8e91-ebd04ae69886-plugins-config-volume\") pod \"2de76834-8ea2-4e53-8e91-ebd04ae69886\" (UID: \"2de76834-8ea2-4e53-8e91-ebd04ae69886\") " Oct 30 02:17:51.182115 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:51.182076 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2de76834-8ea2-4e53-8e91-ebd04ae69886-kube-api-access-qw7qr\" (UniqueName: \"kubernetes.io/projected/2de76834-8ea2-4e53-8e91-ebd04ae69886-kube-api-access-qw7qr\") pod \"2de76834-8ea2-4e53-8e91-ebd04ae69886\" (UID: \"2de76834-8ea2-4e53-8e91-ebd04ae69886\") " Oct 30 02:17:51.185364 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:51.185328 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2de76834-8ea2-4e53-8e91-ebd04ae69886-plugins-config-volume" pod "2de76834-8ea2-4e53-8e91-ebd04ae69886" (UID: "2de76834-8ea2-4e53-8e91-ebd04ae69886"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:17:51.188585 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:51.188557 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2de76834-8ea2-4e53-8e91-ebd04ae69886-kube-api-access-qw7qr" pod "2de76834-8ea2-4e53-8e91-ebd04ae69886" (UID: "2de76834-8ea2-4e53-8e91-ebd04ae69886"). InnerVolumeSpecName "kube-api-access-qw7qr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:51.190022 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:51.189997 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eda881-67d94b78bc-6n9dq" Oct 30 02:17:51.284434 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:51.284395 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d6c050f-1131-4874-b064-3ef68d3b990b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d6c050f-1131-4874-b064-3ef68d3b990b-plugins-config-volume\") pod \"6d6c050f-1131-4874-b064-3ef68d3b990b\" (UID: \"6d6c050f-1131-4874-b064-3ef68d3b990b\") " Oct 30 02:17:51.284984 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:51.284636 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d6c050f-1131-4874-b064-3ef68d3b990b-kube-api-access-7x5mn\" (UniqueName: \"kubernetes.io/projected/6d6c050f-1131-4874-b064-3ef68d3b990b-kube-api-access-7x5mn\") pod \"6d6c050f-1131-4874-b064-3ef68d3b990b\" (UID: \"6d6c050f-1131-4874-b064-3ef68d3b990b\") " Oct 30 02:17:51.284984 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:51.284828 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d6c050f-1131-4874-b064-3ef68d3b990b-plugins-config-volume" pod "6d6c050f-1131-4874-b064-3ef68d3b990b" (UID: "6d6c050f-1131-4874-b064-3ef68d3b990b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:17:51.285180 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:51.285158 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de76834-8ea2-4e53-8e91-ebd04ae69886-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:51.285243 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:51.285184 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw7qr\" (UniqueName: \"kubernetes.io/projected/2de76834-8ea2-4e53-8e91-ebd04ae69886-kube-api-access-qw7qr\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:51.285243 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:51.285199 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d6c050f-1131-4874-b064-3ef68d3b990b-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:51.286759 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:51.286729 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d6c050f-1131-4874-b064-3ef68d3b990b-kube-api-access-7x5mn" pod "6d6c050f-1131-4874-b064-3ef68d3b990b" (UID: "6d6c050f-1131-4874-b064-3ef68d3b990b"). InnerVolumeSpecName "kube-api-access-7x5mn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:51.381903 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:51.381815 6572 generic.go:356] "Generic (PLEG): container finished" podID="2de76834-8ea2-4e53-8e91-ebd04ae69886" containerID="ef418e27639b8c25bc243d572d9415a6b4434f39f26edd2ce9aabbf9af948479" exitCode=0 Oct 30 02:17:51.382072 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:51.381905 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-eda881-5c7cd4d9b-hqnjh" event={"ID":"2de76834-8ea2-4e53-8e91-ebd04ae69886","Type":"ContainerDied","Data":"ef418e27639b8c25bc243d572d9415a6b4434f39f26edd2ce9aabbf9af948479"} Oct 30 02:17:51.382072 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:51.381919 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-eda881-5c7cd4d9b-hqnjh" Oct 30 02:17:51.382072 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:51.381996 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-eda881-5c7cd4d9b-hqnjh" event={"ID":"2de76834-8ea2-4e53-8e91-ebd04ae69886","Type":"ContainerDied","Data":"5b3eb5123329e0cbf0385683507ad72f0f152da6da1b1753259c5776ce9538a8"} Oct 30 02:17:51.382072 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:51.382025 6572 scope.go:122] "RemoveContainer" containerID="ef418e27639b8c25bc243d572d9415a6b4434f39f26edd2ce9aabbf9af948479" Oct 30 02:17:51.383669 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:51.383639 6572 generic.go:356] "Generic (PLEG): container finished" podID="6d6c050f-1131-4874-b064-3ef68d3b990b" containerID="6737b3a464cecce2a8a42b64ad22a0549c0a53b74b0a127f19e40fb215c6e941" exitCode=0 Oct 30 02:17:51.383825 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:51.383675 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eda881-67d94b78bc-6n9dq" Oct 30 02:17:51.383825 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:51.383673 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eda881-67d94b78bc-6n9dq" event={"ID":"6d6c050f-1131-4874-b064-3ef68d3b990b","Type":"ContainerDied","Data":"6737b3a464cecce2a8a42b64ad22a0549c0a53b74b0a127f19e40fb215c6e941"} Oct 30 02:17:51.383825 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:51.383734 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-eda881-67d94b78bc-6n9dq" event={"ID":"6d6c050f-1131-4874-b064-3ef68d3b990b","Type":"ContainerDied","Data":"d97118b851aaef57b838be4ec92d930b3ee59a1b90c372278b32a8d7114ff9b4"} Oct 30 02:17:51.386175 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:51.386138 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x5mn\" (UniqueName: \"kubernetes.io/projected/6d6c050f-1131-4874-b064-3ef68d3b990b-kube-api-access-7x5mn\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:51.397038 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:51.397015 6572 scope.go:122] "RemoveContainer" containerID="ef418e27639b8c25bc243d572d9415a6b4434f39f26edd2ce9aabbf9af948479" Oct 30 02:17:51.397372 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:17:51.397349 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef418e27639b8c25bc243d572d9415a6b4434f39f26edd2ce9aabbf9af948479\": container with ID starting with ef418e27639b8c25bc243d572d9415a6b4434f39f26edd2ce9aabbf9af948479 not found: ID does not exist" containerID="ef418e27639b8c25bc243d572d9415a6b4434f39f26edd2ce9aabbf9af948479" Oct 30 02:17:51.397436 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:51.397380 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef418e27639b8c25bc243d572d9415a6b4434f39f26edd2ce9aabbf9af948479"} err="failed to get container status \"ef418e27639b8c25bc243d572d9415a6b4434f39f26edd2ce9aabbf9af948479\": rpc error: code = NotFound desc = could not find container \"ef418e27639b8c25bc243d572d9415a6b4434f39f26edd2ce9aabbf9af948479\": container with ID starting with ef418e27639b8c25bc243d572d9415a6b4434f39f26edd2ce9aabbf9af948479 not found: ID does not exist" Oct 30 02:17:51.397436 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:51.397396 6572 scope.go:122] "RemoveContainer" containerID="6737b3a464cecce2a8a42b64ad22a0549c0a53b74b0a127f19e40fb215c6e941" Oct 30 02:17:51.408172 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:51.408148 6572 scope.go:122] "RemoveContainer" containerID="6737b3a464cecce2a8a42b64ad22a0549c0a53b74b0a127f19e40fb215c6e941" Oct 30 02:17:51.408456 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:17:51.408434 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6737b3a464cecce2a8a42b64ad22a0549c0a53b74b0a127f19e40fb215c6e941\": container with ID starting with 6737b3a464cecce2a8a42b64ad22a0549c0a53b74b0a127f19e40fb215c6e941 not found: ID does not exist" containerID="6737b3a464cecce2a8a42b64ad22a0549c0a53b74b0a127f19e40fb215c6e941" Oct 30 02:17:51.408510 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:51.408465 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6737b3a464cecce2a8a42b64ad22a0549c0a53b74b0a127f19e40fb215c6e941"} err="failed to get container status \"6737b3a464cecce2a8a42b64ad22a0549c0a53b74b0a127f19e40fb215c6e941\": rpc error: code = NotFound desc = could not find container \"6737b3a464cecce2a8a42b64ad22a0549c0a53b74b0a127f19e40fb215c6e941\": container with ID starting with 6737b3a464cecce2a8a42b64ad22a0549c0a53b74b0a127f19e40fb215c6e941 not found: ID does not exist" Oct 30 02:17:51.448620 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:51.448588 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-eda881-67d94b78bc-6n9dq"] Oct 30 02:17:51.451822 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:51.451800 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-eda881-67d94b78bc-6n9dq"] Oct 30 02:17:51.481121 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:51.481080 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-eda881-5c7cd4d9b-hqnjh"] Oct 30 02:17:51.484983 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:51.484951 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-eda881-5c7cd4d9b-hqnjh"] Oct 30 02:17:51.641327 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:51.641246 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:52.542480 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:52.542441 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2de76834-8ea2-4e53-8e91-ebd04ae69886" path="/var/lib/kubelet/pods/2de76834-8ea2-4e53-8e91-ebd04ae69886/volumes" Oct 30 02:17:52.542931 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:52.542910 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d6c050f-1131-4874-b064-3ef68d3b990b" path="/var/lib/kubelet/pods/6d6c050f-1131-4874-b064-3ef68d3b990b/volumes" Oct 30 02:17:53.032509 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:53.032476 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:54.436731 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:54.436694 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:54.868824 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:54.868701 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:55.066440 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.066405 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:55.095827 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.095789 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7"] Oct 30 02:17:55.096122 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.096087 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" podUID="a9562e58-739e-44e4-a2ca-448684bddd75" containerName="istio-proxy" containerID="cri-o://40ed7f4ae4dc6cf2f2f4eaf65012a6383221347affc60499a9bc46e91171d0c6" gracePeriod=30 Oct 30 02:17:55.267875 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.267841 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:55.501884 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.501838 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:55.639718 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.639623 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a19e-dqfj8"] Oct 30 02:17:55.640240 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.640226 6572 cpu_manager.go:414] "containerMap: removing container" podUID="2de76834-8ea2-4e53-8e91-ebd04ae69886" containerName="payload-processing" Oct 30 02:17:55.640292 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.640241 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2de76834-8ea2-4e53-8e91-ebd04ae69886" containerName="payload-processing" Oct 30 02:17:55.640292 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.640255 6572 cpu_manager.go:414] "containerMap: removing container" podUID="a36c6723-df85-48fa-9b09-e466c2c28500" containerName="revoke-keys" Oct 30 02:17:55.640292 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.640260 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a36c6723-df85-48fa-9b09-e466c2c28500" containerName="revoke-keys" Oct 30 02:17:55.640292 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.640279 6572 cpu_manager.go:414] "containerMap: removing container" podUID="af284c10-c406-439b-98ad-c0a5a35b9a8e" containerName="revoke-keys" Oct 30 02:17:55.640292 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.640284 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af284c10-c406-439b-98ad-c0a5a35b9a8e" containerName="revoke-keys" Oct 30 02:17:55.640906 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.640300 6572 cpu_manager.go:414] "containerMap: removing container" podUID="1e5005f7-fccd-4447-b09d-dd991c1c7733" containerName="payload-processing" Oct 30 02:17:55.640906 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.640306 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e5005f7-fccd-4447-b09d-dd991c1c7733" containerName="payload-processing" Oct 30 02:17:55.640906 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.640350 6572 cpu_manager.go:414] "containerMap: removing container" podUID="6d6c050f-1131-4874-b064-3ef68d3b990b" containerName="payload-pre-processing" Oct 30 02:17:55.640906 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.640356 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d6c050f-1131-4874-b064-3ef68d3b990b" containerName="payload-pre-processing" Oct 30 02:17:55.640906 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.640366 6572 cpu_manager.go:414] "containerMap: removing container" podUID="96c5b66f-d98b-491a-93ae-4b2587cf9809" containerName="payload-pre-processing" Oct 30 02:17:55.640906 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.640371 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96c5b66f-d98b-491a-93ae-4b2587cf9809" containerName="payload-pre-processing" Oct 30 02:17:55.640906 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.640446 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e5005f7-fccd-4447-b09d-dd991c1c7733" containerName="payload-processing" Oct 30 02:17:55.640906 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.640460 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d6c050f-1131-4874-b064-3ef68d3b990b" containerName="payload-pre-processing" Oct 30 02:17:55.640906 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.640467 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="96c5b66f-d98b-491a-93ae-4b2587cf9809" containerName="payload-pre-processing" Oct 30 02:17:55.640906 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.640474 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="2de76834-8ea2-4e53-8e91-ebd04ae69886" containerName="payload-processing" Oct 30 02:17:55.640906 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.640480 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="a36c6723-df85-48fa-9b09-e466c2c28500" containerName="revoke-keys" Oct 30 02:17:55.640906 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.640486 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="af284c10-c406-439b-98ad-c0a5a35b9a8e" containerName="revoke-keys" Oct 30 02:17:55.645808 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.645737 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a19e-dqfj8" Oct 30 02:17:55.664643 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.664591 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a19e-dqfj8"] Oct 30 02:17:55.669538 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.669497 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:17:55.669538 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.669517 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:17:55.685698 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.685664 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:17:55.685698 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.685694 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:17:55.701150 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.700803 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:55.773963 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.773927 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6r4l\" (UniqueName: \"kubernetes.io/projected/0a57999a-c713-4bc5-ae55-36d06200627e-kube-api-access-j6r4l\") pod \"maas-api-revoke-keys-e2e-extmod-a-76a19e-dqfj8\" (UID: \"0a57999a-c713-4bc5-ae55-36d06200627e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a19e-dqfj8" Oct 30 02:17:55.774155 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.773995 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a57999a-c713-4bc5-ae55-36d06200627e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-76a19e-dqfj8\" (UID: \"0a57999a-c713-4bc5-ae55-36d06200627e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a19e-dqfj8" Oct 30 02:17:55.774213 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.774171 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a57999a-c713-4bc5-ae55-36d06200627e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-76a19e-dqfj8\" (UID: \"0a57999a-c713-4bc5-ae55-36d06200627e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a19e-dqfj8" Oct 30 02:17:55.879770 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.876519 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a57999a-c713-4bc5-ae55-36d06200627e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-76a19e-dqfj8\" (UID: \"0a57999a-c713-4bc5-ae55-36d06200627e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a19e-dqfj8" Oct 30 02:17:55.879770 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.876698 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a57999a-c713-4bc5-ae55-36d06200627e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-76a19e-dqfj8\" (UID: \"0a57999a-c713-4bc5-ae55-36d06200627e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a19e-dqfj8" Oct 30 02:17:55.879770 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.876809 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6r4l\" (UniqueName: \"kubernetes.io/projected/0a57999a-c713-4bc5-ae55-36d06200627e-kube-api-access-j6r4l\") pod \"maas-api-revoke-keys-e2e-extmod-a-76a19e-dqfj8\" (UID: \"0a57999a-c713-4bc5-ae55-36d06200627e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a19e-dqfj8" Oct 30 02:17:55.886767 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.882700 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a57999a-c713-4bc5-ae55-36d06200627e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-76a19e-dqfj8\" (UID: \"0a57999a-c713-4bc5-ae55-36d06200627e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a19e-dqfj8" Oct 30 02:17:55.894323 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.890338 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:55.900645 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.900452 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a57999a-c713-4bc5-ae55-36d06200627e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-76a19e-dqfj8\" (UID: \"0a57999a-c713-4bc5-ae55-36d06200627e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a19e-dqfj8" Oct 30 02:17:55.904449 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.903472 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6r4l\" (UniqueName: \"kubernetes.io/projected/0a57999a-c713-4bc5-ae55-36d06200627e-kube-api-access-j6r4l\") pod \"maas-api-revoke-keys-e2e-extmod-a-76a19e-dqfj8\" (UID: \"0a57999a-c713-4bc5-ae55-36d06200627e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a19e-dqfj8" Oct 30 02:17:55.961163 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:55.961125 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a19e-dqfj8" Oct 30 02:17:56.080267 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:56.080230 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:56.271651 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:56.271507 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-093d23-tw952"] Oct 30 02:17:56.277070 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:56.277045 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-093d23-tw952" Oct 30 02:17:56.294155 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:56.294125 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-093d23-tw952"] Oct 30 02:17:56.309146 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:56.309114 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a19e-dqfj8"] Oct 30 02:17:56.310665 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:17:56.310641 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a57999a_c713_4bc5_ae55_36d06200627e.slice/crio-3c0372583ddf3799938c10c173da81ab17cb74ffb89e5f01b96643fb3a28694a WatchSource:0}: Error finding container 3c0372583ddf3799938c10c173da81ab17cb74ffb89e5f01b96643fb3a28694a: Status 404 returned error can't find the container with id 3c0372583ddf3799938c10c173da81ab17cb74ffb89e5f01b96643fb3a28694a Oct 30 02:17:56.383507 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:56.383470 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dd06944-23a9-493a-8974-e67076019ee2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-093d23-tw952\" (UID: \"5dd06944-23a9-493a-8974-e67076019ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-093d23-tw952" Oct 30 02:17:56.383507 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:56.383511 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dd06944-23a9-493a-8974-e67076019ee2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-093d23-tw952\" (UID: \"5dd06944-23a9-493a-8974-e67076019ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-093d23-tw952" Oct 30 02:17:56.383867 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:56.383545 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2glnk\" (UniqueName: \"kubernetes.io/projected/5dd06944-23a9-493a-8974-e67076019ee2-kube-api-access-2glnk\") pod \"maas-api-revoke-keys-e2e-extmod-b-093d23-tw952\" (UID: \"5dd06944-23a9-493a-8974-e67076019ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-093d23-tw952" Oct 30 02:17:56.439488 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:56.439451 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a19e-dqfj8" event={"ID":"0a57999a-c713-4bc5-ae55-36d06200627e","Type":"ContainerStarted","Data":"3c0372583ddf3799938c10c173da81ab17cb74ffb89e5f01b96643fb3a28694a"} Oct 30 02:17:56.484601 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:56.484567 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dd06944-23a9-493a-8974-e67076019ee2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-093d23-tw952\" (UID: \"5dd06944-23a9-493a-8974-e67076019ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-093d23-tw952" Oct 30 02:17:56.484601 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:56.484610 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dd06944-23a9-493a-8974-e67076019ee2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-093d23-tw952\" (UID: \"5dd06944-23a9-493a-8974-e67076019ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-093d23-tw952" Oct 30 02:17:56.484861 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:56.484658 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2glnk\" (UniqueName: \"kubernetes.io/projected/5dd06944-23a9-493a-8974-e67076019ee2-kube-api-access-2glnk\") pod \"maas-api-revoke-keys-e2e-extmod-b-093d23-tw952\" (UID: \"5dd06944-23a9-493a-8974-e67076019ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-093d23-tw952" Oct 30 02:17:56.485331 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:56.485303 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dd06944-23a9-493a-8974-e67076019ee2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-093d23-tw952\" (UID: \"5dd06944-23a9-493a-8974-e67076019ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-093d23-tw952" Oct 30 02:17:56.487019 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:56.487000 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dd06944-23a9-493a-8974-e67076019ee2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-093d23-tw952\" (UID: \"5dd06944-23a9-493a-8974-e67076019ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-093d23-tw952" Oct 30 02:17:56.510969 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:56.510932 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2glnk\" (UniqueName: \"kubernetes.io/projected/5dd06944-23a9-493a-8974-e67076019ee2-kube-api-access-2glnk\") pod \"maas-api-revoke-keys-e2e-extmod-b-093d23-tw952\" (UID: \"5dd06944-23a9-493a-8974-e67076019ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-093d23-tw952" Oct 30 02:17:56.592789 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:56.592736 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-093d23-tw952" Oct 30 02:17:56.895978 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:56.895947 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-093d23-tw952"] Oct 30 02:17:56.896469 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:17:56.896445 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd06944_23a9_493a_8974_e67076019ee2.slice/crio-69263ddedce9822f2371933065ed1d1fbf8fcfb2c9dfd40f889c09286c0f1d4d WatchSource:0}: Error finding container 69263ddedce9822f2371933065ed1d1fbf8fcfb2c9dfd40f889c09286c0f1d4d: Status 404 returned error can't find the container with id 69263ddedce9822f2371933065ed1d1fbf8fcfb2c9dfd40f889c09286c0f1d4d Oct 30 02:17:57.451902 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:57.451860 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-093d23-tw952" event={"ID":"5dd06944-23a9-493a-8974-e67076019ee2","Type":"ContainerStarted","Data":"fd33921159015ef49bffddae29631c4bd559adead35b13c8cf924fa2ae6a3aec"} Oct 30 02:17:57.451902 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:57.451929 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-093d23-tw952" event={"ID":"5dd06944-23a9-493a-8974-e67076019ee2","Type":"ContainerStarted","Data":"69263ddedce9822f2371933065ed1d1fbf8fcfb2c9dfd40f889c09286c0f1d4d"} Oct 30 02:17:57.454160 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:57.454130 6572 generic.go:356] "Generic (PLEG): container finished" podID="0a57999a-c713-4bc5-ae55-36d06200627e" containerID="8607ccd9fb535a1036384337bde613b600daf2a49889128da45a0aa8f9218b1d" exitCode=0 Oct 30 02:17:57.454360 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:57.454211 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a19e-dqfj8" event={"ID":"0a57999a-c713-4bc5-ae55-36d06200627e","Type":"ContainerDied","Data":"8607ccd9fb535a1036384337bde613b600daf2a49889128da45a0aa8f9218b1d"} Oct 30 02:17:57.490677 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:57.490553 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-093d23-tw952" podStartSLOduration=1.49053408 podStartE2EDuration="1.49053408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:17:57.487969235 +0000 UTC m=+3603.064394683" watchObservedRunningTime="2026-10-30 02:17:57.49053408 +0000 UTC m=+3603.066959481" Oct 30 02:17:57.609950 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:57.609918 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:58.464681 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:58.464636 6572 generic.go:356] "Generic (PLEG): container finished" podID="5dd06944-23a9-493a-8974-e67076019ee2" containerID="fd33921159015ef49bffddae29631c4bd559adead35b13c8cf924fa2ae6a3aec" exitCode=0 Oct 30 02:17:58.464916 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:58.464726 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-093d23-tw952" event={"ID":"5dd06944-23a9-493a-8974-e67076019ee2","Type":"ContainerDied","Data":"fd33921159015ef49bffddae29631c4bd559adead35b13c8cf924fa2ae6a3aec"} Oct 30 02:17:58.638204 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:58.638172 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a19e-dqfj8" Oct 30 02:17:58.716679 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:58.716567 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a57999a-c713-4bc5-ae55-36d06200627e-kube-api-access-j6r4l\" (UniqueName: \"kubernetes.io/projected/0a57999a-c713-4bc5-ae55-36d06200627e-kube-api-access-j6r4l\") pod \"0a57999a-c713-4bc5-ae55-36d06200627e\" (UID: \"0a57999a-c713-4bc5-ae55-36d06200627e\") " Oct 30 02:17:58.716679 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:58.716664 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a57999a-c713-4bc5-ae55-36d06200627e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a57999a-c713-4bc5-ae55-36d06200627e-maas-api-xks-ca\") pod \"0a57999a-c713-4bc5-ae55-36d06200627e\" (UID: \"0a57999a-c713-4bc5-ae55-36d06200627e\") " Oct 30 02:17:58.717009 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:58.716727 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a57999a-c713-4bc5-ae55-36d06200627e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a57999a-c713-4bc5-ae55-36d06200627e-maas-api-openshift-ca\") pod \"0a57999a-c713-4bc5-ae55-36d06200627e\" (UID: \"0a57999a-c713-4bc5-ae55-36d06200627e\") " Oct 30 02:17:58.717164 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:58.717140 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a57999a-c713-4bc5-ae55-36d06200627e-maas-api-openshift-ca" pod "0a57999a-c713-4bc5-ae55-36d06200627e" (UID: "0a57999a-c713-4bc5-ae55-36d06200627e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:17:58.717395 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:58.717374 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a57999a-c713-4bc5-ae55-36d06200627e-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:58.719158 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:58.719126 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a57999a-c713-4bc5-ae55-36d06200627e-maas-api-xks-ca" pod "0a57999a-c713-4bc5-ae55-36d06200627e" (UID: "0a57999a-c713-4bc5-ae55-36d06200627e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:17:58.719158 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:58.719145 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a57999a-c713-4bc5-ae55-36d06200627e-kube-api-access-j6r4l" pod "0a57999a-c713-4bc5-ae55-36d06200627e" (UID: "0a57999a-c713-4bc5-ae55-36d06200627e"). InnerVolumeSpecName "kube-api-access-j6r4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:58.819132 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:58.819096 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a57999a-c713-4bc5-ae55-36d06200627e-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:58.819132 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:58.819131 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6r4l\" (UniqueName: \"kubernetes.io/projected/0a57999a-c713-4bc5-ae55-36d06200627e-kube-api-access-j6r4l\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:58.963073 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:58.963038 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:59.475128 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:59.475095 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a19e-dqfj8" Oct 30 02:17:59.475324 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:59.475134 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a19e-dqfj8" event={"ID":"0a57999a-c713-4bc5-ae55-36d06200627e","Type":"ContainerDied","Data":"3c0372583ddf3799938c10c173da81ab17cb74ffb89e5f01b96643fb3a28694a"} Oct 30 02:17:59.475324 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:59.475193 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c0372583ddf3799938c10c173da81ab17cb74ffb89e5f01b96643fb3a28694a" Oct 30 02:17:59.493066 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:59.493035 6572 scope.go:122] "RemoveContainer" containerID="7486ce9f73778af1673217a6d2e89ed7f8c09a043082594bd2c334b8ac034518" Oct 30 02:17:59.637883 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:59.637850 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-093d23-tw952" Oct 30 02:17:59.736717 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:59.736616 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dd06944-23a9-493a-8974-e67076019ee2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dd06944-23a9-493a-8974-e67076019ee2-maas-api-openshift-ca\") pod \"5dd06944-23a9-493a-8974-e67076019ee2\" (UID: \"5dd06944-23a9-493a-8974-e67076019ee2\") " Oct 30 02:17:59.737245 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:59.736828 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dd06944-23a9-493a-8974-e67076019ee2-kube-api-access-2glnk\" (UniqueName: \"kubernetes.io/projected/5dd06944-23a9-493a-8974-e67076019ee2-kube-api-access-2glnk\") pod \"5dd06944-23a9-493a-8974-e67076019ee2\" (UID: \"5dd06944-23a9-493a-8974-e67076019ee2\") " Oct 30 02:17:59.737245 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:59.736909 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5dd06944-23a9-493a-8974-e67076019ee2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dd06944-23a9-493a-8974-e67076019ee2-maas-api-xks-ca\") pod \"5dd06944-23a9-493a-8974-e67076019ee2\" (UID: \"5dd06944-23a9-493a-8974-e67076019ee2\") " Oct 30 02:17:59.737245 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:59.737096 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dd06944-23a9-493a-8974-e67076019ee2-maas-api-openshift-ca" pod "5dd06944-23a9-493a-8974-e67076019ee2" (UID: "5dd06944-23a9-493a-8974-e67076019ee2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:17:59.737388 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:59.737290 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dd06944-23a9-493a-8974-e67076019ee2-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:59.738996 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:59.738968 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dd06944-23a9-493a-8974-e67076019ee2-maas-api-xks-ca" pod "5dd06944-23a9-493a-8974-e67076019ee2" (UID: "5dd06944-23a9-493a-8974-e67076019ee2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:17:59.739121 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:59.739054 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dd06944-23a9-493a-8974-e67076019ee2-kube-api-access-2glnk" pod "5dd06944-23a9-493a-8974-e67076019ee2" (UID: "5dd06944-23a9-493a-8974-e67076019ee2"). InnerVolumeSpecName "kube-api-access-2glnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:59.839061 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:59.839011 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dd06944-23a9-493a-8974-e67076019ee2-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:17:59.839061 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:17:59.839058 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2glnk\" (UniqueName: \"kubernetes.io/projected/5dd06944-23a9-493a-8974-e67076019ee2-kube-api-access-2glnk\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.290700 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.290667 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:00.373126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.373099 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" Oct 30 02:18:00.487966 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.487922 6572 generic.go:356] "Generic (PLEG): container finished" podID="a9562e58-739e-44e4-a2ca-448684bddd75" containerID="40ed7f4ae4dc6cf2f2f4eaf65012a6383221347affc60499a9bc46e91171d0c6" exitCode=0 Oct 30 02:18:00.488304 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.488047 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" event={"ID":"a9562e58-739e-44e4-a2ca-448684bddd75","Type":"ContainerDied","Data":"40ed7f4ae4dc6cf2f2f4eaf65012a6383221347affc60499a9bc46e91171d0c6"} Oct 30 02:18:00.488304 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.488102 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" event={"ID":"a9562e58-739e-44e4-a2ca-448684bddd75","Type":"ContainerDied","Data":"91ea36b2b356575e5990f1c9b9c8905048809ddf83d1f3389983b8485bfd10fa"} Oct 30 02:18:00.488304 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.488134 6572 scope.go:122] "RemoveContainer" containerID="40ed7f4ae4dc6cf2f2f4eaf65012a6383221347affc60499a9bc46e91171d0c6" Oct 30 02:18:00.488545 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.488309 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7" Oct 30 02:18:00.491227 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.491193 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-093d23-tw952" event={"ID":"5dd06944-23a9-493a-8974-e67076019ee2","Type":"ContainerDied","Data":"69263ddedce9822f2371933065ed1d1fbf8fcfb2c9dfd40f889c09286c0f1d4d"} Oct 30 02:18:00.491388 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.491254 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69263ddedce9822f2371933065ed1d1fbf8fcfb2c9dfd40f889c09286c0f1d4d" Oct 30 02:18:00.491388 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.491329 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-093d23-tw952" Oct 30 02:18:00.514496 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.514464 6572 scope.go:122] "RemoveContainer" containerID="40ed7f4ae4dc6cf2f2f4eaf65012a6383221347affc60499a9bc46e91171d0c6" Oct 30 02:18:00.514941 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:18:00.514910 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40ed7f4ae4dc6cf2f2f4eaf65012a6383221347affc60499a9bc46e91171d0c6\": container with ID starting with 40ed7f4ae4dc6cf2f2f4eaf65012a6383221347affc60499a9bc46e91171d0c6 not found: ID does not exist" containerID="40ed7f4ae4dc6cf2f2f4eaf65012a6383221347affc60499a9bc46e91171d0c6" Oct 30 02:18:00.515074 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.514949 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40ed7f4ae4dc6cf2f2f4eaf65012a6383221347affc60499a9bc46e91171d0c6"} err="failed to get container status \"40ed7f4ae4dc6cf2f2f4eaf65012a6383221347affc60499a9bc46e91171d0c6\": rpc error: code = NotFound desc = could not find container \"40ed7f4ae4dc6cf2f2f4eaf65012a6383221347affc60499a9bc46e91171d0c6\": container with ID starting with 40ed7f4ae4dc6cf2f2f4eaf65012a6383221347affc60499a9bc46e91171d0c6 not found: ID does not exist" Oct 30 02:18:00.549032 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.548977 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9562e58-739e-44e4-a2ca-448684bddd75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9562e58-739e-44e4-a2ca-448684bddd75-istio-envoy\") pod \"a9562e58-739e-44e4-a2ca-448684bddd75\" (UID: \"a9562e58-739e-44e4-a2ca-448684bddd75\") " Oct 30 02:18:00.549244 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.549216 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9562e58-739e-44e4-a2ca-448684bddd75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9562e58-739e-44e4-a2ca-448684bddd75-istio-data\") pod \"a9562e58-739e-44e4-a2ca-448684bddd75\" (UID: \"a9562e58-739e-44e4-a2ca-448684bddd75\") " Oct 30 02:18:00.549324 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.549277 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9562e58-739e-44e4-a2ca-448684bddd75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9562e58-739e-44e4-a2ca-448684bddd75-workload-socket\") pod \"a9562e58-739e-44e4-a2ca-448684bddd75\" (UID: \"a9562e58-739e-44e4-a2ca-448684bddd75\") " Oct 30 02:18:00.549545 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.549517 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9562e58-739e-44e4-a2ca-448684bddd75-istio-data" pod "a9562e58-739e-44e4-a2ca-448684bddd75" (UID: "a9562e58-739e-44e4-a2ca-448684bddd75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:00.549674 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.549530 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9562e58-739e-44e4-a2ca-448684bddd75-workload-socket" pod "a9562e58-739e-44e4-a2ca-448684bddd75" (UID: "a9562e58-739e-44e4-a2ca-448684bddd75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:00.550584 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.550290 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9562e58-739e-44e4-a2ca-448684bddd75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9562e58-739e-44e4-a2ca-448684bddd75-istiod-ca-cert\") pod \"a9562e58-739e-44e4-a2ca-448684bddd75\" (UID: \"a9562e58-739e-44e4-a2ca-448684bddd75\") " Oct 30 02:18:00.550584 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.550348 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9562e58-739e-44e4-a2ca-448684bddd75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9562e58-739e-44e4-a2ca-448684bddd75-istio-podinfo\") pod \"a9562e58-739e-44e4-a2ca-448684bddd75\" (UID: \"a9562e58-739e-44e4-a2ca-448684bddd75\") " Oct 30 02:18:00.550584 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.550388 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9562e58-739e-44e4-a2ca-448684bddd75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9562e58-739e-44e4-a2ca-448684bddd75-workload-certs\") pod \"a9562e58-739e-44e4-a2ca-448684bddd75\" (UID: \"a9562e58-739e-44e4-a2ca-448684bddd75\") " Oct 30 02:18:00.550584 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.550474 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9562e58-739e-44e4-a2ca-448684bddd75-istio-token\" (UniqueName: \"kubernetes.io/projected/a9562e58-739e-44e4-a2ca-448684bddd75-istio-token\") pod \"a9562e58-739e-44e4-a2ca-448684bddd75\" (UID: \"a9562e58-739e-44e4-a2ca-448684bddd75\") " Oct 30 02:18:00.550584 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.550527 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9562e58-739e-44e4-a2ca-448684bddd75-kube-api-access-vq7tc\" (UniqueName: \"kubernetes.io/projected/a9562e58-739e-44e4-a2ca-448684bddd75-kube-api-access-vq7tc\") pod \"a9562e58-739e-44e4-a2ca-448684bddd75\" (UID: \"a9562e58-739e-44e4-a2ca-448684bddd75\") " Oct 30 02:18:00.550584 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.550563 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9562e58-739e-44e4-a2ca-448684bddd75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9562e58-739e-44e4-a2ca-448684bddd75-credential-socket\") pod \"a9562e58-739e-44e4-a2ca-448684bddd75\" (UID: \"a9562e58-739e-44e4-a2ca-448684bddd75\") " Oct 30 02:18:00.551465 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.551201 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9562e58-739e-44e4-a2ca-448684bddd75-credential-socket" pod "a9562e58-739e-44e4-a2ca-448684bddd75" (UID: "a9562e58-739e-44e4-a2ca-448684bddd75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:00.551465 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.551225 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9562e58-739e-44e4-a2ca-448684bddd75-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.551465 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.551246 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9562e58-739e-44e4-a2ca-448684bddd75-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.551465 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.551324 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9562e58-739e-44e4-a2ca-448684bddd75-workload-certs" pod "a9562e58-739e-44e4-a2ca-448684bddd75" (UID: "a9562e58-739e-44e4-a2ca-448684bddd75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:00.551919 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.551895 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9562e58-739e-44e4-a2ca-448684bddd75-istiod-ca-cert" pod "a9562e58-739e-44e4-a2ca-448684bddd75" (UID: "a9562e58-739e-44e4-a2ca-448684bddd75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:00.553926 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.552591 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9562e58-739e-44e4-a2ca-448684bddd75-istio-envoy" pod "a9562e58-739e-44e4-a2ca-448684bddd75" (UID: "a9562e58-739e-44e4-a2ca-448684bddd75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:00.553926 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.553732 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9562e58-739e-44e4-a2ca-448684bddd75-kube-api-access-vq7tc" pod "a9562e58-739e-44e4-a2ca-448684bddd75" (UID: "a9562e58-739e-44e4-a2ca-448684bddd75"). InnerVolumeSpecName "kube-api-access-vq7tc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:00.554159 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.554132 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9562e58-739e-44e4-a2ca-448684bddd75-istio-token" pod "a9562e58-739e-44e4-a2ca-448684bddd75" (UID: "a9562e58-739e-44e4-a2ca-448684bddd75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:00.555071 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.555051 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9562e58-739e-44e4-a2ca-448684bddd75-istio-podinfo" pod "a9562e58-739e-44e4-a2ca-448684bddd75" (UID: "a9562e58-739e-44e4-a2ca-448684bddd75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:18:00.652960 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.652928 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9562e58-739e-44e4-a2ca-448684bddd75-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.652960 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.652956 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq7tc\" (UniqueName: \"kubernetes.io/projected/a9562e58-739e-44e4-a2ca-448684bddd75-kube-api-access-vq7tc\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.652960 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.652966 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9562e58-739e-44e4-a2ca-448684bddd75-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.653221 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.652975 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9562e58-739e-44e4-a2ca-448684bddd75-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.653221 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.652984 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9562e58-739e-44e4-a2ca-448684bddd75-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.653221 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.652992 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9562e58-739e-44e4-a2ca-448684bddd75-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.653221 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.652999 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9562e58-739e-44e4-a2ca-448684bddd75-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.843626 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.843591 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7"] Oct 30 02:18:00.847354 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.847317 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-4vxl7"] Oct 30 02:18:00.862972 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.862937 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-601d4b-b48cfb7b-zk29s" Oct 30 02:18:00.898145 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.898118 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-601d4b-565cc7b475-mhc5s" Oct 30 02:18:00.957134 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.957093 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c580bbf-cee6-4937-8ea7-fd44837e076d-kube-api-access-pwmkg\" (UniqueName: \"kubernetes.io/projected/7c580bbf-cee6-4937-8ea7-fd44837e076d-kube-api-access-pwmkg\") pod \"7c580bbf-cee6-4937-8ea7-fd44837e076d\" (UID: \"7c580bbf-cee6-4937-8ea7-fd44837e076d\") " Oct 30 02:18:00.957360 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.957176 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c580bbf-cee6-4937-8ea7-fd44837e076d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c580bbf-cee6-4937-8ea7-fd44837e076d-plugins-config-volume\") pod \"7c580bbf-cee6-4937-8ea7-fd44837e076d\" (UID: \"7c580bbf-cee6-4937-8ea7-fd44837e076d\") " Oct 30 02:18:00.957585 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.957561 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c580bbf-cee6-4937-8ea7-fd44837e076d-plugins-config-volume" pod "7c580bbf-cee6-4937-8ea7-fd44837e076d" (UID: "7c580bbf-cee6-4937-8ea7-fd44837e076d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:00.959438 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:00.959404 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c580bbf-cee6-4937-8ea7-fd44837e076d-kube-api-access-pwmkg" pod "7c580bbf-cee6-4937-8ea7-fd44837e076d" (UID: "7c580bbf-cee6-4937-8ea7-fd44837e076d"). InnerVolumeSpecName "kube-api-access-pwmkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:01.058252 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:01.058212 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7f30150-5a1b-4c5b-8044-cfe1774eb76b-kube-api-access-cnzcr\" (UniqueName: \"kubernetes.io/projected/f7f30150-5a1b-4c5b-8044-cfe1774eb76b-kube-api-access-cnzcr\") pod \"f7f30150-5a1b-4c5b-8044-cfe1774eb76b\" (UID: \"f7f30150-5a1b-4c5b-8044-cfe1774eb76b\") " Oct 30 02:18:01.058462 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:01.058367 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7f30150-5a1b-4c5b-8044-cfe1774eb76b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7f30150-5a1b-4c5b-8044-cfe1774eb76b-plugins-config-volume\") pod \"f7f30150-5a1b-4c5b-8044-cfe1774eb76b\" (UID: \"f7f30150-5a1b-4c5b-8044-cfe1774eb76b\") " Oct 30 02:18:01.058836 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:01.058778 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7f30150-5a1b-4c5b-8044-cfe1774eb76b-plugins-config-volume" pod "f7f30150-5a1b-4c5b-8044-cfe1774eb76b" (UID: "f7f30150-5a1b-4c5b-8044-cfe1774eb76b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:01.059001 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:01.058980 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwmkg\" (UniqueName: \"kubernetes.io/projected/7c580bbf-cee6-4937-8ea7-fd44837e076d-kube-api-access-pwmkg\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:01.059070 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:01.059002 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c580bbf-cee6-4937-8ea7-fd44837e076d-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:01.059070 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:01.059016 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7f30150-5a1b-4c5b-8044-cfe1774eb76b-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:01.060578 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:01.060554 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f30150-5a1b-4c5b-8044-cfe1774eb76b-kube-api-access-cnzcr" pod "f7f30150-5a1b-4c5b-8044-cfe1774eb76b" (UID: "f7f30150-5a1b-4c5b-8044-cfe1774eb76b"). InnerVolumeSpecName "kube-api-access-cnzcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:01.160214 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:01.160181 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnzcr\" (UniqueName: \"kubernetes.io/projected/f7f30150-5a1b-4c5b-8044-cfe1774eb76b-kube-api-access-cnzcr\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:01.500603 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:01.500511 6572 generic.go:356] "Generic (PLEG): container finished" podID="f7f30150-5a1b-4c5b-8044-cfe1774eb76b" containerID="6433cfff0722adfd087c6fa1004d86ed4ebb409fe889485bccce6d7c835f9a05" exitCode=0 Oct 30 02:18:01.500603 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:01.500582 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-601d4b-565cc7b475-mhc5s" Oct 30 02:18:01.500603 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:01.500582 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-601d4b-565cc7b475-mhc5s" event={"ID":"f7f30150-5a1b-4c5b-8044-cfe1774eb76b","Type":"ContainerDied","Data":"6433cfff0722adfd087c6fa1004d86ed4ebb409fe889485bccce6d7c835f9a05"} Oct 30 02:18:01.500982 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:01.500677 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-601d4b-565cc7b475-mhc5s" event={"ID":"f7f30150-5a1b-4c5b-8044-cfe1774eb76b","Type":"ContainerDied","Data":"be991266a78995a43ff2c607186b9594db6bb4ed2fd0b6d19842a371c869fb72"} Oct 30 02:18:01.500982 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:01.500708 6572 scope.go:122] "RemoveContainer" containerID="6433cfff0722adfd087c6fa1004d86ed4ebb409fe889485bccce6d7c835f9a05" Oct 30 02:18:01.502490 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:01.502459 6572 generic.go:356] "Generic (PLEG): container finished" podID="7c580bbf-cee6-4937-8ea7-fd44837e076d" containerID="7bcf40de3cb91740af4e29865456475c612e1a3ef6a87dc9fa9aa259f69d165e" exitCode=0 Oct 30 02:18:01.502610 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:01.502522 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-601d4b-b48cfb7b-zk29s" event={"ID":"7c580bbf-cee6-4937-8ea7-fd44837e076d","Type":"ContainerDied","Data":"7bcf40de3cb91740af4e29865456475c612e1a3ef6a87dc9fa9aa259f69d165e"} Oct 30 02:18:01.502610 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:01.502537 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-601d4b-b48cfb7b-zk29s" Oct 30 02:18:01.502610 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:01.502577 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-601d4b-b48cfb7b-zk29s" event={"ID":"7c580bbf-cee6-4937-8ea7-fd44837e076d","Type":"ContainerDied","Data":"a186c539ebe4b1ac879ae3add7b014acf5b55b133c221e4210c16a83ab6b08c1"} Oct 30 02:18:01.513650 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:01.513621 6572 scope.go:122] "RemoveContainer" containerID="6433cfff0722adfd087c6fa1004d86ed4ebb409fe889485bccce6d7c835f9a05" Oct 30 02:18:01.513988 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:18:01.513965 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6433cfff0722adfd087c6fa1004d86ed4ebb409fe889485bccce6d7c835f9a05\": container with ID starting with 6433cfff0722adfd087c6fa1004d86ed4ebb409fe889485bccce6d7c835f9a05 not found: ID does not exist" containerID="6433cfff0722adfd087c6fa1004d86ed4ebb409fe889485bccce6d7c835f9a05" Oct 30 02:18:01.514053 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:01.513997 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6433cfff0722adfd087c6fa1004d86ed4ebb409fe889485bccce6d7c835f9a05"} err="failed to get container status \"6433cfff0722adfd087c6fa1004d86ed4ebb409fe889485bccce6d7c835f9a05\": rpc error: code = NotFound desc = could not find container \"6433cfff0722adfd087c6fa1004d86ed4ebb409fe889485bccce6d7c835f9a05\": container with ID starting with 6433cfff0722adfd087c6fa1004d86ed4ebb409fe889485bccce6d7c835f9a05 not found: ID does not exist" Oct 30 02:18:01.514053 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:01.514016 6572 scope.go:122] "RemoveContainer" containerID="7bcf40de3cb91740af4e29865456475c612e1a3ef6a87dc9fa9aa259f69d165e" Oct 30 02:18:01.525077 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:01.525053 6572 scope.go:122] "RemoveContainer" containerID="7bcf40de3cb91740af4e29865456475c612e1a3ef6a87dc9fa9aa259f69d165e" Oct 30 02:18:01.525386 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:18:01.525363 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bcf40de3cb91740af4e29865456475c612e1a3ef6a87dc9fa9aa259f69d165e\": container with ID starting with 7bcf40de3cb91740af4e29865456475c612e1a3ef6a87dc9fa9aa259f69d165e not found: ID does not exist" containerID="7bcf40de3cb91740af4e29865456475c612e1a3ef6a87dc9fa9aa259f69d165e" Oct 30 02:18:01.525467 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:01.525398 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bcf40de3cb91740af4e29865456475c612e1a3ef6a87dc9fa9aa259f69d165e"} err="failed to get container status \"7bcf40de3cb91740af4e29865456475c612e1a3ef6a87dc9fa9aa259f69d165e\": rpc error: code = NotFound desc = could not find container \"7bcf40de3cb91740af4e29865456475c612e1a3ef6a87dc9fa9aa259f69d165e\": container with ID starting with 7bcf40de3cb91740af4e29865456475c612e1a3ef6a87dc9fa9aa259f69d165e not found: ID does not exist" Oct 30 02:18:01.550356 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:01.550321 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-601d4b-b48cfb7b-zk29s"] Oct 30 02:18:01.561674 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:01.561641 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-601d4b-b48cfb7b-zk29s"] Oct 30 02:18:01.589405 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:01.589363 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-601d4b-565cc7b475-mhc5s"] Oct 30 02:18:01.593564 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:01.593528 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-601d4b-565cc7b475-mhc5s"] Oct 30 02:18:01.771349 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:01.771243 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:02.543940 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:02.543903 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c580bbf-cee6-4937-8ea7-fd44837e076d" path="/var/lib/kubelet/pods/7c580bbf-cee6-4937-8ea7-fd44837e076d/volumes" Oct 30 02:18:02.544420 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:02.544405 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9562e58-739e-44e4-a2ca-448684bddd75" path="/var/lib/kubelet/pods/a9562e58-739e-44e4-a2ca-448684bddd75/volumes" Oct 30 02:18:02.544814 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:02.544798 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7f30150-5a1b-4c5b-8044-cfe1774eb76b" path="/var/lib/kubelet/pods/f7f30150-5a1b-4c5b-8044-cfe1774eb76b/volumes" Oct 30 02:18:03.108448 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:03.108416 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:04.580673 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:04.580639 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:05.584021 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:05.583988 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:05.799638 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:05.799607 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:06.068175 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:06.068138 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:06.341879 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:06.341785 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:06.542004 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:06.541968 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:06.740367 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:06.740333 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:06.852587 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:06.852547 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-093d23-6d8dccf8c9-428qh"] Oct 30 02:18:07.060547 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:07.060456 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:07.255375 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:07.255342 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:07.618902 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:07.618864 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:09.185550 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:09.185507 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:09.384006 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:09.383970 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:10.576211 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:10.576175 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:11.715468 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:11.715427 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:11.853418 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:11.853366 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-093d23-6d8dccf8c9-428qh" podUID="cd6ba6df-cc18-462b-a9f5-d34f1e4591fb" containerName="payload-pre-processing" containerID="cri-o://b36df82b679949861a63ecdccb59610809ce1c5a2c12f11a7ca89a3337329712" gracePeriod=25 Oct 30 02:18:12.129398 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:12.129295 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:12.179792 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:12.179737 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eda881-qqklx"] Oct 30 02:18:12.190894 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:12.190858 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-eda881-qqklx"] Oct 30 02:18:12.323918 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:12.320924 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:12.515167 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:12.515124 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:12.542677 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:12.542631 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="685f53b7-90dd-46cd-8d51-abc73d483531" path="/var/lib/kubelet/pods/685f53b7-90dd-46cd-8d51-abc73d483531/volumes" Oct 30 02:18:12.743339 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:12.743298 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:13.188952 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:13.188909 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:13.493866 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:13.493733 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:13.699878 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:13.699139 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:14.170857 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:14.170820 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:14.186620 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:14.186586 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a19e-dqfj8"] Oct 30 02:18:14.197359 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:14.197326 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-76a19e-dqfj8"] Oct 30 02:18:14.370536 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:14.370472 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:14.542541 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:14.542448 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a57999a-c713-4bc5-ae55-36d06200627e" path="/var/lib/kubelet/pods/0a57999a-c713-4bc5-ae55-36d06200627e/volumes" Oct 30 02:18:14.689530 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:18:14.689494 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:18:14.747871 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:14.747818 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:15.752374 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:15.752339 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:17.229343 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:17.229316 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:17.663580 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:17.663545 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-093d23-6d8dccf8c9-428qh" podUID="cd6ba6df-cc18-462b-a9f5-d34f1e4591fb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:18:18.350638 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:18.350606 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:18.772694 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:18.772652 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:18.885233 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:18.885189 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-093d23-tw952"] Oct 30 02:18:18.895337 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:18.895304 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-093d23-tw952"] Oct 30 02:18:18.976206 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:18.976170 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:19.269945 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:19.269903 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:19.498771 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:19.498722 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:20.563442 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:20.563406 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dd06944-23a9-493a-8974-e67076019ee2" path="/var/lib/kubelet/pods/5dd06944-23a9-493a-8974-e67076019ee2/volumes" Oct 30 02:18:20.577670 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:20.577627 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:21.158676 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:21.158639 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:21.384535 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:21.384442 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:21.594445 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:21.594407 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:21.898868 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:21.898760 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2e618f-nfzxc"] Oct 30 02:18:21.899604 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:21.899579 6572 cpu_manager.go:414] "containerMap: removing container" podUID="f7f30150-5a1b-4c5b-8044-cfe1774eb76b" containerName="payload-pre-processing" Oct 30 02:18:21.899604 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:21.899604 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7f30150-5a1b-4c5b-8044-cfe1774eb76b" containerName="payload-pre-processing" Oct 30 02:18:21.899825 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:21.899630 6572 cpu_manager.go:414] "containerMap: removing container" podUID="0a57999a-c713-4bc5-ae55-36d06200627e" containerName="revoke-keys" Oct 30 02:18:21.899825 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:21.899636 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a57999a-c713-4bc5-ae55-36d06200627e" containerName="revoke-keys" Oct 30 02:18:21.899825 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:21.899650 6572 cpu_manager.go:414] "containerMap: removing container" podUID="5dd06944-23a9-493a-8974-e67076019ee2" containerName="revoke-keys" Oct 30 02:18:21.899825 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:21.899655 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dd06944-23a9-493a-8974-e67076019ee2" containerName="revoke-keys" Oct 30 02:18:21.899825 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:21.899673 6572 cpu_manager.go:414] "containerMap: removing container" podUID="7c580bbf-cee6-4937-8ea7-fd44837e076d" containerName="payload-processing" Oct 30 02:18:21.899825 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:21.899678 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c580bbf-cee6-4937-8ea7-fd44837e076d" containerName="payload-processing" Oct 30 02:18:21.899825 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:21.899687 6572 cpu_manager.go:414] "containerMap: removing container" podUID="a9562e58-739e-44e4-a2ca-448684bddd75" containerName="istio-proxy" Oct 30 02:18:21.899825 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:21.899692 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9562e58-739e-44e4-a2ca-448684bddd75" containerName="istio-proxy" Oct 30 02:18:21.899825 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:21.899810 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c580bbf-cee6-4937-8ea7-fd44837e076d" containerName="payload-processing" Oct 30 02:18:21.899825 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:21.899822 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9562e58-739e-44e4-a2ca-448684bddd75" containerName="istio-proxy" Oct 30 02:18:21.900367 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:21.899834 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7f30150-5a1b-4c5b-8044-cfe1774eb76b" containerName="payload-pre-processing" Oct 30 02:18:21.900367 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:21.899845 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a57999a-c713-4bc5-ae55-36d06200627e" containerName="revoke-keys" Oct 30 02:18:21.900367 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:21.899852 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dd06944-23a9-493a-8974-e67076019ee2" containerName="revoke-keys" Oct 30 02:18:21.905559 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:21.905532 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2e618f-nfzxc" Oct 30 02:18:21.920688 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:21.920648 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2e618f-nfzxc"] Oct 30 02:18:21.927731 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:21.927691 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:18:21.927731 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:21.927692 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:18:21.928027 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:21.927818 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:18:21.928027 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:21.927946 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:18:22.054493 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:22.054455 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:22.075877 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:22.075829 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghhqt\" (UniqueName: \"kubernetes.io/projected/175defcb-b8af-4917-973b-efaa5ee61efe-kube-api-access-ghhqt\") pod \"maas-api-revoke-keys-e2e-extmod-b-2e618f-nfzxc\" (UID: \"175defcb-b8af-4917-973b-efaa5ee61efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2e618f-nfzxc" Oct 30 02:18:22.076095 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:22.075889 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175defcb-b8af-4917-973b-efaa5ee61efe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-2e618f-nfzxc\" (UID: \"175defcb-b8af-4917-973b-efaa5ee61efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2e618f-nfzxc" Oct 30 02:18:22.076169 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:22.076133 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175defcb-b8af-4917-973b-efaa5ee61efe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-2e618f-nfzxc\" (UID: \"175defcb-b8af-4917-973b-efaa5ee61efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2e618f-nfzxc" Oct 30 02:18:22.177958 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:22.177900 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175defcb-b8af-4917-973b-efaa5ee61efe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-2e618f-nfzxc\" (UID: \"175defcb-b8af-4917-973b-efaa5ee61efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2e618f-nfzxc" Oct 30 02:18:22.178191 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:22.178017 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghhqt\" (UniqueName: \"kubernetes.io/projected/175defcb-b8af-4917-973b-efaa5ee61efe-kube-api-access-ghhqt\") pod \"maas-api-revoke-keys-e2e-extmod-b-2e618f-nfzxc\" (UID: \"175defcb-b8af-4917-973b-efaa5ee61efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2e618f-nfzxc" Oct 30 02:18:22.178191 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:22.178056 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175defcb-b8af-4917-973b-efaa5ee61efe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-2e618f-nfzxc\" (UID: \"175defcb-b8af-4917-973b-efaa5ee61efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2e618f-nfzxc" Oct 30 02:18:22.178596 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:22.178577 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175defcb-b8af-4917-973b-efaa5ee61efe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-2e618f-nfzxc\" (UID: \"175defcb-b8af-4917-973b-efaa5ee61efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2e618f-nfzxc" Oct 30 02:18:22.180621 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:22.180601 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175defcb-b8af-4917-973b-efaa5ee61efe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-2e618f-nfzxc\" (UID: \"175defcb-b8af-4917-973b-efaa5ee61efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2e618f-nfzxc" Oct 30 02:18:22.196650 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:22.196611 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghhqt\" (UniqueName: \"kubernetes.io/projected/175defcb-b8af-4917-973b-efaa5ee61efe-kube-api-access-ghhqt\") pod \"maas-api-revoke-keys-e2e-extmod-b-2e618f-nfzxc\" (UID: \"175defcb-b8af-4917-973b-efaa5ee61efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2e618f-nfzxc" Oct 30 02:18:22.221662 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:22.221626 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2e618f-nfzxc" Oct 30 02:18:22.521480 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:22.521447 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2e618f-nfzxc"] Oct 30 02:18:22.522271 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:18:22.522240 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod175defcb_b8af_4917_973b_efaa5ee61efe.slice/crio-703b3c5f8754240af13eb607e922db6d2d8a69e37bfc55ffc2e11cd599dadc4a WatchSource:0}: Error finding container 703b3c5f8754240af13eb607e922db6d2d8a69e37bfc55ffc2e11cd599dadc4a: Status 404 returned error can't find the container with id 703b3c5f8754240af13eb607e922db6d2d8a69e37bfc55ffc2e11cd599dadc4a Oct 30 02:18:22.701129 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:22.701089 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2e618f-nfzxc" event={"ID":"175defcb-b8af-4917-973b-efaa5ee61efe","Type":"ContainerStarted","Data":"703b3c5f8754240af13eb607e922db6d2d8a69e37bfc55ffc2e11cd599dadc4a"} Oct 30 02:18:23.573077 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:23.573044 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:23.711446 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:23.711409 6572 generic.go:356] "Generic (PLEG): container finished" podID="175defcb-b8af-4917-973b-efaa5ee61efe" containerID="77f953dd53bd6dc7ec51814f02dda020e7bec47891b99f25e97e553164604ce8" exitCode=0 Oct 30 02:18:23.711982 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:23.711536 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2e618f-nfzxc" event={"ID":"175defcb-b8af-4917-973b-efaa5ee61efe","Type":"ContainerDied","Data":"77f953dd53bd6dc7ec51814f02dda020e7bec47891b99f25e97e553164604ce8"} Oct 30 02:18:24.866369 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:24.866343 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2e618f-nfzxc" Oct 30 02:18:25.013243 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:25.013146 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/175defcb-b8af-4917-973b-efaa5ee61efe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175defcb-b8af-4917-973b-efaa5ee61efe-maas-api-openshift-ca\") pod \"175defcb-b8af-4917-973b-efaa5ee61efe\" (UID: \"175defcb-b8af-4917-973b-efaa5ee61efe\") " Oct 30 02:18:25.013419 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:25.013370 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/175defcb-b8af-4917-973b-efaa5ee61efe-kube-api-access-ghhqt\" (UniqueName: \"kubernetes.io/projected/175defcb-b8af-4917-973b-efaa5ee61efe-kube-api-access-ghhqt\") pod \"175defcb-b8af-4917-973b-efaa5ee61efe\" (UID: \"175defcb-b8af-4917-973b-efaa5ee61efe\") " Oct 30 02:18:25.013419 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:25.013405 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/175defcb-b8af-4917-973b-efaa5ee61efe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175defcb-b8af-4917-973b-efaa5ee61efe-maas-api-xks-ca\") pod \"175defcb-b8af-4917-973b-efaa5ee61efe\" (UID: \"175defcb-b8af-4917-973b-efaa5ee61efe\") " Oct 30 02:18:25.013547 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:25.013525 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/175defcb-b8af-4917-973b-efaa5ee61efe-maas-api-openshift-ca" pod "175defcb-b8af-4917-973b-efaa5ee61efe" (UID: "175defcb-b8af-4917-973b-efaa5ee61efe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:25.013731 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:25.013717 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/175defcb-b8af-4917-973b-efaa5ee61efe-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:25.015828 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:25.015787 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/175defcb-b8af-4917-973b-efaa5ee61efe-kube-api-access-ghhqt" pod "175defcb-b8af-4917-973b-efaa5ee61efe" (UID: "175defcb-b8af-4917-973b-efaa5ee61efe"). InnerVolumeSpecName "kube-api-access-ghhqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:25.015970 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:25.015849 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/175defcb-b8af-4917-973b-efaa5ee61efe-maas-api-xks-ca" pod "175defcb-b8af-4917-973b-efaa5ee61efe" (UID: "175defcb-b8af-4917-973b-efaa5ee61efe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:18:25.025529 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:25.025497 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:25.114645 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:25.114600 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghhqt\" (UniqueName: \"kubernetes.io/projected/175defcb-b8af-4917-973b-efaa5ee61efe-kube-api-access-ghhqt\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:25.114645 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:25.114638 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/175defcb-b8af-4917-973b-efaa5ee61efe-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:25.421326 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:25.421287 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:25.617382 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:25.617344 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:25.670467 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:25.670430 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns"] Oct 30 02:18:25.670740 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:25.670712 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" podUID="6d64c9fe-26e0-455a-9200-cbc6c4af615f" containerName="istio-proxy" containerID="cri-o://5bf477f73c3ee2494fc39442aa1b0aa6bf8b7074497e57d7b237f0c4213c6dcc" gracePeriod=30 Oct 30 02:18:25.733811 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:25.733654 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2e618f-nfzxc" Oct 30 02:18:25.733811 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:25.733677 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2e618f-nfzxc" event={"ID":"175defcb-b8af-4917-973b-efaa5ee61efe","Type":"ContainerDied","Data":"703b3c5f8754240af13eb607e922db6d2d8a69e37bfc55ffc2e11cd599dadc4a"} Oct 30 02:18:25.734236 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:25.733921 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="703b3c5f8754240af13eb607e922db6d2d8a69e37bfc55ffc2e11cd599dadc4a" Oct 30 02:18:25.876988 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:25.876957 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:26.222329 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:26.222292 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5fb950-hzzkr"] Oct 30 02:18:26.222869 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:26.222855 6572 cpu_manager.go:414] "containerMap: removing container" podUID="175defcb-b8af-4917-973b-efaa5ee61efe" containerName="revoke-keys" Oct 30 02:18:26.222917 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:26.222871 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="175defcb-b8af-4917-973b-efaa5ee61efe" containerName="revoke-keys" Oct 30 02:18:26.222997 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:26.222988 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="175defcb-b8af-4917-973b-efaa5ee61efe" containerName="revoke-keys" Oct 30 02:18:26.226702 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:26.226682 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5fb950-hzzkr" Oct 30 02:18:26.240967 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:26.240922 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:18:26.241514 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:26.241487 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:18:26.244325 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:26.244302 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5fb950-hzzkr"] Oct 30 02:18:26.271515 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:26.271480 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:18:26.271700 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:26.271530 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:18:26.330259 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:26.330220 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3efccd53-ca2e-4604-8e66-94ef4cf65af6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-5fb950-hzzkr\" (UID: \"3efccd53-ca2e-4604-8e66-94ef4cf65af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5fb950-hzzkr" Oct 30 02:18:26.330496 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:26.330280 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkz8w\" (UniqueName: \"kubernetes.io/projected/3efccd53-ca2e-4604-8e66-94ef4cf65af6-kube-api-access-dkz8w\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-5fb950-hzzkr\" (UID: \"3efccd53-ca2e-4604-8e66-94ef4cf65af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5fb950-hzzkr" Oct 30 02:18:26.330496 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:26.330475 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3efccd53-ca2e-4604-8e66-94ef4cf65af6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-5fb950-hzzkr\" (UID: \"3efccd53-ca2e-4604-8e66-94ef4cf65af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5fb950-hzzkr" Oct 30 02:18:26.432147 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:26.432101 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3efccd53-ca2e-4604-8e66-94ef4cf65af6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-5fb950-hzzkr\" (UID: \"3efccd53-ca2e-4604-8e66-94ef4cf65af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5fb950-hzzkr" Oct 30 02:18:26.432147 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:26.432146 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkz8w\" (UniqueName: \"kubernetes.io/projected/3efccd53-ca2e-4604-8e66-94ef4cf65af6-kube-api-access-dkz8w\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-5fb950-hzzkr\" (UID: \"3efccd53-ca2e-4604-8e66-94ef4cf65af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5fb950-hzzkr" Oct 30 02:18:26.432407 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:26.432271 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3efccd53-ca2e-4604-8e66-94ef4cf65af6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-5fb950-hzzkr\" (UID: \"3efccd53-ca2e-4604-8e66-94ef4cf65af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5fb950-hzzkr" Oct 30 02:18:26.432892 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:26.432869 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3efccd53-ca2e-4604-8e66-94ef4cf65af6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-5fb950-hzzkr\" (UID: \"3efccd53-ca2e-4604-8e66-94ef4cf65af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5fb950-hzzkr" Oct 30 02:18:26.434689 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:26.434664 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3efccd53-ca2e-4604-8e66-94ef4cf65af6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-5fb950-hzzkr\" (UID: \"3efccd53-ca2e-4604-8e66-94ef4cf65af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5fb950-hzzkr" Oct 30 02:18:26.451412 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:26.451381 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkz8w\" (UniqueName: \"kubernetes.io/projected/3efccd53-ca2e-4604-8e66-94ef4cf65af6-kube-api-access-dkz8w\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-5fb950-hzzkr\" (UID: \"3efccd53-ca2e-4604-8e66-94ef4cf65af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5fb950-hzzkr" Oct 30 02:18:26.540124 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:26.540036 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5fb950-hzzkr" Oct 30 02:18:26.707051 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:26.707018 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:26.841960 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:26.841931 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5fb950-hzzkr"] Oct 30 02:18:26.844582 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:18:26.844554 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3efccd53_ca2e_4604_8e66_94ef4cf65af6.slice/crio-429348e4a5530a632302694234ac8abf7450a2cf5f928f0631d462ec556302f3 WatchSource:0}: Error finding container 429348e4a5530a632302694234ac8abf7450a2cf5f928f0631d462ec556302f3: Status 404 returned error can't find the container with id 429348e4a5530a632302694234ac8abf7450a2cf5f928f0631d462ec556302f3 Oct 30 02:18:27.662902 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:27.662858 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-093d23-6d8dccf8c9-428qh" podUID="cd6ba6df-cc18-462b-a9f5-d34f1e4591fb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:18:27.762153 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:27.762119 6572 generic.go:356] "Generic (PLEG): container finished" podID="3efccd53-ca2e-4604-8e66-94ef4cf65af6" containerID="9e6891751a52554a502485eeda680d2819c3cf2811c1a12e02ce5b742d593841" exitCode=0 Oct 30 02:18:27.762338 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:27.762201 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5fb950-hzzkr" event={"ID":"3efccd53-ca2e-4604-8e66-94ef4cf65af6","Type":"ContainerDied","Data":"9e6891751a52554a502485eeda680d2819c3cf2811c1a12e02ce5b742d593841"} Oct 30 02:18:27.762338 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:27.762278 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5fb950-hzzkr" event={"ID":"3efccd53-ca2e-4604-8e66-94ef4cf65af6","Type":"ContainerStarted","Data":"429348e4a5530a632302694234ac8abf7450a2cf5f928f0631d462ec556302f3"} Oct 30 02:18:28.075626 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:28.075542 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:28.918934 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:28.918904 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5fb950-hzzkr" Oct 30 02:18:28.967965 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:28.967931 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3efccd53-ca2e-4604-8e66-94ef4cf65af6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3efccd53-ca2e-4604-8e66-94ef4cf65af6-maas-api-xks-ca\") pod \"3efccd53-ca2e-4604-8e66-94ef4cf65af6\" (UID: \"3efccd53-ca2e-4604-8e66-94ef4cf65af6\") " Oct 30 02:18:28.968165 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:28.967986 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3efccd53-ca2e-4604-8e66-94ef4cf65af6-kube-api-access-dkz8w\" (UniqueName: \"kubernetes.io/projected/3efccd53-ca2e-4604-8e66-94ef4cf65af6-kube-api-access-dkz8w\") pod \"3efccd53-ca2e-4604-8e66-94ef4cf65af6\" (UID: \"3efccd53-ca2e-4604-8e66-94ef4cf65af6\") " Oct 30 02:18:28.968165 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:28.968060 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3efccd53-ca2e-4604-8e66-94ef4cf65af6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3efccd53-ca2e-4604-8e66-94ef4cf65af6-maas-api-openshift-ca\") pod \"3efccd53-ca2e-4604-8e66-94ef4cf65af6\" (UID: \"3efccd53-ca2e-4604-8e66-94ef4cf65af6\") " Oct 30 02:18:28.968931 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:28.968896 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3efccd53-ca2e-4604-8e66-94ef4cf65af6-maas-api-openshift-ca" pod "3efccd53-ca2e-4604-8e66-94ef4cf65af6" (UID: "3efccd53-ca2e-4604-8e66-94ef4cf65af6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:28.970254 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:28.970226 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3efccd53-ca2e-4604-8e66-94ef4cf65af6-maas-api-xks-ca" pod "3efccd53-ca2e-4604-8e66-94ef4cf65af6" (UID: "3efccd53-ca2e-4604-8e66-94ef4cf65af6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:18:28.970792 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:28.970770 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3efccd53-ca2e-4604-8e66-94ef4cf65af6-kube-api-access-dkz8w" pod "3efccd53-ca2e-4604-8e66-94ef4cf65af6" (UID: "3efccd53-ca2e-4604-8e66-94ef4cf65af6"). InnerVolumeSpecName "kube-api-access-dkz8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:29.069534 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:29.069441 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3efccd53-ca2e-4604-8e66-94ef4cf65af6-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:29.069534 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:29.069479 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkz8w\" (UniqueName: \"kubernetes.io/projected/3efccd53-ca2e-4604-8e66-94ef4cf65af6-kube-api-access-dkz8w\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:29.069534 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:29.069495 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3efccd53-ca2e-4604-8e66-94ef4cf65af6-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:29.471342 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:29.471309 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:29.781837 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:29.781723 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5fb950-hzzkr" Oct 30 02:18:29.781837 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:29.781768 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5fb950-hzzkr" event={"ID":"3efccd53-ca2e-4604-8e66-94ef4cf65af6","Type":"ContainerDied","Data":"429348e4a5530a632302694234ac8abf7450a2cf5f928f0631d462ec556302f3"} Oct 30 02:18:29.781837 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:29.781820 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="429348e4a5530a632302694234ac8abf7450a2cf5f928f0631d462ec556302f3" Oct 30 02:18:30.770813 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:30.770737 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:30.793505 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:30.793471 6572 generic.go:356] "Generic (PLEG): container finished" podID="6d64c9fe-26e0-455a-9200-cbc6c4af615f" containerID="5bf477f73c3ee2494fc39442aa1b0aa6bf8b7074497e57d7b237f0c4213c6dcc" exitCode=0 Oct 30 02:18:30.793645 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:30.793552 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" event={"ID":"6d64c9fe-26e0-455a-9200-cbc6c4af615f","Type":"ContainerDied","Data":"5bf477f73c3ee2494fc39442aa1b0aa6bf8b7074497e57d7b237f0c4213c6dcc"} Oct 30 02:18:30.793645 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:30.793635 6572 scope.go:122] "RemoveContainer" containerID="a4cee2ef6ff23279e71e847d8abc9eb923541ae7755b1444569447768ddaa202" Oct 30 02:18:30.950640 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:30.950563 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 02:18:30.994582 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:30.994532 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d64c9fe-26e0-455a-9200-cbc6c4af615f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d64c9fe-26e0-455a-9200-cbc6c4af615f-workload-socket\") pod \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\" (UID: \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\") " Oct 30 02:18:30.994983 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:30.994944 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d64c9fe-26e0-455a-9200-cbc6c4af615f-workload-socket" pod "6d64c9fe-26e0-455a-9200-cbc6c4af615f" (UID: "6d64c9fe-26e0-455a-9200-cbc6c4af615f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:30.995122 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:30.994958 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d64c9fe-26e0-455a-9200-cbc6c4af615f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d64c9fe-26e0-455a-9200-cbc6c4af615f-workload-certs\") pod \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\" (UID: \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\") " Oct 30 02:18:30.995122 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:30.995062 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d64c9fe-26e0-455a-9200-cbc6c4af615f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d64c9fe-26e0-455a-9200-cbc6c4af615f-credential-socket\") pod \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\" (UID: \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\") " Oct 30 02:18:30.995122 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:30.995097 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d64c9fe-26e0-455a-9200-cbc6c4af615f-istio-token\" (UniqueName: \"kubernetes.io/projected/6d64c9fe-26e0-455a-9200-cbc6c4af615f-istio-token\") pod \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\" (UID: \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\") " Oct 30 02:18:30.995317 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:30.995156 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d64c9fe-26e0-455a-9200-cbc6c4af615f-kube-api-access-5qxh6\" (UniqueName: \"kubernetes.io/projected/6d64c9fe-26e0-455a-9200-cbc6c4af615f-kube-api-access-5qxh6\") pod \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\" (UID: \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\") " Oct 30 02:18:30.995317 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:30.995191 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d64c9fe-26e0-455a-9200-cbc6c4af615f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d64c9fe-26e0-455a-9200-cbc6c4af615f-istiod-ca-cert\") pod \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\" (UID: \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\") " Oct 30 02:18:30.995317 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:30.995237 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d64c9fe-26e0-455a-9200-cbc6c4af615f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d64c9fe-26e0-455a-9200-cbc6c4af615f-istio-envoy\") pod \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\" (UID: \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\") " Oct 30 02:18:30.995492 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:30.995301 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d64c9fe-26e0-455a-9200-cbc6c4af615f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d64c9fe-26e0-455a-9200-cbc6c4af615f-istio-podinfo\") pod \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\" (UID: \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\") " Oct 30 02:18:30.995492 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:30.995404 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d64c9fe-26e0-455a-9200-cbc6c4af615f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d64c9fe-26e0-455a-9200-cbc6c4af615f-istio-data\") pod \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\" (UID: \"6d64c9fe-26e0-455a-9200-cbc6c4af615f\") " Oct 30 02:18:30.996006 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:30.995977 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d64c9fe-26e0-455a-9200-cbc6c4af615f-workload-certs" pod "6d64c9fe-26e0-455a-9200-cbc6c4af615f" (UID: "6d64c9fe-26e0-455a-9200-cbc6c4af615f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:30.997687 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:30.997656 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d64c9fe-26e0-455a-9200-cbc6c4af615f-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:30.997687 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:30.997686 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d64c9fe-26e0-455a-9200-cbc6c4af615f-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:31.012908 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:30.998500 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d64c9fe-26e0-455a-9200-cbc6c4af615f-istio-data" pod "6d64c9fe-26e0-455a-9200-cbc6c4af615f" (UID: "6d64c9fe-26e0-455a-9200-cbc6c4af615f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:31.012908 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:30.999124 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d64c9fe-26e0-455a-9200-cbc6c4af615f-credential-socket" pod "6d64c9fe-26e0-455a-9200-cbc6c4af615f" (UID: "6d64c9fe-26e0-455a-9200-cbc6c4af615f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:31.012908 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:30.999442 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d64c9fe-26e0-455a-9200-cbc6c4af615f-istiod-ca-cert" pod "6d64c9fe-26e0-455a-9200-cbc6c4af615f" (UID: "6d64c9fe-26e0-455a-9200-cbc6c4af615f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:31.040794 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:31.033176 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d64c9fe-26e0-455a-9200-cbc6c4af615f-istio-token" pod "6d64c9fe-26e0-455a-9200-cbc6c4af615f" (UID: "6d64c9fe-26e0-455a-9200-cbc6c4af615f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:31.046275 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:31.046196 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d64c9fe-26e0-455a-9200-cbc6c4af615f-istio-podinfo" pod "6d64c9fe-26e0-455a-9200-cbc6c4af615f" (UID: "6d64c9fe-26e0-455a-9200-cbc6c4af615f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:18:31.057696 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:31.057602 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d64c9fe-26e0-455a-9200-cbc6c4af615f-kube-api-access-5qxh6" pod "6d64c9fe-26e0-455a-9200-cbc6c4af615f" (UID: "6d64c9fe-26e0-455a-9200-cbc6c4af615f"). InnerVolumeSpecName "kube-api-access-5qxh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:31.075078 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:31.075034 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d64c9fe-26e0-455a-9200-cbc6c4af615f-istio-envoy" pod "6d64c9fe-26e0-455a-9200-cbc6c4af615f" (UID: "6d64c9fe-26e0-455a-9200-cbc6c4af615f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:31.100895 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:31.100802 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d64c9fe-26e0-455a-9200-cbc6c4af615f-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:31.101278 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:31.100856 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d64c9fe-26e0-455a-9200-cbc6c4af615f-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:31.101278 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:31.101182 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d64c9fe-26e0-455a-9200-cbc6c4af615f-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:31.101278 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:31.101200 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qxh6\" (UniqueName: \"kubernetes.io/projected/6d64c9fe-26e0-455a-9200-cbc6c4af615f-kube-api-access-5qxh6\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:31.101278 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:31.101227 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d64c9fe-26e0-455a-9200-cbc6c4af615f-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:31.101278 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:31.101244 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d64c9fe-26e0-455a-9200-cbc6c4af615f-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:31.101278 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:31.101257 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d64c9fe-26e0-455a-9200-cbc6c4af615f-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:31.803694 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:31.803658 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" Oct 30 02:18:31.804217 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:31.803699 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns" event={"ID":"6d64c9fe-26e0-455a-9200-cbc6c4af615f","Type":"ContainerDied","Data":"6897c9f32b2496fc1e9bc98ef49f5342fde920581d433aad66b38cadd484d9e0"} Oct 30 02:18:31.804217 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:31.803775 6572 scope.go:122] "RemoveContainer" containerID="5bf477f73c3ee2494fc39442aa1b0aa6bf8b7074497e57d7b237f0c4213c6dcc" Oct 30 02:18:31.866538 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:31.866502 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns"] Oct 30 02:18:31.877519 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:31.877487 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-093d23-openshift-default-9576b4fd9-5lqns"] Oct 30 02:18:32.121716 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:32.121686 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-093d23-6d8dccf8c9-428qh" Oct 30 02:18:32.216429 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:32.216371 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd6ba6df-cc18-462b-a9f5-d34f1e4591fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd6ba6df-cc18-462b-a9f5-d34f1e4591fb-plugins-config-volume\") pod \"cd6ba6df-cc18-462b-a9f5-d34f1e4591fb\" (UID: \"cd6ba6df-cc18-462b-a9f5-d34f1e4591fb\") " Oct 30 02:18:32.218004 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:32.216532 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd6ba6df-cc18-462b-a9f5-d34f1e4591fb-kube-api-access-7z92l\" (UniqueName: \"kubernetes.io/projected/cd6ba6df-cc18-462b-a9f5-d34f1e4591fb-kube-api-access-7z92l\") pod \"cd6ba6df-cc18-462b-a9f5-d34f1e4591fb\" (UID: \"cd6ba6df-cc18-462b-a9f5-d34f1e4591fb\") " Oct 30 02:18:32.218832 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:32.218801 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd6ba6df-cc18-462b-a9f5-d34f1e4591fb-plugins-config-volume" pod "cd6ba6df-cc18-462b-a9f5-d34f1e4591fb" (UID: "cd6ba6df-cc18-462b-a9f5-d34f1e4591fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:32.226894 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:32.226702 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd6ba6df-cc18-462b-a9f5-d34f1e4591fb-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:32.231507 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:32.226811 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd6ba6df-cc18-462b-a9f5-d34f1e4591fb-kube-api-access-7z92l" pod "cd6ba6df-cc18-462b-a9f5-d34f1e4591fb" (UID: "cd6ba6df-cc18-462b-a9f5-d34f1e4591fb"). InnerVolumeSpecName "kube-api-access-7z92l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:32.254632 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:32.254593 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:32.329083 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:32.329049 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z92l\" (UniqueName: \"kubernetes.io/projected/cd6ba6df-cc18-462b-a9f5-d34f1e4591fb-kube-api-access-7z92l\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:32.542522 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:32.542436 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d64c9fe-26e0-455a-9200-cbc6c4af615f" path="/var/lib/kubelet/pods/6d64c9fe-26e0-455a-9200-cbc6c4af615f/volumes" Oct 30 02:18:32.596430 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:32.596384 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:32.797613 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:32.797516 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:32.813629 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:32.813597 6572 generic.go:356] "Generic (PLEG): container finished" podID="cd6ba6df-cc18-462b-a9f5-d34f1e4591fb" containerID="b36df82b679949861a63ecdccb59610809ce1c5a2c12f11a7ca89a3337329712" exitCode=0 Oct 30 02:18:32.814138 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:32.813664 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-093d23-6d8dccf8c9-428qh" Oct 30 02:18:32.814138 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:32.813690 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-093d23-6d8dccf8c9-428qh" event={"ID":"cd6ba6df-cc18-462b-a9f5-d34f1e4591fb","Type":"ContainerDied","Data":"b36df82b679949861a63ecdccb59610809ce1c5a2c12f11a7ca89a3337329712"} Oct 30 02:18:32.814138 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:32.813743 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-093d23-6d8dccf8c9-428qh" event={"ID":"cd6ba6df-cc18-462b-a9f5-d34f1e4591fb","Type":"ContainerDied","Data":"c8892552528a5fd104351e0977db4cc7cfe2f9f590fddfeaeead4cd5d9575bc1"} Oct 30 02:18:32.814138 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:32.813791 6572 scope.go:122] "RemoveContainer" containerID="b36df82b679949861a63ecdccb59610809ce1c5a2c12f11a7ca89a3337329712" Oct 30 02:18:32.827287 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:32.827262 6572 scope.go:122] "RemoveContainer" containerID="b36df82b679949861a63ecdccb59610809ce1c5a2c12f11a7ca89a3337329712" Oct 30 02:18:32.827661 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:18:32.827636 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b36df82b679949861a63ecdccb59610809ce1c5a2c12f11a7ca89a3337329712\": container with ID starting with b36df82b679949861a63ecdccb59610809ce1c5a2c12f11a7ca89a3337329712 not found: ID does not exist" containerID="b36df82b679949861a63ecdccb59610809ce1c5a2c12f11a7ca89a3337329712" Oct 30 02:18:32.827821 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:32.827672 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b36df82b679949861a63ecdccb59610809ce1c5a2c12f11a7ca89a3337329712"} err="failed to get container status \"b36df82b679949861a63ecdccb59610809ce1c5a2c12f11a7ca89a3337329712\": rpc error: code = NotFound desc = could not find container \"b36df82b679949861a63ecdccb59610809ce1c5a2c12f11a7ca89a3337329712\": container with ID starting with b36df82b679949861a63ecdccb59610809ce1c5a2c12f11a7ca89a3337329712 not found: ID does not exist" Oct 30 02:18:32.865605 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:32.865550 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-093d23-6d8dccf8c9-428qh"] Oct 30 02:18:32.876861 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:32.876822 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-093d23-6d8dccf8c9-428qh"] Oct 30 02:18:33.684031 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:33.683979 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:34.543000 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:34.542961 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd6ba6df-cc18-462b-a9f5-d34f1e4591fb" path="/var/lib/kubelet/pods/cd6ba6df-cc18-462b-a9f5-d34f1e4591fb/volumes" Oct 30 02:18:35.175891 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:35.175854 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:36.585500 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:36.585462 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:36.644157 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:36.644115 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5fb950-7479858b6d-sn7h4"] Oct 30 02:18:36.669082 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:36.669044 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5fb950-79c94597b7-jh8xt"] Oct 30 02:18:36.860248 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:36.860148 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:37.053446 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:37.053415 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:38.018775 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:38.018707 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:39.388196 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:39.388161 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:40.780640 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:40.780608 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:41.645289 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:41.645229 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5fb950-7479858b6d-sn7h4" podUID="6f409677-36e8-4781-8208-841f96df1888" containerName="payload-processing" containerID="cri-o://7e757d727cd6000faa0fe250f37455fb72dbf40d8e95e7b8e366fa561ea62bb2" gracePeriod=25 Oct 30 02:18:41.669931 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:41.669880 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5fb950-79c94597b7-jh8xt" podUID="895641ba-c480-4aa8-81d6-a831b4fc93e9" containerName="payload-pre-processing" containerID="cri-o://d069e3ca5fce8e59a31ff3239ce06b54e75611f7f8f691a4ae86c897e20026ad" gracePeriod=25 Oct 30 02:18:41.897259 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:41.897161 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:41.898468 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:41.898429 6572 generic.go:356] "Generic (PLEG): container finished" podID="895641ba-c480-4aa8-81d6-a831b4fc93e9" containerID="d069e3ca5fce8e59a31ff3239ce06b54e75611f7f8f691a4ae86c897e20026ad" exitCode=0 Oct 30 02:18:41.898596 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:41.898512 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5fb950-79c94597b7-jh8xt" event={"ID":"895641ba-c480-4aa8-81d6-a831b4fc93e9","Type":"ContainerDied","Data":"d069e3ca5fce8e59a31ff3239ce06b54e75611f7f8f691a4ae86c897e20026ad"} Oct 30 02:18:41.901172 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:41.901122 6572 generic.go:356] "Generic (PLEG): container finished" podID="6f409677-36e8-4781-8208-841f96df1888" containerID="7e757d727cd6000faa0fe250f37455fb72dbf40d8e95e7b8e366fa561ea62bb2" exitCode=0 Oct 30 02:18:41.901338 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:41.901294 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5fb950-7479858b6d-sn7h4" event={"ID":"6f409677-36e8-4781-8208-841f96df1888","Type":"ContainerDied","Data":"7e757d727cd6000faa0fe250f37455fb72dbf40d8e95e7b8e366fa561ea62bb2"} Oct 30 02:18:41.948679 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:41.948651 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5fb950-7479858b6d-sn7h4" Oct 30 02:18:42.038280 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:42.038237 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5fb950-79c94597b7-jh8xt" Oct 30 02:18:42.052814 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:42.052091 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f409677-36e8-4781-8208-841f96df1888-kube-api-access-mj2w6\" (UniqueName: \"kubernetes.io/projected/6f409677-36e8-4781-8208-841f96df1888-kube-api-access-mj2w6\") pod \"6f409677-36e8-4781-8208-841f96df1888\" (UID: \"6f409677-36e8-4781-8208-841f96df1888\") " Oct 30 02:18:42.052814 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:42.052247 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f409677-36e8-4781-8208-841f96df1888-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f409677-36e8-4781-8208-841f96df1888-plugins-config-volume\") pod \"6f409677-36e8-4781-8208-841f96df1888\" (UID: \"6f409677-36e8-4781-8208-841f96df1888\") " Oct 30 02:18:42.053189 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:42.053166 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f409677-36e8-4781-8208-841f96df1888-plugins-config-volume" pod "6f409677-36e8-4781-8208-841f96df1888" (UID: "6f409677-36e8-4781-8208-841f96df1888"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:42.054991 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:42.054965 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f409677-36e8-4781-8208-841f96df1888-kube-api-access-mj2w6" pod "6f409677-36e8-4781-8208-841f96df1888" (UID: "6f409677-36e8-4781-8208-841f96df1888"). InnerVolumeSpecName "kube-api-access-mj2w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:42.154184 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:42.154056 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/895641ba-c480-4aa8-81d6-a831b4fc93e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/895641ba-c480-4aa8-81d6-a831b4fc93e9-plugins-config-volume\") pod \"895641ba-c480-4aa8-81d6-a831b4fc93e9\" (UID: \"895641ba-c480-4aa8-81d6-a831b4fc93e9\") " Oct 30 02:18:42.154391 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:42.154281 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/895641ba-c480-4aa8-81d6-a831b4fc93e9-kube-api-access-pr5bt\" (UniqueName: \"kubernetes.io/projected/895641ba-c480-4aa8-81d6-a831b4fc93e9-kube-api-access-pr5bt\") pod \"895641ba-c480-4aa8-81d6-a831b4fc93e9\" (UID: \"895641ba-c480-4aa8-81d6-a831b4fc93e9\") " Oct 30 02:18:42.154898 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:42.154873 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f409677-36e8-4781-8208-841f96df1888-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:42.155109 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:42.154903 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj2w6\" (UniqueName: \"kubernetes.io/projected/6f409677-36e8-4781-8208-841f96df1888-kube-api-access-mj2w6\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:42.157216 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:42.157182 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/895641ba-c480-4aa8-81d6-a831b4fc93e9-plugins-config-volume" pod "895641ba-c480-4aa8-81d6-a831b4fc93e9" (UID: "895641ba-c480-4aa8-81d6-a831b4fc93e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:42.158618 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:42.158587 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/895641ba-c480-4aa8-81d6-a831b4fc93e9-kube-api-access-pr5bt" pod "895641ba-c480-4aa8-81d6-a831b4fc93e9" (UID: "895641ba-c480-4aa8-81d6-a831b4fc93e9"). InnerVolumeSpecName "kube-api-access-pr5bt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:42.256289 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:42.256254 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr5bt\" (UniqueName: \"kubernetes.io/projected/895641ba-c480-4aa8-81d6-a831b4fc93e9-kube-api-access-pr5bt\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:42.256289 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:42.256286 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/895641ba-c480-4aa8-81d6-a831b4fc93e9-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:42.451848 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:42.451810 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:42.465273 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:42.465235 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2e618f-nfzxc"] Oct 30 02:18:42.468724 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:42.468690 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2e618f-nfzxc"] Oct 30 02:18:42.560143 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:42.560107 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="175defcb-b8af-4917-973b-efaa5ee61efe" path="/var/lib/kubelet/pods/175defcb-b8af-4917-973b-efaa5ee61efe/volumes" Oct 30 02:18:42.653003 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:42.652968 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:42.911997 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:42.911967 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5fb950-79c94597b7-jh8xt" Oct 30 02:18:42.912490 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:42.911967 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5fb950-79c94597b7-jh8xt" event={"ID":"895641ba-c480-4aa8-81d6-a831b4fc93e9","Type":"ContainerDied","Data":"2a9d1e177f24f2faccbc4aa96b88eac605baa146c4c225fb444d3dedca875a2d"} Oct 30 02:18:42.912490 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:42.912115 6572 scope.go:122] "RemoveContainer" containerID="d069e3ca5fce8e59a31ff3239ce06b54e75611f7f8f691a4ae86c897e20026ad" Oct 30 02:18:42.914007 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:42.913981 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5fb950-7479858b6d-sn7h4" event={"ID":"6f409677-36e8-4781-8208-841f96df1888","Type":"ContainerDied","Data":"5c0607ebfe29c553d9cad664ed420fddea318cd1a08c897ecfadc9a8f4e109ce"} Oct 30 02:18:42.914101 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:42.914013 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5fb950-7479858b6d-sn7h4" Oct 30 02:18:42.924633 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:42.924606 6572 scope.go:122] "RemoveContainer" containerID="7e757d727cd6000faa0fe250f37455fb72dbf40d8e95e7b8e366fa561ea62bb2" Oct 30 02:18:42.962582 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:42.962532 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5fb950-7479858b6d-sn7h4"] Oct 30 02:18:42.973919 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:42.973884 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5fb950-7479858b6d-sn7h4"] Oct 30 02:18:42.998600 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:42.998553 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5fb950-79c94597b7-jh8xt"] Oct 30 02:18:43.006633 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:43.006594 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5fb950-79c94597b7-jh8xt"] Oct 30 02:18:43.066399 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:43.066360 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:43.949459 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:43.949418 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:44.375784 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:44.375653 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:44.432709 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:44.432672 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5fb950-hzzkr"] Oct 30 02:18:44.444122 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:44.444084 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5fb950-hzzkr"] Oct 30 02:18:44.552335 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:44.552295 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3efccd53-ca2e-4604-8e66-94ef4cf65af6" path="/var/lib/kubelet/pods/3efccd53-ca2e-4604-8e66-94ef4cf65af6/volumes" Oct 30 02:18:44.552740 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:44.552723 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f409677-36e8-4781-8208-841f96df1888" path="/var/lib/kubelet/pods/6f409677-36e8-4781-8208-841f96df1888/volumes" Oct 30 02:18:44.553333 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:44.553314 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="895641ba-c480-4aa8-81d6-a831b4fc93e9" path="/var/lib/kubelet/pods/895641ba-c480-4aa8-81d6-a831b4fc93e9/volumes" Oct 30 02:18:44.606140 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:44.606103 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:44.690725 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:18:44.690681 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:18:45.013020 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:45.012926 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:45.988674 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:45.988642 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:47.480357 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:47.480319 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:48.853170 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:48.853131 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:49.357538 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:49.357493 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:49.553404 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:49.553367 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:49.743061 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:49.743030 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:50.145666 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:50.145593 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-396f8e-sb4l6"] Oct 30 02:18:50.146128 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:50.146113 6572 cpu_manager.go:414] "containerMap: removing container" podUID="895641ba-c480-4aa8-81d6-a831b4fc93e9" containerName="payload-pre-processing" Oct 30 02:18:50.146175 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:50.146129 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="895641ba-c480-4aa8-81d6-a831b4fc93e9" containerName="payload-pre-processing" Oct 30 02:18:50.146175 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:50.146138 6572 cpu_manager.go:414] "containerMap: removing container" podUID="6f409677-36e8-4781-8208-841f96df1888" containerName="payload-processing" Oct 30 02:18:50.146175 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:50.146143 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f409677-36e8-4781-8208-841f96df1888" containerName="payload-processing" Oct 30 02:18:50.146175 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:50.146155 6572 cpu_manager.go:414] "containerMap: removing container" podUID="6d64c9fe-26e0-455a-9200-cbc6c4af615f" containerName="istio-proxy" Oct 30 02:18:50.146175 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:50.146160 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d64c9fe-26e0-455a-9200-cbc6c4af615f" containerName="istio-proxy" Oct 30 02:18:50.146175 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:50.146175 6572 cpu_manager.go:414] "containerMap: removing container" podUID="6d64c9fe-26e0-455a-9200-cbc6c4af615f" containerName="istio-proxy" Oct 30 02:18:50.146391 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:50.146180 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d64c9fe-26e0-455a-9200-cbc6c4af615f" containerName="istio-proxy" Oct 30 02:18:50.146391 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:50.146199 6572 cpu_manager.go:414] "containerMap: removing container" podUID="cd6ba6df-cc18-462b-a9f5-d34f1e4591fb" containerName="payload-pre-processing" Oct 30 02:18:50.146391 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:50.146204 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd6ba6df-cc18-462b-a9f5-d34f1e4591fb" containerName="payload-pre-processing" Oct 30 02:18:50.146391 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:50.146214 6572 cpu_manager.go:414] "containerMap: removing container" podUID="3efccd53-ca2e-4604-8e66-94ef4cf65af6" containerName="revoke-keys" Oct 30 02:18:50.146391 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:50.146218 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3efccd53-ca2e-4604-8e66-94ef4cf65af6" containerName="revoke-keys" Oct 30 02:18:50.146391 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:50.146327 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d64c9fe-26e0-455a-9200-cbc6c4af615f" containerName="istio-proxy" Oct 30 02:18:50.146391 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:50.146339 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd6ba6df-cc18-462b-a9f5-d34f1e4591fb" containerName="payload-pre-processing" Oct 30 02:18:50.146391 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:50.146345 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="3efccd53-ca2e-4604-8e66-94ef4cf65af6" containerName="revoke-keys" Oct 30 02:18:50.146391 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:50.146355 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="895641ba-c480-4aa8-81d6-a831b4fc93e9" containerName="payload-pre-processing" Oct 30 02:18:50.146391 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:50.146362 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f409677-36e8-4781-8208-841f96df1888" containerName="payload-processing" Oct 30 02:18:50.151203 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:50.151186 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-396f8e-sb4l6" Oct 30 02:18:50.176992 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:50.176957 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:18:50.177149 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:50.177034 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:18:50.182330 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:50.182302 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-396f8e-sb4l6"] Oct 30 02:18:50.200266 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:50.200237 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:18:50.200426 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:50.200330 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:18:50.259089 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:50.259057 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c3683b7-bc0c-4a95-98dd-054a06ab137a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-396f8e-sb4l6\" (UID: \"9c3683b7-bc0c-4a95-98dd-054a06ab137a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-396f8e-sb4l6" Oct 30 02:18:50.259327 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:50.259243 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c3683b7-bc0c-4a95-98dd-054a06ab137a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-396f8e-sb4l6\" (UID: \"9c3683b7-bc0c-4a95-98dd-054a06ab137a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-396f8e-sb4l6" Oct 30 02:18:50.259417 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:50.259375 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r56p7\" (UniqueName: \"kubernetes.io/projected/9c3683b7-bc0c-4a95-98dd-054a06ab137a-kube-api-access-r56p7\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-396f8e-sb4l6\" (UID: \"9c3683b7-bc0c-4a95-98dd-054a06ab137a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-396f8e-sb4l6" Oct 30 02:18:50.360363 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:50.360316 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r56p7\" (UniqueName: \"kubernetes.io/projected/9c3683b7-bc0c-4a95-98dd-054a06ab137a-kube-api-access-r56p7\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-396f8e-sb4l6\" (UID: \"9c3683b7-bc0c-4a95-98dd-054a06ab137a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-396f8e-sb4l6" Oct 30 02:18:50.360554 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:50.360408 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c3683b7-bc0c-4a95-98dd-054a06ab137a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-396f8e-sb4l6\" (UID: \"9c3683b7-bc0c-4a95-98dd-054a06ab137a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-396f8e-sb4l6" Oct 30 02:18:50.360554 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:50.360528 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c3683b7-bc0c-4a95-98dd-054a06ab137a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-396f8e-sb4l6\" (UID: \"9c3683b7-bc0c-4a95-98dd-054a06ab137a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-396f8e-sb4l6" Oct 30 02:18:50.361251 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:50.361230 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c3683b7-bc0c-4a95-98dd-054a06ab137a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-396f8e-sb4l6\" (UID: \"9c3683b7-bc0c-4a95-98dd-054a06ab137a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-396f8e-sb4l6" Oct 30 02:18:50.362806 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:50.362785 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c3683b7-bc0c-4a95-98dd-054a06ab137a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-396f8e-sb4l6\" (UID: \"9c3683b7-bc0c-4a95-98dd-054a06ab137a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-396f8e-sb4l6" Oct 30 02:18:50.381817 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:50.381781 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r56p7\" (UniqueName: \"kubernetes.io/projected/9c3683b7-bc0c-4a95-98dd-054a06ab137a-kube-api-access-r56p7\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-396f8e-sb4l6\" (UID: \"9c3683b7-bc0c-4a95-98dd-054a06ab137a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-396f8e-sb4l6" Oct 30 02:18:50.464429 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:50.464391 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-396f8e-sb4l6" Oct 30 02:18:50.528387 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:50.528358 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:50.767173 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:50.767145 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-396f8e-sb4l6"] Oct 30 02:18:50.769269 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:18:50.769234 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c3683b7_bc0c_4a95_98dd_054a06ab137a.slice/crio-5d124fdb8218f3b3171ddffbc7f851c03b8757da64f4fd0bf8b51de104dd7fd0 WatchSource:0}: Error finding container 5d124fdb8218f3b3171ddffbc7f851c03b8757da64f4fd0bf8b51de104dd7fd0: Status 404 returned error can't find the container with id 5d124fdb8218f3b3171ddffbc7f851c03b8757da64f4fd0bf8b51de104dd7fd0 Oct 30 02:18:50.983904 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:50.983855 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-396f8e-sb4l6" event={"ID":"9c3683b7-bc0c-4a95-98dd-054a06ab137a","Type":"ContainerStarted","Data":"5d124fdb8218f3b3171ddffbc7f851c03b8757da64f4fd0bf8b51de104dd7fd0"} Oct 30 02:18:51.107769 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:51.107619 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:51.293026 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:51.292939 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv"] Oct 30 02:18:51.293528 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:51.293189 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" podUID="223ead86-cf13-4c05-9655-893ffc0b9907" containerName="istio-proxy" containerID="cri-o://904eed12435e699e4bdab08f527f0c4a450cf2611c545b27b9ce2ebad112bb3e" gracePeriod=30 Oct 30 02:18:51.300398 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:51.300366 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:51.498813 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:51.498780 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:51.860669 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:51.860584 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f7628-fw9d6"] Oct 30 02:18:51.861292 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:51.861275 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d64c9fe-26e0-455a-9200-cbc6c4af615f" containerName="istio-proxy" Oct 30 02:18:51.864875 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:51.864848 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f7628-fw9d6" Oct 30 02:18:51.886525 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:51.886488 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f7628-fw9d6"] Oct 30 02:18:51.993588 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:51.993554 6572 generic.go:356] "Generic (PLEG): container finished" podID="9c3683b7-bc0c-4a95-98dd-054a06ab137a" containerID="579b59083a399db96c4c0f29bd15e14acf2cc28e578079a6017a2f18dd1848c1" exitCode=0 Oct 30 02:18:51.993815 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:51.993631 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-396f8e-sb4l6" event={"ID":"9c3683b7-bc0c-4a95-98dd-054a06ab137a","Type":"ContainerDied","Data":"579b59083a399db96c4c0f29bd15e14acf2cc28e578079a6017a2f18dd1848c1"} Oct 30 02:18:52.011917 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:52.011869 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4748ea17-f686-4836-b3f4-0960ec022a05-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4f7628-fw9d6\" (UID: \"4748ea17-f686-4836-b3f4-0960ec022a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f7628-fw9d6" Oct 30 02:18:52.012218 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:52.011991 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbnx7\" (UniqueName: \"kubernetes.io/projected/4748ea17-f686-4836-b3f4-0960ec022a05-kube-api-access-cbnx7\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4f7628-fw9d6\" (UID: \"4748ea17-f686-4836-b3f4-0960ec022a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f7628-fw9d6" Oct 30 02:18:52.012218 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:52.012136 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4748ea17-f686-4836-b3f4-0960ec022a05-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4f7628-fw9d6\" (UID: \"4748ea17-f686-4836-b3f4-0960ec022a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f7628-fw9d6" Oct 30 02:18:52.113715 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:52.113623 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4748ea17-f686-4836-b3f4-0960ec022a05-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4f7628-fw9d6\" (UID: \"4748ea17-f686-4836-b3f4-0960ec022a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f7628-fw9d6" Oct 30 02:18:52.113715 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:52.113677 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbnx7\" (UniqueName: \"kubernetes.io/projected/4748ea17-f686-4836-b3f4-0960ec022a05-kube-api-access-cbnx7\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4f7628-fw9d6\" (UID: \"4748ea17-f686-4836-b3f4-0960ec022a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f7628-fw9d6" Oct 30 02:18:52.114022 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:52.113736 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4748ea17-f686-4836-b3f4-0960ec022a05-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4f7628-fw9d6\" (UID: \"4748ea17-f686-4836-b3f4-0960ec022a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f7628-fw9d6" Oct 30 02:18:52.114479 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:52.114452 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4748ea17-f686-4836-b3f4-0960ec022a05-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4f7628-fw9d6\" (UID: \"4748ea17-f686-4836-b3f4-0960ec022a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f7628-fw9d6" Oct 30 02:18:52.116166 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:52.116140 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4748ea17-f686-4836-b3f4-0960ec022a05-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4f7628-fw9d6\" (UID: \"4748ea17-f686-4836-b3f4-0960ec022a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f7628-fw9d6" Oct 30 02:18:52.132647 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:52.132608 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbnx7\" (UniqueName: \"kubernetes.io/projected/4748ea17-f686-4836-b3f4-0960ec022a05-kube-api-access-cbnx7\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4f7628-fw9d6\" (UID: \"4748ea17-f686-4836-b3f4-0960ec022a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f7628-fw9d6" Oct 30 02:18:52.180379 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:52.180338 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f7628-fw9d6" Oct 30 02:18:52.189005 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:52.188971 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:52.485711 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:52.485677 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f7628-fw9d6"] Oct 30 02:18:52.487188 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:18:52.487159 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4748ea17_f686_4836_b3f4_0960ec022a05.slice/crio-279edc3ace4f857e0ece10bf24df65c5b48d5af3b271acc5e6705c0939ff7062 WatchSource:0}: Error finding container 279edc3ace4f857e0ece10bf24df65c5b48d5af3b271acc5e6705c0939ff7062: Status 404 returned error can't find the container with id 279edc3ace4f857e0ece10bf24df65c5b48d5af3b271acc5e6705c0939ff7062 Oct 30 02:18:53.003495 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:53.003450 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f7628-fw9d6" event={"ID":"4748ea17-f686-4836-b3f4-0960ec022a05","Type":"ContainerStarted","Data":"40984e14414e47efe28b325153cfc87f0f381c4233f8fd4dd7f0c002faa7766d"} Oct 30 02:18:53.003684 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:53.003517 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f7628-fw9d6" event={"ID":"4748ea17-f686-4836-b3f4-0960ec022a05","Type":"ContainerStarted","Data":"279edc3ace4f857e0ece10bf24df65c5b48d5af3b271acc5e6705c0939ff7062"} Oct 30 02:18:53.042100 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:53.042034 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f7628-fw9d6" podStartSLOduration=2.042019352 podStartE2EDuration="2.042019352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:18:53.040728603 +0000 UTC m=+3658.617154008" watchObservedRunningTime="2026-10-30 02:18:53.042019352 +0000 UTC m=+3658.618444754" Oct 30 02:18:53.163810 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:53.163780 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-396f8e-sb4l6" Oct 30 02:18:53.328615 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:53.328573 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c3683b7-bc0c-4a95-98dd-054a06ab137a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c3683b7-bc0c-4a95-98dd-054a06ab137a-maas-api-xks-ca\") pod \"9c3683b7-bc0c-4a95-98dd-054a06ab137a\" (UID: \"9c3683b7-bc0c-4a95-98dd-054a06ab137a\") " Oct 30 02:18:53.328830 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:53.328815 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c3683b7-bc0c-4a95-98dd-054a06ab137a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c3683b7-bc0c-4a95-98dd-054a06ab137a-maas-api-openshift-ca\") pod \"9c3683b7-bc0c-4a95-98dd-054a06ab137a\" (UID: \"9c3683b7-bc0c-4a95-98dd-054a06ab137a\") " Oct 30 02:18:53.328889 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:53.328850 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c3683b7-bc0c-4a95-98dd-054a06ab137a-kube-api-access-r56p7\" (UniqueName: \"kubernetes.io/projected/9c3683b7-bc0c-4a95-98dd-054a06ab137a-kube-api-access-r56p7\") pod \"9c3683b7-bc0c-4a95-98dd-054a06ab137a\" (UID: \"9c3683b7-bc0c-4a95-98dd-054a06ab137a\") " Oct 30 02:18:53.329216 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:53.329189 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c3683b7-bc0c-4a95-98dd-054a06ab137a-maas-api-openshift-ca" pod "9c3683b7-bc0c-4a95-98dd-054a06ab137a" (UID: "9c3683b7-bc0c-4a95-98dd-054a06ab137a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:53.330789 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:53.330734 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c3683b7-bc0c-4a95-98dd-054a06ab137a-maas-api-xks-ca" pod "9c3683b7-bc0c-4a95-98dd-054a06ab137a" (UID: "9c3683b7-bc0c-4a95-98dd-054a06ab137a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:18:53.330958 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:53.330936 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c3683b7-bc0c-4a95-98dd-054a06ab137a-kube-api-access-r56p7" pod "9c3683b7-bc0c-4a95-98dd-054a06ab137a" (UID: "9c3683b7-bc0c-4a95-98dd-054a06ab137a"). InnerVolumeSpecName "kube-api-access-r56p7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:53.430450 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:53.430409 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c3683b7-bc0c-4a95-98dd-054a06ab137a-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:53.430450 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:53.430452 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r56p7\" (UniqueName: \"kubernetes.io/projected/9c3683b7-bc0c-4a95-98dd-054a06ab137a-kube-api-access-r56p7\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:53.430648 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:53.430463 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c3683b7-bc0c-4a95-98dd-054a06ab137a-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:53.565449 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:53.565399 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:54.013180 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:54.013146 6572 generic.go:356] "Generic (PLEG): container finished" podID="4748ea17-f686-4836-b3f4-0960ec022a05" containerID="40984e14414e47efe28b325153cfc87f0f381c4233f8fd4dd7f0c002faa7766d" exitCode=0 Oct 30 02:18:54.013386 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:54.013238 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f7628-fw9d6" event={"ID":"4748ea17-f686-4836-b3f4-0960ec022a05","Type":"ContainerDied","Data":"40984e14414e47efe28b325153cfc87f0f381c4233f8fd4dd7f0c002faa7766d"} Oct 30 02:18:54.014778 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:54.014739 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-396f8e-sb4l6" event={"ID":"9c3683b7-bc0c-4a95-98dd-054a06ab137a","Type":"ContainerDied","Data":"5d124fdb8218f3b3171ddffbc7f851c03b8757da64f4fd0bf8b51de104dd7fd0"} Oct 30 02:18:54.014778 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:54.014770 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-396f8e-sb4l6" Oct 30 02:18:54.014954 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:54.014793 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d124fdb8218f3b3171ddffbc7f851c03b8757da64f4fd0bf8b51de104dd7fd0" Oct 30 02:18:54.125394 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:54.125355 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fdbaa16-bhwkq"] Oct 30 02:18:54.125963 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:54.125945 6572 cpu_manager.go:414] "containerMap: removing container" podUID="9c3683b7-bc0c-4a95-98dd-054a06ab137a" containerName="revoke-keys" Oct 30 02:18:54.125963 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:54.125963 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c3683b7-bc0c-4a95-98dd-054a06ab137a" containerName="revoke-keys" Oct 30 02:18:54.126140 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:54.126088 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c3683b7-bc0c-4a95-98dd-054a06ab137a" containerName="revoke-keys" Oct 30 02:18:54.129716 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:54.129695 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fdbaa16-bhwkq" Oct 30 02:18:54.138425 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:54.138399 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1229a151-ffd5-4691-bc3a-7b4afdd0c5c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fdbaa16-bhwkq\" (UID: \"1229a151-ffd5-4691-bc3a-7b4afdd0c5c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fdbaa16-bhwkq" Oct 30 02:18:54.138540 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:54.138461 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmm7f\" (UniqueName: \"kubernetes.io/projected/1229a151-ffd5-4691-bc3a-7b4afdd0c5c3-kube-api-access-qmm7f\") pod \"maas-api-revoke-sub-8fdbaa16-bhwkq\" (UID: \"1229a151-ffd5-4691-bc3a-7b4afdd0c5c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fdbaa16-bhwkq" Oct 30 02:18:54.138612 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:54.138590 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1229a151-ffd5-4691-bc3a-7b4afdd0c5c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fdbaa16-bhwkq\" (UID: \"1229a151-ffd5-4691-bc3a-7b4afdd0c5c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fdbaa16-bhwkq" Oct 30 02:18:54.147988 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:54.147953 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fdbaa16-bhwkq"] Oct 30 02:18:54.239330 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:54.239294 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1229a151-ffd5-4691-bc3a-7b4afdd0c5c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fdbaa16-bhwkq\" (UID: \"1229a151-ffd5-4691-bc3a-7b4afdd0c5c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fdbaa16-bhwkq" Oct 30 02:18:54.239535 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:54.239357 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmm7f\" (UniqueName: \"kubernetes.io/projected/1229a151-ffd5-4691-bc3a-7b4afdd0c5c3-kube-api-access-qmm7f\") pod \"maas-api-revoke-sub-8fdbaa16-bhwkq\" (UID: \"1229a151-ffd5-4691-bc3a-7b4afdd0c5c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fdbaa16-bhwkq" Oct 30 02:18:54.239535 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:54.239407 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1229a151-ffd5-4691-bc3a-7b4afdd0c5c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fdbaa16-bhwkq\" (UID: \"1229a151-ffd5-4691-bc3a-7b4afdd0c5c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fdbaa16-bhwkq" Oct 30 02:18:54.240099 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:54.240071 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1229a151-ffd5-4691-bc3a-7b4afdd0c5c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fdbaa16-bhwkq\" (UID: \"1229a151-ffd5-4691-bc3a-7b4afdd0c5c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fdbaa16-bhwkq" Oct 30 02:18:54.241851 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:54.241829 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1229a151-ffd5-4691-bc3a-7b4afdd0c5c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fdbaa16-bhwkq\" (UID: \"1229a151-ffd5-4691-bc3a-7b4afdd0c5c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fdbaa16-bhwkq" Oct 30 02:18:54.258387 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:54.258356 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmm7f\" (UniqueName: \"kubernetes.io/projected/1229a151-ffd5-4691-bc3a-7b4afdd0c5c3-kube-api-access-qmm7f\") pod \"maas-api-revoke-sub-8fdbaa16-bhwkq\" (UID: \"1229a151-ffd5-4691-bc3a-7b4afdd0c5c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fdbaa16-bhwkq" Oct 30 02:18:54.443507 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:54.443474 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fdbaa16-bhwkq" Oct 30 02:18:54.753283 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:54.753253 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fdbaa16-bhwkq"] Oct 30 02:18:54.754872 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:18:54.754844 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1229a151_ffd5_4691_bc3a_7b4afdd0c5c3.slice/crio-99d8af4e281bc5b3f88f4a3365d3f0cd001b0c08037d1fb8a20930d787eabd12 WatchSource:0}: Error finding container 99d8af4e281bc5b3f88f4a3365d3f0cd001b0c08037d1fb8a20930d787eabd12: Status 404 returned error can't find the container with id 99d8af4e281bc5b3f88f4a3365d3f0cd001b0c08037d1fb8a20930d787eabd12 Oct 30 02:18:54.950603 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:54.950566 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:55.025180 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:55.025140 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fdbaa16-bhwkq" event={"ID":"1229a151-ffd5-4691-bc3a-7b4afdd0c5c3","Type":"ContainerStarted","Data":"99d8af4e281bc5b3f88f4a3365d3f0cd001b0c08037d1fb8a20930d787eabd12"} Oct 30 02:18:55.182723 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:55.182694 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f7628-fw9d6" Oct 30 02:18:55.249982 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:55.249942 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4748ea17-f686-4836-b3f4-0960ec022a05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4748ea17-f686-4836-b3f4-0960ec022a05-maas-api-xks-ca\") pod \"4748ea17-f686-4836-b3f4-0960ec022a05\" (UID: \"4748ea17-f686-4836-b3f4-0960ec022a05\") " Oct 30 02:18:55.250198 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:55.249998 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4748ea17-f686-4836-b3f4-0960ec022a05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4748ea17-f686-4836-b3f4-0960ec022a05-maas-api-openshift-ca\") pod \"4748ea17-f686-4836-b3f4-0960ec022a05\" (UID: \"4748ea17-f686-4836-b3f4-0960ec022a05\") " Oct 30 02:18:55.250198 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:55.250081 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4748ea17-f686-4836-b3f4-0960ec022a05-kube-api-access-cbnx7\" (UniqueName: \"kubernetes.io/projected/4748ea17-f686-4836-b3f4-0960ec022a05-kube-api-access-cbnx7\") pod \"4748ea17-f686-4836-b3f4-0960ec022a05\" (UID: \"4748ea17-f686-4836-b3f4-0960ec022a05\") " Oct 30 02:18:55.250493 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:55.250464 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4748ea17-f686-4836-b3f4-0960ec022a05-maas-api-openshift-ca" pod "4748ea17-f686-4836-b3f4-0960ec022a05" (UID: "4748ea17-f686-4836-b3f4-0960ec022a05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:55.250620 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:55.250601 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4748ea17-f686-4836-b3f4-0960ec022a05-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:55.252360 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:55.252330 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4748ea17-f686-4836-b3f4-0960ec022a05-kube-api-access-cbnx7" pod "4748ea17-f686-4836-b3f4-0960ec022a05" (UID: "4748ea17-f686-4836-b3f4-0960ec022a05"). InnerVolumeSpecName "kube-api-access-cbnx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:55.252517 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:55.252364 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4748ea17-f686-4836-b3f4-0960ec022a05-maas-api-xks-ca" pod "4748ea17-f686-4836-b3f4-0960ec022a05" (UID: "4748ea17-f686-4836-b3f4-0960ec022a05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:18:55.351939 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:55.351835 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbnx7\" (UniqueName: \"kubernetes.io/projected/4748ea17-f686-4836-b3f4-0960ec022a05-kube-api-access-cbnx7\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:55.351939 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:55.351875 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4748ea17-f686-4836-b3f4-0960ec022a05-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:56.035887 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:56.035792 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f7628-fw9d6" Oct 30 02:18:56.036397 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:56.035783 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f7628-fw9d6" event={"ID":"4748ea17-f686-4836-b3f4-0960ec022a05","Type":"ContainerDied","Data":"279edc3ace4f857e0ece10bf24df65c5b48d5af3b271acc5e6705c0939ff7062"} Oct 30 02:18:56.036397 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:56.035934 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="279edc3ace4f857e0ece10bf24df65c5b48d5af3b271acc5e6705c0939ff7062" Oct 30 02:18:56.037919 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:56.037891 6572 generic.go:356] "Generic (PLEG): container finished" podID="1229a151-ffd5-4691-bc3a-7b4afdd0c5c3" containerID="218d59830482cb008d1c1561dbd0d1d2c905cd4c14273317d6c054171f5ea076" exitCode=0 Oct 30 02:18:56.038054 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:56.037939 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fdbaa16-bhwkq" event={"ID":"1229a151-ffd5-4691-bc3a-7b4afdd0c5c3","Type":"ContainerDied","Data":"218d59830482cb008d1c1561dbd0d1d2c905cd4c14273317d6c054171f5ea076"} Oct 30 02:18:56.369292 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:56.369251 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:56.599321 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:56.599289 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 02:18:56.665970 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:56.665930 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/223ead86-cf13-4c05-9655-893ffc0b9907-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/223ead86-cf13-4c05-9655-893ffc0b9907-istio-envoy\") pod \"223ead86-cf13-4c05-9655-893ffc0b9907\" (UID: \"223ead86-cf13-4c05-9655-893ffc0b9907\") " Oct 30 02:18:56.666163 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:56.665995 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/223ead86-cf13-4c05-9655-893ffc0b9907-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/223ead86-cf13-4c05-9655-893ffc0b9907-istio-data\") pod \"223ead86-cf13-4c05-9655-893ffc0b9907\" (UID: \"223ead86-cf13-4c05-9655-893ffc0b9907\") " Oct 30 02:18:56.666163 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:56.666045 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/223ead86-cf13-4c05-9655-893ffc0b9907-kube-api-access-qvcw4\" (UniqueName: \"kubernetes.io/projected/223ead86-cf13-4c05-9655-893ffc0b9907-kube-api-access-qvcw4\") pod \"223ead86-cf13-4c05-9655-893ffc0b9907\" (UID: \"223ead86-cf13-4c05-9655-893ffc0b9907\") " Oct 30 02:18:56.666163 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:56.666108 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/223ead86-cf13-4c05-9655-893ffc0b9907-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/223ead86-cf13-4c05-9655-893ffc0b9907-istiod-ca-cert\") pod \"223ead86-cf13-4c05-9655-893ffc0b9907\" (UID: \"223ead86-cf13-4c05-9655-893ffc0b9907\") " Oct 30 02:18:56.666323 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:56.666184 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/223ead86-cf13-4c05-9655-893ffc0b9907-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/223ead86-cf13-4c05-9655-893ffc0b9907-credential-socket\") pod \"223ead86-cf13-4c05-9655-893ffc0b9907\" (UID: \"223ead86-cf13-4c05-9655-893ffc0b9907\") " Oct 30 02:18:56.666323 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:56.666309 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/223ead86-cf13-4c05-9655-893ffc0b9907-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/223ead86-cf13-4c05-9655-893ffc0b9907-istio-podinfo\") pod \"223ead86-cf13-4c05-9655-893ffc0b9907\" (UID: \"223ead86-cf13-4c05-9655-893ffc0b9907\") " Oct 30 02:18:56.666453 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:56.666348 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/223ead86-cf13-4c05-9655-893ffc0b9907-istio-data" pod "223ead86-cf13-4c05-9655-893ffc0b9907" (UID: "223ead86-cf13-4c05-9655-893ffc0b9907"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:56.666453 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:56.666368 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/223ead86-cf13-4c05-9655-893ffc0b9907-istio-token\" (UniqueName: \"kubernetes.io/projected/223ead86-cf13-4c05-9655-893ffc0b9907-istio-token\") pod \"223ead86-cf13-4c05-9655-893ffc0b9907\" (UID: \"223ead86-cf13-4c05-9655-893ffc0b9907\") " Oct 30 02:18:56.666453 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:56.666397 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/223ead86-cf13-4c05-9655-893ffc0b9907-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/223ead86-cf13-4c05-9655-893ffc0b9907-workload-socket\") pod \"223ead86-cf13-4c05-9655-893ffc0b9907\" (UID: \"223ead86-cf13-4c05-9655-893ffc0b9907\") " Oct 30 02:18:56.666453 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:56.666437 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/223ead86-cf13-4c05-9655-893ffc0b9907-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/223ead86-cf13-4c05-9655-893ffc0b9907-workload-certs\") pod \"223ead86-cf13-4c05-9655-893ffc0b9907\" (UID: \"223ead86-cf13-4c05-9655-893ffc0b9907\") " Oct 30 02:18:56.666691 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:56.666600 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/223ead86-cf13-4c05-9655-893ffc0b9907-istiod-ca-cert" pod "223ead86-cf13-4c05-9655-893ffc0b9907" (UID: "223ead86-cf13-4c05-9655-893ffc0b9907"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:56.666691 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:56.666640 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/223ead86-cf13-4c05-9655-893ffc0b9907-credential-socket" pod "223ead86-cf13-4c05-9655-893ffc0b9907" (UID: "223ead86-cf13-4c05-9655-893ffc0b9907"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:56.667175 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:56.666869 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/223ead86-cf13-4c05-9655-893ffc0b9907-workload-certs" pod "223ead86-cf13-4c05-9655-893ffc0b9907" (UID: "223ead86-cf13-4c05-9655-893ffc0b9907"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:56.667175 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:56.667086 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/223ead86-cf13-4c05-9655-893ffc0b9907-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:56.667175 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:56.667110 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/223ead86-cf13-4c05-9655-893ffc0b9907-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:56.667175 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:56.667126 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/223ead86-cf13-4c05-9655-893ffc0b9907-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:56.667175 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:56.667140 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/223ead86-cf13-4c05-9655-893ffc0b9907-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:56.667601 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:56.667436 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/223ead86-cf13-4c05-9655-893ffc0b9907-workload-socket" pod "223ead86-cf13-4c05-9655-893ffc0b9907" (UID: "223ead86-cf13-4c05-9655-893ffc0b9907"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:56.668876 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:56.668846 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/223ead86-cf13-4c05-9655-893ffc0b9907-kube-api-access-qvcw4" pod "223ead86-cf13-4c05-9655-893ffc0b9907" (UID: "223ead86-cf13-4c05-9655-893ffc0b9907"). InnerVolumeSpecName "kube-api-access-qvcw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:56.668876 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:56.668869 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/223ead86-cf13-4c05-9655-893ffc0b9907-istio-envoy" pod "223ead86-cf13-4c05-9655-893ffc0b9907" (UID: "223ead86-cf13-4c05-9655-893ffc0b9907"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:56.669538 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:56.669503 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/223ead86-cf13-4c05-9655-893ffc0b9907-istio-podinfo" pod "223ead86-cf13-4c05-9655-893ffc0b9907" (UID: "223ead86-cf13-4c05-9655-893ffc0b9907"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:18:56.669875 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:56.669855 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/223ead86-cf13-4c05-9655-893ffc0b9907-istio-token" pod "223ead86-cf13-4c05-9655-893ffc0b9907" (UID: "223ead86-cf13-4c05-9655-893ffc0b9907"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:56.768575 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:56.768530 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/223ead86-cf13-4c05-9655-893ffc0b9907-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:56.768575 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:56.768567 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvcw4\" (UniqueName: \"kubernetes.io/projected/223ead86-cf13-4c05-9655-893ffc0b9907-kube-api-access-qvcw4\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:56.768575 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:56.768577 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/223ead86-cf13-4c05-9655-893ffc0b9907-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:56.768907 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:56.768589 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/223ead86-cf13-4c05-9655-893ffc0b9907-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:56.768907 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:56.768600 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/223ead86-cf13-4c05-9655-893ffc0b9907-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:57.048733 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:57.048698 6572 generic.go:356] "Generic (PLEG): container finished" podID="223ead86-cf13-4c05-9655-893ffc0b9907" containerID="904eed12435e699e4bdab08f527f0c4a450cf2611c545b27b9ce2ebad112bb3e" exitCode=0 Oct 30 02:18:57.049265 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:57.048774 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" Oct 30 02:18:57.049265 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:57.048780 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" event={"ID":"223ead86-cf13-4c05-9655-893ffc0b9907","Type":"ContainerDied","Data":"904eed12435e699e4bdab08f527f0c4a450cf2611c545b27b9ce2ebad112bb3e"} Oct 30 02:18:57.049265 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:57.048833 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv" event={"ID":"223ead86-cf13-4c05-9655-893ffc0b9907","Type":"ContainerDied","Data":"2ceae912aa7728a7e0f8a4b8490f75f749fc939dd28a00d2bc537196e8f4ff8f"} Oct 30 02:18:57.049265 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:57.048855 6572 scope.go:122] "RemoveContainer" containerID="904eed12435e699e4bdab08f527f0c4a450cf2611c545b27b9ce2ebad112bb3e" Oct 30 02:18:57.065703 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:57.065677 6572 scope.go:122] "RemoveContainer" containerID="03fce063a6260553f3b174cfb950a99452488a18c1f8906d3b4262e1c4de7878" Oct 30 02:18:57.078450 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:57.078418 6572 scope.go:122] "RemoveContainer" containerID="904eed12435e699e4bdab08f527f0c4a450cf2611c545b27b9ce2ebad112bb3e" Oct 30 02:18:57.078869 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:18:57.078838 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"904eed12435e699e4bdab08f527f0c4a450cf2611c545b27b9ce2ebad112bb3e\": container with ID starting with 904eed12435e699e4bdab08f527f0c4a450cf2611c545b27b9ce2ebad112bb3e not found: ID does not exist" containerID="904eed12435e699e4bdab08f527f0c4a450cf2611c545b27b9ce2ebad112bb3e" Oct 30 02:18:57.078943 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:57.078888 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"904eed12435e699e4bdab08f527f0c4a450cf2611c545b27b9ce2ebad112bb3e"} err="failed to get container status \"904eed12435e699e4bdab08f527f0c4a450cf2611c545b27b9ce2ebad112bb3e\": rpc error: code = NotFound desc = could not find container \"904eed12435e699e4bdab08f527f0c4a450cf2611c545b27b9ce2ebad112bb3e\": container with ID starting with 904eed12435e699e4bdab08f527f0c4a450cf2611c545b27b9ce2ebad112bb3e not found: ID does not exist" Oct 30 02:18:57.078943 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:57.078914 6572 scope.go:122] "RemoveContainer" containerID="03fce063a6260553f3b174cfb950a99452488a18c1f8906d3b4262e1c4de7878" Oct 30 02:18:57.079257 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:18:57.079236 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03fce063a6260553f3b174cfb950a99452488a18c1f8906d3b4262e1c4de7878\": container with ID starting with 03fce063a6260553f3b174cfb950a99452488a18c1f8906d3b4262e1c4de7878 not found: ID does not exist" containerID="03fce063a6260553f3b174cfb950a99452488a18c1f8906d3b4262e1c4de7878" Oct 30 02:18:57.079317 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:57.079265 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03fce063a6260553f3b174cfb950a99452488a18c1f8906d3b4262e1c4de7878"} err="failed to get container status \"03fce063a6260553f3b174cfb950a99452488a18c1f8906d3b4262e1c4de7878\": rpc error: code = NotFound desc = could not find container \"03fce063a6260553f3b174cfb950a99452488a18c1f8906d3b4262e1c4de7878\": container with ID starting with 03fce063a6260553f3b174cfb950a99452488a18c1f8906d3b4262e1c4de7878 not found: ID does not exist" Oct 30 02:18:57.100678 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:57.100640 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv"] Oct 30 02:18:57.113959 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:57.113921 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5fb950-openshift-default-5fb6d597d7-fgwkv"] Oct 30 02:18:57.199227 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:57.199198 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fdbaa16-bhwkq" Oct 30 02:18:57.274601 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:57.274557 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1229a151-ffd5-4691-bc3a-7b4afdd0c5c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1229a151-ffd5-4691-bc3a-7b4afdd0c5c3-maas-api-xks-ca\") pod \"1229a151-ffd5-4691-bc3a-7b4afdd0c5c3\" (UID: \"1229a151-ffd5-4691-bc3a-7b4afdd0c5c3\") " Oct 30 02:18:57.274800 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:57.274785 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1229a151-ffd5-4691-bc3a-7b4afdd0c5c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1229a151-ffd5-4691-bc3a-7b4afdd0c5c3-maas-api-openshift-ca\") pod \"1229a151-ffd5-4691-bc3a-7b4afdd0c5c3\" (UID: \"1229a151-ffd5-4691-bc3a-7b4afdd0c5c3\") " Oct 30 02:18:57.274878 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:57.274819 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1229a151-ffd5-4691-bc3a-7b4afdd0c5c3-kube-api-access-qmm7f\" (UniqueName: \"kubernetes.io/projected/1229a151-ffd5-4691-bc3a-7b4afdd0c5c3-kube-api-access-qmm7f\") pod \"1229a151-ffd5-4691-bc3a-7b4afdd0c5c3\" (UID: \"1229a151-ffd5-4691-bc3a-7b4afdd0c5c3\") " Oct 30 02:18:57.275237 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:57.275208 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1229a151-ffd5-4691-bc3a-7b4afdd0c5c3-maas-api-openshift-ca" pod "1229a151-ffd5-4691-bc3a-7b4afdd0c5c3" (UID: "1229a151-ffd5-4691-bc3a-7b4afdd0c5c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:57.276975 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:57.276942 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1229a151-ffd5-4691-bc3a-7b4afdd0c5c3-maas-api-xks-ca" pod "1229a151-ffd5-4691-bc3a-7b4afdd0c5c3" (UID: "1229a151-ffd5-4691-bc3a-7b4afdd0c5c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:18:57.277105 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:57.277080 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1229a151-ffd5-4691-bc3a-7b4afdd0c5c3-kube-api-access-qmm7f" pod "1229a151-ffd5-4691-bc3a-7b4afdd0c5c3" (UID: "1229a151-ffd5-4691-bc3a-7b4afdd0c5c3"). InnerVolumeSpecName "kube-api-access-qmm7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:57.376111 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:57.376017 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1229a151-ffd5-4691-bc3a-7b4afdd0c5c3-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:57.376111 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:57.376050 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmm7f\" (UniqueName: \"kubernetes.io/projected/1229a151-ffd5-4691-bc3a-7b4afdd0c5c3-kube-api-access-qmm7f\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:57.376111 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:57.376061 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1229a151-ffd5-4691-bc3a-7b4afdd0c5c3-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:18:57.658472 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:57.658368 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:58.059550 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:58.059516 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fdbaa16-bhwkq" Oct 30 02:18:58.060018 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:58.059555 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fdbaa16-bhwkq" event={"ID":"1229a151-ffd5-4691-bc3a-7b4afdd0c5c3","Type":"ContainerDied","Data":"99d8af4e281bc5b3f88f4a3365d3f0cd001b0c08037d1fb8a20930d787eabd12"} Oct 30 02:18:58.060018 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:58.059607 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99d8af4e281bc5b3f88f4a3365d3f0cd001b0c08037d1fb8a20930d787eabd12" Oct 30 02:18:58.541779 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:58.541710 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="223ead86-cf13-4c05-9655-893ffc0b9907" path="/var/lib/kubelet/pods/223ead86-cf13-4c05-9655-893ffc0b9907/volumes" Oct 30 02:18:59.114051 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:59.114014 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:59.653933 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:59.653901 6572 scope.go:122] "RemoveContainer" containerID="351149746ac04315c86c2e99e4fad956d61874677b1683b62bb8259e030906f6" Oct 30 02:18:59.666062 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:59.665966 6572 scope.go:122] "RemoveContainer" containerID="11a77fac2bdfb09297c3636c11ce40cf79069a4dd4bcefa8d36db5f1058655d0" Oct 30 02:18:59.678868 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:59.678846 6572 scope.go:122] "RemoveContainer" containerID="73fd74bfb4ab98c1cb424ec0e390c283f20a15383f35cf6eb3fa5549ca3aa7a4" Oct 30 02:18:59.689067 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:59.689046 6572 scope.go:122] "RemoveContainer" containerID="47f1efe4a4f4a15f4e0fc6666deacf9383ce3f3063116397289e9acdc009be1f" Oct 30 02:18:59.698686 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:59.698662 6572 scope.go:122] "RemoveContainer" containerID="108ca68d130758ca46835ec61951cd311b5aed54c53c1f406a1e9c1ce85c220a" Oct 30 02:18:59.708476 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:18:59.708458 6572 scope.go:122] "RemoveContainer" containerID="e1b71fb9c00d0cf0e10eab9c758e5da21009f41fa0035989497f827999f92e83" Oct 30 02:19:00.439689 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:00.439653 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:00.844015 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:00.843075 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:01.024129 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:01.024094 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:01.845792 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:01.833897 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:02.550230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:02.549287 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:02.736361 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:02.736317 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:03.267121 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:03.267087 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:04.694588 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:04.694540 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:05.065226 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:05.065135 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv"] Oct 30 02:19:05.337550 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:05.337448 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:05.617467 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:05.617383 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a56df390-2dpvv"] Oct 30 02:19:05.618049 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:05.618028 6572 cpu_manager.go:414] "containerMap: removing container" podUID="4748ea17-f686-4836-b3f4-0960ec022a05" containerName="revoke-keys" Oct 30 02:19:05.618049 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:05.618048 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4748ea17-f686-4836-b3f4-0960ec022a05" containerName="revoke-keys" Oct 30 02:19:05.618200 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:05.618075 6572 cpu_manager.go:414] "containerMap: removing container" podUID="1229a151-ffd5-4691-bc3a-7b4afdd0c5c3" containerName="revoke-keys" Oct 30 02:19:05.618200 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:05.618080 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1229a151-ffd5-4691-bc3a-7b4afdd0c5c3" containerName="revoke-keys" Oct 30 02:19:05.618200 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:05.618098 6572 cpu_manager.go:414] "containerMap: removing container" podUID="223ead86-cf13-4c05-9655-893ffc0b9907" containerName="istio-proxy" Oct 30 02:19:05.618200 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:05.618105 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="223ead86-cf13-4c05-9655-893ffc0b9907" containerName="istio-proxy" Oct 30 02:19:05.618200 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:05.618128 6572 cpu_manager.go:414] "containerMap: removing container" podUID="223ead86-cf13-4c05-9655-893ffc0b9907" containerName="istio-proxy" Oct 30 02:19:05.618200 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:05.618133 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="223ead86-cf13-4c05-9655-893ffc0b9907" containerName="istio-proxy" Oct 30 02:19:05.618419 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:05.618215 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="223ead86-cf13-4c05-9655-893ffc0b9907" containerName="istio-proxy" Oct 30 02:19:05.618419 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:05.618223 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="4748ea17-f686-4836-b3f4-0960ec022a05" containerName="revoke-keys" Oct 30 02:19:05.618419 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:05.618234 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="1229a151-ffd5-4691-bc3a-7b4afdd0c5c3" containerName="revoke-keys" Oct 30 02:19:05.624231 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:05.624204 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56df390-2dpvv" Oct 30 02:19:05.638366 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:05.638328 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:19:05.638555 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:05.638423 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:19:05.641108 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:05.641075 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a56df390-2dpvv"] Oct 30 02:19:05.686341 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:05.686299 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjvv6\" (UniqueName: \"kubernetes.io/projected/70e0bbb2-2d70-4f55-9aa1-91e0929b53c2-kube-api-access-vjvv6\") pod \"maas-api-revoke-sub-a56df390-2dpvv\" (UID: \"70e0bbb2-2d70-4f55-9aa1-91e0929b53c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56df390-2dpvv" Oct 30 02:19:05.686530 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:05.686473 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70e0bbb2-2d70-4f55-9aa1-91e0929b53c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a56df390-2dpvv\" (UID: \"70e0bbb2-2d70-4f55-9aa1-91e0929b53c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56df390-2dpvv" Oct 30 02:19:05.686635 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:05.686611 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70e0bbb2-2d70-4f55-9aa1-91e0929b53c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a56df390-2dpvv\" (UID: \"70e0bbb2-2d70-4f55-9aa1-91e0929b53c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56df390-2dpvv" Oct 30 02:19:05.687311 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:05.687292 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:19:05.687510 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:05.687494 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:19:05.788214 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:05.788174 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjvv6\" (UniqueName: \"kubernetes.io/projected/70e0bbb2-2d70-4f55-9aa1-91e0929b53c2-kube-api-access-vjvv6\") pod \"maas-api-revoke-sub-a56df390-2dpvv\" (UID: \"70e0bbb2-2d70-4f55-9aa1-91e0929b53c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56df390-2dpvv" Oct 30 02:19:05.788719 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:05.788298 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70e0bbb2-2d70-4f55-9aa1-91e0929b53c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a56df390-2dpvv\" (UID: \"70e0bbb2-2d70-4f55-9aa1-91e0929b53c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56df390-2dpvv" Oct 30 02:19:05.788719 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:05.788389 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70e0bbb2-2d70-4f55-9aa1-91e0929b53c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a56df390-2dpvv\" (UID: \"70e0bbb2-2d70-4f55-9aa1-91e0929b53c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56df390-2dpvv" Oct 30 02:19:05.789220 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:05.789195 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70e0bbb2-2d70-4f55-9aa1-91e0929b53c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a56df390-2dpvv\" (UID: \"70e0bbb2-2d70-4f55-9aa1-91e0929b53c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56df390-2dpvv" Oct 30 02:19:05.790949 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:05.790926 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70e0bbb2-2d70-4f55-9aa1-91e0929b53c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a56df390-2dpvv\" (UID: \"70e0bbb2-2d70-4f55-9aa1-91e0929b53c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56df390-2dpvv" Oct 30 02:19:05.807434 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:05.807394 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjvv6\" (UniqueName: \"kubernetes.io/projected/70e0bbb2-2d70-4f55-9aa1-91e0929b53c2-kube-api-access-vjvv6\") pod \"maas-api-revoke-sub-a56df390-2dpvv\" (UID: \"70e0bbb2-2d70-4f55-9aa1-91e0929b53c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56df390-2dpvv" Oct 30 02:19:05.938457 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:05.938417 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56df390-2dpvv" Oct 30 02:19:06.092671 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:06.092632 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:06.241934 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:19:06.241877 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70e0bbb2_2d70_4f55_9aa1_91e0929b53c2.slice/crio-c7465601f58d3007f4e1e1ee3153e38d3854b24027ae4215e6fd280f1796d4bd WatchSource:0}: Error finding container c7465601f58d3007f4e1e1ee3153e38d3854b24027ae4215e6fd280f1796d4bd: Status 404 returned error can't find the container with id c7465601f58d3007f4e1e1ee3153e38d3854b24027ae4215e6fd280f1796d4bd Oct 30 02:19:06.243436 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:06.243413 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a56df390-2dpvv"] Oct 30 02:19:07.131921 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:07.131883 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56df390-2dpvv" event={"ID":"70e0bbb2-2d70-4f55-9aa1-91e0929b53c2","Type":"ContainerStarted","Data":"aa9fbed665ad97d579f18376f5f985c2c975df2e7393df64df03ddbf050e3104"} Oct 30 02:19:07.132330 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:07.131944 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56df390-2dpvv" event={"ID":"70e0bbb2-2d70-4f55-9aa1-91e0929b53c2","Type":"ContainerStarted","Data":"c7465601f58d3007f4e1e1ee3153e38d3854b24027ae4215e6fd280f1796d4bd"} Oct 30 02:19:07.170031 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:07.169966 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56df390-2dpvv" podStartSLOduration=2.1699465079999998 podStartE2EDuration="2.169946508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:19:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:19:07.168075059 +0000 UTC m=+3672.744500460" watchObservedRunningTime="2026-10-30 02:19:07.169946508 +0000 UTC m=+3672.746371912" Oct 30 02:19:07.349563 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:07.349525 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:07.660670 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:07.660623 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:07.807460 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:07.807416 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f7628-fw9d6"] Oct 30 02:19:07.817299 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:07.817261 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4f7628-fw9d6"] Oct 30 02:19:07.849828 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:07.849789 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:08.051612 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:08.051517 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:08.141352 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:08.141316 6572 generic.go:356] "Generic (PLEG): container finished" podID="70e0bbb2-2d70-4f55-9aa1-91e0929b53c2" containerID="aa9fbed665ad97d579f18376f5f985c2c975df2e7393df64df03ddbf050e3104" exitCode=0 Oct 30 02:19:08.141793 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:08.141403 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56df390-2dpvv" event={"ID":"70e0bbb2-2d70-4f55-9aa1-91e0929b53c2","Type":"ContainerDied","Data":"aa9fbed665ad97d579f18376f5f985c2c975df2e7393df64df03ddbf050e3104"} Oct 30 02:19:08.350848 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:08.350740 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:08.362464 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:08.362428 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-396f8e-sb4l6"] Oct 30 02:19:08.365501 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:08.365474 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-396f8e-sb4l6"] Oct 30 02:19:08.544568 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:08.544533 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4748ea17-f686-4836-b3f4-0960ec022a05" path="/var/lib/kubelet/pods/4748ea17-f686-4836-b3f4-0960ec022a05/volumes" Oct 30 02:19:08.545128 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:08.545108 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c3683b7-bc0c-4a95-98dd-054a06ab137a" path="/var/lib/kubelet/pods/9c3683b7-bc0c-4a95-98dd-054a06ab137a/volumes" Oct 30 02:19:08.551275 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:08.551237 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:08.987703 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:08.987670 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:09.318265 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:09.318233 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56df390-2dpvv" Oct 30 02:19:09.435466 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:09.435428 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70e0bbb2-2d70-4f55-9aa1-91e0929b53c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70e0bbb2-2d70-4f55-9aa1-91e0929b53c2-maas-api-xks-ca\") pod \"70e0bbb2-2d70-4f55-9aa1-91e0929b53c2\" (UID: \"70e0bbb2-2d70-4f55-9aa1-91e0929b53c2\") " Oct 30 02:19:09.435695 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:09.435500 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70e0bbb2-2d70-4f55-9aa1-91e0929b53c2-kube-api-access-vjvv6\" (UniqueName: \"kubernetes.io/projected/70e0bbb2-2d70-4f55-9aa1-91e0929b53c2-kube-api-access-vjvv6\") pod \"70e0bbb2-2d70-4f55-9aa1-91e0929b53c2\" (UID: \"70e0bbb2-2d70-4f55-9aa1-91e0929b53c2\") " Oct 30 02:19:09.435695 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:09.435659 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70e0bbb2-2d70-4f55-9aa1-91e0929b53c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70e0bbb2-2d70-4f55-9aa1-91e0929b53c2-maas-api-openshift-ca\") pod \"70e0bbb2-2d70-4f55-9aa1-91e0929b53c2\" (UID: \"70e0bbb2-2d70-4f55-9aa1-91e0929b53c2\") " Oct 30 02:19:09.436086 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:09.436063 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70e0bbb2-2d70-4f55-9aa1-91e0929b53c2-maas-api-openshift-ca" pod "70e0bbb2-2d70-4f55-9aa1-91e0929b53c2" (UID: "70e0bbb2-2d70-4f55-9aa1-91e0929b53c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:09.436230 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:09.436214 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70e0bbb2-2d70-4f55-9aa1-91e0929b53c2-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:09.437718 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:09.437698 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70e0bbb2-2d70-4f55-9aa1-91e0929b53c2-maas-api-xks-ca" pod "70e0bbb2-2d70-4f55-9aa1-91e0929b53c2" (UID: "70e0bbb2-2d70-4f55-9aa1-91e0929b53c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:19:09.437804 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:09.437713 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70e0bbb2-2d70-4f55-9aa1-91e0929b53c2-kube-api-access-vjvv6" pod "70e0bbb2-2d70-4f55-9aa1-91e0929b53c2" (UID: "70e0bbb2-2d70-4f55-9aa1-91e0929b53c2"). InnerVolumeSpecName "kube-api-access-vjvv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:09.448373 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:09.448334 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:09.537696 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:09.537647 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70e0bbb2-2d70-4f55-9aa1-91e0929b53c2-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:09.537696 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:09.537683 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjvv6\" (UniqueName: \"kubernetes.io/projected/70e0bbb2-2d70-4f55-9aa1-91e0929b53c2-kube-api-access-vjvv6\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:10.050701 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:10.050658 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:10.158931 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:10.158894 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56df390-2dpvv" event={"ID":"70e0bbb2-2d70-4f55-9aa1-91e0929b53c2","Type":"ContainerDied","Data":"c7465601f58d3007f4e1e1ee3153e38d3854b24027ae4215e6fd280f1796d4bd"} Oct 30 02:19:10.158931 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:10.158916 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56df390-2dpvv" Oct 30 02:19:10.159171 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:10.158951 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7465601f58d3007f4e1e1ee3153e38d3854b24027ae4215e6fd280f1796d4bd" Oct 30 02:19:11.359405 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:11.359366 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:12.649662 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:12.649627 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:13.975680 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:13.975646 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:14.377460 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:14.377379 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:14.597493 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:14.597457 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:14.688217 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:19:14.688171 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:19:14.793735 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:14.793689 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:15.196481 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:15.196447 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:15.318435 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:15.318389 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-558327-pvkqq"] Oct 30 02:19:15.319158 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:15.319129 6572 cpu_manager.go:414] "containerMap: removing container" podUID="70e0bbb2-2d70-4f55-9aa1-91e0929b53c2" containerName="revoke-keys" Oct 30 02:19:15.319358 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:15.319328 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70e0bbb2-2d70-4f55-9aa1-91e0929b53c2" containerName="revoke-keys" Oct 30 02:19:15.319590 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:15.319570 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="223ead86-cf13-4c05-9655-893ffc0b9907" containerName="istio-proxy" Oct 30 02:19:15.319773 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:15.319741 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="70e0bbb2-2d70-4f55-9aa1-91e0929b53c2" containerName="revoke-keys" Oct 30 02:19:15.324062 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:15.324041 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-558327-pvkqq" Oct 30 02:19:15.339161 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:15.339139 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:19:15.350519 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:15.350485 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:19:15.350677 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:15.350537 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:19:15.350677 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:15.350664 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:19:15.354403 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:15.354379 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-558327-pvkqq"] Oct 30 02:19:15.402610 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:15.402580 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:15.424396 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:15.424364 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcbaf9cf-86a4-4eb9-86d7-263c84c928e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-558327-pvkqq\" (UID: \"bcbaf9cf-86a4-4eb9-86d7-263c84c928e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-558327-pvkqq" Oct 30 02:19:15.424572 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:15.424402 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scjsk\" (UniqueName: \"kubernetes.io/projected/bcbaf9cf-86a4-4eb9-86d7-263c84c928e8-kube-api-access-scjsk\") pod \"maas-api-revoke-keys-e2e-worker-w3-558327-pvkqq\" (UID: \"bcbaf9cf-86a4-4eb9-86d7-263c84c928e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-558327-pvkqq" Oct 30 02:19:15.424572 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:15.424428 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcbaf9cf-86a4-4eb9-86d7-263c84c928e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-558327-pvkqq\" (UID: \"bcbaf9cf-86a4-4eb9-86d7-263c84c928e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-558327-pvkqq" Oct 30 02:19:15.525676 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:15.525585 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcbaf9cf-86a4-4eb9-86d7-263c84c928e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-558327-pvkqq\" (UID: \"bcbaf9cf-86a4-4eb9-86d7-263c84c928e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-558327-pvkqq" Oct 30 02:19:15.525676 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:15.525626 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scjsk\" (UniqueName: \"kubernetes.io/projected/bcbaf9cf-86a4-4eb9-86d7-263c84c928e8-kube-api-access-scjsk\") pod \"maas-api-revoke-keys-e2e-worker-w3-558327-pvkqq\" (UID: \"bcbaf9cf-86a4-4eb9-86d7-263c84c928e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-558327-pvkqq" Oct 30 02:19:15.525676 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:15.525659 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcbaf9cf-86a4-4eb9-86d7-263c84c928e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-558327-pvkqq\" (UID: \"bcbaf9cf-86a4-4eb9-86d7-263c84c928e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-558327-pvkqq" Oct 30 02:19:15.526497 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:15.526466 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcbaf9cf-86a4-4eb9-86d7-263c84c928e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-558327-pvkqq\" (UID: \"bcbaf9cf-86a4-4eb9-86d7-263c84c928e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-558327-pvkqq" Oct 30 02:19:15.530154 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:15.530109 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcbaf9cf-86a4-4eb9-86d7-263c84c928e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-558327-pvkqq\" (UID: \"bcbaf9cf-86a4-4eb9-86d7-263c84c928e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-558327-pvkqq" Oct 30 02:19:15.548637 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:15.548603 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scjsk\" (UniqueName: \"kubernetes.io/projected/bcbaf9cf-86a4-4eb9-86d7-263c84c928e8-kube-api-access-scjsk\") pod \"maas-api-revoke-keys-e2e-worker-w3-558327-pvkqq\" (UID: \"bcbaf9cf-86a4-4eb9-86d7-263c84c928e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-558327-pvkqq" Oct 30 02:19:15.563383 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:15.559337 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k"] Oct 30 02:19:15.563383 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:15.559593 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" podUID="7e60f1dc-1188-4053-8f31-fcd08da0ef1c" containerName="istio-proxy" containerID="cri-o://801d83f6ed507e7b49a1d63784214ef842b44a7c295e4d9a9ada9963d1a2367e" gracePeriod=30 Oct 30 02:19:15.596626 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:15.596580 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:15.638956 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:15.638910 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-558327-pvkqq" Oct 30 02:19:15.797411 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:15.797294 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:16.016392 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:16.016354 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-558327-pvkqq"] Oct 30 02:19:16.018178 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:19:16.018147 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcbaf9cf_86a4_4eb9_86d7_263c84c928e8.slice/crio-447fb8165350f64b2bc2244fe3a72d8988eb67561df02ea491b74263f1f009b2 WatchSource:0}: Error finding container 447fb8165350f64b2bc2244fe3a72d8988eb67561df02ea491b74263f1f009b2: Status 404 returned error can't find the container with id 447fb8165350f64b2bc2244fe3a72d8988eb67561df02ea491b74263f1f009b2 Oct 30 02:19:16.200817 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:16.200772 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d376-rtvsf"] Oct 30 02:19:16.228052 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:16.228012 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d376-rtvsf"] Oct 30 02:19:16.228256 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:16.228167 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d376-rtvsf" Oct 30 02:19:16.255299 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:16.255262 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-558327-pvkqq" event={"ID":"bcbaf9cf-86a4-4eb9-86d7-263c84c928e8","Type":"ContainerStarted","Data":"447fb8165350f64b2bc2244fe3a72d8988eb67561df02ea491b74263f1f009b2"} Oct 30 02:19:16.342561 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:16.342434 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9b5008a-8219-4a63-b0fd-6c718235f05e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c8d376-rtvsf\" (UID: \"c9b5008a-8219-4a63-b0fd-6c718235f05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d376-rtvsf" Oct 30 02:19:16.342561 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:16.342495 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtczl\" (UniqueName: \"kubernetes.io/projected/c9b5008a-8219-4a63-b0fd-6c718235f05e-kube-api-access-jtczl\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c8d376-rtvsf\" (UID: \"c9b5008a-8219-4a63-b0fd-6c718235f05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d376-rtvsf" Oct 30 02:19:16.342846 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:16.342613 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9b5008a-8219-4a63-b0fd-6c718235f05e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c8d376-rtvsf\" (UID: \"c9b5008a-8219-4a63-b0fd-6c718235f05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d376-rtvsf" Oct 30 02:19:16.444502 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:16.444468 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9b5008a-8219-4a63-b0fd-6c718235f05e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c8d376-rtvsf\" (UID: \"c9b5008a-8219-4a63-b0fd-6c718235f05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d376-rtvsf" Oct 30 02:19:16.444690 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:16.444643 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9b5008a-8219-4a63-b0fd-6c718235f05e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c8d376-rtvsf\" (UID: \"c9b5008a-8219-4a63-b0fd-6c718235f05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d376-rtvsf" Oct 30 02:19:16.444690 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:16.444671 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtczl\" (UniqueName: \"kubernetes.io/projected/c9b5008a-8219-4a63-b0fd-6c718235f05e-kube-api-access-jtczl\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c8d376-rtvsf\" (UID: \"c9b5008a-8219-4a63-b0fd-6c718235f05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d376-rtvsf" Oct 30 02:19:16.445516 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:16.445484 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9b5008a-8219-4a63-b0fd-6c718235f05e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c8d376-rtvsf\" (UID: \"c9b5008a-8219-4a63-b0fd-6c718235f05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d376-rtvsf" Oct 30 02:19:16.446971 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:16.446951 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9b5008a-8219-4a63-b0fd-6c718235f05e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c8d376-rtvsf\" (UID: \"c9b5008a-8219-4a63-b0fd-6c718235f05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d376-rtvsf" Oct 30 02:19:16.467822 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:16.467793 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtczl\" (UniqueName: \"kubernetes.io/projected/c9b5008a-8219-4a63-b0fd-6c718235f05e-kube-api-access-jtczl\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c8d376-rtvsf\" (UID: \"c9b5008a-8219-4a63-b0fd-6c718235f05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d376-rtvsf" Oct 30 02:19:16.550914 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:16.550884 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d376-rtvsf" Oct 30 02:19:16.858523 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:16.858438 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d376-rtvsf"] Oct 30 02:19:16.861185 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:19:16.861156 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b5008a_8219_4a63_b0fd_6c718235f05e.slice/crio-692c32f8466d16914e12af30c6869d1969423881629dc8ca03188f7ce837db99 WatchSource:0}: Error finding container 692c32f8466d16914e12af30c6869d1969423881629dc8ca03188f7ce837db99: Status 404 returned error can't find the container with id 692c32f8466d16914e12af30c6869d1969423881629dc8ca03188f7ce837db99 Oct 30 02:19:17.266472 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:17.266436 6572 generic.go:356] "Generic (PLEG): container finished" podID="bcbaf9cf-86a4-4eb9-86d7-263c84c928e8" containerID="0ebafb655a0efb262cc447bee4610964442d762cdf98ff1caf33e30ea797da61" exitCode=0 Oct 30 02:19:17.266989 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:17.266581 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-558327-pvkqq" event={"ID":"bcbaf9cf-86a4-4eb9-86d7-263c84c928e8","Type":"ContainerDied","Data":"0ebafb655a0efb262cc447bee4610964442d762cdf98ff1caf33e30ea797da61"} Oct 30 02:19:17.270216 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:17.270178 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d376-rtvsf" event={"ID":"c9b5008a-8219-4a63-b0fd-6c718235f05e","Type":"ContainerStarted","Data":"4c1640258360b985da7c4731f092e7c29cbacf4bf0a25e72c8d9eb326004f34a"} Oct 30 02:19:17.270409 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:17.270249 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d376-rtvsf" event={"ID":"c9b5008a-8219-4a63-b0fd-6c718235f05e","Type":"ContainerStarted","Data":"692c32f8466d16914e12af30c6869d1969423881629dc8ca03188f7ce837db99"} Oct 30 02:19:17.324551 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:17.324516 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:17.344482 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:17.344423 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d376-rtvsf" podStartSLOduration=1.344408433 podStartE2EDuration="1.344408433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:19:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:19:17.343091897 +0000 UTC m=+3682.919517298" watchObservedRunningTime="2026-10-30 02:19:17.344408433 +0000 UTC m=+3682.920833862" Oct 30 02:19:18.279499 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:18.279466 6572 generic.go:356] "Generic (PLEG): container finished" podID="c9b5008a-8219-4a63-b0fd-6c718235f05e" containerID="4c1640258360b985da7c4731f092e7c29cbacf4bf0a25e72c8d9eb326004f34a" exitCode=0 Oct 30 02:19:18.279985 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:18.279542 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d376-rtvsf" event={"ID":"c9b5008a-8219-4a63-b0fd-6c718235f05e","Type":"ContainerDied","Data":"4c1640258360b985da7c4731f092e7c29cbacf4bf0a25e72c8d9eb326004f34a"} Oct 30 02:19:18.451998 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:18.451972 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-558327-pvkqq" Oct 30 02:19:18.569202 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:18.569117 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcbaf9cf-86a4-4eb9-86d7-263c84c928e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcbaf9cf-86a4-4eb9-86d7-263c84c928e8-maas-api-xks-ca\") pod \"bcbaf9cf-86a4-4eb9-86d7-263c84c928e8\" (UID: \"bcbaf9cf-86a4-4eb9-86d7-263c84c928e8\") " Oct 30 02:19:18.569374 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:18.569297 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcbaf9cf-86a4-4eb9-86d7-263c84c928e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcbaf9cf-86a4-4eb9-86d7-263c84c928e8-maas-api-openshift-ca\") pod \"bcbaf9cf-86a4-4eb9-86d7-263c84c928e8\" (UID: \"bcbaf9cf-86a4-4eb9-86d7-263c84c928e8\") " Oct 30 02:19:18.569430 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:18.569393 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcbaf9cf-86a4-4eb9-86d7-263c84c928e8-kube-api-access-scjsk\" (UniqueName: \"kubernetes.io/projected/bcbaf9cf-86a4-4eb9-86d7-263c84c928e8-kube-api-access-scjsk\") pod \"bcbaf9cf-86a4-4eb9-86d7-263c84c928e8\" (UID: \"bcbaf9cf-86a4-4eb9-86d7-263c84c928e8\") " Oct 30 02:19:18.569954 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:18.569908 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcbaf9cf-86a4-4eb9-86d7-263c84c928e8-maas-api-openshift-ca" pod "bcbaf9cf-86a4-4eb9-86d7-263c84c928e8" (UID: "bcbaf9cf-86a4-4eb9-86d7-263c84c928e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:18.570149 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:18.570131 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcbaf9cf-86a4-4eb9-86d7-263c84c928e8-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:18.571504 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:18.571474 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcbaf9cf-86a4-4eb9-86d7-263c84c928e8-maas-api-xks-ca" pod "bcbaf9cf-86a4-4eb9-86d7-263c84c928e8" (UID: "bcbaf9cf-86a4-4eb9-86d7-263c84c928e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:19:18.571627 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:18.571513 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcbaf9cf-86a4-4eb9-86d7-263c84c928e8-kube-api-access-scjsk" pod "bcbaf9cf-86a4-4eb9-86d7-263c84c928e8" (UID: "bcbaf9cf-86a4-4eb9-86d7-263c84c928e8"). InnerVolumeSpecName "kube-api-access-scjsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:18.629739 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:18.629708 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:18.671064 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:18.671027 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scjsk\" (UniqueName: \"kubernetes.io/projected/bcbaf9cf-86a4-4eb9-86d7-263c84c928e8-kube-api-access-scjsk\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:18.671064 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:18.671061 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcbaf9cf-86a4-4eb9-86d7-263c84c928e8-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:19.288664 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:19.288633 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-558327-pvkqq" Oct 30 02:19:19.288664 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:19.288653 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-558327-pvkqq" event={"ID":"bcbaf9cf-86a4-4eb9-86d7-263c84c928e8","Type":"ContainerDied","Data":"447fb8165350f64b2bc2244fe3a72d8988eb67561df02ea491b74263f1f009b2"} Oct 30 02:19:19.289129 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:19.288703 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="447fb8165350f64b2bc2244fe3a72d8988eb67561df02ea491b74263f1f009b2" Oct 30 02:19:19.370788 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:19.370737 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j"] Oct 30 02:19:19.458953 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:19.458926 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d376-rtvsf" Oct 30 02:19:19.490319 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:19.490266 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9b5008a-8219-4a63-b0fd-6c718235f05e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9b5008a-8219-4a63-b0fd-6c718235f05e-maas-api-xks-ca\") pod \"c9b5008a-8219-4a63-b0fd-6c718235f05e\" (UID: \"c9b5008a-8219-4a63-b0fd-6c718235f05e\") " Oct 30 02:19:19.490548 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:19.490464 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9b5008a-8219-4a63-b0fd-6c718235f05e-kube-api-access-jtczl\" (UniqueName: \"kubernetes.io/projected/c9b5008a-8219-4a63-b0fd-6c718235f05e-kube-api-access-jtczl\") pod \"c9b5008a-8219-4a63-b0fd-6c718235f05e\" (UID: \"c9b5008a-8219-4a63-b0fd-6c718235f05e\") " Oct 30 02:19:19.490620 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:19.490554 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9b5008a-8219-4a63-b0fd-6c718235f05e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9b5008a-8219-4a63-b0fd-6c718235f05e-maas-api-openshift-ca\") pod \"c9b5008a-8219-4a63-b0fd-6c718235f05e\" (UID: \"c9b5008a-8219-4a63-b0fd-6c718235f05e\") " Oct 30 02:19:19.491462 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:19.491415 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9b5008a-8219-4a63-b0fd-6c718235f05e-maas-api-openshift-ca" pod "c9b5008a-8219-4a63-b0fd-6c718235f05e" (UID: "c9b5008a-8219-4a63-b0fd-6c718235f05e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:19.499222 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:19.499186 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9b5008a-8219-4a63-b0fd-6c718235f05e-maas-api-xks-ca" pod "c9b5008a-8219-4a63-b0fd-6c718235f05e" (UID: "c9b5008a-8219-4a63-b0fd-6c718235f05e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:19:19.505319 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:19.504943 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9b5008a-8219-4a63-b0fd-6c718235f05e-kube-api-access-jtczl" pod "c9b5008a-8219-4a63-b0fd-6c718235f05e" (UID: "c9b5008a-8219-4a63-b0fd-6c718235f05e"). InnerVolumeSpecName "kube-api-access-jtczl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:19.517810 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:19.517774 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:19.591872 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:19.591781 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9b5008a-8219-4a63-b0fd-6c718235f05e-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:19.591872 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:19.591813 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtczl\" (UniqueName: \"kubernetes.io/projected/c9b5008a-8219-4a63-b0fd-6c718235f05e-kube-api-access-jtczl\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:19.591872 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:19.591823 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9b5008a-8219-4a63-b0fd-6c718235f05e-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:19.605898 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:19.605854 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-340ae6-kjnzn"] Oct 30 02:19:19.606497 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:19.606474 6572 cpu_manager.go:414] "containerMap: removing container" podUID="c9b5008a-8219-4a63-b0fd-6c718235f05e" containerName="revoke-keys" Oct 30 02:19:19.606497 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:19.606497 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9b5008a-8219-4a63-b0fd-6c718235f05e" containerName="revoke-keys" Oct 30 02:19:19.606680 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:19.606550 6572 cpu_manager.go:414] "containerMap: removing container" podUID="bcbaf9cf-86a4-4eb9-86d7-263c84c928e8" containerName="revoke-keys" Oct 30 02:19:19.606680 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:19.606558 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcbaf9cf-86a4-4eb9-86d7-263c84c928e8" containerName="revoke-keys" Oct 30 02:19:19.606838 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:19.606692 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9b5008a-8219-4a63-b0fd-6c718235f05e" containerName="revoke-keys" Oct 30 02:19:19.606838 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:19.606717 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcbaf9cf-86a4-4eb9-86d7-263c84c928e8" containerName="revoke-keys" Oct 30 02:19:19.610323 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:19.610301 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-340ae6-kjnzn" Oct 30 02:19:19.627579 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:19.627550 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-340ae6-kjnzn"] Oct 30 02:19:19.692927 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:19.692892 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52e134be-8aac-4af1-a95d-3cef6eb20a93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-340ae6-kjnzn\" (UID: \"52e134be-8aac-4af1-a95d-3cef6eb20a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-340ae6-kjnzn" Oct 30 02:19:19.693187 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:19.692931 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52e134be-8aac-4af1-a95d-3cef6eb20a93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-340ae6-kjnzn\" (UID: \"52e134be-8aac-4af1-a95d-3cef6eb20a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-340ae6-kjnzn" Oct 30 02:19:19.693187 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:19.693101 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsrbt\" (UniqueName: \"kubernetes.io/projected/52e134be-8aac-4af1-a95d-3cef6eb20a93-kube-api-access-zsrbt\") pod \"maas-api-revoke-keys-e2e-inf-a-340ae6-kjnzn\" (UID: \"52e134be-8aac-4af1-a95d-3cef6eb20a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-340ae6-kjnzn" Oct 30 02:19:19.793905 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:19.793866 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsrbt\" (UniqueName: \"kubernetes.io/projected/52e134be-8aac-4af1-a95d-3cef6eb20a93-kube-api-access-zsrbt\") pod \"maas-api-revoke-keys-e2e-inf-a-340ae6-kjnzn\" (UID: \"52e134be-8aac-4af1-a95d-3cef6eb20a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-340ae6-kjnzn" Oct 30 02:19:19.794100 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:19.793968 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52e134be-8aac-4af1-a95d-3cef6eb20a93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-340ae6-kjnzn\" (UID: \"52e134be-8aac-4af1-a95d-3cef6eb20a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-340ae6-kjnzn" Oct 30 02:19:19.794100 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:19.793994 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52e134be-8aac-4af1-a95d-3cef6eb20a93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-340ae6-kjnzn\" (UID: \"52e134be-8aac-4af1-a95d-3cef6eb20a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-340ae6-kjnzn" Oct 30 02:19:19.794603 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:19.794582 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52e134be-8aac-4af1-a95d-3cef6eb20a93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-340ae6-kjnzn\" (UID: \"52e134be-8aac-4af1-a95d-3cef6eb20a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-340ae6-kjnzn" Oct 30 02:19:19.796420 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:19.796395 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52e134be-8aac-4af1-a95d-3cef6eb20a93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-340ae6-kjnzn\" (UID: \"52e134be-8aac-4af1-a95d-3cef6eb20a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-340ae6-kjnzn" Oct 30 02:19:19.812310 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:19.812287 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsrbt\" (UniqueName: \"kubernetes.io/projected/52e134be-8aac-4af1-a95d-3cef6eb20a93-kube-api-access-zsrbt\") pod \"maas-api-revoke-keys-e2e-inf-a-340ae6-kjnzn\" (UID: \"52e134be-8aac-4af1-a95d-3cef6eb20a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-340ae6-kjnzn" Oct 30 02:19:19.928907 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:19.928873 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-340ae6-kjnzn" Oct 30 02:19:20.078106 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.078047 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6"] Oct 30 02:19:20.086269 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.086197 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv" podUID="dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea" containerName="main" containerID="cri-o://bdb43767b55fc3e3e3f5d543d7899171d470a11ea3bc3c68b55d107ce68b3f6e" gracePeriod=45 Oct 30 02:19:20.117080 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.117050 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:20.234985 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.234952 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-340ae6-kjnzn"] Oct 30 02:19:20.235472 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:19:20.235446 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52e134be_8aac_4af1_a95d_3cef6eb20a93.slice/crio-d33c57d2d3674596a9b868f60407c770a59edcb2fbcd1d388e98dd6be6d8e77d WatchSource:0}: Error finding container d33c57d2d3674596a9b868f60407c770a59edcb2fbcd1d388e98dd6be6d8e77d: Status 404 returned error can't find the container with id d33c57d2d3674596a9b868f60407c770a59edcb2fbcd1d388e98dd6be6d8e77d Oct 30 02:19:20.301435 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.301396 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-340ae6-kjnzn" event={"ID":"52e134be-8aac-4af1-a95d-3cef6eb20a93","Type":"ContainerStarted","Data":"d33c57d2d3674596a9b868f60407c770a59edcb2fbcd1d388e98dd6be6d8e77d"} Oct 30 02:19:20.303509 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.303482 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d376-rtvsf" Oct 30 02:19:20.303632 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.303512 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d376-rtvsf" event={"ID":"c9b5008a-8219-4a63-b0fd-6c718235f05e","Type":"ContainerDied","Data":"692c32f8466d16914e12af30c6869d1969423881629dc8ca03188f7ce837db99"} Oct 30 02:19:20.303632 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.303569 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="692c32f8466d16914e12af30c6869d1969423881629dc8ca03188f7ce837db99" Oct 30 02:19:20.305713 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.305676 6572 generic.go:356] "Generic (PLEG): container finished" podID="dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea" containerID="bdb43767b55fc3e3e3f5d543d7899171d470a11ea3bc3c68b55d107ce68b3f6e" exitCode=0 Oct 30 02:19:20.305856 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.305784 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv" event={"ID":"dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea","Type":"ContainerDied","Data":"bdb43767b55fc3e3e3f5d543d7899171d470a11ea3bc3c68b55d107ce68b3f6e"} Oct 30 02:19:20.317130 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.317096 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:20.348900 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.348879 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv" Oct 30 02:19:20.411294 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.411268 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-tmp-dir\") pod \"dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea\" (UID: \"dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea\") " Oct 30 02:19:20.411449 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.411345 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-dshm\") pod \"dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea\" (UID: \"dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea\") " Oct 30 02:19:20.411449 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.411423 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-home\" (UniqueName: \"kubernetes.io/empty-dir/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-home\") pod \"dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea\" (UID: \"dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea\") " Oct 30 02:19:20.411449 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.411455 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-kube-api-access-c789c\" (UniqueName: \"kubernetes.io/projected/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-kube-api-access-c789c\") pod \"dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea\" (UID: \"dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea\") " Oct 30 02:19:20.411648 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.411518 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-tmp-dir" pod "dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea" (UID: "dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:20.411648 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.411529 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-tls-certs\" (UniqueName: \"kubernetes.io/secret/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-tls-certs\") pod \"dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea\" (UID: \"dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea\") " Oct 30 02:19:20.411648 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.411617 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-model-cache\") pod \"dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea\" (UID: \"dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea\") " Oct 30 02:19:20.411880 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.411730 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-home" pod "dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea" (UID: "dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:20.412001 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.411977 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-model-cache" pod "dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea" (UID: "dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:20.412221 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.412200 6572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-home\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:20.412322 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.412226 6572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-model-cache\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:20.412322 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.412264 6572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-tmp-dir\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:20.413786 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.413758 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-dshm" pod "dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea" (UID: "dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:20.413918 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.413851 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-tls-certs" pod "dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea" (UID: "dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:19:20.414005 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.413973 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-kube-api-access-c789c" pod "dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea" (UID: "dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea"). InnerVolumeSpecName "kube-api-access-c789c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:20.513518 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.513487 6572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-dshm\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:20.513518 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.513515 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c789c\" (UniqueName: \"kubernetes.io/projected/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-kube-api-access-c789c\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:20.513731 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.513527 6572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea-tls-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:20.818703 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.818677 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 02:19:20.917501 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.917468 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-istio-token\" (UniqueName: \"kubernetes.io/projected/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-istio-token\") pod \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\" (UID: \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\") " Oct 30 02:19:20.917501 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.917504 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-kube-api-access-xql4f\" (UniqueName: \"kubernetes.io/projected/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-kube-api-access-xql4f\") pod \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\" (UID: \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\") " Oct 30 02:19:20.917765 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.917711 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-istio-data\") pod \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\" (UID: \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\") " Oct 30 02:19:20.917843 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.917787 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-istio-podinfo\") pod \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\" (UID: \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\") " Oct 30 02:19:20.917909 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.917883 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-workload-socket\") pod \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\" (UID: \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\") " Oct 30 02:19:20.917968 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.917929 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-istiod-ca-cert\") pod \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\" (UID: \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\") " Oct 30 02:19:20.917968 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.917954 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-istio-data" pod "7e60f1dc-1188-4053-8f31-fcd08da0ef1c" (UID: "7e60f1dc-1188-4053-8f31-fcd08da0ef1c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:20.918075 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.918043 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-workload-certs\") pod \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\" (UID: \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\") " Oct 30 02:19:20.918144 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.918087 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-istio-envoy\") pod \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\" (UID: \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\") " Oct 30 02:19:20.918144 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.918126 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-credential-socket\") pod \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\" (UID: \"7e60f1dc-1188-4053-8f31-fcd08da0ef1c\") " Oct 30 02:19:20.918144 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.918134 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-workload-socket" pod "7e60f1dc-1188-4053-8f31-fcd08da0ef1c" (UID: "7e60f1dc-1188-4053-8f31-fcd08da0ef1c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:20.918432 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.918407 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-workload-certs" pod "7e60f1dc-1188-4053-8f31-fcd08da0ef1c" (UID: "7e60f1dc-1188-4053-8f31-fcd08da0ef1c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:20.918587 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.918566 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:20.918820 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.918804 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:20.918901 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.918823 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:20.918901 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.918660 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-credential-socket" pod "7e60f1dc-1188-4053-8f31-fcd08da0ef1c" (UID: "7e60f1dc-1188-4053-8f31-fcd08da0ef1c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:20.918901 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.918737 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-istiod-ca-cert" pod "7e60f1dc-1188-4053-8f31-fcd08da0ef1c" (UID: "7e60f1dc-1188-4053-8f31-fcd08da0ef1c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:20.919901 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.919875 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-istio-podinfo" pod "7e60f1dc-1188-4053-8f31-fcd08da0ef1c" (UID: "7e60f1dc-1188-4053-8f31-fcd08da0ef1c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:19:20.920140 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.920118 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-kube-api-access-xql4f" pod "7e60f1dc-1188-4053-8f31-fcd08da0ef1c" (UID: "7e60f1dc-1188-4053-8f31-fcd08da0ef1c"). InnerVolumeSpecName "kube-api-access-xql4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:20.920205 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.920151 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-istio-token" pod "7e60f1dc-1188-4053-8f31-fcd08da0ef1c" (UID: "7e60f1dc-1188-4053-8f31-fcd08da0ef1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:20.920251 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:20.920228 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-istio-envoy" pod "7e60f1dc-1188-4053-8f31-fcd08da0ef1c" (UID: "7e60f1dc-1188-4053-8f31-fcd08da0ef1c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:21.020480 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:21.020445 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:21.020480 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:21.020477 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:21.020717 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:21.020490 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:21.020717 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:21.020508 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:21.020717 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:21.020521 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xql4f\" (UniqueName: \"kubernetes.io/projected/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-kube-api-access-xql4f\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:21.020717 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:21.020532 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e60f1dc-1188-4053-8f31-fcd08da0ef1c-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:21.316160 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:21.316124 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv" Oct 30 02:19:21.316634 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:21.316150 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv" event={"ID":"dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea","Type":"ContainerDied","Data":"cd0913bfa41d89bb751ca6e423bba3869a6b0e6b66097d8891a71e9ebfe88d14"} Oct 30 02:19:21.316634 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:21.316222 6572 scope.go:122] "RemoveContainer" containerID="bdb43767b55fc3e3e3f5d543d7899171d470a11ea3bc3c68b55d107ce68b3f6e" Oct 30 02:19:21.318428 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:21.318398 6572 generic.go:356] "Generic (PLEG): container finished" podID="7e60f1dc-1188-4053-8f31-fcd08da0ef1c" containerID="801d83f6ed507e7b49a1d63784214ef842b44a7c295e4d9a9ada9963d1a2367e" exitCode=0 Oct 30 02:19:21.318678 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:21.318657 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" Oct 30 02:19:21.318971 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:21.318951 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" event={"ID":"7e60f1dc-1188-4053-8f31-fcd08da0ef1c","Type":"ContainerDied","Data":"801d83f6ed507e7b49a1d63784214ef842b44a7c295e4d9a9ada9963d1a2367e"} Oct 30 02:19:21.319065 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:21.319032 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k" event={"ID":"7e60f1dc-1188-4053-8f31-fcd08da0ef1c","Type":"ContainerDied","Data":"a0c1643fe9ac9146f6494e41f7a53f5c7281bc69ed8f11725457f67e91f20437"} Oct 30 02:19:21.320798 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:21.320739 6572 generic.go:356] "Generic (PLEG): container finished" podID="52e134be-8aac-4af1-a95d-3cef6eb20a93" containerID="d46de24608ddbded499ac1c78a89812d3f2cb26c46052f6399718021f3b491bd" exitCode=0 Oct 30 02:19:21.320904 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:21.320871 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-340ae6-kjnzn" event={"ID":"52e134be-8aac-4af1-a95d-3cef6eb20a93","Type":"ContainerDied","Data":"d46de24608ddbded499ac1c78a89812d3f2cb26c46052f6399718021f3b491bd"} Oct 30 02:19:21.328030 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:21.328003 6572 scope.go:122] "RemoveContainer" containerID="801d83f6ed507e7b49a1d63784214ef842b44a7c295e4d9a9ada9963d1a2367e" Oct 30 02:19:21.339919 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:21.339730 6572 scope.go:122] "RemoveContainer" containerID="63e3d2a4a936608d5d3466dd4111cb9af461b13e9ee684ef2b45ac6229096912" Oct 30 02:19:21.351886 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:21.351867 6572 scope.go:122] "RemoveContainer" containerID="801d83f6ed507e7b49a1d63784214ef842b44a7c295e4d9a9ada9963d1a2367e" Oct 30 02:19:21.352193 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:19:21.352173 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"801d83f6ed507e7b49a1d63784214ef842b44a7c295e4d9a9ada9963d1a2367e\": container with ID starting with 801d83f6ed507e7b49a1d63784214ef842b44a7c295e4d9a9ada9963d1a2367e not found: ID does not exist" containerID="801d83f6ed507e7b49a1d63784214ef842b44a7c295e4d9a9ada9963d1a2367e" Oct 30 02:19:21.352253 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:21.352203 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"801d83f6ed507e7b49a1d63784214ef842b44a7c295e4d9a9ada9963d1a2367e"} err="failed to get container status \"801d83f6ed507e7b49a1d63784214ef842b44a7c295e4d9a9ada9963d1a2367e\": rpc error: code = NotFound desc = could not find container \"801d83f6ed507e7b49a1d63784214ef842b44a7c295e4d9a9ada9963d1a2367e\": container with ID starting with 801d83f6ed507e7b49a1d63784214ef842b44a7c295e4d9a9ada9963d1a2367e not found: ID does not exist" Oct 30 02:19:21.352253 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:21.352223 6572 scope.go:122] "RemoveContainer" containerID="63e3d2a4a936608d5d3466dd4111cb9af461b13e9ee684ef2b45ac6229096912" Oct 30 02:19:21.352475 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:19:21.352457 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63e3d2a4a936608d5d3466dd4111cb9af461b13e9ee684ef2b45ac6229096912\": container with ID starting with 63e3d2a4a936608d5d3466dd4111cb9af461b13e9ee684ef2b45ac6229096912 not found: ID does not exist" containerID="63e3d2a4a936608d5d3466dd4111cb9af461b13e9ee684ef2b45ac6229096912" Oct 30 02:19:21.352535 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:21.352485 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63e3d2a4a936608d5d3466dd4111cb9af461b13e9ee684ef2b45ac6229096912"} err="failed to get container status \"63e3d2a4a936608d5d3466dd4111cb9af461b13e9ee684ef2b45ac6229096912\": rpc error: code = NotFound desc = could not find container \"63e3d2a4a936608d5d3466dd4111cb9af461b13e9ee684ef2b45ac6229096912\": container with ID starting with 63e3d2a4a936608d5d3466dd4111cb9af461b13e9ee684ef2b45ac6229096912 not found: ID does not exist" Oct 30 02:19:21.412825 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:21.412792 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv"] Oct 30 02:19:21.415991 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:21.415965 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-340ae6/test-model-340ae6-kserve-6bc96674b5-z2bcv"] Oct 30 02:19:21.447522 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:21.447491 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k"] Oct 30 02:19:21.453106 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:21.453077 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-396f8e-openshift-default-d58d99f57-zvj7k"] Oct 30 02:19:21.658313 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:21.658273 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:22.480787 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:22.480739 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-340ae6-kjnzn" Oct 30 02:19:22.542331 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:22.542295 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e60f1dc-1188-4053-8f31-fcd08da0ef1c" path="/var/lib/kubelet/pods/7e60f1dc-1188-4053-8f31-fcd08da0ef1c/volumes" Oct 30 02:19:22.542878 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:22.542857 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea" path="/var/lib/kubelet/pods/dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea/volumes" Oct 30 02:19:22.546197 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:22.546169 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52e134be-8aac-4af1-a95d-3cef6eb20a93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52e134be-8aac-4af1-a95d-3cef6eb20a93-maas-api-openshift-ca\") pod \"52e134be-8aac-4af1-a95d-3cef6eb20a93\" (UID: \"52e134be-8aac-4af1-a95d-3cef6eb20a93\") " Oct 30 02:19:22.546306 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:22.546222 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52e134be-8aac-4af1-a95d-3cef6eb20a93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52e134be-8aac-4af1-a95d-3cef6eb20a93-maas-api-xks-ca\") pod \"52e134be-8aac-4af1-a95d-3cef6eb20a93\" (UID: \"52e134be-8aac-4af1-a95d-3cef6eb20a93\") " Oct 30 02:19:22.546427 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:22.546409 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52e134be-8aac-4af1-a95d-3cef6eb20a93-kube-api-access-zsrbt\" (UniqueName: \"kubernetes.io/projected/52e134be-8aac-4af1-a95d-3cef6eb20a93-kube-api-access-zsrbt\") pod \"52e134be-8aac-4af1-a95d-3cef6eb20a93\" (UID: \"52e134be-8aac-4af1-a95d-3cef6eb20a93\") " Oct 30 02:19:22.546585 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:22.546558 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52e134be-8aac-4af1-a95d-3cef6eb20a93-maas-api-openshift-ca" pod "52e134be-8aac-4af1-a95d-3cef6eb20a93" (UID: "52e134be-8aac-4af1-a95d-3cef6eb20a93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:22.546881 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:22.546857 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52e134be-8aac-4af1-a95d-3cef6eb20a93-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:22.548514 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:22.548492 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52e134be-8aac-4af1-a95d-3cef6eb20a93-maas-api-xks-ca" pod "52e134be-8aac-4af1-a95d-3cef6eb20a93" (UID: "52e134be-8aac-4af1-a95d-3cef6eb20a93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:19:22.548913 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:22.548888 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52e134be-8aac-4af1-a95d-3cef6eb20a93-kube-api-access-zsrbt" pod "52e134be-8aac-4af1-a95d-3cef6eb20a93" (UID: "52e134be-8aac-4af1-a95d-3cef6eb20a93"). InnerVolumeSpecName "kube-api-access-zsrbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:22.647966 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:22.647888 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52e134be-8aac-4af1-a95d-3cef6eb20a93-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:22.647966 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:22.647917 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsrbt\" (UniqueName: \"kubernetes.io/projected/52e134be-8aac-4af1-a95d-3cef6eb20a93-kube-api-access-zsrbt\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:23.221805 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:23.221728 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:23.341353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:23.341324 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-340ae6-kjnzn" Oct 30 02:19:23.341505 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:23.341352 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-340ae6-kjnzn" event={"ID":"52e134be-8aac-4af1-a95d-3cef6eb20a93","Type":"ContainerDied","Data":"d33c57d2d3674596a9b868f60407c770a59edcb2fbcd1d388e98dd6be6d8e77d"} Oct 30 02:19:23.341505 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:23.341397 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d33c57d2d3674596a9b868f60407c770a59edcb2fbcd1d388e98dd6be6d8e77d" Oct 30 02:19:24.772307 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:24.772271 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:25.743220 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:25.743188 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-558327-6bc8566f5f-gbqkb"] Oct 30 02:19:25.941216 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:25.941177 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:26.135174 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:26.135071 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:26.499286 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:26.499248 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:26.685764 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:26.685722 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:26.882030 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:26.881940 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:28.081462 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:28.081426 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:29.682197 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:29.682158 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:30.019716 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:30.019684 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-340ae6-58897f9598-pv72n"] Oct 30 02:19:30.253689 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:30.253651 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:30.441586 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:30.441547 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:30.744093 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:30.743989 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-558327-6bc8566f5f-gbqkb" podUID="e07b1c7e-fe5d-4dff-b4aa-6d03ce285792" containerName="payload-pre-processing" containerID="cri-o://1bfa6bea1affb9adc3567789ae6772fa2789b92b5c8dc869859208be9cf3a785" gracePeriod=25 Oct 30 02:19:30.955902 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:30.955864 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:31.308341 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:31.308305 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:31.447088 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:31.447049 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d376-rtvsf"] Oct 30 02:19:31.456559 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:31.456528 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c8d376-rtvsf"] Oct 30 02:19:31.605367 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:31.605279 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:31.826110 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:31.826067 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:31.987180 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:31.987051 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-340ae6-kjnzn"] Oct 30 02:19:32.013529 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:32.013496 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-340ae6-kjnzn"] Oct 30 02:19:32.020987 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:32.020958 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:32.206039 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:32.206004 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:32.539305 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:32.539270 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:32.542762 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:32.542721 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52e134be-8aac-4af1-a95d-3cef6eb20a93" path="/var/lib/kubelet/pods/52e134be-8aac-4af1-a95d-3cef6eb20a93/volumes" Oct 30 02:19:32.543204 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:32.543189 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9b5008a-8219-4a63-b0fd-6c718235f05e" path="/var/lib/kubelet/pods/c9b5008a-8219-4a63-b0fd-6c718235f05e/volumes" Oct 30 02:19:33.677173 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:33.677086 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:34.289846 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:34.289805 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-558327-6bc8566f5f-gbqkb" podUID="e07b1c7e-fe5d-4dff-b4aa-6d03ce285792" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:19:34.393908 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:34.393851 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j" podUID="d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9" containerName="main" containerID="cri-o://f1c5c2b48433dee0ca2668ff481054d06c63273b57d6ed9605c888149da3cd2e" gracePeriod=45 Oct 30 02:19:34.555440 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:34.555340 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j" podUID="d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9" containerName="main" probeResult="failure" output="Get \"https://10.133.1.141:8000/ready\": dial tcp 10.133.1.141:8000: connect: connection refused" Oct 30 02:19:34.672909 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:34.672885 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j" Oct 30 02:19:34.809121 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:34.809029 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-tmp-dir\") pod \"d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9\" (UID: \"d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9\") " Oct 30 02:19:34.809121 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:34.809106 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-home\" (UniqueName: \"kubernetes.io/empty-dir/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-home\") pod \"d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9\" (UID: \"d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9\") " Oct 30 02:19:34.809608 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:34.809161 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-dshm\") pod \"d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9\" (UID: \"d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9\") " Oct 30 02:19:34.809608 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:34.809207 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-model-cache\") pod \"d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9\" (UID: \"d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9\") " Oct 30 02:19:34.809608 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:34.809355 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-tls-certs\" (UniqueName: \"kubernetes.io/secret/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-tls-certs\") pod \"d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9\" (UID: \"d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9\") " Oct 30 02:19:34.809608 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:34.809366 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-tmp-dir" pod "d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9" (UID: "d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:34.809608 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:34.809384 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-home" pod "d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9" (UID: "d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:34.809608 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:34.809399 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-kube-api-access-rn8zr\" (UniqueName: \"kubernetes.io/projected/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-kube-api-access-rn8zr\") pod \"d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9\" (UID: \"d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9\") " Oct 30 02:19:34.809608 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:34.809528 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-model-cache" pod "d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9" (UID: "d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:34.809917 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:34.809884 6572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-home\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:34.809917 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:34.809908 6572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-model-cache\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:34.810010 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:34.809925 6572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-tmp-dir\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:34.811621 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:34.811599 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-dshm" pod "d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9" (UID: "d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:34.811621 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:34.811609 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-tls-certs" pod "d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9" (UID: "d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:19:34.811787 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:34.811637 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-kube-api-access-rn8zr" pod "d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9" (UID: "d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9"). InnerVolumeSpecName "kube-api-access-rn8zr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:34.910729 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:34.910683 6572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-dshm\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:34.910944 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:34.910738 6572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-tls-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:34.910944 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:34.910779 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn8zr\" (UniqueName: \"kubernetes.io/projected/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9-kube-api-access-rn8zr\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:35.020174 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.020106 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-340ae6-58897f9598-pv72n" podUID="90887a7d-bb7c-4c14-9e63-0866a137918b" containerName="payload-processing" containerID="cri-o://7140f05e250959aca84f6f44e72dced7ca12dedc0ab92f8a0d336adb8aa18e6d" gracePeriod=25 Oct 30 02:19:35.104960 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.104877 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" podUID="317505ef-238c-458e-a74e-1198806b94d2" containerName="main" containerID="cri-o://9a962bf0d349c460708f4489ecc07c1a6d5c6365fc749b0b5328ca390ace6af7" gracePeriod=45 Oct 30 02:19:35.176500 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.176456 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:35.349849 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.349798 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-340ae6-58897f9598-pv72n" Oct 30 02:19:35.394457 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.394427 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" Oct 30 02:19:35.416294 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.416258 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90887a7d-bb7c-4c14-9e63-0866a137918b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90887a7d-bb7c-4c14-9e63-0866a137918b-plugins-config-volume\") pod \"90887a7d-bb7c-4c14-9e63-0866a137918b\" (UID: \"90887a7d-bb7c-4c14-9e63-0866a137918b\") " Oct 30 02:19:35.416489 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.416373 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90887a7d-bb7c-4c14-9e63-0866a137918b-kube-api-access-wxz79\" (UniqueName: \"kubernetes.io/projected/90887a7d-bb7c-4c14-9e63-0866a137918b-kube-api-access-wxz79\") pod \"90887a7d-bb7c-4c14-9e63-0866a137918b\" (UID: \"90887a7d-bb7c-4c14-9e63-0866a137918b\") " Oct 30 02:19:35.416725 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.416692 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90887a7d-bb7c-4c14-9e63-0866a137918b-plugins-config-volume" pod "90887a7d-bb7c-4c14-9e63-0866a137918b" (UID: "90887a7d-bb7c-4c14-9e63-0866a137918b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:35.417097 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.417040 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90887a7d-bb7c-4c14-9e63-0866a137918b-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:35.420572 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.420538 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90887a7d-bb7c-4c14-9e63-0866a137918b-kube-api-access-wxz79" pod "90887a7d-bb7c-4c14-9e63-0866a137918b" (UID: "90887a7d-bb7c-4c14-9e63-0866a137918b"). InnerVolumeSpecName "kube-api-access-wxz79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:35.454737 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.454701 6572 generic.go:356] "Generic (PLEG): container finished" podID="d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9" containerID="f1c5c2b48433dee0ca2668ff481054d06c63273b57d6ed9605c888149da3cd2e" exitCode=0 Oct 30 02:19:35.454933 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.454802 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j" Oct 30 02:19:35.454933 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.454806 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j" event={"ID":"d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9","Type":"ContainerDied","Data":"f1c5c2b48433dee0ca2668ff481054d06c63273b57d6ed9605c888149da3cd2e"} Oct 30 02:19:35.454933 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.454851 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j" event={"ID":"d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9","Type":"ContainerDied","Data":"ef0f8782ee3cd5e27b71d68af888ce952feca689d9ab1ea760a26bf13f9c6375"} Oct 30 02:19:35.454933 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.454874 6572 scope.go:122] "RemoveContainer" containerID="f1c5c2b48433dee0ca2668ff481054d06c63273b57d6ed9605c888149da3cd2e" Oct 30 02:19:35.456586 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.456564 6572 generic.go:356] "Generic (PLEG): container finished" podID="90887a7d-bb7c-4c14-9e63-0866a137918b" containerID="7140f05e250959aca84f6f44e72dced7ca12dedc0ab92f8a0d336adb8aa18e6d" exitCode=0 Oct 30 02:19:35.456902 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.456878 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-340ae6-58897f9598-pv72n" event={"ID":"90887a7d-bb7c-4c14-9e63-0866a137918b","Type":"ContainerDied","Data":"7140f05e250959aca84f6f44e72dced7ca12dedc0ab92f8a0d336adb8aa18e6d"} Oct 30 02:19:35.457016 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.456980 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-340ae6-58897f9598-pv72n" event={"ID":"90887a7d-bb7c-4c14-9e63-0866a137918b","Type":"ContainerDied","Data":"48b02ba3e409cb71716fab213b45327a608fdbc8743f4d5e1a7e9d5698c1f5e4"} Oct 30 02:19:35.457082 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.457051 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-340ae6-58897f9598-pv72n" Oct 30 02:19:35.468595 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.468563 6572 generic.go:356] "Generic (PLEG): container finished" podID="317505ef-238c-458e-a74e-1198806b94d2" containerID="9a962bf0d349c460708f4489ecc07c1a6d5c6365fc749b0b5328ca390ace6af7" exitCode=0 Oct 30 02:19:35.468717 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.468631 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" Oct 30 02:19:35.468717 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.468653 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" event={"ID":"317505ef-238c-458e-a74e-1198806b94d2","Type":"ContainerDied","Data":"9a962bf0d349c460708f4489ecc07c1a6d5c6365fc749b0b5328ca390ace6af7"} Oct 30 02:19:35.468717 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.468687 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6" event={"ID":"317505ef-238c-458e-a74e-1198806b94d2","Type":"ContainerDied","Data":"fe02478f2c89ab73c0d020cd3e732269d00de83c7fc4249920d4a97ae568dd42"} Oct 30 02:19:35.470406 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.470387 6572 scope.go:122] "RemoveContainer" containerID="f1c5c2b48433dee0ca2668ff481054d06c63273b57d6ed9605c888149da3cd2e" Oct 30 02:19:35.470726 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:19:35.470704 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1c5c2b48433dee0ca2668ff481054d06c63273b57d6ed9605c888149da3cd2e\": container with ID starting with f1c5c2b48433dee0ca2668ff481054d06c63273b57d6ed9605c888149da3cd2e not found: ID does not exist" containerID="f1c5c2b48433dee0ca2668ff481054d06c63273b57d6ed9605c888149da3cd2e" Oct 30 02:19:35.470876 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.470735 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1c5c2b48433dee0ca2668ff481054d06c63273b57d6ed9605c888149da3cd2e"} err="failed to get container status \"f1c5c2b48433dee0ca2668ff481054d06c63273b57d6ed9605c888149da3cd2e\": rpc error: code = NotFound desc = could not find container \"f1c5c2b48433dee0ca2668ff481054d06c63273b57d6ed9605c888149da3cd2e\": container with ID starting with f1c5c2b48433dee0ca2668ff481054d06c63273b57d6ed9605c888149da3cd2e not found: ID does not exist" Oct 30 02:19:35.470876 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.470770 6572 scope.go:122] "RemoveContainer" containerID="7140f05e250959aca84f6f44e72dced7ca12dedc0ab92f8a0d336adb8aa18e6d" Oct 30 02:19:35.483527 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.483506 6572 scope.go:122] "RemoveContainer" containerID="7140f05e250959aca84f6f44e72dced7ca12dedc0ab92f8a0d336adb8aa18e6d" Oct 30 02:19:35.483868 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:19:35.483844 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7140f05e250959aca84f6f44e72dced7ca12dedc0ab92f8a0d336adb8aa18e6d\": container with ID starting with 7140f05e250959aca84f6f44e72dced7ca12dedc0ab92f8a0d336adb8aa18e6d not found: ID does not exist" containerID="7140f05e250959aca84f6f44e72dced7ca12dedc0ab92f8a0d336adb8aa18e6d" Oct 30 02:19:35.483948 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.483873 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7140f05e250959aca84f6f44e72dced7ca12dedc0ab92f8a0d336adb8aa18e6d"} err="failed to get container status \"7140f05e250959aca84f6f44e72dced7ca12dedc0ab92f8a0d336adb8aa18e6d\": rpc error: code = NotFound desc = could not find container \"7140f05e250959aca84f6f44e72dced7ca12dedc0ab92f8a0d336adb8aa18e6d\": container with ID starting with 7140f05e250959aca84f6f44e72dced7ca12dedc0ab92f8a0d336adb8aa18e6d not found: ID does not exist" Oct 30 02:19:35.483948 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.483890 6572 scope.go:122] "RemoveContainer" containerID="9a962bf0d349c460708f4489ecc07c1a6d5c6365fc749b0b5328ca390ace6af7" Oct 30 02:19:35.496417 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.496397 6572 scope.go:122] "RemoveContainer" containerID="9a962bf0d349c460708f4489ecc07c1a6d5c6365fc749b0b5328ca390ace6af7" Oct 30 02:19:35.496701 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:19:35.496681 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a962bf0d349c460708f4489ecc07c1a6d5c6365fc749b0b5328ca390ace6af7\": container with ID starting with 9a962bf0d349c460708f4489ecc07c1a6d5c6365fc749b0b5328ca390ace6af7 not found: ID does not exist" containerID="9a962bf0d349c460708f4489ecc07c1a6d5c6365fc749b0b5328ca390ace6af7" Oct 30 02:19:35.496850 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.496706 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a962bf0d349c460708f4489ecc07c1a6d5c6365fc749b0b5328ca390ace6af7"} err="failed to get container status \"9a962bf0d349c460708f4489ecc07c1a6d5c6365fc749b0b5328ca390ace6af7\": rpc error: code = NotFound desc = could not find container \"9a962bf0d349c460708f4489ecc07c1a6d5c6365fc749b0b5328ca390ace6af7\": container with ID starting with 9a962bf0d349c460708f4489ecc07c1a6d5c6365fc749b0b5328ca390ace6af7 not found: ID does not exist" Oct 30 02:19:35.504914 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.504887 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j"] Oct 30 02:19:35.508109 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.508086 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-2d5aa9/test-model-2d5aa9-kserve-648bdb9d6c-9nf5j"] Oct 30 02:19:35.517578 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.517557 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/317505ef-238c-458e-a74e-1198806b94d2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/317505ef-238c-458e-a74e-1198806b94d2-tmp-dir\") pod \"317505ef-238c-458e-a74e-1198806b94d2\" (UID: \"317505ef-238c-458e-a74e-1198806b94d2\") " Oct 30 02:19:35.517654 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.517609 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/317505ef-238c-458e-a74e-1198806b94d2-tls-certs\" (UniqueName: \"kubernetes.io/secret/317505ef-238c-458e-a74e-1198806b94d2-tls-certs\") pod \"317505ef-238c-458e-a74e-1198806b94d2\" (UID: \"317505ef-238c-458e-a74e-1198806b94d2\") " Oct 30 02:19:35.517654 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.517643 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/317505ef-238c-458e-a74e-1198806b94d2-kube-api-access-kj8b9\" (UniqueName: \"kubernetes.io/projected/317505ef-238c-458e-a74e-1198806b94d2-kube-api-access-kj8b9\") pod \"317505ef-238c-458e-a74e-1198806b94d2\" (UID: \"317505ef-238c-458e-a74e-1198806b94d2\") " Oct 30 02:19:35.517743 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.517673 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/317505ef-238c-458e-a74e-1198806b94d2-home\" (UniqueName: \"kubernetes.io/empty-dir/317505ef-238c-458e-a74e-1198806b94d2-home\") pod \"317505ef-238c-458e-a74e-1198806b94d2\" (UID: \"317505ef-238c-458e-a74e-1198806b94d2\") " Oct 30 02:19:35.517743 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.517692 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/317505ef-238c-458e-a74e-1198806b94d2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/317505ef-238c-458e-a74e-1198806b94d2-dshm\") pod \"317505ef-238c-458e-a74e-1198806b94d2\" (UID: \"317505ef-238c-458e-a74e-1198806b94d2\") " Oct 30 02:19:35.517900 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.517817 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/317505ef-238c-458e-a74e-1198806b94d2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/317505ef-238c-458e-a74e-1198806b94d2-model-cache\") pod \"317505ef-238c-458e-a74e-1198806b94d2\" (UID: \"317505ef-238c-458e-a74e-1198806b94d2\") " Oct 30 02:19:35.517956 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.517933 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/317505ef-238c-458e-a74e-1198806b94d2-tmp-dir" pod "317505ef-238c-458e-a74e-1198806b94d2" (UID: "317505ef-238c-458e-a74e-1198806b94d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:35.518397 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.518065 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/317505ef-238c-458e-a74e-1198806b94d2-home" pod "317505ef-238c-458e-a74e-1198806b94d2" (UID: "317505ef-238c-458e-a74e-1198806b94d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:35.518397 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.518321 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/317505ef-238c-458e-a74e-1198806b94d2-model-cache" pod "317505ef-238c-458e-a74e-1198806b94d2" (UID: "317505ef-238c-458e-a74e-1198806b94d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:35.518397 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.518339 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxz79\" (UniqueName: \"kubernetes.io/projected/90887a7d-bb7c-4c14-9e63-0866a137918b-kube-api-access-wxz79\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:35.518397 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.518363 6572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/317505ef-238c-458e-a74e-1198806b94d2-tmp-dir\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:35.518397 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.518378 6572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/317505ef-238c-458e-a74e-1198806b94d2-home\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:35.520131 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.520097 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/317505ef-238c-458e-a74e-1198806b94d2-tls-certs" pod "317505ef-238c-458e-a74e-1198806b94d2" (UID: "317505ef-238c-458e-a74e-1198806b94d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:19:35.520339 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.520296 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/317505ef-238c-458e-a74e-1198806b94d2-dshm" pod "317505ef-238c-458e-a74e-1198806b94d2" (UID: "317505ef-238c-458e-a74e-1198806b94d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:35.520480 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.520348 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/317505ef-238c-458e-a74e-1198806b94d2-kube-api-access-kj8b9" pod "317505ef-238c-458e-a74e-1198806b94d2" (UID: "317505ef-238c-458e-a74e-1198806b94d2"). InnerVolumeSpecName "kube-api-access-kj8b9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:35.570899 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.570868 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-340ae6-58897f9598-pv72n"] Oct 30 02:19:35.573617 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.573592 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-340ae6-58897f9598-pv72n"] Oct 30 02:19:35.619937 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.619907 6572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/317505ef-238c-458e-a74e-1198806b94d2-tls-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:35.619937 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.619934 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj8b9\" (UniqueName: \"kubernetes.io/projected/317505ef-238c-458e-a74e-1198806b94d2-kube-api-access-kj8b9\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:35.620124 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.619947 6572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/317505ef-238c-458e-a74e-1198806b94d2-dshm\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:35.620124 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.619958 6572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/317505ef-238c-458e-a74e-1198806b94d2-model-cache\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:35.813502 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.813466 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6"] Oct 30 02:19:35.816273 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:35.816249 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-558327/premium-w3-558327-kserve-5dc6b96f5-z9wx6"] Oct 30 02:19:36.541686 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:36.541657 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="317505ef-238c-458e-a74e-1198806b94d2" path="/var/lib/kubelet/pods/317505ef-238c-458e-a74e-1198806b94d2/volumes" Oct 30 02:19:36.542080 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:36.542066 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90887a7d-bb7c-4c14-9e63-0866a137918b" path="/var/lib/kubelet/pods/90887a7d-bb7c-4c14-9e63-0866a137918b/volumes" Oct 30 02:19:36.542447 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:36.542435 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9" path="/var/lib/kubelet/pods/d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9/volumes" Oct 30 02:19:36.785492 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:36.785444 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:38.076555 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:38.076516 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:38.305576 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:38.300788 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq"] Oct 30 02:19:38.305576 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:38.300843 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:38.305576 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:38.301026 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" podUID="322cef03-4f70-42c8-8571-42c939fba68b" containerName="istio-proxy" containerID="cri-o://a47a8d4c2caa136ea3ab6db923cb02b3b0bc5dee668d52aae61b5d4773314b0f" gracePeriod=30 Oct 30 02:19:38.506009 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:38.505813 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:38.893632 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:38.893452 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:39.021781 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.021720 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a3a2ac-7kn8z"] Oct 30 02:19:39.022475 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.022428 6572 cpu_manager.go:414] "containerMap: removing container" podUID="317505ef-238c-458e-a74e-1198806b94d2" containerName="main" Oct 30 02:19:39.022475 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.022473 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="317505ef-238c-458e-a74e-1198806b94d2" containerName="main" Oct 30 02:19:39.022711 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.022493 6572 cpu_manager.go:414] "containerMap: removing container" podUID="7e60f1dc-1188-4053-8f31-fcd08da0ef1c" containerName="istio-proxy" Oct 30 02:19:39.022711 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.022501 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e60f1dc-1188-4053-8f31-fcd08da0ef1c" containerName="istio-proxy" Oct 30 02:19:39.022711 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.022518 6572 cpu_manager.go:414] "containerMap: removing container" podUID="dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea" containerName="main" Oct 30 02:19:39.022711 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.022527 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea" containerName="main" Oct 30 02:19:39.022711 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.022544 6572 cpu_manager.go:414] "containerMap: removing container" podUID="90887a7d-bb7c-4c14-9e63-0866a137918b" containerName="payload-processing" Oct 30 02:19:39.022711 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.022549 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90887a7d-bb7c-4c14-9e63-0866a137918b" containerName="payload-processing" Oct 30 02:19:39.022711 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.022557 6572 cpu_manager.go:414] "containerMap: removing container" podUID="52e134be-8aac-4af1-a95d-3cef6eb20a93" containerName="revoke-keys" Oct 30 02:19:39.022711 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.022562 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52e134be-8aac-4af1-a95d-3cef6eb20a93" containerName="revoke-keys" Oct 30 02:19:39.022711 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.022579 6572 cpu_manager.go:414] "containerMap: removing container" podUID="d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9" containerName="main" Oct 30 02:19:39.022711 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.022587 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9" containerName="main" Oct 30 02:19:39.022711 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.022675 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1ab3c1f-ad36-4a68-b0d0-66e95e88ecb9" containerName="main" Oct 30 02:19:39.022711 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.022687 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e60f1dc-1188-4053-8f31-fcd08da0ef1c" containerName="istio-proxy" Oct 30 02:19:39.022711 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.022695 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="317505ef-238c-458e-a74e-1198806b94d2" containerName="main" Oct 30 02:19:39.022711 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.022704 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e60f1dc-1188-4053-8f31-fcd08da0ef1c" containerName="istio-proxy" Oct 30 02:19:39.022711 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.022714 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="dac0f19b-d0bd-45d7-9e36-a6e0ddf7e7ea" containerName="main" Oct 30 02:19:39.023248 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.022726 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="90887a7d-bb7c-4c14-9e63-0866a137918b" containerName="payload-processing" Oct 30 02:19:39.023248 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.022733 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="52e134be-8aac-4af1-a95d-3cef6eb20a93" containerName="revoke-keys" Oct 30 02:19:39.029293 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.029260 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a3a2ac-7kn8z" Oct 30 02:19:39.045543 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.045513 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a3a2ac-7kn8z"] Oct 30 02:19:39.045703 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.045691 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:19:39.045837 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.045821 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:19:39.058583 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.058557 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:19:39.058778 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.058694 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:19:39.082039 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.082006 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:39.164583 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.164539 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e9ec3c-fe81-4e11-906d-6a2dd69d0cee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a3a2ac-7kn8z\" (UID: \"94e9ec3c-fe81-4e11-906d-6a2dd69d0cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a3a2ac-7kn8z" Oct 30 02:19:39.164799 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.164776 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e9ec3c-fe81-4e11-906d-6a2dd69d0cee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a3a2ac-7kn8z\" (UID: \"94e9ec3c-fe81-4e11-906d-6a2dd69d0cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a3a2ac-7kn8z" Oct 30 02:19:39.164871 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.164811 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcjc4\" (UniqueName: \"kubernetes.io/projected/94e9ec3c-fe81-4e11-906d-6a2dd69d0cee-kube-api-access-hcjc4\") pod \"maas-api-revoke-keys-e2e-worker-w0-a3a2ac-7kn8z\" (UID: \"94e9ec3c-fe81-4e11-906d-6a2dd69d0cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a3a2ac-7kn8z" Oct 30 02:19:39.258006 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.257966 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw"] Oct 30 02:19:39.258266 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.258230 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" podUID="d32bbe46-7de7-4307-a62d-ee16d8b59741" containerName="istio-proxy" containerID="cri-o://f2f7c87a7516025f1fe86fe1725973ee44ea731d37473fe7c122fea190ce1cff" gracePeriod=30 Oct 30 02:19:39.266253 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.266209 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e9ec3c-fe81-4e11-906d-6a2dd69d0cee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a3a2ac-7kn8z\" (UID: \"94e9ec3c-fe81-4e11-906d-6a2dd69d0cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a3a2ac-7kn8z" Oct 30 02:19:39.266672 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.266392 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e9ec3c-fe81-4e11-906d-6a2dd69d0cee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a3a2ac-7kn8z\" (UID: \"94e9ec3c-fe81-4e11-906d-6a2dd69d0cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a3a2ac-7kn8z" Oct 30 02:19:39.266672 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.266429 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcjc4\" (UniqueName: \"kubernetes.io/projected/94e9ec3c-fe81-4e11-906d-6a2dd69d0cee-kube-api-access-hcjc4\") pod \"maas-api-revoke-keys-e2e-worker-w0-a3a2ac-7kn8z\" (UID: \"94e9ec3c-fe81-4e11-906d-6a2dd69d0cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a3a2ac-7kn8z" Oct 30 02:19:39.267155 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.267131 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e9ec3c-fe81-4e11-906d-6a2dd69d0cee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a3a2ac-7kn8z\" (UID: \"94e9ec3c-fe81-4e11-906d-6a2dd69d0cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a3a2ac-7kn8z" Oct 30 02:19:39.269412 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.269389 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e9ec3c-fe81-4e11-906d-6a2dd69d0cee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a3a2ac-7kn8z\" (UID: \"94e9ec3c-fe81-4e11-906d-6a2dd69d0cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a3a2ac-7kn8z" Oct 30 02:19:39.286522 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.286496 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcjc4\" (UniqueName: \"kubernetes.io/projected/94e9ec3c-fe81-4e11-906d-6a2dd69d0cee-kube-api-access-hcjc4\") pod \"maas-api-revoke-keys-e2e-worker-w0-a3a2ac-7kn8z\" (UID: \"94e9ec3c-fe81-4e11-906d-6a2dd69d0cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a3a2ac-7kn8z" Oct 30 02:19:39.348301 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.348264 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a3a2ac-7kn8z" Oct 30 02:19:39.388788 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.388191 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:39.608720 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.608678 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-543807da-gxx84"] Oct 30 02:19:39.609346 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.609323 6572 cpu_manager.go:414] "containerMap: removing container" podUID="7e60f1dc-1188-4053-8f31-fcd08da0ef1c" containerName="istio-proxy" Oct 30 02:19:39.609346 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.609346 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e60f1dc-1188-4053-8f31-fcd08da0ef1c" containerName="istio-proxy" Oct 30 02:19:39.613181 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.613160 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-543807da-gxx84" Oct 30 02:19:39.628481 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.628449 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4abfaf4e-nvxw8"] Oct 30 02:19:39.634262 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.634241 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-543807da-gxx84"] Oct 30 02:19:39.634360 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.634345 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abfaf4e-nvxw8" Oct 30 02:19:39.652876 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.652850 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4abfaf4e-nvxw8"] Oct 30 02:19:39.673854 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.673826 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a3a2ac-7kn8z"] Oct 30 02:19:39.676727 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:19:39.676687 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94e9ec3c_fe81_4e11_906d_6a2dd69d0cee.slice/crio-ad5606a699883553380ceace6bbe685d374f69f8df8d383ee398dd13bb146a70 WatchSource:0}: Error finding container ad5606a699883553380ceace6bbe685d374f69f8df8d383ee398dd13bb146a70: Status 404 returned error can't find the container with id ad5606a699883553380ceace6bbe685d374f69f8df8d383ee398dd13bb146a70 Oct 30 02:19:39.678050 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.678019 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e203d4b8-3f30-4d07-9c6f-431af65d9ec2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-543807da-gxx84\" (UID: \"e203d4b8-3f30-4d07-9c6f-431af65d9ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-543807da-gxx84" Oct 30 02:19:39.678164 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.678102 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e203d4b8-3f30-4d07-9c6f-431af65d9ec2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-543807da-gxx84\" (UID: \"e203d4b8-3f30-4d07-9c6f-431af65d9ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-543807da-gxx84" Oct 30 02:19:39.678244 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.678163 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shdn9\" (UniqueName: \"kubernetes.io/projected/e203d4b8-3f30-4d07-9c6f-431af65d9ec2-kube-api-access-shdn9\") pod \"maas-api-revoke-sub-543807da-gxx84\" (UID: \"e203d4b8-3f30-4d07-9c6f-431af65d9ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-543807da-gxx84" Oct 30 02:19:39.779698 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.779663 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kthbr\" (UniqueName: \"kubernetes.io/projected/ed6145a7-d13c-4efa-ad6d-eaa6c5d90974-kube-api-access-kthbr\") pod \"maas-api-revoke-sub-4abfaf4e-nvxw8\" (UID: \"ed6145a7-d13c-4efa-ad6d-eaa6c5d90974\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abfaf4e-nvxw8" Oct 30 02:19:39.779698 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.779711 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed6145a7-d13c-4efa-ad6d-eaa6c5d90974-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4abfaf4e-nvxw8\" (UID: \"ed6145a7-d13c-4efa-ad6d-eaa6c5d90974\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abfaf4e-nvxw8" Oct 30 02:19:39.779974 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.779885 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e203d4b8-3f30-4d07-9c6f-431af65d9ec2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-543807da-gxx84\" (UID: \"e203d4b8-3f30-4d07-9c6f-431af65d9ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-543807da-gxx84" Oct 30 02:19:39.779974 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.779916 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e203d4b8-3f30-4d07-9c6f-431af65d9ec2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-543807da-gxx84\" (UID: \"e203d4b8-3f30-4d07-9c6f-431af65d9ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-543807da-gxx84" Oct 30 02:19:39.779974 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.779937 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shdn9\" (UniqueName: \"kubernetes.io/projected/e203d4b8-3f30-4d07-9c6f-431af65d9ec2-kube-api-access-shdn9\") pod \"maas-api-revoke-sub-543807da-gxx84\" (UID: \"e203d4b8-3f30-4d07-9c6f-431af65d9ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-543807da-gxx84" Oct 30 02:19:39.779974 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.779967 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed6145a7-d13c-4efa-ad6d-eaa6c5d90974-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4abfaf4e-nvxw8\" (UID: \"ed6145a7-d13c-4efa-ad6d-eaa6c5d90974\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abfaf4e-nvxw8" Oct 30 02:19:39.780656 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.780636 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e203d4b8-3f30-4d07-9c6f-431af65d9ec2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-543807da-gxx84\" (UID: \"e203d4b8-3f30-4d07-9c6f-431af65d9ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-543807da-gxx84" Oct 30 02:19:39.782350 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.782326 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e203d4b8-3f30-4d07-9c6f-431af65d9ec2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-543807da-gxx84\" (UID: \"e203d4b8-3f30-4d07-9c6f-431af65d9ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-543807da-gxx84" Oct 30 02:19:39.798374 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.798342 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shdn9\" (UniqueName: \"kubernetes.io/projected/e203d4b8-3f30-4d07-9c6f-431af65d9ec2-kube-api-access-shdn9\") pod \"maas-api-revoke-sub-543807da-gxx84\" (UID: \"e203d4b8-3f30-4d07-9c6f-431af65d9ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-543807da-gxx84" Oct 30 02:19:39.843785 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.843725 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d5aa9-l87rv"] Oct 30 02:19:39.848983 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.848963 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d5aa9-l87rv" Oct 30 02:19:39.866859 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.866830 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d5aa9-l87rv"] Oct 30 02:19:39.881693 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.881656 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kthbr\" (UniqueName: \"kubernetes.io/projected/ed6145a7-d13c-4efa-ad6d-eaa6c5d90974-kube-api-access-kthbr\") pod \"maas-api-revoke-sub-4abfaf4e-nvxw8\" (UID: \"ed6145a7-d13c-4efa-ad6d-eaa6c5d90974\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abfaf4e-nvxw8" Oct 30 02:19:39.881903 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.881720 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed6145a7-d13c-4efa-ad6d-eaa6c5d90974-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4abfaf4e-nvxw8\" (UID: \"ed6145a7-d13c-4efa-ad6d-eaa6c5d90974\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abfaf4e-nvxw8" Oct 30 02:19:39.881903 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.881864 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed6145a7-d13c-4efa-ad6d-eaa6c5d90974-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4abfaf4e-nvxw8\" (UID: \"ed6145a7-d13c-4efa-ad6d-eaa6c5d90974\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abfaf4e-nvxw8" Oct 30 02:19:39.882470 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.882446 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed6145a7-d13c-4efa-ad6d-eaa6c5d90974-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4abfaf4e-nvxw8\" (UID: \"ed6145a7-d13c-4efa-ad6d-eaa6c5d90974\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abfaf4e-nvxw8" Oct 30 02:19:39.884314 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.884293 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed6145a7-d13c-4efa-ad6d-eaa6c5d90974-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4abfaf4e-nvxw8\" (UID: \"ed6145a7-d13c-4efa-ad6d-eaa6c5d90974\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abfaf4e-nvxw8" Oct 30 02:19:39.900363 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.900335 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kthbr\" (UniqueName: \"kubernetes.io/projected/ed6145a7-d13c-4efa-ad6d-eaa6c5d90974-kube-api-access-kthbr\") pod \"maas-api-revoke-sub-4abfaf4e-nvxw8\" (UID: \"ed6145a7-d13c-4efa-ad6d-eaa6c5d90974\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abfaf4e-nvxw8" Oct 30 02:19:39.927734 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.927700 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-543807da-gxx84" Oct 30 02:19:39.950018 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.949986 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abfaf4e-nvxw8" Oct 30 02:19:39.983008 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.982968 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a8f46a4-ea68-466e-94ed-db3d9b5872de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2d5aa9-l87rv\" (UID: \"9a8f46a4-ea68-466e-94ed-db3d9b5872de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d5aa9-l87rv" Oct 30 02:19:39.983355 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.983258 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tt6v\" (UniqueName: \"kubernetes.io/projected/9a8f46a4-ea68-466e-94ed-db3d9b5872de-kube-api-access-4tt6v\") pod \"maas-api-revoke-keys-e2e-inf-b-2d5aa9-l87rv\" (UID: \"9a8f46a4-ea68-466e-94ed-db3d9b5872de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d5aa9-l87rv" Oct 30 02:19:39.983355 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:39.983325 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a8f46a4-ea68-466e-94ed-db3d9b5872de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2d5aa9-l87rv\" (UID: \"9a8f46a4-ea68-466e-94ed-db3d9b5872de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d5aa9-l87rv" Oct 30 02:19:40.085111 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:40.085054 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tt6v\" (UniqueName: \"kubernetes.io/projected/9a8f46a4-ea68-466e-94ed-db3d9b5872de-kube-api-access-4tt6v\") pod \"maas-api-revoke-keys-e2e-inf-b-2d5aa9-l87rv\" (UID: \"9a8f46a4-ea68-466e-94ed-db3d9b5872de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d5aa9-l87rv" Oct 30 02:19:40.086231 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:40.085130 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a8f46a4-ea68-466e-94ed-db3d9b5872de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2d5aa9-l87rv\" (UID: \"9a8f46a4-ea68-466e-94ed-db3d9b5872de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d5aa9-l87rv" Oct 30 02:19:40.086231 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:40.085211 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a8f46a4-ea68-466e-94ed-db3d9b5872de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2d5aa9-l87rv\" (UID: \"9a8f46a4-ea68-466e-94ed-db3d9b5872de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d5aa9-l87rv" Oct 30 02:19:40.086507 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:40.086450 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a8f46a4-ea68-466e-94ed-db3d9b5872de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2d5aa9-l87rv\" (UID: \"9a8f46a4-ea68-466e-94ed-db3d9b5872de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d5aa9-l87rv" Oct 30 02:19:40.088163 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:40.088140 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a8f46a4-ea68-466e-94ed-db3d9b5872de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2d5aa9-l87rv\" (UID: \"9a8f46a4-ea68-466e-94ed-db3d9b5872de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d5aa9-l87rv" Oct 30 02:19:40.104741 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:40.104697 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tt6v\" (UniqueName: \"kubernetes.io/projected/9a8f46a4-ea68-466e-94ed-db3d9b5872de-kube-api-access-4tt6v\") pod \"maas-api-revoke-keys-e2e-inf-b-2d5aa9-l87rv\" (UID: \"9a8f46a4-ea68-466e-94ed-db3d9b5872de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d5aa9-l87rv" Oct 30 02:19:40.162907 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:40.162869 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d5aa9-l87rv" Oct 30 02:19:40.244472 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:40.244438 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-543807da-gxx84"] Oct 30 02:19:40.251353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:40.251162 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:40.272704 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:40.272671 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4abfaf4e-nvxw8"] Oct 30 02:19:40.276567 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:19:40.276539 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded6145a7_d13c_4efa_ad6d_eaa6c5d90974.slice/crio-159ba61e32e5df7cf6167afe5f54a0df9f0ef90e0bc2f3ec37600f24d54bc194 WatchSource:0}: Error finding container 159ba61e32e5df7cf6167afe5f54a0df9f0ef90e0bc2f3ec37600f24d54bc194: Status 404 returned error can't find the container with id 159ba61e32e5df7cf6167afe5f54a0df9f0ef90e0bc2f3ec37600f24d54bc194 Oct 30 02:19:40.494611 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:40.494583 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d5aa9-l87rv"] Oct 30 02:19:40.496235 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:19:40.496206 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a8f46a4_ea68_466e_94ed_db3d9b5872de.slice/crio-af78c6623b1ddafded16d32bb6d184525f299556230cc5f2b470e7a130a366b8 WatchSource:0}: Error finding container af78c6623b1ddafded16d32bb6d184525f299556230cc5f2b470e7a130a366b8: Status 404 returned error can't find the container with id af78c6623b1ddafded16d32bb6d184525f299556230cc5f2b470e7a130a366b8 Oct 30 02:19:40.526998 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:40.526966 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a3a2ac-7kn8z" event={"ID":"94e9ec3c-fe81-4e11-906d-6a2dd69d0cee","Type":"ContainerStarted","Data":"72637ffe616ea5e10ea34b377fbec99092e7b305da8609e91047dcae28d746bf"} Oct 30 02:19:40.527113 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:40.527027 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a3a2ac-7kn8z" event={"ID":"94e9ec3c-fe81-4e11-906d-6a2dd69d0cee","Type":"ContainerStarted","Data":"ad5606a699883553380ceace6bbe685d374f69f8df8d383ee398dd13bb146a70"} Oct 30 02:19:40.528435 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:40.528413 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d5aa9-l87rv" event={"ID":"9a8f46a4-ea68-466e-94ed-db3d9b5872de","Type":"ContainerStarted","Data":"af78c6623b1ddafded16d32bb6d184525f299556230cc5f2b470e7a130a366b8"} Oct 30 02:19:40.529949 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:40.529929 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abfaf4e-nvxw8" event={"ID":"ed6145a7-d13c-4efa-ad6d-eaa6c5d90974","Type":"ContainerStarted","Data":"159ba61e32e5df7cf6167afe5f54a0df9f0ef90e0bc2f3ec37600f24d54bc194"} Oct 30 02:19:40.531708 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:40.531687 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-543807da-gxx84" event={"ID":"e203d4b8-3f30-4d07-9c6f-431af65d9ec2","Type":"ContainerStarted","Data":"3197bd12ac98eae91ebc78a48cd5d9c097374fe28a7c2a42a472b6c8f9a4f3c5"} Oct 30 02:19:40.568717 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:40.568671 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a3a2ac-7kn8z" podStartSLOduration=2.568657917 podStartE2EDuration="2.568657917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:19:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:19:40.568009119 +0000 UTC m=+3706.144434532" watchObservedRunningTime="2026-10-30 02:19:40.568657917 +0000 UTC m=+3706.145083318" Oct 30 02:19:40.610626 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:40.610578 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-543807da-gxx84" podStartSLOduration=1.61056371 podStartE2EDuration="1.61056371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:19:40.607315089 +0000 UTC m=+3706.183740490" watchObservedRunningTime="2026-10-30 02:19:40.61056371 +0000 UTC m=+3706.186989112" Oct 30 02:19:41.542175 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:41.542139 6572 generic.go:356] "Generic (PLEG): container finished" podID="ed6145a7-d13c-4efa-ad6d-eaa6c5d90974" containerID="dc8918d5374d444102b690a92812bdbd1ca293652042891ca2b0a141d1470237" exitCode=0 Oct 30 02:19:41.542656 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:41.542228 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abfaf4e-nvxw8" event={"ID":"ed6145a7-d13c-4efa-ad6d-eaa6c5d90974","Type":"ContainerDied","Data":"dc8918d5374d444102b690a92812bdbd1ca293652042891ca2b0a141d1470237"} Oct 30 02:19:41.543963 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:41.543940 6572 generic.go:356] "Generic (PLEG): container finished" podID="e203d4b8-3f30-4d07-9c6f-431af65d9ec2" containerID="1cac6a4d5561afbf850d1d88bf0f021fa04edc73a77ab887caa213424a2dc5ad" exitCode=0 Oct 30 02:19:41.544100 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:41.544024 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-543807da-gxx84" event={"ID":"e203d4b8-3f30-4d07-9c6f-431af65d9ec2","Type":"ContainerDied","Data":"1cac6a4d5561afbf850d1d88bf0f021fa04edc73a77ab887caa213424a2dc5ad"} Oct 30 02:19:41.545811 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:41.545789 6572 generic.go:356] "Generic (PLEG): container finished" podID="94e9ec3c-fe81-4e11-906d-6a2dd69d0cee" containerID="72637ffe616ea5e10ea34b377fbec99092e7b305da8609e91047dcae28d746bf" exitCode=0 Oct 30 02:19:41.545910 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:41.545848 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a3a2ac-7kn8z" event={"ID":"94e9ec3c-fe81-4e11-906d-6a2dd69d0cee","Type":"ContainerDied","Data":"72637ffe616ea5e10ea34b377fbec99092e7b305da8609e91047dcae28d746bf"} Oct 30 02:19:41.547452 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:41.547428 6572 generic.go:356] "Generic (PLEG): container finished" podID="9a8f46a4-ea68-466e-94ed-db3d9b5872de" containerID="74ef9280f95ab58f05aae941260403fe9338b8a2ae30ddd7c99931e3a85033c5" exitCode=0 Oct 30 02:19:41.547556 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:41.547453 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d5aa9-l87rv" event={"ID":"9a8f46a4-ea68-466e-94ed-db3d9b5872de","Type":"ContainerDied","Data":"74ef9280f95ab58f05aae941260403fe9338b8a2ae30ddd7c99931e3a85033c5"} Oct 30 02:19:41.893777 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:41.893726 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:42.720188 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:42.720152 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abfaf4e-nvxw8" Oct 30 02:19:42.791253 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:42.791222 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d5aa9-l87rv" Oct 30 02:19:42.795582 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:42.795559 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a3a2ac-7kn8z" Oct 30 02:19:42.800553 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:42.800527 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-543807da-gxx84" Oct 30 02:19:42.815545 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:42.815484 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed6145a7-d13c-4efa-ad6d-eaa6c5d90974-kube-api-access-kthbr\" (UniqueName: \"kubernetes.io/projected/ed6145a7-d13c-4efa-ad6d-eaa6c5d90974-kube-api-access-kthbr\") pod \"ed6145a7-d13c-4efa-ad6d-eaa6c5d90974\" (UID: \"ed6145a7-d13c-4efa-ad6d-eaa6c5d90974\") " Oct 30 02:19:42.815545 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:42.815534 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed6145a7-d13c-4efa-ad6d-eaa6c5d90974-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed6145a7-d13c-4efa-ad6d-eaa6c5d90974-maas-api-openshift-ca\") pod \"ed6145a7-d13c-4efa-ad6d-eaa6c5d90974\" (UID: \"ed6145a7-d13c-4efa-ad6d-eaa6c5d90974\") " Oct 30 02:19:42.815695 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:42.815553 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed6145a7-d13c-4efa-ad6d-eaa6c5d90974-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed6145a7-d13c-4efa-ad6d-eaa6c5d90974-maas-api-xks-ca\") pod \"ed6145a7-d13c-4efa-ad6d-eaa6c5d90974\" (UID: \"ed6145a7-d13c-4efa-ad6d-eaa6c5d90974\") " Oct 30 02:19:42.815982 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:42.815958 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed6145a7-d13c-4efa-ad6d-eaa6c5d90974-maas-api-openshift-ca" pod "ed6145a7-d13c-4efa-ad6d-eaa6c5d90974" (UID: "ed6145a7-d13c-4efa-ad6d-eaa6c5d90974"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:42.817617 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:42.817598 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed6145a7-d13c-4efa-ad6d-eaa6c5d90974-maas-api-xks-ca" pod "ed6145a7-d13c-4efa-ad6d-eaa6c5d90974" (UID: "ed6145a7-d13c-4efa-ad6d-eaa6c5d90974"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:19:42.817701 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:42.817681 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed6145a7-d13c-4efa-ad6d-eaa6c5d90974-kube-api-access-kthbr" pod "ed6145a7-d13c-4efa-ad6d-eaa6c5d90974" (UID: "ed6145a7-d13c-4efa-ad6d-eaa6c5d90974"). InnerVolumeSpecName "kube-api-access-kthbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:42.916380 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:42.916345 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94e9ec3c-fe81-4e11-906d-6a2dd69d0cee-kube-api-access-hcjc4\" (UniqueName: \"kubernetes.io/projected/94e9ec3c-fe81-4e11-906d-6a2dd69d0cee-kube-api-access-hcjc4\") pod \"94e9ec3c-fe81-4e11-906d-6a2dd69d0cee\" (UID: \"94e9ec3c-fe81-4e11-906d-6a2dd69d0cee\") " Oct 30 02:19:42.916613 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:42.916420 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94e9ec3c-fe81-4e11-906d-6a2dd69d0cee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e9ec3c-fe81-4e11-906d-6a2dd69d0cee-maas-api-openshift-ca\") pod \"94e9ec3c-fe81-4e11-906d-6a2dd69d0cee\" (UID: \"94e9ec3c-fe81-4e11-906d-6a2dd69d0cee\") " Oct 30 02:19:42.916613 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:42.916484 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a8f46a4-ea68-466e-94ed-db3d9b5872de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a8f46a4-ea68-466e-94ed-db3d9b5872de-maas-api-openshift-ca\") pod \"9a8f46a4-ea68-466e-94ed-db3d9b5872de\" (UID: \"9a8f46a4-ea68-466e-94ed-db3d9b5872de\") " Oct 30 02:19:42.916613 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:42.916548 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e203d4b8-3f30-4d07-9c6f-431af65d9ec2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e203d4b8-3f30-4d07-9c6f-431af65d9ec2-maas-api-xks-ca\") pod \"e203d4b8-3f30-4d07-9c6f-431af65d9ec2\" (UID: \"e203d4b8-3f30-4d07-9c6f-431af65d9ec2\") " Oct 30 02:19:42.916613 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:42.916582 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94e9ec3c-fe81-4e11-906d-6a2dd69d0cee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e9ec3c-fe81-4e11-906d-6a2dd69d0cee-maas-api-xks-ca\") pod \"94e9ec3c-fe81-4e11-906d-6a2dd69d0cee\" (UID: \"94e9ec3c-fe81-4e11-906d-6a2dd69d0cee\") " Oct 30 02:19:42.916921 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:42.916642 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a8f46a4-ea68-466e-94ed-db3d9b5872de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a8f46a4-ea68-466e-94ed-db3d9b5872de-maas-api-xks-ca\") pod \"9a8f46a4-ea68-466e-94ed-db3d9b5872de\" (UID: \"9a8f46a4-ea68-466e-94ed-db3d9b5872de\") " Oct 30 02:19:42.916921 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:42.916697 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e203d4b8-3f30-4d07-9c6f-431af65d9ec2-kube-api-access-shdn9\" (UniqueName: \"kubernetes.io/projected/e203d4b8-3f30-4d07-9c6f-431af65d9ec2-kube-api-access-shdn9\") pod \"e203d4b8-3f30-4d07-9c6f-431af65d9ec2\" (UID: \"e203d4b8-3f30-4d07-9c6f-431af65d9ec2\") " Oct 30 02:19:42.916921 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:42.916818 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e203d4b8-3f30-4d07-9c6f-431af65d9ec2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e203d4b8-3f30-4d07-9c6f-431af65d9ec2-maas-api-openshift-ca\") pod \"e203d4b8-3f30-4d07-9c6f-431af65d9ec2\" (UID: \"e203d4b8-3f30-4d07-9c6f-431af65d9ec2\") " Oct 30 02:19:42.916921 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:42.916833 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94e9ec3c-fe81-4e11-906d-6a2dd69d0cee-maas-api-openshift-ca" pod "94e9ec3c-fe81-4e11-906d-6a2dd69d0cee" (UID: "94e9ec3c-fe81-4e11-906d-6a2dd69d0cee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:42.916921 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:42.916859 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a8f46a4-ea68-466e-94ed-db3d9b5872de-kube-api-access-4tt6v\" (UniqueName: \"kubernetes.io/projected/9a8f46a4-ea68-466e-94ed-db3d9b5872de-kube-api-access-4tt6v\") pod \"9a8f46a4-ea68-466e-94ed-db3d9b5872de\" (UID: \"9a8f46a4-ea68-466e-94ed-db3d9b5872de\") " Oct 30 02:19:42.917232 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:42.916936 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a8f46a4-ea68-466e-94ed-db3d9b5872de-maas-api-openshift-ca" pod "9a8f46a4-ea68-466e-94ed-db3d9b5872de" (UID: "9a8f46a4-ea68-466e-94ed-db3d9b5872de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:42.917347 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:42.917309 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e203d4b8-3f30-4d07-9c6f-431af65d9ec2-maas-api-openshift-ca" pod "e203d4b8-3f30-4d07-9c6f-431af65d9ec2" (UID: "e203d4b8-3f30-4d07-9c6f-431af65d9ec2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:42.917347 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:42.917318 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kthbr\" (UniqueName: \"kubernetes.io/projected/ed6145a7-d13c-4efa-ad6d-eaa6c5d90974-kube-api-access-kthbr\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:42.917510 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:42.917364 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed6145a7-d13c-4efa-ad6d-eaa6c5d90974-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:42.917510 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:42.917381 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed6145a7-d13c-4efa-ad6d-eaa6c5d90974-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:42.917510 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:42.917399 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e9ec3c-fe81-4e11-906d-6a2dd69d0cee-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:42.917510 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:42.917416 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a8f46a4-ea68-466e-94ed-db3d9b5872de-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:42.919371 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:42.919337 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94e9ec3c-fe81-4e11-906d-6a2dd69d0cee-maas-api-xks-ca" pod "94e9ec3c-fe81-4e11-906d-6a2dd69d0cee" (UID: "94e9ec3c-fe81-4e11-906d-6a2dd69d0cee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:19:42.919476 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:42.919450 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e203d4b8-3f30-4d07-9c6f-431af65d9ec2-maas-api-xks-ca" pod "e203d4b8-3f30-4d07-9c6f-431af65d9ec2" (UID: "e203d4b8-3f30-4d07-9c6f-431af65d9ec2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:19:42.919476 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:42.919463 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e203d4b8-3f30-4d07-9c6f-431af65d9ec2-kube-api-access-shdn9" pod "e203d4b8-3f30-4d07-9c6f-431af65d9ec2" (UID: "e203d4b8-3f30-4d07-9c6f-431af65d9ec2"). InnerVolumeSpecName "kube-api-access-shdn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:42.919560 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:42.919546 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e9ec3c-fe81-4e11-906d-6a2dd69d0cee-kube-api-access-hcjc4" pod "94e9ec3c-fe81-4e11-906d-6a2dd69d0cee" (UID: "94e9ec3c-fe81-4e11-906d-6a2dd69d0cee"). InnerVolumeSpecName "kube-api-access-hcjc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:42.919689 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:42.919668 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a8f46a4-ea68-466e-94ed-db3d9b5872de-maas-api-xks-ca" pod "9a8f46a4-ea68-466e-94ed-db3d9b5872de" (UID: "9a8f46a4-ea68-466e-94ed-db3d9b5872de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:19:42.919778 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:42.919689 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8f46a4-ea68-466e-94ed-db3d9b5872de-kube-api-access-4tt6v" pod "9a8f46a4-ea68-466e-94ed-db3d9b5872de" (UID: "9a8f46a4-ea68-466e-94ed-db3d9b5872de"). InnerVolumeSpecName "kube-api-access-4tt6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:43.019129 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.019089 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shdn9\" (UniqueName: \"kubernetes.io/projected/e203d4b8-3f30-4d07-9c6f-431af65d9ec2-kube-api-access-shdn9\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:43.019129 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.019124 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e203d4b8-3f30-4d07-9c6f-431af65d9ec2-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:43.019129 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.019135 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tt6v\" (UniqueName: \"kubernetes.io/projected/9a8f46a4-ea68-466e-94ed-db3d9b5872de-kube-api-access-4tt6v\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:43.019409 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.019144 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcjc4\" (UniqueName: \"kubernetes.io/projected/94e9ec3c-fe81-4e11-906d-6a2dd69d0cee-kube-api-access-hcjc4\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:43.019409 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.019154 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e203d4b8-3f30-4d07-9c6f-431af65d9ec2-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:43.019409 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.019163 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e9ec3c-fe81-4e11-906d-6a2dd69d0cee-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:43.019409 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.019173 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a8f46a4-ea68-466e-94ed-db3d9b5872de-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:43.381163 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.381130 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:43.554790 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.554765 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 02:19:43.562985 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.562962 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abfaf4e-nvxw8" Oct 30 02:19:43.563139 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.562999 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abfaf4e-nvxw8" event={"ID":"ed6145a7-d13c-4efa-ad6d-eaa6c5d90974","Type":"ContainerDied","Data":"159ba61e32e5df7cf6167afe5f54a0df9f0ef90e0bc2f3ec37600f24d54bc194"} Oct 30 02:19:43.563139 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.563058 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="159ba61e32e5df7cf6167afe5f54a0df9f0ef90e0bc2f3ec37600f24d54bc194" Oct 30 02:19:43.564435 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.564414 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-543807da-gxx84" Oct 30 02:19:43.564581 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.564446 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-543807da-gxx84" event={"ID":"e203d4b8-3f30-4d07-9c6f-431af65d9ec2","Type":"ContainerDied","Data":"3197bd12ac98eae91ebc78a48cd5d9c097374fe28a7c2a42a472b6c8f9a4f3c5"} Oct 30 02:19:43.564581 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.564496 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3197bd12ac98eae91ebc78a48cd5d9c097374fe28a7c2a42a472b6c8f9a4f3c5" Oct 30 02:19:43.566496 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.566473 6572 generic.go:356] "Generic (PLEG): container finished" podID="322cef03-4f70-42c8-8571-42c939fba68b" containerID="a47a8d4c2caa136ea3ab6db923cb02b3b0bc5dee668d52aae61b5d4773314b0f" exitCode=0 Oct 30 02:19:43.566642 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.566630 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" event={"ID":"322cef03-4f70-42c8-8571-42c939fba68b","Type":"ContainerDied","Data":"a47a8d4c2caa136ea3ab6db923cb02b3b0bc5dee668d52aae61b5d4773314b0f"} Oct 30 02:19:43.566724 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.566659 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" Oct 30 02:19:43.566724 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.566678 6572 scope.go:122] "RemoveContainer" containerID="a47a8d4c2caa136ea3ab6db923cb02b3b0bc5dee668d52aae61b5d4773314b0f" Oct 30 02:19:43.566882 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.566664 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq" event={"ID":"322cef03-4f70-42c8-8571-42c939fba68b","Type":"ContainerDied","Data":"28d64368f89b4a1d196e36d4c40ce1955a6836b3a1c129de1a975716adf52856"} Oct 30 02:19:43.568553 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.568531 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a3a2ac-7kn8z" event={"ID":"94e9ec3c-fe81-4e11-906d-6a2dd69d0cee","Type":"ContainerDied","Data":"ad5606a699883553380ceace6bbe685d374f69f8df8d383ee398dd13bb146a70"} Oct 30 02:19:43.568668 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.568574 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad5606a699883553380ceace6bbe685d374f69f8df8d383ee398dd13bb146a70" Oct 30 02:19:43.568668 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.568536 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a3a2ac-7kn8z" Oct 30 02:19:43.570309 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.570291 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d5aa9-l87rv" Oct 30 02:19:43.570416 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.570337 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d5aa9-l87rv" event={"ID":"9a8f46a4-ea68-466e-94ed-db3d9b5872de","Type":"ContainerDied","Data":"af78c6623b1ddafded16d32bb6d184525f299556230cc5f2b470e7a130a366b8"} Oct 30 02:19:43.570416 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.570377 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af78c6623b1ddafded16d32bb6d184525f299556230cc5f2b470e7a130a366b8" Oct 30 02:19:43.580101 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.580024 6572 scope.go:122] "RemoveContainer" containerID="fe9b85c6a37107c277e18fb1db2f2d60c35aa9f585467f2d4ef2cae370a26036" Oct 30 02:19:43.594494 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.594469 6572 scope.go:122] "RemoveContainer" containerID="a47a8d4c2caa136ea3ab6db923cb02b3b0bc5dee668d52aae61b5d4773314b0f" Oct 30 02:19:43.594835 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:19:43.594812 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a47a8d4c2caa136ea3ab6db923cb02b3b0bc5dee668d52aae61b5d4773314b0f\": container with ID starting with a47a8d4c2caa136ea3ab6db923cb02b3b0bc5dee668d52aae61b5d4773314b0f not found: ID does not exist" containerID="a47a8d4c2caa136ea3ab6db923cb02b3b0bc5dee668d52aae61b5d4773314b0f" Oct 30 02:19:43.594918 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.594843 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a47a8d4c2caa136ea3ab6db923cb02b3b0bc5dee668d52aae61b5d4773314b0f"} err="failed to get container status \"a47a8d4c2caa136ea3ab6db923cb02b3b0bc5dee668d52aae61b5d4773314b0f\": rpc error: code = NotFound desc = could not find container \"a47a8d4c2caa136ea3ab6db923cb02b3b0bc5dee668d52aae61b5d4773314b0f\": container with ID starting with a47a8d4c2caa136ea3ab6db923cb02b3b0bc5dee668d52aae61b5d4773314b0f not found: ID does not exist" Oct 30 02:19:43.594918 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.594861 6572 scope.go:122] "RemoveContainer" containerID="fe9b85c6a37107c277e18fb1db2f2d60c35aa9f585467f2d4ef2cae370a26036" Oct 30 02:19:43.595105 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:19:43.595086 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe9b85c6a37107c277e18fb1db2f2d60c35aa9f585467f2d4ef2cae370a26036\": container with ID starting with fe9b85c6a37107c277e18fb1db2f2d60c35aa9f585467f2d4ef2cae370a26036 not found: ID does not exist" containerID="fe9b85c6a37107c277e18fb1db2f2d60c35aa9f585467f2d4ef2cae370a26036" Oct 30 02:19:43.595151 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.595112 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe9b85c6a37107c277e18fb1db2f2d60c35aa9f585467f2d4ef2cae370a26036"} err="failed to get container status \"fe9b85c6a37107c277e18fb1db2f2d60c35aa9f585467f2d4ef2cae370a26036\": rpc error: code = NotFound desc = could not find container \"fe9b85c6a37107c277e18fb1db2f2d60c35aa9f585467f2d4ef2cae370a26036\": container with ID starting with fe9b85c6a37107c277e18fb1db2f2d60c35aa9f585467f2d4ef2cae370a26036 not found: ID does not exist" Oct 30 02:19:43.625228 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.625195 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/322cef03-4f70-42c8-8571-42c939fba68b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/322cef03-4f70-42c8-8571-42c939fba68b-workload-certs\") pod \"322cef03-4f70-42c8-8571-42c939fba68b\" (UID: \"322cef03-4f70-42c8-8571-42c939fba68b\") " Oct 30 02:19:43.625396 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.625237 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/322cef03-4f70-42c8-8571-42c939fba68b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/322cef03-4f70-42c8-8571-42c939fba68b-credential-socket\") pod \"322cef03-4f70-42c8-8571-42c939fba68b\" (UID: \"322cef03-4f70-42c8-8571-42c939fba68b\") " Oct 30 02:19:43.625396 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.625325 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/322cef03-4f70-42c8-8571-42c939fba68b-kube-api-access-qwsvk\" (UniqueName: \"kubernetes.io/projected/322cef03-4f70-42c8-8571-42c939fba68b-kube-api-access-qwsvk\") pod \"322cef03-4f70-42c8-8571-42c939fba68b\" (UID: \"322cef03-4f70-42c8-8571-42c939fba68b\") " Oct 30 02:19:43.625396 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.625362 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/322cef03-4f70-42c8-8571-42c939fba68b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/322cef03-4f70-42c8-8571-42c939fba68b-istio-envoy\") pod \"322cef03-4f70-42c8-8571-42c939fba68b\" (UID: \"322cef03-4f70-42c8-8571-42c939fba68b\") " Oct 30 02:19:43.625588 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.625398 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/322cef03-4f70-42c8-8571-42c939fba68b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/322cef03-4f70-42c8-8571-42c939fba68b-istio-podinfo\") pod \"322cef03-4f70-42c8-8571-42c939fba68b\" (UID: \"322cef03-4f70-42c8-8571-42c939fba68b\") " Oct 30 02:19:43.625588 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.625537 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/322cef03-4f70-42c8-8571-42c939fba68b-workload-certs" pod "322cef03-4f70-42c8-8571-42c939fba68b" (UID: "322cef03-4f70-42c8-8571-42c939fba68b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:43.625588 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.625584 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/322cef03-4f70-42c8-8571-42c939fba68b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/322cef03-4f70-42c8-8571-42c939fba68b-workload-socket\") pod \"322cef03-4f70-42c8-8571-42c939fba68b\" (UID: \"322cef03-4f70-42c8-8571-42c939fba68b\") " Oct 30 02:19:43.625728 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.625666 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/322cef03-4f70-42c8-8571-42c939fba68b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/322cef03-4f70-42c8-8571-42c939fba68b-istiod-ca-cert\") pod \"322cef03-4f70-42c8-8571-42c939fba68b\" (UID: \"322cef03-4f70-42c8-8571-42c939fba68b\") " Oct 30 02:19:43.625728 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.625713 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/322cef03-4f70-42c8-8571-42c939fba68b-istio-token\" (UniqueName: \"kubernetes.io/projected/322cef03-4f70-42c8-8571-42c939fba68b-istio-token\") pod \"322cef03-4f70-42c8-8571-42c939fba68b\" (UID: \"322cef03-4f70-42c8-8571-42c939fba68b\") " Oct 30 02:19:43.625886 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.625866 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/322cef03-4f70-42c8-8571-42c939fba68b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/322cef03-4f70-42c8-8571-42c939fba68b-istio-data\") pod \"322cef03-4f70-42c8-8571-42c939fba68b\" (UID: \"322cef03-4f70-42c8-8571-42c939fba68b\") " Oct 30 02:19:43.625976 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.625954 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/322cef03-4f70-42c8-8571-42c939fba68b-workload-socket" pod "322cef03-4f70-42c8-8571-42c939fba68b" (UID: "322cef03-4f70-42c8-8571-42c939fba68b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:43.626189 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.626164 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/322cef03-4f70-42c8-8571-42c939fba68b-istiod-ca-cert" pod "322cef03-4f70-42c8-8571-42c939fba68b" (UID: "322cef03-4f70-42c8-8571-42c939fba68b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:43.626329 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.626310 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/322cef03-4f70-42c8-8571-42c939fba68b-istio-data" pod "322cef03-4f70-42c8-8571-42c939fba68b" (UID: "322cef03-4f70-42c8-8571-42c939fba68b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:43.626487 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.626472 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/322cef03-4f70-42c8-8571-42c939fba68b-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:43.626563 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.626493 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/322cef03-4f70-42c8-8571-42c939fba68b-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:43.626563 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.626508 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/322cef03-4f70-42c8-8571-42c939fba68b-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:43.626563 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.626522 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/322cef03-4f70-42c8-8571-42c939fba68b-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:43.626775 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.626727 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/322cef03-4f70-42c8-8571-42c939fba68b-credential-socket" pod "322cef03-4f70-42c8-8571-42c939fba68b" (UID: "322cef03-4f70-42c8-8571-42c939fba68b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:43.627855 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.627830 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/322cef03-4f70-42c8-8571-42c939fba68b-istio-podinfo" pod "322cef03-4f70-42c8-8571-42c939fba68b" (UID: "322cef03-4f70-42c8-8571-42c939fba68b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:19:43.628027 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.628011 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/322cef03-4f70-42c8-8571-42c939fba68b-kube-api-access-qwsvk" pod "322cef03-4f70-42c8-8571-42c939fba68b" (UID: "322cef03-4f70-42c8-8571-42c939fba68b"). InnerVolumeSpecName "kube-api-access-qwsvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:43.628917 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.628882 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/322cef03-4f70-42c8-8571-42c939fba68b-istio-envoy" pod "322cef03-4f70-42c8-8571-42c939fba68b" (UID: "322cef03-4f70-42c8-8571-42c939fba68b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:43.629024 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.628916 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/322cef03-4f70-42c8-8571-42c939fba68b-istio-token" pod "322cef03-4f70-42c8-8571-42c939fba68b" (UID: "322cef03-4f70-42c8-8571-42c939fba68b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:43.727616 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.727584 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/322cef03-4f70-42c8-8571-42c939fba68b-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:43.727616 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.727613 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwsvk\" (UniqueName: \"kubernetes.io/projected/322cef03-4f70-42c8-8571-42c939fba68b-kube-api-access-qwsvk\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:43.728061 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.727624 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/322cef03-4f70-42c8-8571-42c939fba68b-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:43.728061 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.727637 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/322cef03-4f70-42c8-8571-42c939fba68b-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:43.728061 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.727650 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/322cef03-4f70-42c8-8571-42c939fba68b-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:43.766412 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.766378 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp"] Oct 30 02:19:43.913148 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.912973 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq"] Oct 30 02:19:43.917250 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.917224 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c8d376-openshift-default-6b7f8fc987-pvkrq"] Oct 30 02:19:43.962385 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:43.962346 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:44.290098 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.290065 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-558327-6bc8566f5f-gbqkb" podUID="e07b1c7e-fe5d-4dff-b4aa-6d03ce285792" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:19:44.522581 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.522517 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:19:44.541248 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.541220 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="322cef03-4f70-42c8-8571-42c939fba68b" path="/var/lib/kubelet/pods/322cef03-4f70-42c8-8571-42c939fba68b/volumes" Oct 30 02:19:44.580941 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.580911 6572 generic.go:356] "Generic (PLEG): container finished" podID="d32bbe46-7de7-4307-a62d-ee16d8b59741" containerID="f2f7c87a7516025f1fe86fe1725973ee44ea731d37473fe7c122fea190ce1cff" exitCode=0 Oct 30 02:19:44.581067 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.580975 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" Oct 30 02:19:44.581067 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.580991 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" event={"ID":"d32bbe46-7de7-4307-a62d-ee16d8b59741","Type":"ContainerDied","Data":"f2f7c87a7516025f1fe86fe1725973ee44ea731d37473fe7c122fea190ce1cff"} Oct 30 02:19:44.581067 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.581042 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw" event={"ID":"d32bbe46-7de7-4307-a62d-ee16d8b59741","Type":"ContainerDied","Data":"5e5b46695b4578f68d4a914dbcf145c16ce9de34356c16da454489233da66424"} Oct 30 02:19:44.581202 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.581073 6572 scope.go:122] "RemoveContainer" containerID="f2f7c87a7516025f1fe86fe1725973ee44ea731d37473fe7c122fea190ce1cff" Oct 30 02:19:44.595215 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.595195 6572 scope.go:122] "RemoveContainer" containerID="bb1dffbbc8f6b7d54e0b3092038eeadb2860e27159c7f4fcde7bb5b692499bee" Oct 30 02:19:44.605337 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.605310 6572 scope.go:122] "RemoveContainer" containerID="f2f7c87a7516025f1fe86fe1725973ee44ea731d37473fe7c122fea190ce1cff" Oct 30 02:19:44.605623 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:19:44.605604 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2f7c87a7516025f1fe86fe1725973ee44ea731d37473fe7c122fea190ce1cff\": container with ID starting with f2f7c87a7516025f1fe86fe1725973ee44ea731d37473fe7c122fea190ce1cff not found: ID does not exist" containerID="f2f7c87a7516025f1fe86fe1725973ee44ea731d37473fe7c122fea190ce1cff" Oct 30 02:19:44.605687 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.605631 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2f7c87a7516025f1fe86fe1725973ee44ea731d37473fe7c122fea190ce1cff"} err="failed to get container status \"f2f7c87a7516025f1fe86fe1725973ee44ea731d37473fe7c122fea190ce1cff\": rpc error: code = NotFound desc = could not find container \"f2f7c87a7516025f1fe86fe1725973ee44ea731d37473fe7c122fea190ce1cff\": container with ID starting with f2f7c87a7516025f1fe86fe1725973ee44ea731d37473fe7c122fea190ce1cff not found: ID does not exist" Oct 30 02:19:44.605687 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.605647 6572 scope.go:122] "RemoveContainer" containerID="bb1dffbbc8f6b7d54e0b3092038eeadb2860e27159c7f4fcde7bb5b692499bee" Oct 30 02:19:44.605938 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:19:44.605922 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb1dffbbc8f6b7d54e0b3092038eeadb2860e27159c7f4fcde7bb5b692499bee\": container with ID starting with bb1dffbbc8f6b7d54e0b3092038eeadb2860e27159c7f4fcde7bb5b692499bee not found: ID does not exist" containerID="bb1dffbbc8f6b7d54e0b3092038eeadb2860e27159c7f4fcde7bb5b692499bee" Oct 30 02:19:44.605995 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.605942 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb1dffbbc8f6b7d54e0b3092038eeadb2860e27159c7f4fcde7bb5b692499bee"} err="failed to get container status \"bb1dffbbc8f6b7d54e0b3092038eeadb2860e27159c7f4fcde7bb5b692499bee\": rpc error: code = NotFound desc = could not find container \"bb1dffbbc8f6b7d54e0b3092038eeadb2860e27159c7f4fcde7bb5b692499bee\": container with ID starting with bb1dffbbc8f6b7d54e0b3092038eeadb2860e27159c7f4fcde7bb5b692499bee not found: ID does not exist" Oct 30 02:19:44.638158 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.638129 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d32bbe46-7de7-4307-a62d-ee16d8b59741-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d32bbe46-7de7-4307-a62d-ee16d8b59741-istio-data\") pod \"d32bbe46-7de7-4307-a62d-ee16d8b59741\" (UID: \"d32bbe46-7de7-4307-a62d-ee16d8b59741\") " Oct 30 02:19:44.638273 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.638196 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d32bbe46-7de7-4307-a62d-ee16d8b59741-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32bbe46-7de7-4307-a62d-ee16d8b59741-credential-socket\") pod \"d32bbe46-7de7-4307-a62d-ee16d8b59741\" (UID: \"d32bbe46-7de7-4307-a62d-ee16d8b59741\") " Oct 30 02:19:44.638273 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.638220 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d32bbe46-7de7-4307-a62d-ee16d8b59741-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d32bbe46-7de7-4307-a62d-ee16d8b59741-istiod-ca-cert\") pod \"d32bbe46-7de7-4307-a62d-ee16d8b59741\" (UID: \"d32bbe46-7de7-4307-a62d-ee16d8b59741\") " Oct 30 02:19:44.638273 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.638246 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d32bbe46-7de7-4307-a62d-ee16d8b59741-istio-token\" (UniqueName: \"kubernetes.io/projected/d32bbe46-7de7-4307-a62d-ee16d8b59741-istio-token\") pod \"d32bbe46-7de7-4307-a62d-ee16d8b59741\" (UID: \"d32bbe46-7de7-4307-a62d-ee16d8b59741\") " Oct 30 02:19:44.638273 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.638263 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d32bbe46-7de7-4307-a62d-ee16d8b59741-kube-api-access-g52cc\" (UniqueName: \"kubernetes.io/projected/d32bbe46-7de7-4307-a62d-ee16d8b59741-kube-api-access-g52cc\") pod \"d32bbe46-7de7-4307-a62d-ee16d8b59741\" (UID: \"d32bbe46-7de7-4307-a62d-ee16d8b59741\") " Oct 30 02:19:44.638613 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.638306 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d32bbe46-7de7-4307-a62d-ee16d8b59741-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d32bbe46-7de7-4307-a62d-ee16d8b59741-workload-certs\") pod \"d32bbe46-7de7-4307-a62d-ee16d8b59741\" (UID: \"d32bbe46-7de7-4307-a62d-ee16d8b59741\") " Oct 30 02:19:44.638613 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.638354 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d32bbe46-7de7-4307-a62d-ee16d8b59741-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32bbe46-7de7-4307-a62d-ee16d8b59741-workload-socket\") pod \"d32bbe46-7de7-4307-a62d-ee16d8b59741\" (UID: \"d32bbe46-7de7-4307-a62d-ee16d8b59741\") " Oct 30 02:19:44.638613 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.638413 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d32bbe46-7de7-4307-a62d-ee16d8b59741-istio-data" pod "d32bbe46-7de7-4307-a62d-ee16d8b59741" (UID: "d32bbe46-7de7-4307-a62d-ee16d8b59741"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:44.638613 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.638451 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d32bbe46-7de7-4307-a62d-ee16d8b59741-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d32bbe46-7de7-4307-a62d-ee16d8b59741-istio-envoy\") pod \"d32bbe46-7de7-4307-a62d-ee16d8b59741\" (UID: \"d32bbe46-7de7-4307-a62d-ee16d8b59741\") " Oct 30 02:19:44.638613 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.638478 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d32bbe46-7de7-4307-a62d-ee16d8b59741-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d32bbe46-7de7-4307-a62d-ee16d8b59741-istio-podinfo\") pod \"d32bbe46-7de7-4307-a62d-ee16d8b59741\" (UID: \"d32bbe46-7de7-4307-a62d-ee16d8b59741\") " Oct 30 02:19:44.638613 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.638514 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d32bbe46-7de7-4307-a62d-ee16d8b59741-credential-socket" pod "d32bbe46-7de7-4307-a62d-ee16d8b59741" (UID: "d32bbe46-7de7-4307-a62d-ee16d8b59741"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:44.638996 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.638667 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d32bbe46-7de7-4307-a62d-ee16d8b59741-workload-socket" pod "d32bbe46-7de7-4307-a62d-ee16d8b59741" (UID: "d32bbe46-7de7-4307-a62d-ee16d8b59741"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:44.638996 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.638673 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d32bbe46-7de7-4307-a62d-ee16d8b59741-istiod-ca-cert" pod "d32bbe46-7de7-4307-a62d-ee16d8b59741" (UID: "d32bbe46-7de7-4307-a62d-ee16d8b59741"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:44.638996 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.638883 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d32bbe46-7de7-4307-a62d-ee16d8b59741-workload-certs" pod "d32bbe46-7de7-4307-a62d-ee16d8b59741" (UID: "d32bbe46-7de7-4307-a62d-ee16d8b59741"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:44.638996 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.638971 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d32bbe46-7de7-4307-a62d-ee16d8b59741-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:44.638996 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.638993 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32bbe46-7de7-4307-a62d-ee16d8b59741-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:44.639299 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.639005 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d32bbe46-7de7-4307-a62d-ee16d8b59741-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:44.639299 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.639014 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d32bbe46-7de7-4307-a62d-ee16d8b59741-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:44.639299 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.639022 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d32bbe46-7de7-4307-a62d-ee16d8b59741-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:44.640722 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.640700 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d32bbe46-7de7-4307-a62d-ee16d8b59741-istio-podinfo" pod "d32bbe46-7de7-4307-a62d-ee16d8b59741" (UID: "d32bbe46-7de7-4307-a62d-ee16d8b59741"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:19:44.640834 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.640728 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d32bbe46-7de7-4307-a62d-ee16d8b59741-istio-token" pod "d32bbe46-7de7-4307-a62d-ee16d8b59741" (UID: "d32bbe46-7de7-4307-a62d-ee16d8b59741"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:44.641044 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.641015 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d32bbe46-7de7-4307-a62d-ee16d8b59741-istio-envoy" pod "d32bbe46-7de7-4307-a62d-ee16d8b59741" (UID: "d32bbe46-7de7-4307-a62d-ee16d8b59741"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:44.641165 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.641025 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d32bbe46-7de7-4307-a62d-ee16d8b59741-kube-api-access-g52cc" pod "d32bbe46-7de7-4307-a62d-ee16d8b59741" (UID: "d32bbe46-7de7-4307-a62d-ee16d8b59741"). InnerVolumeSpecName "kube-api-access-g52cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:44.688219 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:19:44.688182 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:19:44.740231 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.740190 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d32bbe46-7de7-4307-a62d-ee16d8b59741-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:44.740231 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.740225 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g52cc\" (UniqueName: \"kubernetes.io/projected/d32bbe46-7de7-4307-a62d-ee16d8b59741-kube-api-access-g52cc\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:44.740231 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.740238 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d32bbe46-7de7-4307-a62d-ee16d8b59741-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:44.740690 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.740250 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d32bbe46-7de7-4307-a62d-ee16d8b59741-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:44.926688 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.926653 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw"] Oct 30 02:19:44.938552 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.938512 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-340ae6-openshift-default-7c68dd85bf-8bvkw"] Oct 30 02:19:44.999547 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:44.999508 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:46.479084 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:46.479051 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:46.541849 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:46.541808 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d32bbe46-7de7-4307-a62d-ee16d8b59741" path="/var/lib/kubelet/pods/d32bbe46-7de7-4307-a62d-ee16d8b59741/volumes" Oct 30 02:19:48.188973 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:48.188917 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:49.683666 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:49.683626 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:49.869211 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:49.869171 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:50.052039 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:50.051915 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:50.174019 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:50.173981 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2d5aa9-66754f6cd7-5sdmc"] Oct 30 02:19:50.366222 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:50.366078 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:50.556586 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:50.556554 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:51.007027 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:51.006959 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-558327-6bc8566f5f-gbqkb" Oct 30 02:19:51.113587 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:51.113542 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e07b1c7e-fe5d-4dff-b4aa-6d03ce285792-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e07b1c7e-fe5d-4dff-b4aa-6d03ce285792-plugins-config-volume\") pod \"e07b1c7e-fe5d-4dff-b4aa-6d03ce285792\" (UID: \"e07b1c7e-fe5d-4dff-b4aa-6d03ce285792\") " Oct 30 02:19:51.113827 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:51.113694 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e07b1c7e-fe5d-4dff-b4aa-6d03ce285792-kube-api-access-229lv\" (UniqueName: \"kubernetes.io/projected/e07b1c7e-fe5d-4dff-b4aa-6d03ce285792-kube-api-access-229lv\") pod \"e07b1c7e-fe5d-4dff-b4aa-6d03ce285792\" (UID: \"e07b1c7e-fe5d-4dff-b4aa-6d03ce285792\") " Oct 30 02:19:51.113964 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:51.113941 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e07b1c7e-fe5d-4dff-b4aa-6d03ce285792-plugins-config-volume" pod "e07b1c7e-fe5d-4dff-b4aa-6d03ce285792" (UID: "e07b1c7e-fe5d-4dff-b4aa-6d03ce285792"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:51.114127 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:51.114107 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e07b1c7e-fe5d-4dff-b4aa-6d03ce285792-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:51.115981 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:51.115953 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e07b1c7e-fe5d-4dff-b4aa-6d03ce285792-kube-api-access-229lv" pod "e07b1c7e-fe5d-4dff-b4aa-6d03ce285792" (UID: "e07b1c7e-fe5d-4dff-b4aa-6d03ce285792"). InnerVolumeSpecName "kube-api-access-229lv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:51.215488 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:51.215448 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-229lv\" (UniqueName: \"kubernetes.io/projected/e07b1c7e-fe5d-4dff-b4aa-6d03ce285792-kube-api-access-229lv\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:51.518906 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:51.518863 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:51.646708 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:51.646674 6572 generic.go:356] "Generic (PLEG): container finished" podID="e07b1c7e-fe5d-4dff-b4aa-6d03ce285792" containerID="1bfa6bea1affb9adc3567789ae6772fa2789b92b5c8dc869859208be9cf3a785" exitCode=0 Oct 30 02:19:51.646931 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:51.646729 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-558327-6bc8566f5f-gbqkb" Oct 30 02:19:51.646931 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:51.646782 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-558327-6bc8566f5f-gbqkb" event={"ID":"e07b1c7e-fe5d-4dff-b4aa-6d03ce285792","Type":"ContainerDied","Data":"1bfa6bea1affb9adc3567789ae6772fa2789b92b5c8dc869859208be9cf3a785"} Oct 30 02:19:51.646931 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:51.646841 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-558327-6bc8566f5f-gbqkb" event={"ID":"e07b1c7e-fe5d-4dff-b4aa-6d03ce285792","Type":"ContainerDied","Data":"e6f858eba8f546bea95f603f9fa04551722e085ade6d7431a151f85aaadf5777"} Oct 30 02:19:51.646931 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:51.646877 6572 scope.go:122] "RemoveContainer" containerID="1bfa6bea1affb9adc3567789ae6772fa2789b92b5c8dc869859208be9cf3a785" Oct 30 02:19:51.658290 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:51.658266 6572 scope.go:122] "RemoveContainer" containerID="1bfa6bea1affb9adc3567789ae6772fa2789b92b5c8dc869859208be9cf3a785" Oct 30 02:19:51.658666 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:19:51.658635 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bfa6bea1affb9adc3567789ae6772fa2789b92b5c8dc869859208be9cf3a785\": container with ID starting with 1bfa6bea1affb9adc3567789ae6772fa2789b92b5c8dc869859208be9cf3a785 not found: ID does not exist" containerID="1bfa6bea1affb9adc3567789ae6772fa2789b92b5c8dc869859208be9cf3a785" Oct 30 02:19:51.658797 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:51.658675 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bfa6bea1affb9adc3567789ae6772fa2789b92b5c8dc869859208be9cf3a785"} err="failed to get container status \"1bfa6bea1affb9adc3567789ae6772fa2789b92b5c8dc869859208be9cf3a785\": rpc error: code = NotFound desc = could not find container \"1bfa6bea1affb9adc3567789ae6772fa2789b92b5c8dc869859208be9cf3a785\": container with ID starting with 1bfa6bea1affb9adc3567789ae6772fa2789b92b5c8dc869859208be9cf3a785 not found: ID does not exist" Oct 30 02:19:51.706433 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:51.706391 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-558327-6bc8566f5f-gbqkb"] Oct 30 02:19:51.715170 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:51.715130 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-558327-6bc8566f5f-gbqkb"] Oct 30 02:19:52.541504 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:52.541467 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e07b1c7e-fe5d-4dff-b4aa-6d03ce285792" path="/var/lib/kubelet/pods/e07b1c7e-fe5d-4dff-b4aa-6d03ce285792/volumes" Oct 30 02:19:52.887777 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:52.887678 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:53.310067 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:53.310033 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:53.331876 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:53.331841 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-558327-pvkqq"] Oct 30 02:19:53.343051 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:53.343007 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-558327-pvkqq"] Oct 30 02:19:53.586077 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:53.585992 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:53.923476 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:53.923438 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:54.542782 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:54.542736 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcbaf9cf-86a4-4eb9-86d7-263c84c928e8" path="/var/lib/kubelet/pods/bcbaf9cf-86a4-4eb9-86d7-263c84c928e8/volumes" Oct 30 02:19:55.174589 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:55.174545 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d5aa9-66754f6cd7-5sdmc" podUID="4d425439-27d7-485c-b992-199e1b587c60" containerName="payload-pre-processing" containerID="cri-o://21f6cf1389d0e4f46ab99ce6b4c2ee8aa44dbbe2ff83ed9716c1ed15648682ad" gracePeriod=25 Oct 30 02:19:55.255949 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:55.255905 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:56.329994 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.329948 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:56.502808 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.502765 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ed7ea96-xhnnm"] Oct 30 02:19:56.503245 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.503232 6572 cpu_manager.go:414] "containerMap: removing container" podUID="d32bbe46-7de7-4307-a62d-ee16d8b59741" containerName="istio-proxy" Oct 30 02:19:56.503534 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.503246 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d32bbe46-7de7-4307-a62d-ee16d8b59741" containerName="istio-proxy" Oct 30 02:19:56.503534 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.503258 6572 cpu_manager.go:414] "containerMap: removing container" podUID="322cef03-4f70-42c8-8571-42c939fba68b" containerName="istio-proxy" Oct 30 02:19:56.503534 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.503263 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="322cef03-4f70-42c8-8571-42c939fba68b" containerName="istio-proxy" Oct 30 02:19:56.503534 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.503274 6572 cpu_manager.go:414] "containerMap: removing container" podUID="e203d4b8-3f30-4d07-9c6f-431af65d9ec2" containerName="revoke-keys" Oct 30 02:19:56.503534 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.503280 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e203d4b8-3f30-4d07-9c6f-431af65d9ec2" containerName="revoke-keys" Oct 30 02:19:56.503534 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.503290 6572 cpu_manager.go:414] "containerMap: removing container" podUID="ed6145a7-d13c-4efa-ad6d-eaa6c5d90974" containerName="revoke-keys" Oct 30 02:19:56.503534 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.503296 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed6145a7-d13c-4efa-ad6d-eaa6c5d90974" containerName="revoke-keys" Oct 30 02:19:56.503534 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.503312 6572 cpu_manager.go:414] "containerMap: removing container" podUID="e07b1c7e-fe5d-4dff-b4aa-6d03ce285792" containerName="payload-pre-processing" Oct 30 02:19:56.503534 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.503318 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e07b1c7e-fe5d-4dff-b4aa-6d03ce285792" containerName="payload-pre-processing" Oct 30 02:19:56.503534 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.503334 6572 cpu_manager.go:414] "containerMap: removing container" podUID="9a8f46a4-ea68-466e-94ed-db3d9b5872de" containerName="revoke-keys" Oct 30 02:19:56.503534 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.503339 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a8f46a4-ea68-466e-94ed-db3d9b5872de" containerName="revoke-keys" Oct 30 02:19:56.503534 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.503371 6572 cpu_manager.go:414] "containerMap: removing container" podUID="94e9ec3c-fe81-4e11-906d-6a2dd69d0cee" containerName="revoke-keys" Oct 30 02:19:56.503534 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.503377 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94e9ec3c-fe81-4e11-906d-6a2dd69d0cee" containerName="revoke-keys" Oct 30 02:19:56.503534 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.503439 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="d32bbe46-7de7-4307-a62d-ee16d8b59741" containerName="istio-proxy" Oct 30 02:19:56.503534 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.503447 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="e07b1c7e-fe5d-4dff-b4aa-6d03ce285792" containerName="payload-pre-processing" Oct 30 02:19:56.503534 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.503458 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a8f46a4-ea68-466e-94ed-db3d9b5872de" containerName="revoke-keys" Oct 30 02:19:56.503534 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.503466 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="94e9ec3c-fe81-4e11-906d-6a2dd69d0cee" containerName="revoke-keys" Oct 30 02:19:56.503534 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.503474 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="322cef03-4f70-42c8-8571-42c939fba68b" containerName="istio-proxy" Oct 30 02:19:56.503534 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.503480 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="e203d4b8-3f30-4d07-9c6f-431af65d9ec2" containerName="revoke-keys" Oct 30 02:19:56.503534 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.503485 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed6145a7-d13c-4efa-ad6d-eaa6c5d90974" containerName="revoke-keys" Oct 30 02:19:56.503534 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.503490 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="322cef03-4f70-42c8-8571-42c939fba68b" containerName="istio-proxy" Oct 30 02:19:56.508149 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.508128 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed7ea96-xhnnm" Oct 30 02:19:56.525359 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.525333 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a4f6408-jc7bj"] Oct 30 02:19:56.526001 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.525974 6572 cpu_manager.go:414] "containerMap: removing container" podUID="d32bbe46-7de7-4307-a62d-ee16d8b59741" containerName="istio-proxy" Oct 30 02:19:56.526001 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.526000 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d32bbe46-7de7-4307-a62d-ee16d8b59741" containerName="istio-proxy" Oct 30 02:19:56.526001 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.526016 6572 cpu_manager.go:414] "containerMap: removing container" podUID="322cef03-4f70-42c8-8571-42c939fba68b" containerName="istio-proxy" Oct 30 02:19:56.526319 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.526025 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="322cef03-4f70-42c8-8571-42c939fba68b" containerName="istio-proxy" Oct 30 02:19:56.526319 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.526200 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="d32bbe46-7de7-4307-a62d-ee16d8b59741" containerName="istio-proxy" Oct 30 02:19:56.529679 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.529657 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a4f6408-jc7bj" Oct 30 02:19:56.533247 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.533218 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:19:56.533384 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.533260 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:19:56.536691 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.536668 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:19:56.536942 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.536797 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:19:56.538503 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.538478 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:56.548606 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.548579 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ed7ea96-xhnnm"] Oct 30 02:19:56.551765 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.551720 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a4f6408-jc7bj"] Oct 30 02:19:56.574949 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.574923 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e87a3a0-b554-423f-b1cf-ff9516ec89b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a4f6408-jc7bj\" (UID: \"8e87a3a0-b554-423f-b1cf-ff9516ec89b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a4f6408-jc7bj" Oct 30 02:19:56.575066 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.574993 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r69n6\" (UniqueName: \"kubernetes.io/projected/8e87a3a0-b554-423f-b1cf-ff9516ec89b6-kube-api-access-r69n6\") pod \"maas-api-revoke-sub-9a4f6408-jc7bj\" (UID: \"8e87a3a0-b554-423f-b1cf-ff9516ec89b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a4f6408-jc7bj" Oct 30 02:19:56.575120 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.575075 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/760b1b1c-26ad-4c65-a696-7f62a9917a93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ed7ea96-xhnnm\" (UID: \"760b1b1c-26ad-4c65-a696-7f62a9917a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed7ea96-xhnnm" Oct 30 02:19:56.575120 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.575113 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wjvt\" (UniqueName: \"kubernetes.io/projected/760b1b1c-26ad-4c65-a696-7f62a9917a93-kube-api-access-8wjvt\") pod \"maas-api-revoke-sub-9ed7ea96-xhnnm\" (UID: \"760b1b1c-26ad-4c65-a696-7f62a9917a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed7ea96-xhnnm" Oct 30 02:19:56.575206 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.575133 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e87a3a0-b554-423f-b1cf-ff9516ec89b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a4f6408-jc7bj\" (UID: \"8e87a3a0-b554-423f-b1cf-ff9516ec89b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a4f6408-jc7bj" Oct 30 02:19:56.575260 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.575226 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/760b1b1c-26ad-4c65-a696-7f62a9917a93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ed7ea96-xhnnm\" (UID: \"760b1b1c-26ad-4c65-a696-7f62a9917a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed7ea96-xhnnm" Oct 30 02:19:56.675760 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.675713 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/760b1b1c-26ad-4c65-a696-7f62a9917a93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ed7ea96-xhnnm\" (UID: \"760b1b1c-26ad-4c65-a696-7f62a9917a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed7ea96-xhnnm" Oct 30 02:19:56.675943 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.675781 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wjvt\" (UniqueName: \"kubernetes.io/projected/760b1b1c-26ad-4c65-a696-7f62a9917a93-kube-api-access-8wjvt\") pod \"maas-api-revoke-sub-9ed7ea96-xhnnm\" (UID: \"760b1b1c-26ad-4c65-a696-7f62a9917a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed7ea96-xhnnm" Oct 30 02:19:56.675943 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.675818 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e87a3a0-b554-423f-b1cf-ff9516ec89b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a4f6408-jc7bj\" (UID: \"8e87a3a0-b554-423f-b1cf-ff9516ec89b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a4f6408-jc7bj" Oct 30 02:19:56.676031 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.675962 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/760b1b1c-26ad-4c65-a696-7f62a9917a93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ed7ea96-xhnnm\" (UID: \"760b1b1c-26ad-4c65-a696-7f62a9917a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed7ea96-xhnnm" Oct 30 02:19:56.676071 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.676056 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e87a3a0-b554-423f-b1cf-ff9516ec89b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a4f6408-jc7bj\" (UID: \"8e87a3a0-b554-423f-b1cf-ff9516ec89b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a4f6408-jc7bj" Oct 30 02:19:56.676202 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.676177 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r69n6\" (UniqueName: \"kubernetes.io/projected/8e87a3a0-b554-423f-b1cf-ff9516ec89b6-kube-api-access-r69n6\") pod \"maas-api-revoke-sub-9a4f6408-jc7bj\" (UID: \"8e87a3a0-b554-423f-b1cf-ff9516ec89b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a4f6408-jc7bj" Oct 30 02:19:56.676603 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.676581 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/760b1b1c-26ad-4c65-a696-7f62a9917a93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ed7ea96-xhnnm\" (UID: \"760b1b1c-26ad-4c65-a696-7f62a9917a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed7ea96-xhnnm" Oct 30 02:19:56.676700 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.676659 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e87a3a0-b554-423f-b1cf-ff9516ec89b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a4f6408-jc7bj\" (UID: \"8e87a3a0-b554-423f-b1cf-ff9516ec89b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a4f6408-jc7bj" Oct 30 02:19:56.678520 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.678500 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e87a3a0-b554-423f-b1cf-ff9516ec89b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a4f6408-jc7bj\" (UID: \"8e87a3a0-b554-423f-b1cf-ff9516ec89b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a4f6408-jc7bj" Oct 30 02:19:56.678623 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.678550 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/760b1b1c-26ad-4c65-a696-7f62a9917a93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ed7ea96-xhnnm\" (UID: \"760b1b1c-26ad-4c65-a696-7f62a9917a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed7ea96-xhnnm" Oct 30 02:19:56.694793 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.694740 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r69n6\" (UniqueName: \"kubernetes.io/projected/8e87a3a0-b554-423f-b1cf-ff9516ec89b6-kube-api-access-r69n6\") pod \"maas-api-revoke-sub-9a4f6408-jc7bj\" (UID: \"8e87a3a0-b554-423f-b1cf-ff9516ec89b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a4f6408-jc7bj" Oct 30 02:19:56.694928 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.694779 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wjvt\" (UniqueName: \"kubernetes.io/projected/760b1b1c-26ad-4c65-a696-7f62a9917a93-kube-api-access-8wjvt\") pod \"maas-api-revoke-sub-9ed7ea96-xhnnm\" (UID: \"760b1b1c-26ad-4c65-a696-7f62a9917a93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed7ea96-xhnnm" Oct 30 02:19:56.822260 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.822223 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed7ea96-xhnnm" Oct 30 02:19:56.846468 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.846443 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a4f6408-jc7bj" Oct 30 02:19:56.988724 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:56.988621 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:57.123789 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:57.123736 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ed7ea96-xhnnm"] Oct 30 02:19:57.132531 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:19:57.132502 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod760b1b1c_26ad_4c65_a696_7f62a9917a93.slice/crio-6739a1f59a4e649b109a11b2dcf667653d52eb12b035bbffb2e3aa0ee7e8bb47 WatchSource:0}: Error finding container 6739a1f59a4e649b109a11b2dcf667653d52eb12b035bbffb2e3aa0ee7e8bb47: Status 404 returned error can't find the container with id 6739a1f59a4e649b109a11b2dcf667653d52eb12b035bbffb2e3aa0ee7e8bb47 Oct 30 02:19:57.166191 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:57.166166 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a4f6408-jc7bj"] Oct 30 02:19:57.167103 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:19:57.167078 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e87a3a0_b554_423f_b1cf_ff9516ec89b6.slice/crio-2e11bd4568c2c0aaa33c0baeb844bc3196547b83a2fb29d49028f3bd7a29fb9b WatchSource:0}: Error finding container 2e11bd4568c2c0aaa33c0baeb844bc3196547b83a2fb29d49028f3bd7a29fb9b: Status 404 returned error can't find the container with id 2e11bd4568c2c0aaa33c0baeb844bc3196547b83a2fb29d49028f3bd7a29fb9b Oct 30 02:19:57.697596 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:57.697561 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed7ea96-xhnnm" event={"ID":"760b1b1c-26ad-4c65-a696-7f62a9917a93","Type":"ContainerStarted","Data":"13e21e15e29eba5295e071479961b02fa38bb24e1d9664561928ea572ce5b3e1"} Oct 30 02:19:57.698036 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:57.697651 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed7ea96-xhnnm" event={"ID":"760b1b1c-26ad-4c65-a696-7f62a9917a93","Type":"ContainerStarted","Data":"6739a1f59a4e649b109a11b2dcf667653d52eb12b035bbffb2e3aa0ee7e8bb47"} Oct 30 02:19:57.699277 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:57.699250 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a4f6408-jc7bj" event={"ID":"8e87a3a0-b554-423f-b1cf-ff9516ec89b6","Type":"ContainerStarted","Data":"77535d53777c7215fd9d3a9bc6f43b10601c9805670cd2f4cd968c271af3620b"} Oct 30 02:19:57.699373 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:57.699302 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a4f6408-jc7bj" event={"ID":"8e87a3a0-b554-423f-b1cf-ff9516ec89b6","Type":"ContainerStarted","Data":"2e11bd4568c2c0aaa33c0baeb844bc3196547b83a2fb29d49028f3bd7a29fb9b"} Oct 30 02:19:57.735869 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:57.735821 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed7ea96-xhnnm" podStartSLOduration=1.735807085 podStartE2EDuration="1.735807085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:19:57.733481353 +0000 UTC m=+3723.309906754" watchObservedRunningTime="2026-10-30 02:19:57.735807085 +0000 UTC m=+3723.312232487" Oct 30 02:19:57.770999 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:57.770946 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a4f6408-jc7bj" podStartSLOduration=1.770932115 podStartE2EDuration="1.770932115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:19:57.7694001 +0000 UTC m=+3723.345825502" watchObservedRunningTime="2026-10-30 02:19:57.770932115 +0000 UTC m=+3723.347357517" Oct 30 02:19:58.477888 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:58.477855 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:58.680389 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:58.680351 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:58.710881 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:58.710849 6572 generic.go:356] "Generic (PLEG): container finished" podID="760b1b1c-26ad-4c65-a696-7f62a9917a93" containerID="13e21e15e29eba5295e071479961b02fa38bb24e1d9664561928ea572ce5b3e1" exitCode=0 Oct 30 02:19:58.711318 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:58.710939 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed7ea96-xhnnm" event={"ID":"760b1b1c-26ad-4c65-a696-7f62a9917a93","Type":"ContainerDied","Data":"13e21e15e29eba5295e071479961b02fa38bb24e1d9664561928ea572ce5b3e1"} Oct 30 02:19:58.712795 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:58.712736 6572 generic.go:356] "Generic (PLEG): container finished" podID="8e87a3a0-b554-423f-b1cf-ff9516ec89b6" containerID="77535d53777c7215fd9d3a9bc6f43b10601c9805670cd2f4cd968c271af3620b" exitCode=0 Oct 30 02:19:58.712928 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:58.712792 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a4f6408-jc7bj" event={"ID":"8e87a3a0-b554-423f-b1cf-ff9516ec89b6","Type":"ContainerDied","Data":"77535d53777c7215fd9d3a9bc6f43b10601c9805670cd2f4cd968c271af3620b"} Oct 30 02:19:58.788268 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:58.788175 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" podUID="cb20484a-51e2-4581-a629-e92d4234784e" containerName="main" containerID="cri-o://5d440c306c4be1f8e0b17057f73633f6d604c476fff8d7ce9ece155aee9c0002" gracePeriod=45 Oct 30 02:19:58.940037 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:58.940005 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:58.958050 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:58.958007 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d5aa9-l87rv"] Oct 30 02:19:58.962471 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:58.962444 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d5aa9-l87rv"] Oct 30 02:19:59.088549 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.088525 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" Oct 30 02:19:59.101065 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.101037 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb20484a-51e2-4581-a629-e92d4234784e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb20484a-51e2-4581-a629-e92d4234784e-tmp-dir\") pod \"cb20484a-51e2-4581-a629-e92d4234784e\" (UID: \"cb20484a-51e2-4581-a629-e92d4234784e\") " Oct 30 02:19:59.101226 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.101146 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb20484a-51e2-4581-a629-e92d4234784e-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb20484a-51e2-4581-a629-e92d4234784e-tls-certs\") pod \"cb20484a-51e2-4581-a629-e92d4234784e\" (UID: \"cb20484a-51e2-4581-a629-e92d4234784e\") " Oct 30 02:19:59.101226 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.101220 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb20484a-51e2-4581-a629-e92d4234784e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb20484a-51e2-4581-a629-e92d4234784e-dshm\") pod \"cb20484a-51e2-4581-a629-e92d4234784e\" (UID: \"cb20484a-51e2-4581-a629-e92d4234784e\") " Oct 30 02:19:59.101369 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.101249 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb20484a-51e2-4581-a629-e92d4234784e-kube-api-access-qsdxs\" (UniqueName: \"kubernetes.io/projected/cb20484a-51e2-4581-a629-e92d4234784e-kube-api-access-qsdxs\") pod \"cb20484a-51e2-4581-a629-e92d4234784e\" (UID: \"cb20484a-51e2-4581-a629-e92d4234784e\") " Oct 30 02:19:59.101369 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.101298 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb20484a-51e2-4581-a629-e92d4234784e-tmp-dir" pod "cb20484a-51e2-4581-a629-e92d4234784e" (UID: "cb20484a-51e2-4581-a629-e92d4234784e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:59.101369 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.101307 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb20484a-51e2-4581-a629-e92d4234784e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb20484a-51e2-4581-a629-e92d4234784e-model-cache\") pod \"cb20484a-51e2-4581-a629-e92d4234784e\" (UID: \"cb20484a-51e2-4581-a629-e92d4234784e\") " Oct 30 02:19:59.101369 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.101355 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb20484a-51e2-4581-a629-e92d4234784e-home\" (UniqueName: \"kubernetes.io/empty-dir/cb20484a-51e2-4581-a629-e92d4234784e-home\") pod \"cb20484a-51e2-4581-a629-e92d4234784e\" (UID: \"cb20484a-51e2-4581-a629-e92d4234784e\") " Oct 30 02:19:59.101609 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.101571 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb20484a-51e2-4581-a629-e92d4234784e-model-cache" pod "cb20484a-51e2-4581-a629-e92d4234784e" (UID: "cb20484a-51e2-4581-a629-e92d4234784e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:59.101677 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.101648 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb20484a-51e2-4581-a629-e92d4234784e-home" pod "cb20484a-51e2-4581-a629-e92d4234784e" (UID: "cb20484a-51e2-4581-a629-e92d4234784e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:59.101806 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.101786 6572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb20484a-51e2-4581-a629-e92d4234784e-model-cache\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:59.101806 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.101804 6572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb20484a-51e2-4581-a629-e92d4234784e-home\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:59.101958 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.101813 6572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb20484a-51e2-4581-a629-e92d4234784e-tmp-dir\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:59.103967 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.103725 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb20484a-51e2-4581-a629-e92d4234784e-tls-certs" pod "cb20484a-51e2-4581-a629-e92d4234784e" (UID: "cb20484a-51e2-4581-a629-e92d4234784e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:19:59.103967 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.103923 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb20484a-51e2-4581-a629-e92d4234784e-dshm" pod "cb20484a-51e2-4581-a629-e92d4234784e" (UID: "cb20484a-51e2-4581-a629-e92d4234784e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:59.104112 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.104044 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb20484a-51e2-4581-a629-e92d4234784e-kube-api-access-qsdxs" pod "cb20484a-51e2-4581-a629-e92d4234784e" (UID: "cb20484a-51e2-4581-a629-e92d4234784e"). InnerVolumeSpecName "kube-api-access-qsdxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:59.123848 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.123815 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:59.203128 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.203081 6572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb20484a-51e2-4581-a629-e92d4234784e-tls-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:59.203128 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.203127 6572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb20484a-51e2-4581-a629-e92d4234784e-dshm\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:59.203360 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.203142 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsdxs\" (UniqueName: \"kubernetes.io/projected/cb20484a-51e2-4581-a629-e92d4234784e-kube-api-access-qsdxs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:19:59.673696 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.673665 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:59.722600 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.722562 6572 generic.go:356] "Generic (PLEG): container finished" podID="cb20484a-51e2-4581-a629-e92d4234784e" containerID="5d440c306c4be1f8e0b17057f73633f6d604c476fff8d7ce9ece155aee9c0002" exitCode=0 Oct 30 02:19:59.723096 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.722615 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" Oct 30 02:19:59.723096 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.722645 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" event={"ID":"cb20484a-51e2-4581-a629-e92d4234784e","Type":"ContainerDied","Data":"5d440c306c4be1f8e0b17057f73633f6d604c476fff8d7ce9ece155aee9c0002"} Oct 30 02:19:59.723096 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.722692 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" event={"ID":"cb20484a-51e2-4581-a629-e92d4234784e","Type":"ContainerDied","Data":"0c61535a350120f9e199f9754a36e49b045e6fddcf6500711e473d15582fc01b"} Oct 30 02:19:59.723096 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.722713 6572 scope.go:122] "RemoveContainer" containerID="5d440c306c4be1f8e0b17057f73633f6d604c476fff8d7ce9ece155aee9c0002" Oct 30 02:19:59.736669 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.736639 6572 scope.go:122] "RemoveContainer" containerID="5d440c306c4be1f8e0b17057f73633f6d604c476fff8d7ce9ece155aee9c0002" Oct 30 02:19:59.737586 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:19:59.737557 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d440c306c4be1f8e0b17057f73633f6d604c476fff8d7ce9ece155aee9c0002\": container with ID starting with 5d440c306c4be1f8e0b17057f73633f6d604c476fff8d7ce9ece155aee9c0002 not found: ID does not exist" containerID="5d440c306c4be1f8e0b17057f73633f6d604c476fff8d7ce9ece155aee9c0002" Oct 30 02:19:59.737694 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.737595 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d440c306c4be1f8e0b17057f73633f6d604c476fff8d7ce9ece155aee9c0002"} err="failed to get container status \"5d440c306c4be1f8e0b17057f73633f6d604c476fff8d7ce9ece155aee9c0002\": rpc error: code = NotFound desc = could not find container \"5d440c306c4be1f8e0b17057f73633f6d604c476fff8d7ce9ece155aee9c0002\": container with ID starting with 5d440c306c4be1f8e0b17057f73633f6d604c476fff8d7ce9ece155aee9c0002 not found: ID does not exist" Oct 30 02:19:59.772889 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.772861 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp"] Oct 30 02:19:59.786853 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.786811 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp"] Oct 30 02:19:59.851559 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.851525 6572 scope.go:122] "RemoveContainer" containerID="b53127a074daf4425c03a11a189306505512b1a062565ff01956a68ac0d67a27" Oct 30 02:19:59.866526 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.866486 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed7ea96-xhnnm" Oct 30 02:19:59.867045 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.867021 6572 scope.go:122] "RemoveContainer" containerID="cf54ded7227dff6130c92f852b027353242aa3c2b699afaba1627e1ec2dd1c6a" Oct 30 02:19:59.909502 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.909476 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/760b1b1c-26ad-4c65-a696-7f62a9917a93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/760b1b1c-26ad-4c65-a696-7f62a9917a93-maas-api-xks-ca\") pod \"760b1b1c-26ad-4c65-a696-7f62a9917a93\" (UID: \"760b1b1c-26ad-4c65-a696-7f62a9917a93\") " Oct 30 02:19:59.909991 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.909625 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/760b1b1c-26ad-4c65-a696-7f62a9917a93-kube-api-access-8wjvt\" (UniqueName: \"kubernetes.io/projected/760b1b1c-26ad-4c65-a696-7f62a9917a93-kube-api-access-8wjvt\") pod \"760b1b1c-26ad-4c65-a696-7f62a9917a93\" (UID: \"760b1b1c-26ad-4c65-a696-7f62a9917a93\") " Oct 30 02:19:59.909991 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.909683 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/760b1b1c-26ad-4c65-a696-7f62a9917a93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/760b1b1c-26ad-4c65-a696-7f62a9917a93-maas-api-openshift-ca\") pod \"760b1b1c-26ad-4c65-a696-7f62a9917a93\" (UID: \"760b1b1c-26ad-4c65-a696-7f62a9917a93\") " Oct 30 02:19:59.910486 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.910459 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/760b1b1c-26ad-4c65-a696-7f62a9917a93-maas-api-openshift-ca" pod "760b1b1c-26ad-4c65-a696-7f62a9917a93" (UID: "760b1b1c-26ad-4c65-a696-7f62a9917a93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:59.915907 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.912535 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/760b1b1c-26ad-4c65-a696-7f62a9917a93-kube-api-access-8wjvt" pod "760b1b1c-26ad-4c65-a696-7f62a9917a93" (UID: "760b1b1c-26ad-4c65-a696-7f62a9917a93"). InnerVolumeSpecName "kube-api-access-8wjvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:59.915907 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.912577 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/760b1b1c-26ad-4c65-a696-7f62a9917a93-maas-api-xks-ca" pod "760b1b1c-26ad-4c65-a696-7f62a9917a93" (UID: "760b1b1c-26ad-4c65-a696-7f62a9917a93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:19:59.918268 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.917848 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a4f6408-jc7bj" Oct 30 02:19:59.970690 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.970648 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d5aa9-66754f6cd7-5sdmc" podUID="4d425439-27d7-485c-b992-199e1b587c60" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:19:59.973261 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:19:59.972899 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-w0-a3a2ac-kserve-5745f4f794-r9qtp" podUID="cb20484a-51e2-4581-a629-e92d4234784e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:20:00.010736 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:00.010700 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e87a3a0-b554-423f-b1cf-ff9516ec89b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e87a3a0-b554-423f-b1cf-ff9516ec89b6-maas-api-xks-ca\") pod \"8e87a3a0-b554-423f-b1cf-ff9516ec89b6\" (UID: \"8e87a3a0-b554-423f-b1cf-ff9516ec89b6\") " Oct 30 02:20:00.010980 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:00.010829 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e87a3a0-b554-423f-b1cf-ff9516ec89b6-kube-api-access-r69n6\" (UniqueName: \"kubernetes.io/projected/8e87a3a0-b554-423f-b1cf-ff9516ec89b6-kube-api-access-r69n6\") pod \"8e87a3a0-b554-423f-b1cf-ff9516ec89b6\" (UID: \"8e87a3a0-b554-423f-b1cf-ff9516ec89b6\") " Oct 30 02:20:00.010980 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:00.010900 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e87a3a0-b554-423f-b1cf-ff9516ec89b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e87a3a0-b554-423f-b1cf-ff9516ec89b6-maas-api-openshift-ca\") pod \"8e87a3a0-b554-423f-b1cf-ff9516ec89b6\" (UID: \"8e87a3a0-b554-423f-b1cf-ff9516ec89b6\") " Oct 30 02:20:00.011258 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:00.011233 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/760b1b1c-26ad-4c65-a696-7f62a9917a93-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:00.011363 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:00.011262 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wjvt\" (UniqueName: \"kubernetes.io/projected/760b1b1c-26ad-4c65-a696-7f62a9917a93-kube-api-access-8wjvt\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:00.011363 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:00.011261 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e87a3a0-b554-423f-b1cf-ff9516ec89b6-maas-api-openshift-ca" pod "8e87a3a0-b554-423f-b1cf-ff9516ec89b6" (UID: "8e87a3a0-b554-423f-b1cf-ff9516ec89b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:20:00.011363 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:00.011276 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/760b1b1c-26ad-4c65-a696-7f62a9917a93-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:00.012955 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:00.012932 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e87a3a0-b554-423f-b1cf-ff9516ec89b6-kube-api-access-r69n6" pod "8e87a3a0-b554-423f-b1cf-ff9516ec89b6" (UID: "8e87a3a0-b554-423f-b1cf-ff9516ec89b6"). InnerVolumeSpecName "kube-api-access-r69n6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:00.013037 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:00.012935 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e87a3a0-b554-423f-b1cf-ff9516ec89b6-maas-api-xks-ca" pod "8e87a3a0-b554-423f-b1cf-ff9516ec89b6" (UID: "8e87a3a0-b554-423f-b1cf-ff9516ec89b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:20:00.112357 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:00.112323 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r69n6\" (UniqueName: \"kubernetes.io/projected/8e87a3a0-b554-423f-b1cf-ff9516ec89b6-kube-api-access-r69n6\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:00.112357 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:00.112352 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e87a3a0-b554-423f-b1cf-ff9516ec89b6-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:00.112357 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:00.112362 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e87a3a0-b554-423f-b1cf-ff9516ec89b6-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:00.549095 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:00.549051 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a8f46a4-ea68-466e-94ed-db3d9b5872de" path="/var/lib/kubelet/pods/9a8f46a4-ea68-466e-94ed-db3d9b5872de/volumes" Oct 30 02:20:00.549583 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:00.549560 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb20484a-51e2-4581-a629-e92d4234784e" path="/var/lib/kubelet/pods/cb20484a-51e2-4581-a629-e92d4234784e/volumes" Oct 30 02:20:00.727209 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:00.727173 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:00.731889 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:00.731867 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed7ea96-xhnnm" Oct 30 02:20:00.732023 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:00.731908 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed7ea96-xhnnm" event={"ID":"760b1b1c-26ad-4c65-a696-7f62a9917a93","Type":"ContainerDied","Data":"6739a1f59a4e649b109a11b2dcf667653d52eb12b035bbffb2e3aa0ee7e8bb47"} Oct 30 02:20:00.732023 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:00.731967 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6739a1f59a4e649b109a11b2dcf667653d52eb12b035bbffb2e3aa0ee7e8bb47" Oct 30 02:20:00.733738 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:00.733716 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a4f6408-jc7bj" Oct 30 02:20:00.733935 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:00.733775 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a4f6408-jc7bj" event={"ID":"8e87a3a0-b554-423f-b1cf-ff9516ec89b6","Type":"ContainerDied","Data":"2e11bd4568c2c0aaa33c0baeb844bc3196547b83a2fb29d49028f3bd7a29fb9b"} Oct 30 02:20:00.733935 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:00.733822 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e11bd4568c2c0aaa33c0baeb844bc3196547b83a2fb29d49028f3bd7a29fb9b" Oct 30 02:20:00.945854 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:00.945821 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:01.139576 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:01.139492 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:02.482178 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:02.482143 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:03.982200 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:03.982168 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:04.160485 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:04.160452 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:05.674663 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:05.674623 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:06.209914 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:06.209876 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:06.416919 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:06.414730 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:06.437125 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:06.437083 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg"] Oct 30 02:20:06.437406 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:06.437378 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" podUID="0e662617-4bbb-4021-8bae-ba979960bb94" containerName="istio-proxy" containerID="cri-o://487304269ac77b713e363b4327e1c9b8dc649bba358bcb82b512c4e79430b6c5" gracePeriod=30 Oct 30 02:20:06.606645 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:06.606555 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:07.062292 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:07.062247 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-92f0cc-wpl9h"] Oct 30 02:20:07.062853 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:07.062720 6572 cpu_manager.go:414] "containerMap: removing container" podUID="8e87a3a0-b554-423f-b1cf-ff9516ec89b6" containerName="revoke-keys" Oct 30 02:20:07.062853 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:07.062734 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e87a3a0-b554-423f-b1cf-ff9516ec89b6" containerName="revoke-keys" Oct 30 02:20:07.062853 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:07.062772 6572 cpu_manager.go:414] "containerMap: removing container" podUID="760b1b1c-26ad-4c65-a696-7f62a9917a93" containerName="revoke-keys" Oct 30 02:20:07.062853 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:07.062779 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="760b1b1c-26ad-4c65-a696-7f62a9917a93" containerName="revoke-keys" Oct 30 02:20:07.062853 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:07.062788 6572 cpu_manager.go:414] "containerMap: removing container" podUID="cb20484a-51e2-4581-a629-e92d4234784e" containerName="main" Oct 30 02:20:07.062853 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:07.062793 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb20484a-51e2-4581-a629-e92d4234784e" containerName="main" Oct 30 02:20:07.063229 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:07.062879 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e87a3a0-b554-423f-b1cf-ff9516ec89b6" containerName="revoke-keys" Oct 30 02:20:07.063229 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:07.062888 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="760b1b1c-26ad-4c65-a696-7f62a9917a93" containerName="revoke-keys" Oct 30 02:20:07.063229 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:07.062894 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb20484a-51e2-4581-a629-e92d4234784e" containerName="main" Oct 30 02:20:07.067989 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:07.067962 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-92f0cc-wpl9h" Oct 30 02:20:07.087733 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:07.087706 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:20:07.087930 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:07.087800 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:20:07.089031 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:07.089007 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-92f0cc-wpl9h"] Oct 30 02:20:07.141597 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:07.141562 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:20:07.141802 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:07.141629 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:20:07.201769 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:07.200580 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:07.201769 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:07.201331 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3012f80-f0c3-4168-a742-1c92cdb07af7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-92f0cc-wpl9h\" (UID: \"e3012f80-f0c3-4168-a742-1c92cdb07af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-92f0cc-wpl9h" Oct 30 02:20:07.201769 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:07.201408 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3012f80-f0c3-4168-a742-1c92cdb07af7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-92f0cc-wpl9h\" (UID: \"e3012f80-f0c3-4168-a742-1c92cdb07af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-92f0cc-wpl9h" Oct 30 02:20:07.201769 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:07.201488 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gtqs\" (UniqueName: \"kubernetes.io/projected/e3012f80-f0c3-4168-a742-1c92cdb07af7-kube-api-access-8gtqs\") pod \"maas-api-revoke-keys-e2e-extmod-a-92f0cc-wpl9h\" (UID: \"e3012f80-f0c3-4168-a742-1c92cdb07af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-92f0cc-wpl9h" Oct 30 02:20:07.319825 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:07.318834 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3012f80-f0c3-4168-a742-1c92cdb07af7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-92f0cc-wpl9h\" (UID: \"e3012f80-f0c3-4168-a742-1c92cdb07af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-92f0cc-wpl9h" Oct 30 02:20:07.319825 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:07.318910 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3012f80-f0c3-4168-a742-1c92cdb07af7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-92f0cc-wpl9h\" (UID: \"e3012f80-f0c3-4168-a742-1c92cdb07af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-92f0cc-wpl9h" Oct 30 02:20:07.319825 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:07.318982 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gtqs\" (UniqueName: \"kubernetes.io/projected/e3012f80-f0c3-4168-a742-1c92cdb07af7-kube-api-access-8gtqs\") pod \"maas-api-revoke-keys-e2e-extmod-a-92f0cc-wpl9h\" (UID: \"e3012f80-f0c3-4168-a742-1c92cdb07af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-92f0cc-wpl9h" Oct 30 02:20:07.353767 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:07.320739 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3012f80-f0c3-4168-a742-1c92cdb07af7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-92f0cc-wpl9h\" (UID: \"e3012f80-f0c3-4168-a742-1c92cdb07af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-92f0cc-wpl9h" Oct 30 02:20:07.353767 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:07.326464 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3012f80-f0c3-4168-a742-1c92cdb07af7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-92f0cc-wpl9h\" (UID: \"e3012f80-f0c3-4168-a742-1c92cdb07af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-92f0cc-wpl9h" Oct 30 02:20:07.401221 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:07.401136 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gtqs\" (UniqueName: \"kubernetes.io/projected/e3012f80-f0c3-4168-a742-1c92cdb07af7-kube-api-access-8gtqs\") pod \"maas-api-revoke-keys-e2e-extmod-a-92f0cc-wpl9h\" (UID: \"e3012f80-f0c3-4168-a742-1c92cdb07af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-92f0cc-wpl9h" Oct 30 02:20:07.682475 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:07.682433 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-92f0cc-wpl9h" Oct 30 02:20:07.777146 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:07.777094 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:07.817861 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:07.817823 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a3a2ac-7kn8z"] Oct 30 02:20:07.822886 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:07.822852 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a3a2ac-7kn8z"] Oct 30 02:20:07.969201 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:07.969109 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:07.995556 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:07.995520 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-92f0cc-wpl9h"] Oct 30 02:20:07.997082 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:20:07.997054 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3012f80_f0c3_4168_a742_1c92cdb07af7.slice/crio-0377173aaa1d620ef7de4b16b412d7fb201c66b342ae1bf7d5aa7c66c6429781 WatchSource:0}: Error finding container 0377173aaa1d620ef7de4b16b412d7fb201c66b342ae1bf7d5aa7c66c6429781: Status 404 returned error can't find the container with id 0377173aaa1d620ef7de4b16b412d7fb201c66b342ae1bf7d5aa7c66c6429781 Oct 30 02:20:08.399723 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:08.399687 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:08.541721 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:08.541687 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94e9ec3c-fe81-4e11-906d-6a2dd69d0cee" path="/var/lib/kubelet/pods/94e9ec3c-fe81-4e11-906d-6a2dd69d0cee/volumes" Oct 30 02:20:08.796732 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:08.796695 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-92f0cc-wpl9h" event={"ID":"e3012f80-f0c3-4168-a742-1c92cdb07af7","Type":"ContainerStarted","Data":"a926bb8c69b9d11ed9a8bf96253e5e54c19d63ca26a0c1fd50aeadde9fc531bf"} Oct 30 02:20:08.796924 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:08.796775 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-92f0cc-wpl9h" event={"ID":"e3012f80-f0c3-4168-a742-1c92cdb07af7","Type":"ContainerStarted","Data":"0377173aaa1d620ef7de4b16b412d7fb201c66b342ae1bf7d5aa7c66c6429781"} Oct 30 02:20:08.837628 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:08.837567 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-92f0cc-wpl9h" podStartSLOduration=1.8375492919999998 podStartE2EDuration="1.837549292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:20:08.833606549 +0000 UTC m=+3734.410031952" watchObservedRunningTime="2026-10-30 02:20:08.837549292 +0000 UTC m=+3734.413974704" Oct 30 02:20:09.469928 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:09.469893 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:09.508305 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:09.508150 6572 patch_prober.go:29] interesting pod/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:20:09.508305 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:09.508211 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" podUID="0e662617-4bbb-4021-8bae-ba979960bb94" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:20:09.681119 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:09.681074 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:09.806465 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:09.806382 6572 generic.go:356] "Generic (PLEG): container finished" podID="e3012f80-f0c3-4168-a742-1c92cdb07af7" containerID="a926bb8c69b9d11ed9a8bf96253e5e54c19d63ca26a0c1fd50aeadde9fc531bf" exitCode=0 Oct 30 02:20:09.806642 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:09.806470 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-92f0cc-wpl9h" event={"ID":"e3012f80-f0c3-4168-a742-1c92cdb07af7","Type":"ContainerDied","Data":"a926bb8c69b9d11ed9a8bf96253e5e54c19d63ca26a0c1fd50aeadde9fc531bf"} Oct 30 02:20:09.968109 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:09.968068 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d5aa9-66754f6cd7-5sdmc" podUID="4d425439-27d7-485c-b992-199e1b587c60" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:20:10.952315 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:10.952287 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-92f0cc-wpl9h" Oct 30 02:20:11.078291 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.078202 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3012f80-f0c3-4168-a742-1c92cdb07af7-kube-api-access-8gtqs\" (UniqueName: \"kubernetes.io/projected/e3012f80-f0c3-4168-a742-1c92cdb07af7-kube-api-access-8gtqs\") pod \"e3012f80-f0c3-4168-a742-1c92cdb07af7\" (UID: \"e3012f80-f0c3-4168-a742-1c92cdb07af7\") " Oct 30 02:20:11.078291 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.078274 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3012f80-f0c3-4168-a742-1c92cdb07af7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3012f80-f0c3-4168-a742-1c92cdb07af7-maas-api-openshift-ca\") pod \"e3012f80-f0c3-4168-a742-1c92cdb07af7\" (UID: \"e3012f80-f0c3-4168-a742-1c92cdb07af7\") " Oct 30 02:20:11.078541 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.078347 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3012f80-f0c3-4168-a742-1c92cdb07af7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3012f80-f0c3-4168-a742-1c92cdb07af7-maas-api-xks-ca\") pod \"e3012f80-f0c3-4168-a742-1c92cdb07af7\" (UID: \"e3012f80-f0c3-4168-a742-1c92cdb07af7\") " Oct 30 02:20:11.078668 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.078640 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3012f80-f0c3-4168-a742-1c92cdb07af7-maas-api-openshift-ca" pod "e3012f80-f0c3-4168-a742-1c92cdb07af7" (UID: "e3012f80-f0c3-4168-a742-1c92cdb07af7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:20:11.080377 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.080357 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3012f80-f0c3-4168-a742-1c92cdb07af7-kube-api-access-8gtqs" pod "e3012f80-f0c3-4168-a742-1c92cdb07af7" (UID: "e3012f80-f0c3-4168-a742-1c92cdb07af7"). InnerVolumeSpecName "kube-api-access-8gtqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:11.080456 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.080426 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3012f80-f0c3-4168-a742-1c92cdb07af7-maas-api-xks-ca" pod "e3012f80-f0c3-4168-a742-1c92cdb07af7" (UID: "e3012f80-f0c3-4168-a742-1c92cdb07af7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:20:11.179790 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.179742 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gtqs\" (UniqueName: \"kubernetes.io/projected/e3012f80-f0c3-4168-a742-1c92cdb07af7-kube-api-access-8gtqs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:11.179790 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.179789 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3012f80-f0c3-4168-a742-1c92cdb07af7-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:11.179973 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.179799 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3012f80-f0c3-4168-a742-1c92cdb07af7-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:11.281247 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.281212 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:11.707386 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.707313 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:20:11.785723 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.785677 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e662617-4bbb-4021-8bae-ba979960bb94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e662617-4bbb-4021-8bae-ba979960bb94-workload-certs\") pod \"0e662617-4bbb-4021-8bae-ba979960bb94\" (UID: \"0e662617-4bbb-4021-8bae-ba979960bb94\") " Oct 30 02:20:11.785950 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.785793 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e662617-4bbb-4021-8bae-ba979960bb94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e662617-4bbb-4021-8bae-ba979960bb94-workload-socket\") pod \"0e662617-4bbb-4021-8bae-ba979960bb94\" (UID: \"0e662617-4bbb-4021-8bae-ba979960bb94\") " Oct 30 02:20:11.785950 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.785856 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e662617-4bbb-4021-8bae-ba979960bb94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e662617-4bbb-4021-8bae-ba979960bb94-istio-data\") pod \"0e662617-4bbb-4021-8bae-ba979960bb94\" (UID: \"0e662617-4bbb-4021-8bae-ba979960bb94\") " Oct 30 02:20:11.785950 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.785931 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e662617-4bbb-4021-8bae-ba979960bb94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e662617-4bbb-4021-8bae-ba979960bb94-credential-socket\") pod \"0e662617-4bbb-4021-8bae-ba979960bb94\" (UID: \"0e662617-4bbb-4021-8bae-ba979960bb94\") " Oct 30 02:20:11.786115 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.785960 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e662617-4bbb-4021-8bae-ba979960bb94-istio-token\" (UniqueName: \"kubernetes.io/projected/0e662617-4bbb-4021-8bae-ba979960bb94-istio-token\") pod \"0e662617-4bbb-4021-8bae-ba979960bb94\" (UID: \"0e662617-4bbb-4021-8bae-ba979960bb94\") " Oct 30 02:20:11.786115 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.786003 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e662617-4bbb-4021-8bae-ba979960bb94-workload-certs" pod "0e662617-4bbb-4021-8bae-ba979960bb94" (UID: "0e662617-4bbb-4021-8bae-ba979960bb94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:20:11.786115 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.786037 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e662617-4bbb-4021-8bae-ba979960bb94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e662617-4bbb-4021-8bae-ba979960bb94-istio-envoy\") pod \"0e662617-4bbb-4021-8bae-ba979960bb94\" (UID: \"0e662617-4bbb-4021-8bae-ba979960bb94\") " Oct 30 02:20:11.786115 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.786066 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e662617-4bbb-4021-8bae-ba979960bb94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e662617-4bbb-4021-8bae-ba979960bb94-istio-podinfo\") pod \"0e662617-4bbb-4021-8bae-ba979960bb94\" (UID: \"0e662617-4bbb-4021-8bae-ba979960bb94\") " Oct 30 02:20:11.786115 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.786100 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e662617-4bbb-4021-8bae-ba979960bb94-kube-api-access-psld8\" (UniqueName: \"kubernetes.io/projected/0e662617-4bbb-4021-8bae-ba979960bb94-kube-api-access-psld8\") pod \"0e662617-4bbb-4021-8bae-ba979960bb94\" (UID: \"0e662617-4bbb-4021-8bae-ba979960bb94\") " Oct 30 02:20:11.786356 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.786132 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e662617-4bbb-4021-8bae-ba979960bb94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e662617-4bbb-4021-8bae-ba979960bb94-istiod-ca-cert\") pod \"0e662617-4bbb-4021-8bae-ba979960bb94\" (UID: \"0e662617-4bbb-4021-8bae-ba979960bb94\") " Oct 30 02:20:11.786356 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.786169 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e662617-4bbb-4021-8bae-ba979960bb94-workload-socket" pod "0e662617-4bbb-4021-8bae-ba979960bb94" (UID: "0e662617-4bbb-4021-8bae-ba979960bb94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:20:11.786356 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.786267 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e662617-4bbb-4021-8bae-ba979960bb94-istio-data" pod "0e662617-4bbb-4021-8bae-ba979960bb94" (UID: "0e662617-4bbb-4021-8bae-ba979960bb94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:20:11.786543 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.786521 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e662617-4bbb-4021-8bae-ba979960bb94-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:11.786543 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.786544 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e662617-4bbb-4021-8bae-ba979960bb94-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:11.786735 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.786557 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e662617-4bbb-4021-8bae-ba979960bb94-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:11.786824 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.786790 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e662617-4bbb-4021-8bae-ba979960bb94-credential-socket" pod "0e662617-4bbb-4021-8bae-ba979960bb94" (UID: "0e662617-4bbb-4021-8bae-ba979960bb94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:20:11.787867 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.787838 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e662617-4bbb-4021-8bae-ba979960bb94-istiod-ca-cert" pod "0e662617-4bbb-4021-8bae-ba979960bb94" (UID: "0e662617-4bbb-4021-8bae-ba979960bb94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:20:11.789298 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.789270 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e662617-4bbb-4021-8bae-ba979960bb94-istio-envoy" pod "0e662617-4bbb-4021-8bae-ba979960bb94" (UID: "0e662617-4bbb-4021-8bae-ba979960bb94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:20:11.789403 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.789278 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e662617-4bbb-4021-8bae-ba979960bb94-istio-podinfo" pod "0e662617-4bbb-4021-8bae-ba979960bb94" (UID: "0e662617-4bbb-4021-8bae-ba979960bb94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:20:11.789403 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.789321 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e662617-4bbb-4021-8bae-ba979960bb94-kube-api-access-psld8" pod "0e662617-4bbb-4021-8bae-ba979960bb94" (UID: "0e662617-4bbb-4021-8bae-ba979960bb94"). InnerVolumeSpecName "kube-api-access-psld8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:11.789560 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.789541 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e662617-4bbb-4021-8bae-ba979960bb94-istio-token" pod "0e662617-4bbb-4021-8bae-ba979960bb94" (UID: "0e662617-4bbb-4021-8bae-ba979960bb94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:11.823058 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.823019 6572 generic.go:356] "Generic (PLEG): container finished" podID="0e662617-4bbb-4021-8bae-ba979960bb94" containerID="487304269ac77b713e363b4327e1c9b8dc649bba358bcb82b512c4e79430b6c5" exitCode=0 Oct 30 02:20:11.823275 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.823072 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" Oct 30 02:20:11.823275 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.823104 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" event={"ID":"0e662617-4bbb-4021-8bae-ba979960bb94","Type":"ContainerDied","Data":"487304269ac77b713e363b4327e1c9b8dc649bba358bcb82b512c4e79430b6c5"} Oct 30 02:20:11.823275 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.823163 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg" event={"ID":"0e662617-4bbb-4021-8bae-ba979960bb94","Type":"ContainerDied","Data":"f6bfa4b20fb2793bbdc2a4f0c1e7e1b8ec1157dab5757a042aa79d39c7c15c04"} Oct 30 02:20:11.823275 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.823188 6572 scope.go:122] "RemoveContainer" containerID="487304269ac77b713e363b4327e1c9b8dc649bba358bcb82b512c4e79430b6c5" Oct 30 02:20:11.825309 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.825291 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-92f0cc-wpl9h" Oct 30 02:20:11.825599 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.825287 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-92f0cc-wpl9h" event={"ID":"e3012f80-f0c3-4168-a742-1c92cdb07af7","Type":"ContainerDied","Data":"0377173aaa1d620ef7de4b16b412d7fb201c66b342ae1bf7d5aa7c66c6429781"} Oct 30 02:20:11.825690 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.825630 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0377173aaa1d620ef7de4b16b412d7fb201c66b342ae1bf7d5aa7c66c6429781" Oct 30 02:20:11.837017 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.836989 6572 scope.go:122] "RemoveContainer" containerID="275743f8bf29b2b7143c534cfc570df9a9e1ca985baed8835a56ac070c838cb9" Oct 30 02:20:11.853083 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.852971 6572 scope.go:122] "RemoveContainer" containerID="487304269ac77b713e363b4327e1c9b8dc649bba358bcb82b512c4e79430b6c5" Oct 30 02:20:11.853512 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:20:11.853400 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"487304269ac77b713e363b4327e1c9b8dc649bba358bcb82b512c4e79430b6c5\": container with ID starting with 487304269ac77b713e363b4327e1c9b8dc649bba358bcb82b512c4e79430b6c5 not found: ID does not exist" containerID="487304269ac77b713e363b4327e1c9b8dc649bba358bcb82b512c4e79430b6c5" Oct 30 02:20:11.853512 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.853436 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"487304269ac77b713e363b4327e1c9b8dc649bba358bcb82b512c4e79430b6c5"} err="failed to get container status \"487304269ac77b713e363b4327e1c9b8dc649bba358bcb82b512c4e79430b6c5\": rpc error: code = NotFound desc = could not find container \"487304269ac77b713e363b4327e1c9b8dc649bba358bcb82b512c4e79430b6c5\": container with ID starting with 487304269ac77b713e363b4327e1c9b8dc649bba358bcb82b512c4e79430b6c5 not found: ID does not exist" Oct 30 02:20:11.853512 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.853457 6572 scope.go:122] "RemoveContainer" containerID="275743f8bf29b2b7143c534cfc570df9a9e1ca985baed8835a56ac070c838cb9" Oct 30 02:20:11.854115 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:20:11.853985 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"275743f8bf29b2b7143c534cfc570df9a9e1ca985baed8835a56ac070c838cb9\": container with ID starting with 275743f8bf29b2b7143c534cfc570df9a9e1ca985baed8835a56ac070c838cb9 not found: ID does not exist" containerID="275743f8bf29b2b7143c534cfc570df9a9e1ca985baed8835a56ac070c838cb9" Oct 30 02:20:11.854115 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.854020 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"275743f8bf29b2b7143c534cfc570df9a9e1ca985baed8835a56ac070c838cb9"} err="failed to get container status \"275743f8bf29b2b7143c534cfc570df9a9e1ca985baed8835a56ac070c838cb9\": rpc error: code = NotFound desc = could not find container \"275743f8bf29b2b7143c534cfc570df9a9e1ca985baed8835a56ac070c838cb9\": container with ID starting with 275743f8bf29b2b7143c534cfc570df9a9e1ca985baed8835a56ac070c838cb9 not found: ID does not exist" Oct 30 02:20:11.876849 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.876820 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg"] Oct 30 02:20:11.885369 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.885344 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2d5aa9-openshift-default-6d4f8c9db5-6chqg"] Oct 30 02:20:11.887662 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.887634 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e662617-4bbb-4021-8bae-ba979960bb94-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:11.887803 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.887665 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e662617-4bbb-4021-8bae-ba979960bb94-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:11.887803 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.887680 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e662617-4bbb-4021-8bae-ba979960bb94-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:11.887803 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.887692 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e662617-4bbb-4021-8bae-ba979960bb94-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:11.887803 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.887703 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psld8\" (UniqueName: \"kubernetes.io/projected/0e662617-4bbb-4021-8bae-ba979960bb94-kube-api-access-psld8\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:11.887803 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:11.887715 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e662617-4bbb-4021-8bae-ba979960bb94-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:12.542324 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:12.542289 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e662617-4bbb-4021-8bae-ba979960bb94" path="/var/lib/kubelet/pods/0e662617-4bbb-4021-8bae-ba979960bb94/volumes" Oct 30 02:20:12.879846 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:12.879738 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:14.304612 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:14.304550 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:14.507277 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:14.507238 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:14.554803 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:14.554656 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf"] Oct 30 02:20:14.555001 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:14.554913 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" podUID="e9764434-6c1a-4889-a18a-6b714f6404fe" containerName="istio-proxy" containerID="cri-o://65d381682ac70a70717bd9e4d16e6baad97ce02fe955f91f819328c24c93cc91" gracePeriod=30 Oct 30 02:20:14.691959 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:20:14.691919 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:20:14.714544 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:14.714506 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:15.429935 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:15.429869 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d5aa9-66754f6cd7-5sdmc" Oct 30 02:20:15.524922 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:15.524892 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d425439-27d7-485c-b992-199e1b587c60-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d425439-27d7-485c-b992-199e1b587c60-plugins-config-volume\") pod \"4d425439-27d7-485c-b992-199e1b587c60\" (UID: \"4d425439-27d7-485c-b992-199e1b587c60\") " Oct 30 02:20:15.525129 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:15.524961 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d425439-27d7-485c-b992-199e1b587c60-kube-api-access-bf8fg\" (UniqueName: \"kubernetes.io/projected/4d425439-27d7-485c-b992-199e1b587c60-kube-api-access-bf8fg\") pod \"4d425439-27d7-485c-b992-199e1b587c60\" (UID: \"4d425439-27d7-485c-b992-199e1b587c60\") " Oct 30 02:20:15.525273 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:15.525253 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d425439-27d7-485c-b992-199e1b587c60-plugins-config-volume" pod "4d425439-27d7-485c-b992-199e1b587c60" (UID: "4d425439-27d7-485c-b992-199e1b587c60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:20:15.527104 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:15.527083 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d425439-27d7-485c-b992-199e1b587c60-kube-api-access-bf8fg" pod "4d425439-27d7-485c-b992-199e1b587c60" (UID: "4d425439-27d7-485c-b992-199e1b587c60"). InnerVolumeSpecName "kube-api-access-bf8fg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:15.626826 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:15.626792 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d425439-27d7-485c-b992-199e1b587c60-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:15.626826 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:15.626821 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf8fg\" (UniqueName: \"kubernetes.io/projected/4d425439-27d7-485c-b992-199e1b587c60-kube-api-access-bf8fg\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:15.882702 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:15.882619 6572 generic.go:356] "Generic (PLEG): container finished" podID="4d425439-27d7-485c-b992-199e1b587c60" containerID="21f6cf1389d0e4f46ab99ce6b4c2ee8aa44dbbe2ff83ed9716c1ed15648682ad" exitCode=0 Oct 30 02:20:15.882702 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:15.882667 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d5aa9-66754f6cd7-5sdmc" Oct 30 02:20:15.882983 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:15.882674 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d5aa9-66754f6cd7-5sdmc" event={"ID":"4d425439-27d7-485c-b992-199e1b587c60","Type":"ContainerDied","Data":"21f6cf1389d0e4f46ab99ce6b4c2ee8aa44dbbe2ff83ed9716c1ed15648682ad"} Oct 30 02:20:15.882983 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:15.882805 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d5aa9-66754f6cd7-5sdmc" event={"ID":"4d425439-27d7-485c-b992-199e1b587c60","Type":"ContainerDied","Data":"c85dd24bd97c568ac054d9abc546ffeec1b6c917308ce254f99b2ceaa8e7a762"} Oct 30 02:20:15.882983 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:15.882835 6572 scope.go:122] "RemoveContainer" containerID="21f6cf1389d0e4f46ab99ce6b4c2ee8aa44dbbe2ff83ed9716c1ed15648682ad" Oct 30 02:20:15.893834 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:15.893817 6572 scope.go:122] "RemoveContainer" containerID="21f6cf1389d0e4f46ab99ce6b4c2ee8aa44dbbe2ff83ed9716c1ed15648682ad" Oct 30 02:20:15.894100 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:20:15.894080 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21f6cf1389d0e4f46ab99ce6b4c2ee8aa44dbbe2ff83ed9716c1ed15648682ad\": container with ID starting with 21f6cf1389d0e4f46ab99ce6b4c2ee8aa44dbbe2ff83ed9716c1ed15648682ad not found: ID does not exist" containerID="21f6cf1389d0e4f46ab99ce6b4c2ee8aa44dbbe2ff83ed9716c1ed15648682ad" Oct 30 02:20:15.894171 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:15.894111 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21f6cf1389d0e4f46ab99ce6b4c2ee8aa44dbbe2ff83ed9716c1ed15648682ad"} err="failed to get container status \"21f6cf1389d0e4f46ab99ce6b4c2ee8aa44dbbe2ff83ed9716c1ed15648682ad\": rpc error: code = NotFound desc = could not find container \"21f6cf1389d0e4f46ab99ce6b4c2ee8aa44dbbe2ff83ed9716c1ed15648682ad\": container with ID starting with 21f6cf1389d0e4f46ab99ce6b4c2ee8aa44dbbe2ff83ed9716c1ed15648682ad not found: ID does not exist" Oct 30 02:20:15.932627 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:15.932604 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2d5aa9-66754f6cd7-5sdmc"] Oct 30 02:20:15.938366 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:15.938340 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2d5aa9-66754f6cd7-5sdmc"] Oct 30 02:20:16.068965 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:16.068933 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:16.541959 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:16.541926 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d425439-27d7-485c-b992-199e1b587c60" path="/var/lib/kubelet/pods/4d425439-27d7-485c-b992-199e1b587c60/volumes" Oct 30 02:20:17.689204 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:17.689172 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:17.877151 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:17.877103 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:19.282830 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:19.282784 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:19.821037 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:19.821015 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" Oct 30 02:20:19.919265 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:19.919231 6572 generic.go:356] "Generic (PLEG): container finished" podID="e9764434-6c1a-4889-a18a-6b714f6404fe" containerID="65d381682ac70a70717bd9e4d16e6baad97ce02fe955f91f819328c24c93cc91" exitCode=0 Oct 30 02:20:19.919497 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:19.919283 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" Oct 30 02:20:19.919497 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:19.919324 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" event={"ID":"e9764434-6c1a-4889-a18a-6b714f6404fe","Type":"ContainerDied","Data":"65d381682ac70a70717bd9e4d16e6baad97ce02fe955f91f819328c24c93cc91"} Oct 30 02:20:19.919497 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:19.919420 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf" event={"ID":"e9764434-6c1a-4889-a18a-6b714f6404fe","Type":"ContainerDied","Data":"288e48deef67fbe5cc8957833e17569f3a1bfd3fd966fff6edfe96ad59cf2a0d"} Oct 30 02:20:19.919497 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:19.919452 6572 scope.go:122] "RemoveContainer" containerID="65d381682ac70a70717bd9e4d16e6baad97ce02fe955f91f819328c24c93cc91" Oct 30 02:20:19.930930 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:19.930909 6572 scope.go:122] "RemoveContainer" containerID="65d381682ac70a70717bd9e4d16e6baad97ce02fe955f91f819328c24c93cc91" Oct 30 02:20:19.931246 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:20:19.931227 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65d381682ac70a70717bd9e4d16e6baad97ce02fe955f91f819328c24c93cc91\": container with ID starting with 65d381682ac70a70717bd9e4d16e6baad97ce02fe955f91f819328c24c93cc91 not found: ID does not exist" containerID="65d381682ac70a70717bd9e4d16e6baad97ce02fe955f91f819328c24c93cc91" Oct 30 02:20:19.931299 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:19.931256 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65d381682ac70a70717bd9e4d16e6baad97ce02fe955f91f819328c24c93cc91"} err="failed to get container status \"65d381682ac70a70717bd9e4d16e6baad97ce02fe955f91f819328c24c93cc91\": rpc error: code = NotFound desc = could not find container \"65d381682ac70a70717bd9e4d16e6baad97ce02fe955f91f819328c24c93cc91\": container with ID starting with 65d381682ac70a70717bd9e4d16e6baad97ce02fe955f91f819328c24c93cc91 not found: ID does not exist" Oct 30 02:20:19.973531 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:19.973487 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9764434-6c1a-4889-a18a-6b714f6404fe-istio-token\" (UniqueName: \"kubernetes.io/projected/e9764434-6c1a-4889-a18a-6b714f6404fe-istio-token\") pod \"e9764434-6c1a-4889-a18a-6b714f6404fe\" (UID: \"e9764434-6c1a-4889-a18a-6b714f6404fe\") " Oct 30 02:20:19.973697 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:19.973582 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9764434-6c1a-4889-a18a-6b714f6404fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9764434-6c1a-4889-a18a-6b714f6404fe-istio-envoy\") pod \"e9764434-6c1a-4889-a18a-6b714f6404fe\" (UID: \"e9764434-6c1a-4889-a18a-6b714f6404fe\") " Oct 30 02:20:19.973697 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:19.973611 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9764434-6c1a-4889-a18a-6b714f6404fe-kube-api-access-fm5bk\" (UniqueName: \"kubernetes.io/projected/e9764434-6c1a-4889-a18a-6b714f6404fe-kube-api-access-fm5bk\") pod \"e9764434-6c1a-4889-a18a-6b714f6404fe\" (UID: \"e9764434-6c1a-4889-a18a-6b714f6404fe\") " Oct 30 02:20:19.973697 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:19.973637 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9764434-6c1a-4889-a18a-6b714f6404fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9764434-6c1a-4889-a18a-6b714f6404fe-workload-socket\") pod \"e9764434-6c1a-4889-a18a-6b714f6404fe\" (UID: \"e9764434-6c1a-4889-a18a-6b714f6404fe\") " Oct 30 02:20:19.973697 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:19.973694 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9764434-6c1a-4889-a18a-6b714f6404fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9764434-6c1a-4889-a18a-6b714f6404fe-workload-certs\") pod \"e9764434-6c1a-4889-a18a-6b714f6404fe\" (UID: \"e9764434-6c1a-4889-a18a-6b714f6404fe\") " Oct 30 02:20:19.973970 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:19.973764 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9764434-6c1a-4889-a18a-6b714f6404fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9764434-6c1a-4889-a18a-6b714f6404fe-istio-podinfo\") pod \"e9764434-6c1a-4889-a18a-6b714f6404fe\" (UID: \"e9764434-6c1a-4889-a18a-6b714f6404fe\") " Oct 30 02:20:19.973970 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:19.973792 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9764434-6c1a-4889-a18a-6b714f6404fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9764434-6c1a-4889-a18a-6b714f6404fe-credential-socket\") pod \"e9764434-6c1a-4889-a18a-6b714f6404fe\" (UID: \"e9764434-6c1a-4889-a18a-6b714f6404fe\") " Oct 30 02:20:19.973970 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:19.973821 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9764434-6c1a-4889-a18a-6b714f6404fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9764434-6c1a-4889-a18a-6b714f6404fe-istio-data\") pod \"e9764434-6c1a-4889-a18a-6b714f6404fe\" (UID: \"e9764434-6c1a-4889-a18a-6b714f6404fe\") " Oct 30 02:20:19.973970 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:19.973917 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9764434-6c1a-4889-a18a-6b714f6404fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9764434-6c1a-4889-a18a-6b714f6404fe-istiod-ca-cert\") pod \"e9764434-6c1a-4889-a18a-6b714f6404fe\" (UID: \"e9764434-6c1a-4889-a18a-6b714f6404fe\") " Oct 30 02:20:19.974342 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:19.974287 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9764434-6c1a-4889-a18a-6b714f6404fe-credential-socket" pod "e9764434-6c1a-4889-a18a-6b714f6404fe" (UID: "e9764434-6c1a-4889-a18a-6b714f6404fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:20:19.974588 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:19.974486 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9764434-6c1a-4889-a18a-6b714f6404fe-workload-certs" pod "e9764434-6c1a-4889-a18a-6b714f6404fe" (UID: "e9764434-6c1a-4889-a18a-6b714f6404fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:20:19.974588 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:19.974557 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9764434-6c1a-4889-a18a-6b714f6404fe-workload-socket" pod "e9764434-6c1a-4889-a18a-6b714f6404fe" (UID: "e9764434-6c1a-4889-a18a-6b714f6404fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:20:19.974766 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:19.974568 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9764434-6c1a-4889-a18a-6b714f6404fe-istio-data" pod "e9764434-6c1a-4889-a18a-6b714f6404fe" (UID: "e9764434-6c1a-4889-a18a-6b714f6404fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:20:19.974766 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:19.974607 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9764434-6c1a-4889-a18a-6b714f6404fe-istiod-ca-cert" pod "e9764434-6c1a-4889-a18a-6b714f6404fe" (UID: "e9764434-6c1a-4889-a18a-6b714f6404fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:20:19.975951 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:19.975923 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9764434-6c1a-4889-a18a-6b714f6404fe-kube-api-access-fm5bk" pod "e9764434-6c1a-4889-a18a-6b714f6404fe" (UID: "e9764434-6c1a-4889-a18a-6b714f6404fe"). InnerVolumeSpecName "kube-api-access-fm5bk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:19.977021 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:19.977004 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9764434-6c1a-4889-a18a-6b714f6404fe-istio-envoy" pod "e9764434-6c1a-4889-a18a-6b714f6404fe" (UID: "e9764434-6c1a-4889-a18a-6b714f6404fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:20:19.977157 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:19.977132 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9764434-6c1a-4889-a18a-6b714f6404fe-istio-podinfo" pod "e9764434-6c1a-4889-a18a-6b714f6404fe" (UID: "e9764434-6c1a-4889-a18a-6b714f6404fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:20:19.977207 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:19.977155 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9764434-6c1a-4889-a18a-6b714f6404fe-istio-token" pod "e9764434-6c1a-4889-a18a-6b714f6404fe" (UID: "e9764434-6c1a-4889-a18a-6b714f6404fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:20.075510 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:20.075470 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9764434-6c1a-4889-a18a-6b714f6404fe-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:20.075510 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:20.075503 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9764434-6c1a-4889-a18a-6b714f6404fe-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:20.075510 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:20.075513 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm5bk\" (UniqueName: \"kubernetes.io/projected/e9764434-6c1a-4889-a18a-6b714f6404fe-kube-api-access-fm5bk\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:20.075790 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:20.075525 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9764434-6c1a-4889-a18a-6b714f6404fe-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:20.075790 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:20.075534 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9764434-6c1a-4889-a18a-6b714f6404fe-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:20.075790 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:20.075542 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9764434-6c1a-4889-a18a-6b714f6404fe-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:20.075790 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:20.075550 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9764434-6c1a-4889-a18a-6b714f6404fe-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:20.075790 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:20.075563 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9764434-6c1a-4889-a18a-6b714f6404fe-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:20.075790 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:20.075575 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9764434-6c1a-4889-a18a-6b714f6404fe-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:20.264838 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:20.264763 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf"] Oct 30 02:20:20.275979 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:20.275949 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-bttzf"] Oct 30 02:20:20.542073 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:20.541975 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9764434-6c1a-4889-a18a-6b714f6404fe" path="/var/lib/kubelet/pods/e9764434-6c1a-4889-a18a-6b714f6404fe/volumes" Oct 30 02:20:20.781366 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:20.781321 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:21.003444 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:21.003410 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:22.001991 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:22.001950 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:22.215985 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:22.215945 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:22.673482 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:22.673442 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:22.737606 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:22.737571 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-92f0cc-wpl9h"] Oct 30 02:20:22.748480 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:22.748451 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-92f0cc-wpl9h"] Oct 30 02:20:22.868784 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:22.868724 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:23.369070 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:23.369031 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:24.542270 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:24.542233 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3012f80-f0c3-4168-a742-1c92cdb07af7" path="/var/lib/kubelet/pods/e3012f80-f0c3-4168-a742-1c92cdb07af7/volumes" Oct 30 02:20:24.581514 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:24.581476 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:26.071520 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:26.071479 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:27.588873 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:27.588835 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:28.328328 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:28.328296 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:29.080845 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:29.080811 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:29.680702 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:29.680665 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:29.887371 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:29.887336 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:30.102407 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:30.102370 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:30.516735 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:30.516698 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1edaf-5bqmd"] Oct 30 02:20:30.517233 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:30.517205 6572 cpu_manager.go:414] "containerMap: removing container" podUID="4d425439-27d7-485c-b992-199e1b587c60" containerName="payload-pre-processing" Oct 30 02:20:30.517233 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:30.517224 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d425439-27d7-485c-b992-199e1b587c60" containerName="payload-pre-processing" Oct 30 02:20:30.517393 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:30.517253 6572 cpu_manager.go:414] "containerMap: removing container" podUID="e9764434-6c1a-4889-a18a-6b714f6404fe" containerName="istio-proxy" Oct 30 02:20:30.517393 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:30.517262 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9764434-6c1a-4889-a18a-6b714f6404fe" containerName="istio-proxy" Oct 30 02:20:30.517393 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:30.517284 6572 cpu_manager.go:414] "containerMap: removing container" podUID="0e662617-4bbb-4021-8bae-ba979960bb94" containerName="istio-proxy" Oct 30 02:20:30.517393 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:30.517289 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e662617-4bbb-4021-8bae-ba979960bb94" containerName="istio-proxy" Oct 30 02:20:30.517393 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:30.517298 6572 cpu_manager.go:414] "containerMap: removing container" podUID="0e662617-4bbb-4021-8bae-ba979960bb94" containerName="istio-proxy" Oct 30 02:20:30.517393 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:30.517302 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e662617-4bbb-4021-8bae-ba979960bb94" containerName="istio-proxy" Oct 30 02:20:30.517393 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:30.517312 6572 cpu_manager.go:414] "containerMap: removing container" podUID="e3012f80-f0c3-4168-a742-1c92cdb07af7" containerName="revoke-keys" Oct 30 02:20:30.517393 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:30.517316 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3012f80-f0c3-4168-a742-1c92cdb07af7" containerName="revoke-keys" Oct 30 02:20:30.517393 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:30.517382 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9764434-6c1a-4889-a18a-6b714f6404fe" containerName="istio-proxy" Oct 30 02:20:30.517393 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:30.517394 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e662617-4bbb-4021-8bae-ba979960bb94" containerName="istio-proxy" Oct 30 02:20:30.517732 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:30.517402 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e662617-4bbb-4021-8bae-ba979960bb94" containerName="istio-proxy" Oct 30 02:20:30.517732 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:30.517407 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3012f80-f0c3-4168-a742-1c92cdb07af7" containerName="revoke-keys" Oct 30 02:20:30.517732 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:30.517415 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d425439-27d7-485c-b992-199e1b587c60" containerName="payload-pre-processing" Oct 30 02:20:30.520885 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:30.520868 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1edaf-5bqmd" Oct 30 02:20:30.540478 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:30.540439 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:20:30.548686 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:30.548651 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:20:30.548904 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:30.548732 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:20:30.548904 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:30.548845 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:20:30.552582 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:30.552547 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1edaf-5bqmd"] Oct 30 02:20:30.690407 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:30.690362 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f769k\" (UniqueName: \"kubernetes.io/projected/2ed7c6a8-cdba-427e-af81-318f69a70d9f-kube-api-access-f769k\") pod \"maas-api-revoke-keys-e2e-extmod-b-f1edaf-5bqmd\" (UID: \"2ed7c6a8-cdba-427e-af81-318f69a70d9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1edaf-5bqmd" Oct 30 02:20:30.690592 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:30.690418 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ed7c6a8-cdba-427e-af81-318f69a70d9f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f1edaf-5bqmd\" (UID: \"2ed7c6a8-cdba-427e-af81-318f69a70d9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1edaf-5bqmd" Oct 30 02:20:30.690592 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:30.690444 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ed7c6a8-cdba-427e-af81-318f69a70d9f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f1edaf-5bqmd\" (UID: \"2ed7c6a8-cdba-427e-af81-318f69a70d9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1edaf-5bqmd" Oct 30 02:20:30.777732 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:30.777643 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:30.791999 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:30.791963 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f769k\" (UniqueName: \"kubernetes.io/projected/2ed7c6a8-cdba-427e-af81-318f69a70d9f-kube-api-access-f769k\") pod \"maas-api-revoke-keys-e2e-extmod-b-f1edaf-5bqmd\" (UID: \"2ed7c6a8-cdba-427e-af81-318f69a70d9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1edaf-5bqmd" Oct 30 02:20:30.792131 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:30.792016 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ed7c6a8-cdba-427e-af81-318f69a70d9f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f1edaf-5bqmd\" (UID: \"2ed7c6a8-cdba-427e-af81-318f69a70d9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1edaf-5bqmd" Oct 30 02:20:30.792131 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:30.792053 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ed7c6a8-cdba-427e-af81-318f69a70d9f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f1edaf-5bqmd\" (UID: \"2ed7c6a8-cdba-427e-af81-318f69a70d9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1edaf-5bqmd" Oct 30 02:20:30.792729 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:30.792704 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ed7c6a8-cdba-427e-af81-318f69a70d9f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f1edaf-5bqmd\" (UID: \"2ed7c6a8-cdba-427e-af81-318f69a70d9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1edaf-5bqmd" Oct 30 02:20:30.794531 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:30.794506 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ed7c6a8-cdba-427e-af81-318f69a70d9f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f1edaf-5bqmd\" (UID: \"2ed7c6a8-cdba-427e-af81-318f69a70d9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1edaf-5bqmd" Oct 30 02:20:30.811559 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:30.811526 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f769k\" (UniqueName: \"kubernetes.io/projected/2ed7c6a8-cdba-427e-af81-318f69a70d9f-kube-api-access-f769k\") pod \"maas-api-revoke-keys-e2e-extmod-b-f1edaf-5bqmd\" (UID: \"2ed7c6a8-cdba-427e-af81-318f69a70d9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1edaf-5bqmd" Oct 30 02:20:30.834688 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:30.834648 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1edaf-5bqmd" Oct 30 02:20:31.139847 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:31.139813 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1edaf-5bqmd"] Oct 30 02:20:31.142176 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:20:31.142145 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ed7c6a8_cdba_427e_af81_318f69a70d9f.slice/crio-4ed9e36742ba3026446b50bf1fdc7944cf3c73baa692033d778d061e0ed0d68d WatchSource:0}: Error finding container 4ed9e36742ba3026446b50bf1fdc7944cf3c73baa692033d778d061e0ed0d68d: Status 404 returned error can't find the container with id 4ed9e36742ba3026446b50bf1fdc7944cf3c73baa692033d778d061e0ed0d68d Oct 30 02:20:32.013389 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:32.013353 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1edaf-5bqmd" event={"ID":"2ed7c6a8-cdba-427e-af81-318f69a70d9f","Type":"ContainerStarted","Data":"7b143d4483986ffa79d5049937755e781fdd148d1fa86ec364e565b9329db4ab"} Oct 30 02:20:32.013622 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:32.013417 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1edaf-5bqmd" event={"ID":"2ed7c6a8-cdba-427e-af81-318f69a70d9f","Type":"ContainerStarted","Data":"4ed9e36742ba3026446b50bf1fdc7944cf3c73baa692033d778d061e0ed0d68d"} Oct 30 02:20:32.051231 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:32.051161 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1edaf-5bqmd" podStartSLOduration=2.051140239 podStartE2EDuration="2.051140239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:20:32.05026139 +0000 UTC m=+3757.626686789" watchObservedRunningTime="2026-10-30 02:20:32.051140239 +0000 UTC m=+3757.627565643" Oct 30 02:20:32.403054 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:32.402958 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:33.027885 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:33.027845 6572 generic.go:356] "Generic (PLEG): container finished" podID="2ed7c6a8-cdba-427e-af81-318f69a70d9f" containerID="7b143d4483986ffa79d5049937755e781fdd148d1fa86ec364e565b9329db4ab" exitCode=0 Oct 30 02:20:33.028090 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:33.027936 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1edaf-5bqmd" event={"ID":"2ed7c6a8-cdba-427e-af81-318f69a70d9f","Type":"ContainerDied","Data":"7b143d4483986ffa79d5049937755e781fdd148d1fa86ec364e565b9329db4ab"} Oct 30 02:20:33.994641 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:33.994602 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:34.180990 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:34.180959 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1edaf-5bqmd" Oct 30 02:20:34.226669 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:34.226636 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ed7c6a8-cdba-427e-af81-318f69a70d9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ed7c6a8-cdba-427e-af81-318f69a70d9f-maas-api-openshift-ca\") pod \"2ed7c6a8-cdba-427e-af81-318f69a70d9f\" (UID: \"2ed7c6a8-cdba-427e-af81-318f69a70d9f\") " Oct 30 02:20:34.227044 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:34.226692 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ed7c6a8-cdba-427e-af81-318f69a70d9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ed7c6a8-cdba-427e-af81-318f69a70d9f-maas-api-xks-ca\") pod \"2ed7c6a8-cdba-427e-af81-318f69a70d9f\" (UID: \"2ed7c6a8-cdba-427e-af81-318f69a70d9f\") " Oct 30 02:20:34.227044 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:34.226863 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ed7c6a8-cdba-427e-af81-318f69a70d9f-kube-api-access-f769k\" (UniqueName: \"kubernetes.io/projected/2ed7c6a8-cdba-427e-af81-318f69a70d9f-kube-api-access-f769k\") pod \"2ed7c6a8-cdba-427e-af81-318f69a70d9f\" (UID: \"2ed7c6a8-cdba-427e-af81-318f69a70d9f\") " Oct 30 02:20:34.227718 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:34.227690 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ed7c6a8-cdba-427e-af81-318f69a70d9f-maas-api-openshift-ca" pod "2ed7c6a8-cdba-427e-af81-318f69a70d9f" (UID: "2ed7c6a8-cdba-427e-af81-318f69a70d9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:20:34.230032 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:34.230001 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ed7c6a8-cdba-427e-af81-318f69a70d9f-kube-api-access-f769k" pod "2ed7c6a8-cdba-427e-af81-318f69a70d9f" (UID: "2ed7c6a8-cdba-427e-af81-318f69a70d9f"). InnerVolumeSpecName "kube-api-access-f769k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:34.230269 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:34.230242 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ed7c6a8-cdba-427e-af81-318f69a70d9f-maas-api-xks-ca" pod "2ed7c6a8-cdba-427e-af81-318f69a70d9f" (UID: "2ed7c6a8-cdba-427e-af81-318f69a70d9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:20:34.328242 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:34.328132 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ed7c6a8-cdba-427e-af81-318f69a70d9f-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:34.328242 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:34.328174 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f769k\" (UniqueName: \"kubernetes.io/projected/2ed7c6a8-cdba-427e-af81-318f69a70d9f-kube-api-access-f769k\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:34.328242 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:34.328195 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ed7c6a8-cdba-427e-af81-318f69a70d9f-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:35.047942 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:35.047907 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1edaf-5bqmd" Oct 30 02:20:35.047942 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:35.047926 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1edaf-5bqmd" event={"ID":"2ed7c6a8-cdba-427e-af81-318f69a70d9f","Type":"ContainerDied","Data":"4ed9e36742ba3026446b50bf1fdc7944cf3c73baa692033d778d061e0ed0d68d"} Oct 30 02:20:35.048482 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:35.047975 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ed9e36742ba3026446b50bf1fdc7944cf3c73baa692033d778d061e0ed0d68d" Oct 30 02:20:35.593548 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:35.593515 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:37.160022 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:37.159983 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:38.587786 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:38.587726 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:40.084039 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:40.084003 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:40.968294 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:40.968251 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f1edaf-55c447f959-nh7w8"] Oct 30 02:20:41.171721 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:41.171681 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:41.370987 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:41.370887 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:41.689371 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:41.689331 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:43.205187 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:43.205149 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:44.688446 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:20:44.688406 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:20:44.792563 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:44.792522 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:45.968913 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:45.968839 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1edaf-55c447f959-nh7w8" podUID="f730cd49-e419-4643-bbaa-c5e7d7ab82eb" containerName="payload-pre-processing" containerID="cri-o://55566dbaf540987bdcd8feab8f1669a9896888c89dfe17575dd7be9de4a6c940" gracePeriod=25 Oct 30 02:20:46.157011 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:46.156974 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:46.662076 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:46.662037 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:46.676757 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:46.676716 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1edaf-5bqmd"] Oct 30 02:20:46.687917 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:46.687881 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1edaf-5bqmd"] Oct 30 02:20:46.866690 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:46.866641 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:47.293937 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:47.293897 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:48.026992 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:48.026952 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95ced13a-prqjw"] Oct 30 02:20:48.037986 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:48.037944 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95ced13a-prqjw"] Oct 30 02:20:48.504283 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:48.504241 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:48.540686 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:48.540652 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ed7c6a8-cdba-427e-af81-318f69a70d9f" path="/var/lib/kubelet/pods/2ed7c6a8-cdba-427e-af81-318f69a70d9f/volumes" Oct 30 02:20:48.541121 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:48.541105 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d91b3759-1119-4059-922c-aa54b8d2922c" path="/var/lib/kubelet/pods/d91b3759-1119-4059-922c-aa54b8d2922c/volumes" Oct 30 02:20:49.977512 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:49.977475 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:50.818305 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:50.818260 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1edaf-55c447f959-nh7w8" podUID="f730cd49-e419-4643-bbaa-c5e7d7ab82eb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:20:51.489676 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:51.489639 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:52.979035 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:52.978998 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:53.373925 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:53.373834 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:53.580991 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:53.580955 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:53.585289 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:53.585260 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh"] Oct 30 02:20:53.585526 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:53.585503 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" podUID="22e0c074-1c61-435e-b086-0c06657d2ab9" containerName="istio-proxy" containerID="cri-o://3de5bdb45184edb4b47cf4041bfbcf0ba9cb5ee42b0383d6f481fa51c99b312d" gracePeriod=30 Oct 30 02:20:53.767381 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:53.767348 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:54.186879 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:54.186845 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-abb30f-rndpf"] Oct 30 02:20:54.187321 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:54.187306 6572 cpu_manager.go:414] "containerMap: removing container" podUID="2ed7c6a8-cdba-427e-af81-318f69a70d9f" containerName="revoke-keys" Oct 30 02:20:54.187368 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:54.187321 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ed7c6a8-cdba-427e-af81-318f69a70d9f" containerName="revoke-keys" Oct 30 02:20:54.187413 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:54.187404 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ed7c6a8-cdba-427e-af81-318f69a70d9f" containerName="revoke-keys" Oct 30 02:20:54.192400 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:54.192372 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-abb30f-rndpf" Oct 30 02:20:54.208776 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:54.208706 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-abb30f-rndpf"] Oct 30 02:20:54.216390 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:54.216351 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:20:54.216390 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:54.216351 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:20:54.216691 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:54.216422 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:20:54.217692 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:54.217662 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:20:54.346764 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:54.346717 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2df25a6-7150-44bd-b38a-6afe82241801-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-abb30f-rndpf\" (UID: \"e2df25a6-7150-44bd-b38a-6afe82241801\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-abb30f-rndpf" Oct 30 02:20:54.346971 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:54.346853 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2df25a6-7150-44bd-b38a-6afe82241801-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-abb30f-rndpf\" (UID: \"e2df25a6-7150-44bd-b38a-6afe82241801\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-abb30f-rndpf" Oct 30 02:20:54.346971 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:54.346915 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vffb\" (UniqueName: \"kubernetes.io/projected/e2df25a6-7150-44bd-b38a-6afe82241801-kube-api-access-8vffb\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-abb30f-rndpf\" (UID: \"e2df25a6-7150-44bd-b38a-6afe82241801\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-abb30f-rndpf" Oct 30 02:20:54.447649 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:54.447553 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2df25a6-7150-44bd-b38a-6afe82241801-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-abb30f-rndpf\" (UID: \"e2df25a6-7150-44bd-b38a-6afe82241801\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-abb30f-rndpf" Oct 30 02:20:54.447649 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:54.447601 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2df25a6-7150-44bd-b38a-6afe82241801-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-abb30f-rndpf\" (UID: \"e2df25a6-7150-44bd-b38a-6afe82241801\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-abb30f-rndpf" Oct 30 02:20:54.447649 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:54.447640 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vffb\" (UniqueName: \"kubernetes.io/projected/e2df25a6-7150-44bd-b38a-6afe82241801-kube-api-access-8vffb\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-abb30f-rndpf\" (UID: \"e2df25a6-7150-44bd-b38a-6afe82241801\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-abb30f-rndpf" Oct 30 02:20:54.448448 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:54.448423 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2df25a6-7150-44bd-b38a-6afe82241801-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-abb30f-rndpf\" (UID: \"e2df25a6-7150-44bd-b38a-6afe82241801\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-abb30f-rndpf" Oct 30 02:20:54.450172 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:54.450147 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2df25a6-7150-44bd-b38a-6afe82241801-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-abb30f-rndpf\" (UID: \"e2df25a6-7150-44bd-b38a-6afe82241801\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-abb30f-rndpf" Oct 30 02:20:54.466779 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:54.466716 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vffb\" (UniqueName: \"kubernetes.io/projected/e2df25a6-7150-44bd-b38a-6afe82241801-kube-api-access-8vffb\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-abb30f-rndpf\" (UID: \"e2df25a6-7150-44bd-b38a-6afe82241801\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-abb30f-rndpf" Oct 30 02:20:54.507044 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:54.507004 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-abb30f-rndpf" Oct 30 02:20:54.719158 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:54.719040 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:54.809033 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:54.809007 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-abb30f-rndpf"] Oct 30 02:20:54.811161 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:20:54.811125 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2df25a6_7150_44bd_b38a_6afe82241801.slice/crio-1bda2436a3ae50423291441114bdeb14f5ccd1a3b7572b8af0be03b42a8d9f96 WatchSource:0}: Error finding container 1bda2436a3ae50423291441114bdeb14f5ccd1a3b7572b8af0be03b42a8d9f96: Status 404 returned error can't find the container with id 1bda2436a3ae50423291441114bdeb14f5ccd1a3b7572b8af0be03b42a8d9f96 Oct 30 02:20:55.201339 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:55.201292 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-abb30f-rndpf" event={"ID":"e2df25a6-7150-44bd-b38a-6afe82241801","Type":"ContainerStarted","Data":"06168b22c867558efb93d70b68021385b51016c9c24aae27bcda4e3a814c474d"} Oct 30 02:20:55.201805 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:55.201361 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-abb30f-rndpf" event={"ID":"e2df25a6-7150-44bd-b38a-6afe82241801","Type":"ContainerStarted","Data":"1bda2436a3ae50423291441114bdeb14f5ccd1a3b7572b8af0be03b42a8d9f96"} Oct 30 02:20:55.240921 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:55.240854 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-abb30f-rndpf" podStartSLOduration=1.240835952 podStartE2EDuration="1.240835952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:20:55.238163654 +0000 UTC m=+3780.814589056" watchObservedRunningTime="2026-10-30 02:20:55.240835952 +0000 UTC m=+3780.817261353" Oct 30 02:20:55.839527 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:55.839489 6572 patch_prober.go:29] interesting pod/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:20:55.839712 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:55.839551 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" podUID="22e0c074-1c61-435e-b086-0c06657d2ab9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:20:56.209683 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:56.209644 6572 generic.go:356] "Generic (PLEG): container finished" podID="e2df25a6-7150-44bd-b38a-6afe82241801" containerID="06168b22c867558efb93d70b68021385b51016c9c24aae27bcda4e3a814c474d" exitCode=0 Oct 30 02:20:56.210197 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:56.209739 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-abb30f-rndpf" event={"ID":"e2df25a6-7150-44bd-b38a-6afe82241801","Type":"ContainerDied","Data":"06168b22c867558efb93d70b68021385b51016c9c24aae27bcda4e3a814c474d"} Oct 30 02:20:56.285897 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:56.285863 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:57.363096 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:57.363064 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-abb30f-rndpf" Oct 30 02:20:57.485843 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:57.485698 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2df25a6-7150-44bd-b38a-6afe82241801-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2df25a6-7150-44bd-b38a-6afe82241801-maas-api-openshift-ca\") pod \"e2df25a6-7150-44bd-b38a-6afe82241801\" (UID: \"e2df25a6-7150-44bd-b38a-6afe82241801\") " Oct 30 02:20:57.485843 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:57.485798 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2df25a6-7150-44bd-b38a-6afe82241801-kube-api-access-8vffb\" (UniqueName: \"kubernetes.io/projected/e2df25a6-7150-44bd-b38a-6afe82241801-kube-api-access-8vffb\") pod \"e2df25a6-7150-44bd-b38a-6afe82241801\" (UID: \"e2df25a6-7150-44bd-b38a-6afe82241801\") " Oct 30 02:20:57.486064 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:57.485991 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2df25a6-7150-44bd-b38a-6afe82241801-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2df25a6-7150-44bd-b38a-6afe82241801-maas-api-xks-ca\") pod \"e2df25a6-7150-44bd-b38a-6afe82241801\" (UID: \"e2df25a6-7150-44bd-b38a-6afe82241801\") " Oct 30 02:20:57.486128 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:57.486101 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2df25a6-7150-44bd-b38a-6afe82241801-maas-api-openshift-ca" pod "e2df25a6-7150-44bd-b38a-6afe82241801" (UID: "e2df25a6-7150-44bd-b38a-6afe82241801"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:20:57.486341 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:57.486318 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2df25a6-7150-44bd-b38a-6afe82241801-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:57.488009 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:57.487987 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2df25a6-7150-44bd-b38a-6afe82241801-maas-api-xks-ca" pod "e2df25a6-7150-44bd-b38a-6afe82241801" (UID: "e2df25a6-7150-44bd-b38a-6afe82241801"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:20:57.488104 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:57.488028 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2df25a6-7150-44bd-b38a-6afe82241801-kube-api-access-8vffb" pod "e2df25a6-7150-44bd-b38a-6afe82241801" (UID: "e2df25a6-7150-44bd-b38a-6afe82241801"). InnerVolumeSpecName "kube-api-access-8vffb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:57.587201 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:57.587164 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vffb\" (UniqueName: \"kubernetes.io/projected/e2df25a6-7150-44bd-b38a-6afe82241801-kube-api-access-8vffb\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:57.587201 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:57.587196 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2df25a6-7150-44bd-b38a-6afe82241801-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:57.812716 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:57.812620 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:58.227125 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:58.227094 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-abb30f-rndpf" Oct 30 02:20:58.227321 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:58.227127 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-abb30f-rndpf" event={"ID":"e2df25a6-7150-44bd-b38a-6afe82241801","Type":"ContainerDied","Data":"1bda2436a3ae50423291441114bdeb14f5ccd1a3b7572b8af0be03b42a8d9f96"} Oct 30 02:20:58.227321 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:58.227173 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bda2436a3ae50423291441114bdeb14f5ccd1a3b7572b8af0be03b42a8d9f96" Oct 30 02:20:58.840423 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:58.840396 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 02:20:59.000207 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.000163 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22e0c074-1c61-435e-b086-0c06657d2ab9-istio-token\" (UniqueName: \"kubernetes.io/projected/22e0c074-1c61-435e-b086-0c06657d2ab9-istio-token\") pod \"22e0c074-1c61-435e-b086-0c06657d2ab9\" (UID: \"22e0c074-1c61-435e-b086-0c06657d2ab9\") " Oct 30 02:20:59.000442 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.000221 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22e0c074-1c61-435e-b086-0c06657d2ab9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e0c074-1c61-435e-b086-0c06657d2ab9-credential-socket\") pod \"22e0c074-1c61-435e-b086-0c06657d2ab9\" (UID: \"22e0c074-1c61-435e-b086-0c06657d2ab9\") " Oct 30 02:20:59.000442 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.000249 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22e0c074-1c61-435e-b086-0c06657d2ab9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22e0c074-1c61-435e-b086-0c06657d2ab9-istio-envoy\") pod \"22e0c074-1c61-435e-b086-0c06657d2ab9\" (UID: \"22e0c074-1c61-435e-b086-0c06657d2ab9\") " Oct 30 02:20:59.000442 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.000306 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22e0c074-1c61-435e-b086-0c06657d2ab9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22e0c074-1c61-435e-b086-0c06657d2ab9-workload-certs\") pod \"22e0c074-1c61-435e-b086-0c06657d2ab9\" (UID: \"22e0c074-1c61-435e-b086-0c06657d2ab9\") " Oct 30 02:20:59.000442 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.000331 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22e0c074-1c61-435e-b086-0c06657d2ab9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22e0c074-1c61-435e-b086-0c06657d2ab9-istio-data\") pod \"22e0c074-1c61-435e-b086-0c06657d2ab9\" (UID: \"22e0c074-1c61-435e-b086-0c06657d2ab9\") " Oct 30 02:20:59.000442 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.000418 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22e0c074-1c61-435e-b086-0c06657d2ab9-kube-api-access-w7nh2\" (UniqueName: \"kubernetes.io/projected/22e0c074-1c61-435e-b086-0c06657d2ab9-kube-api-access-w7nh2\") pod \"22e0c074-1c61-435e-b086-0c06657d2ab9\" (UID: \"22e0c074-1c61-435e-b086-0c06657d2ab9\") " Oct 30 02:20:59.000731 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.000540 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22e0c074-1c61-435e-b086-0c06657d2ab9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22e0c074-1c61-435e-b086-0c06657d2ab9-istio-podinfo\") pod \"22e0c074-1c61-435e-b086-0c06657d2ab9\" (UID: \"22e0c074-1c61-435e-b086-0c06657d2ab9\") " Oct 30 02:20:59.000731 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.000576 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22e0c074-1c61-435e-b086-0c06657d2ab9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e0c074-1c61-435e-b086-0c06657d2ab9-workload-socket\") pod \"22e0c074-1c61-435e-b086-0c06657d2ab9\" (UID: \"22e0c074-1c61-435e-b086-0c06657d2ab9\") " Oct 30 02:20:59.000731 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.000592 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e0c074-1c61-435e-b086-0c06657d2ab9-credential-socket" pod "22e0c074-1c61-435e-b086-0c06657d2ab9" (UID: "22e0c074-1c61-435e-b086-0c06657d2ab9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:20:59.000731 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.000613 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22e0c074-1c61-435e-b086-0c06657d2ab9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22e0c074-1c61-435e-b086-0c06657d2ab9-istiod-ca-cert\") pod \"22e0c074-1c61-435e-b086-0c06657d2ab9\" (UID: \"22e0c074-1c61-435e-b086-0c06657d2ab9\") " Oct 30 02:20:59.001013 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.000810 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e0c074-1c61-435e-b086-0c06657d2ab9-workload-certs" pod "22e0c074-1c61-435e-b086-0c06657d2ab9" (UID: "22e0c074-1c61-435e-b086-0c06657d2ab9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:20:59.001013 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.000826 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e0c074-1c61-435e-b086-0c06657d2ab9-istio-data" pod "22e0c074-1c61-435e-b086-0c06657d2ab9" (UID: "22e0c074-1c61-435e-b086-0c06657d2ab9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:20:59.001145 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.001075 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e0c074-1c61-435e-b086-0c06657d2ab9-workload-socket" pod "22e0c074-1c61-435e-b086-0c06657d2ab9" (UID: "22e0c074-1c61-435e-b086-0c06657d2ab9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:20:59.001145 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.001086 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e0c074-1c61-435e-b086-0c06657d2ab9-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:59.001145 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.001111 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22e0c074-1c61-435e-b086-0c06657d2ab9-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:59.001145 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.001126 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22e0c074-1c61-435e-b086-0c06657d2ab9-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:59.001345 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.001168 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22e0c074-1c61-435e-b086-0c06657d2ab9-istiod-ca-cert" pod "22e0c074-1c61-435e-b086-0c06657d2ab9" (UID: "22e0c074-1c61-435e-b086-0c06657d2ab9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:20:59.002687 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.002660 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e0c074-1c61-435e-b086-0c06657d2ab9-kube-api-access-w7nh2" pod "22e0c074-1c61-435e-b086-0c06657d2ab9" (UID: "22e0c074-1c61-435e-b086-0c06657d2ab9"). InnerVolumeSpecName "kube-api-access-w7nh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:59.002687 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.002667 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e0c074-1c61-435e-b086-0c06657d2ab9-istio-token" pod "22e0c074-1c61-435e-b086-0c06657d2ab9" (UID: "22e0c074-1c61-435e-b086-0c06657d2ab9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:59.002976 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.002950 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e0c074-1c61-435e-b086-0c06657d2ab9-istio-envoy" pod "22e0c074-1c61-435e-b086-0c06657d2ab9" (UID: "22e0c074-1c61-435e-b086-0c06657d2ab9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:20:59.002976 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.002960 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22e0c074-1c61-435e-b086-0c06657d2ab9-istio-podinfo" pod "22e0c074-1c61-435e-b086-0c06657d2ab9" (UID: "22e0c074-1c61-435e-b086-0c06657d2ab9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:20:59.028397 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.028363 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b1ffb61-dsvfq"] Oct 30 02:20:59.031187 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.031156 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b1ffb61-dsvfq"] Oct 30 02:20:59.101716 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.101675 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7nh2\" (UniqueName: \"kubernetes.io/projected/22e0c074-1c61-435e-b086-0c06657d2ab9-kube-api-access-w7nh2\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:59.101716 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.101710 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22e0c074-1c61-435e-b086-0c06657d2ab9-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:59.101716 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.101722 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22e0c074-1c61-435e-b086-0c06657d2ab9-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:59.102011 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.101731 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22e0c074-1c61-435e-b086-0c06657d2ab9-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:59.102011 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.101740 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22e0c074-1c61-435e-b086-0c06657d2ab9-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:59.102011 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.101783 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22e0c074-1c61-435e-b086-0c06657d2ab9-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:20:59.237523 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.237481 6572 generic.go:356] "Generic (PLEG): container finished" podID="22e0c074-1c61-435e-b086-0c06657d2ab9" containerID="3de5bdb45184edb4b47cf4041bfbcf0ba9cb5ee42b0383d6f481fa51c99b312d" exitCode=0 Oct 30 02:20:59.237728 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.237543 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" Oct 30 02:20:59.237728 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.237563 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" event={"ID":"22e0c074-1c61-435e-b086-0c06657d2ab9","Type":"ContainerDied","Data":"3de5bdb45184edb4b47cf4041bfbcf0ba9cb5ee42b0383d6f481fa51c99b312d"} Oct 30 02:20:59.237728 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.237632 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh" event={"ID":"22e0c074-1c61-435e-b086-0c06657d2ab9","Type":"ContainerDied","Data":"06dcee2519e219fd4c58718f6204a075a1d47930701a55583af3d35751cbcc6d"} Oct 30 02:20:59.237728 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.237662 6572 scope.go:122] "RemoveContainer" containerID="3de5bdb45184edb4b47cf4041bfbcf0ba9cb5ee42b0383d6f481fa51c99b312d" Oct 30 02:20:59.254349 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.254323 6572 scope.go:122] "RemoveContainer" containerID="a61525cca28467846d8814fad0e50b1779a391008b10a753031c8f7c52162699" Oct 30 02:20:59.287973 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.287790 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:59.288788 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.288741 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh"] Oct 30 02:20:59.291433 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.291411 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-f1edaf-openshift-default-7df7c5c97-5ndqh"] Oct 30 02:20:59.293831 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.293811 6572 scope.go:122] "RemoveContainer" containerID="3de5bdb45184edb4b47cf4041bfbcf0ba9cb5ee42b0383d6f481fa51c99b312d" Oct 30 02:20:59.294213 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:20:59.294193 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3de5bdb45184edb4b47cf4041bfbcf0ba9cb5ee42b0383d6f481fa51c99b312d\": container with ID starting with 3de5bdb45184edb4b47cf4041bfbcf0ba9cb5ee42b0383d6f481fa51c99b312d not found: ID does not exist" containerID="3de5bdb45184edb4b47cf4041bfbcf0ba9cb5ee42b0383d6f481fa51c99b312d" Oct 30 02:20:59.294277 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.294222 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3de5bdb45184edb4b47cf4041bfbcf0ba9cb5ee42b0383d6f481fa51c99b312d"} err="failed to get container status \"3de5bdb45184edb4b47cf4041bfbcf0ba9cb5ee42b0383d6f481fa51c99b312d\": rpc error: code = NotFound desc = could not find container \"3de5bdb45184edb4b47cf4041bfbcf0ba9cb5ee42b0383d6f481fa51c99b312d\": container with ID starting with 3de5bdb45184edb4b47cf4041bfbcf0ba9cb5ee42b0383d6f481fa51c99b312d not found: ID does not exist" Oct 30 02:20:59.294277 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.294240 6572 scope.go:122] "RemoveContainer" containerID="a61525cca28467846d8814fad0e50b1779a391008b10a753031c8f7c52162699" Oct 30 02:20:59.294549 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:20:59.294531 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a61525cca28467846d8814fad0e50b1779a391008b10a753031c8f7c52162699\": container with ID starting with a61525cca28467846d8814fad0e50b1779a391008b10a753031c8f7c52162699 not found: ID does not exist" containerID="a61525cca28467846d8814fad0e50b1779a391008b10a753031c8f7c52162699" Oct 30 02:20:59.294625 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.294555 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a61525cca28467846d8814fad0e50b1779a391008b10a753031c8f7c52162699"} err="failed to get container status \"a61525cca28467846d8814fad0e50b1779a391008b10a753031c8f7c52162699\": rpc error: code = NotFound desc = could not find container \"a61525cca28467846d8814fad0e50b1779a391008b10a753031c8f7c52162699\": container with ID starting with a61525cca28467846d8814fad0e50b1779a391008b10a753031c8f7c52162699 not found: ID does not exist" Oct 30 02:20:59.982709 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.982675 6572 scope.go:122] "RemoveContainer" containerID="ace2cc1ea69c1a546fec74f78528f9a27c384c067ef9f13dd04c0ebd40e5b58e" Oct 30 02:20:59.993272 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:20:59.993248 6572 scope.go:122] "RemoveContainer" containerID="64d073a7bb553ff2ad0b96082836e87142299b8974a9c3161ad4e03929bad220" Oct 30 02:21:00.542219 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:00.542180 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22e0c074-1c61-435e-b086-0c06657d2ab9" path="/var/lib/kubelet/pods/22e0c074-1c61-435e-b086-0c06657d2ab9/volumes" Oct 30 02:21:00.542613 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:00.542599 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475" path="/var/lib/kubelet/pods/c3bd2ccc-4ec5-4e6b-ba5b-5df3dc003475/volumes" Oct 30 02:21:00.781843 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:00.781805 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:00.817937 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:00.817835 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1edaf-55c447f959-nh7w8" podUID="f730cd49-e419-4643-bbaa-c5e7d7ab82eb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:21:02.295504 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:02.295465 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:03.794568 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:03.794469 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:04.840236 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:04.840144 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:05.027404 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:05.027365 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:05.363475 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:05.363440 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:06.243835 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:06.243805 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1edaf-55c447f959-nh7w8" Oct 30 02:21:06.271078 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:06.271043 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f730cd49-e419-4643-bbaa-c5e7d7ab82eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f730cd49-e419-4643-bbaa-c5e7d7ab82eb-plugins-config-volume\") pod \"f730cd49-e419-4643-bbaa-c5e7d7ab82eb\" (UID: \"f730cd49-e419-4643-bbaa-c5e7d7ab82eb\") " Oct 30 02:21:06.271253 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:06.271189 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f730cd49-e419-4643-bbaa-c5e7d7ab82eb-kube-api-access-pvpkl\" (UniqueName: \"kubernetes.io/projected/f730cd49-e419-4643-bbaa-c5e7d7ab82eb-kube-api-access-pvpkl\") pod \"f730cd49-e419-4643-bbaa-c5e7d7ab82eb\" (UID: \"f730cd49-e419-4643-bbaa-c5e7d7ab82eb\") " Oct 30 02:21:06.271450 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:06.271424 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f730cd49-e419-4643-bbaa-c5e7d7ab82eb-plugins-config-volume" pod "f730cd49-e419-4643-bbaa-c5e7d7ab82eb" (UID: "f730cd49-e419-4643-bbaa-c5e7d7ab82eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:21:06.271609 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:06.271588 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f730cd49-e419-4643-bbaa-c5e7d7ab82eb-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:21:06.273372 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:06.273351 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f730cd49-e419-4643-bbaa-c5e7d7ab82eb-kube-api-access-pvpkl" pod "f730cd49-e419-4643-bbaa-c5e7d7ab82eb" (UID: "f730cd49-e419-4643-bbaa-c5e7d7ab82eb"). InnerVolumeSpecName "kube-api-access-pvpkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:21:06.290291 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:06.290263 6572 generic.go:356] "Generic (PLEG): container finished" podID="f730cd49-e419-4643-bbaa-c5e7d7ab82eb" containerID="55566dbaf540987bdcd8feab8f1669a9896888c89dfe17575dd7be9de4a6c940" exitCode=0 Oct 30 02:21:06.290479 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:06.290335 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1edaf-55c447f959-nh7w8" Oct 30 02:21:06.290479 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:06.290356 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1edaf-55c447f959-nh7w8" event={"ID":"f730cd49-e419-4643-bbaa-c5e7d7ab82eb","Type":"ContainerDied","Data":"55566dbaf540987bdcd8feab8f1669a9896888c89dfe17575dd7be9de4a6c940"} Oct 30 02:21:06.290479 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:06.290416 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1edaf-55c447f959-nh7w8" event={"ID":"f730cd49-e419-4643-bbaa-c5e7d7ab82eb","Type":"ContainerDied","Data":"d7e1546974e0a0d5b7302293c3798df14982ef778cd6d9cbc28cf0c58ac494fa"} Oct 30 02:21:06.290479 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:06.290447 6572 scope.go:122] "RemoveContainer" containerID="55566dbaf540987bdcd8feab8f1669a9896888c89dfe17575dd7be9de4a6c940" Oct 30 02:21:06.302959 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:06.302940 6572 scope.go:122] "RemoveContainer" containerID="55566dbaf540987bdcd8feab8f1669a9896888c89dfe17575dd7be9de4a6c940" Oct 30 02:21:06.303282 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:21:06.303257 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55566dbaf540987bdcd8feab8f1669a9896888c89dfe17575dd7be9de4a6c940\": container with ID starting with 55566dbaf540987bdcd8feab8f1669a9896888c89dfe17575dd7be9de4a6c940 not found: ID does not exist" containerID="55566dbaf540987bdcd8feab8f1669a9896888c89dfe17575dd7be9de4a6c940" Oct 30 02:21:06.303346 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:06.303296 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55566dbaf540987bdcd8feab8f1669a9896888c89dfe17575dd7be9de4a6c940"} err="failed to get container status \"55566dbaf540987bdcd8feab8f1669a9896888c89dfe17575dd7be9de4a6c940\": rpc error: code = NotFound desc = could not find container \"55566dbaf540987bdcd8feab8f1669a9896888c89dfe17575dd7be9de4a6c940\": container with ID starting with 55566dbaf540987bdcd8feab8f1669a9896888c89dfe17575dd7be9de4a6c940 not found: ID does not exist" Oct 30 02:21:06.342598 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:06.342559 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f1edaf-55c447f959-nh7w8"] Oct 30 02:21:06.348947 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:06.348916 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f1edaf-55c447f959-nh7w8"] Oct 30 02:21:06.372767 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:06.372710 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvpkl\" (UniqueName: \"kubernetes.io/projected/f730cd49-e419-4643-bbaa-c5e7d7ab82eb-kube-api-access-pvpkl\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:21:06.541724 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:06.541643 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f730cd49-e419-4643-bbaa-c5e7d7ab82eb" path="/var/lib/kubelet/pods/f730cd49-e419-4643-bbaa-c5e7d7ab82eb/volumes" Oct 30 02:21:06.981709 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:06.981674 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:08.660689 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:08.660653 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:09.592204 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:09.592167 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:09.878488 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:09.878405 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:10.372123 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:10.372088 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:10.391422 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:10.391388 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-abb30f-rndpf"] Oct 30 02:21:10.404909 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:10.404883 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-abb30f-rndpf"] Oct 30 02:21:10.541225 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:10.541187 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2df25a6-7150-44bd-b38a-6afe82241801" path="/var/lib/kubelet/pods/e2df25a6-7150-44bd-b38a-6afe82241801/volumes" Oct 30 02:21:10.559302 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:10.559266 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:11.009886 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:11.009858 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:12.099233 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:12.099192 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:13.699866 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:13.699834 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:14.688118 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:21:14.688075 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:21:15.200652 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:15.200596 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:16.799297 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:16.799257 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:17.172963 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:17.172928 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:17.390714 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:17.390677 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:17.397175 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:17.397146 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p"] Oct 30 02:21:17.397631 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:17.397601 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" podUID="414b871d-0343-42a6-b560-0be9f9ff4fce" containerName="istio-proxy" containerID="cri-o://8f2d924a182f67c7f12aab304b2d722eac721eced21b25f79feab7136924d149" gracePeriod=30 Oct 30 02:21:17.624271 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:17.624171 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:17.958923 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:17.958883 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-861cd2-g5bmz"] Oct 30 02:21:17.959554 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:17.959536 6572 cpu_manager.go:414] "containerMap: removing container" podUID="22e0c074-1c61-435e-b086-0c06657d2ab9" containerName="istio-proxy" Oct 30 02:21:17.959617 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:17.959558 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22e0c074-1c61-435e-b086-0c06657d2ab9" containerName="istio-proxy" Oct 30 02:21:17.959617 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:17.959575 6572 cpu_manager.go:414] "containerMap: removing container" podUID="e2df25a6-7150-44bd-b38a-6afe82241801" containerName="revoke-keys" Oct 30 02:21:17.959617 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:17.959583 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2df25a6-7150-44bd-b38a-6afe82241801" containerName="revoke-keys" Oct 30 02:21:17.959617 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:17.959596 6572 cpu_manager.go:414] "containerMap: removing container" podUID="f730cd49-e419-4643-bbaa-c5e7d7ab82eb" containerName="payload-pre-processing" Oct 30 02:21:17.959617 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:17.959605 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f730cd49-e419-4643-bbaa-c5e7d7ab82eb" containerName="payload-pre-processing" Oct 30 02:21:17.959892 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:17.959723 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="22e0c074-1c61-435e-b086-0c06657d2ab9" containerName="istio-proxy" Oct 30 02:21:17.959892 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:17.959741 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="22e0c074-1c61-435e-b086-0c06657d2ab9" containerName="istio-proxy" Oct 30 02:21:17.959892 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:17.959769 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2df25a6-7150-44bd-b38a-6afe82241801" containerName="revoke-keys" Oct 30 02:21:17.959892 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:17.959780 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="f730cd49-e419-4643-bbaa-c5e7d7ab82eb" containerName="payload-pre-processing" Oct 30 02:21:17.964820 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:17.964794 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-861cd2-g5bmz" Oct 30 02:21:17.981525 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:17.981491 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-861cd2-g5bmz"] Oct 30 02:21:17.989041 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:17.988988 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:21:17.989041 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:17.989007 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:21:18.003257 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:18.003222 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:21:18.003257 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:18.003230 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:21:18.087498 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:18.087458 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twpkc\" (UniqueName: \"kubernetes.io/projected/3cea286f-8506-49b5-b530-d2fc80ffd298-kube-api-access-twpkc\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-861cd2-g5bmz\" (UID: \"3cea286f-8506-49b5-b530-d2fc80ffd298\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-861cd2-g5bmz" Oct 30 02:21:18.087690 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:18.087537 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cea286f-8506-49b5-b530-d2fc80ffd298-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-861cd2-g5bmz\" (UID: \"3cea286f-8506-49b5-b530-d2fc80ffd298\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-861cd2-g5bmz" Oct 30 02:21:18.087690 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:18.087655 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cea286f-8506-49b5-b530-d2fc80ffd298-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-861cd2-g5bmz\" (UID: \"3cea286f-8506-49b5-b530-d2fc80ffd298\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-861cd2-g5bmz" Oct 30 02:21:18.188924 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:18.188883 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twpkc\" (UniqueName: \"kubernetes.io/projected/3cea286f-8506-49b5-b530-d2fc80ffd298-kube-api-access-twpkc\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-861cd2-g5bmz\" (UID: \"3cea286f-8506-49b5-b530-d2fc80ffd298\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-861cd2-g5bmz" Oct 30 02:21:18.189119 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:18.188954 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cea286f-8506-49b5-b530-d2fc80ffd298-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-861cd2-g5bmz\" (UID: \"3cea286f-8506-49b5-b530-d2fc80ffd298\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-861cd2-g5bmz" Oct 30 02:21:18.189119 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:18.188989 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cea286f-8506-49b5-b530-d2fc80ffd298-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-861cd2-g5bmz\" (UID: \"3cea286f-8506-49b5-b530-d2fc80ffd298\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-861cd2-g5bmz" Oct 30 02:21:18.189661 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:18.189639 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cea286f-8506-49b5-b530-d2fc80ffd298-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-861cd2-g5bmz\" (UID: \"3cea286f-8506-49b5-b530-d2fc80ffd298\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-861cd2-g5bmz" Oct 30 02:21:18.191452 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:18.191429 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cea286f-8506-49b5-b530-d2fc80ffd298-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-861cd2-g5bmz\" (UID: \"3cea286f-8506-49b5-b530-d2fc80ffd298\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-861cd2-g5bmz" Oct 30 02:21:18.207495 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:18.207467 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twpkc\" (UniqueName: \"kubernetes.io/projected/3cea286f-8506-49b5-b530-d2fc80ffd298-kube-api-access-twpkc\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-861cd2-g5bmz\" (UID: \"3cea286f-8506-49b5-b530-d2fc80ffd298\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-861cd2-g5bmz" Oct 30 02:21:18.279254 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:18.279158 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-861cd2-g5bmz" Oct 30 02:21:18.473082 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:18.473040 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:18.578516 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:18.578491 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-861cd2-g5bmz"] Oct 30 02:21:18.580795 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:21:18.580737 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cea286f_8506_49b5_b530_d2fc80ffd298.slice/crio-1100d7a2dafb571bb33d1f4468836e110388c553ae7f5f5ed28e87c8c6165449 WatchSource:0}: Error finding container 1100d7a2dafb571bb33d1f4468836e110388c553ae7f5f5ed28e87c8c6165449: Status 404 returned error can't find the container with id 1100d7a2dafb571bb33d1f4468836e110388c553ae7f5f5ed28e87c8c6165449 Oct 30 02:21:19.390201 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:19.390109 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-861cd2-g5bmz" event={"ID":"3cea286f-8506-49b5-b530-d2fc80ffd298","Type":"ContainerStarted","Data":"98e87df7d17143645feb49b83157d2a35c2be32896107a9fb9c72d72a429e883"} Oct 30 02:21:19.390201 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:19.390159 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-861cd2-g5bmz" event={"ID":"3cea286f-8506-49b5-b530-d2fc80ffd298","Type":"ContainerStarted","Data":"1100d7a2dafb571bb33d1f4468836e110388c553ae7f5f5ed28e87c8c6165449"} Oct 30 02:21:19.432017 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:19.431954 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-861cd2-g5bmz" podStartSLOduration=2.431937104 podStartE2EDuration="2.431937104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:21:19.428849583 +0000 UTC m=+3805.005274985" watchObservedRunningTime="2026-10-30 02:21:19.431937104 +0000 UTC m=+3805.008362506" Oct 30 02:21:20.000802 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:20.000743 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:20.400891 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:20.400801 6572 generic.go:356] "Generic (PLEG): container finished" podID="3cea286f-8506-49b5-b530-d2fc80ffd298" containerID="98e87df7d17143645feb49b83157d2a35c2be32896107a9fb9c72d72a429e883" exitCode=0 Oct 30 02:21:20.400891 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:20.400863 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-861cd2-g5bmz" event={"ID":"3cea286f-8506-49b5-b530-d2fc80ffd298","Type":"ContainerDied","Data":"98e87df7d17143645feb49b83157d2a35c2be32896107a9fb9c72d72a429e883"} Oct 30 02:21:21.554657 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:21.554628 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-861cd2-g5bmz" Oct 30 02:21:21.621857 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:21.621820 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cea286f-8506-49b5-b530-d2fc80ffd298-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cea286f-8506-49b5-b530-d2fc80ffd298-maas-api-xks-ca\") pod \"3cea286f-8506-49b5-b530-d2fc80ffd298\" (UID: \"3cea286f-8506-49b5-b530-d2fc80ffd298\") " Oct 30 02:21:21.622087 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:21.621988 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cea286f-8506-49b5-b530-d2fc80ffd298-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cea286f-8506-49b5-b530-d2fc80ffd298-maas-api-openshift-ca\") pod \"3cea286f-8506-49b5-b530-d2fc80ffd298\" (UID: \"3cea286f-8506-49b5-b530-d2fc80ffd298\") " Oct 30 02:21:21.622087 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:21.622019 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cea286f-8506-49b5-b530-d2fc80ffd298-kube-api-access-twpkc\" (UniqueName: \"kubernetes.io/projected/3cea286f-8506-49b5-b530-d2fc80ffd298-kube-api-access-twpkc\") pod \"3cea286f-8506-49b5-b530-d2fc80ffd298\" (UID: \"3cea286f-8506-49b5-b530-d2fc80ffd298\") " Oct 30 02:21:21.622419 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:21.622388 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cea286f-8506-49b5-b530-d2fc80ffd298-maas-api-openshift-ca" pod "3cea286f-8506-49b5-b530-d2fc80ffd298" (UID: "3cea286f-8506-49b5-b530-d2fc80ffd298"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:21:21.623990 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:21.623963 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cea286f-8506-49b5-b530-d2fc80ffd298-maas-api-xks-ca" pod "3cea286f-8506-49b5-b530-d2fc80ffd298" (UID: "3cea286f-8506-49b5-b530-d2fc80ffd298"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:21:21.624110 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:21.624068 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cea286f-8506-49b5-b530-d2fc80ffd298-kube-api-access-twpkc" pod "3cea286f-8506-49b5-b530-d2fc80ffd298" (UID: "3cea286f-8506-49b5-b530-d2fc80ffd298"). InnerVolumeSpecName "kube-api-access-twpkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:21:21.723111 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:21.723070 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cea286f-8506-49b5-b530-d2fc80ffd298-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:21:21.723111 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:21.723104 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twpkc\" (UniqueName: \"kubernetes.io/projected/3cea286f-8506-49b5-b530-d2fc80ffd298-kube-api-access-twpkc\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:21:21.723111 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:21.723117 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cea286f-8506-49b5-b530-d2fc80ffd298-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:21:22.417732 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:22.417703 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-861cd2-g5bmz" Oct 30 02:21:22.417732 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:22.417714 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-861cd2-g5bmz" event={"ID":"3cea286f-8506-49b5-b530-d2fc80ffd298","Type":"ContainerDied","Data":"1100d7a2dafb571bb33d1f4468836e110388c553ae7f5f5ed28e87c8c6165449"} Oct 30 02:21:22.417945 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:22.417822 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1100d7a2dafb571bb33d1f4468836e110388c553ae7f5f5ed28e87c8c6165449" Oct 30 02:21:22.544645 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:22.544614 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:22.653091 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:22.653065 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 02:21:22.731614 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:22.731510 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/414b871d-0343-42a6-b560-0be9f9ff4fce-kube-api-access-ghnnn\" (UniqueName: \"kubernetes.io/projected/414b871d-0343-42a6-b560-0be9f9ff4fce-kube-api-access-ghnnn\") pod \"414b871d-0343-42a6-b560-0be9f9ff4fce\" (UID: \"414b871d-0343-42a6-b560-0be9f9ff4fce\") " Oct 30 02:21:22.731614 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:22.731558 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/414b871d-0343-42a6-b560-0be9f9ff4fce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/414b871d-0343-42a6-b560-0be9f9ff4fce-workload-socket\") pod \"414b871d-0343-42a6-b560-0be9f9ff4fce\" (UID: \"414b871d-0343-42a6-b560-0be9f9ff4fce\") " Oct 30 02:21:22.731614 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:22.731596 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/414b871d-0343-42a6-b560-0be9f9ff4fce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/414b871d-0343-42a6-b560-0be9f9ff4fce-istio-podinfo\") pod \"414b871d-0343-42a6-b560-0be9f9ff4fce\" (UID: \"414b871d-0343-42a6-b560-0be9f9ff4fce\") " Oct 30 02:21:22.731614 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:22.731616 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/414b871d-0343-42a6-b560-0be9f9ff4fce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/414b871d-0343-42a6-b560-0be9f9ff4fce-credential-socket\") pod \"414b871d-0343-42a6-b560-0be9f9ff4fce\" (UID: \"414b871d-0343-42a6-b560-0be9f9ff4fce\") " Oct 30 02:21:22.732033 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:22.731636 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/414b871d-0343-42a6-b560-0be9f9ff4fce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/414b871d-0343-42a6-b560-0be9f9ff4fce-istio-data\") pod \"414b871d-0343-42a6-b560-0be9f9ff4fce\" (UID: \"414b871d-0343-42a6-b560-0be9f9ff4fce\") " Oct 30 02:21:22.732033 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:22.731656 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/414b871d-0343-42a6-b560-0be9f9ff4fce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/414b871d-0343-42a6-b560-0be9f9ff4fce-istiod-ca-cert\") pod \"414b871d-0343-42a6-b560-0be9f9ff4fce\" (UID: \"414b871d-0343-42a6-b560-0be9f9ff4fce\") " Oct 30 02:21:22.732033 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:22.731673 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/414b871d-0343-42a6-b560-0be9f9ff4fce-istio-token\" (UniqueName: \"kubernetes.io/projected/414b871d-0343-42a6-b560-0be9f9ff4fce-istio-token\") pod \"414b871d-0343-42a6-b560-0be9f9ff4fce\" (UID: \"414b871d-0343-42a6-b560-0be9f9ff4fce\") " Oct 30 02:21:22.732033 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:22.731787 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/414b871d-0343-42a6-b560-0be9f9ff4fce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/414b871d-0343-42a6-b560-0be9f9ff4fce-istio-envoy\") pod \"414b871d-0343-42a6-b560-0be9f9ff4fce\" (UID: \"414b871d-0343-42a6-b560-0be9f9ff4fce\") " Oct 30 02:21:22.732033 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:22.731857 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/414b871d-0343-42a6-b560-0be9f9ff4fce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/414b871d-0343-42a6-b560-0be9f9ff4fce-workload-certs\") pod \"414b871d-0343-42a6-b560-0be9f9ff4fce\" (UID: \"414b871d-0343-42a6-b560-0be9f9ff4fce\") " Oct 30 02:21:22.732328 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:22.732033 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/414b871d-0343-42a6-b560-0be9f9ff4fce-istio-data" pod "414b871d-0343-42a6-b560-0be9f9ff4fce" (UID: "414b871d-0343-42a6-b560-0be9f9ff4fce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:21:22.732328 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:22.732051 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/414b871d-0343-42a6-b560-0be9f9ff4fce-workload-socket" pod "414b871d-0343-42a6-b560-0be9f9ff4fce" (UID: "414b871d-0343-42a6-b560-0be9f9ff4fce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:21:22.732328 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:22.732143 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/414b871d-0343-42a6-b560-0be9f9ff4fce-credential-socket" pod "414b871d-0343-42a6-b560-0be9f9ff4fce" (UID: "414b871d-0343-42a6-b560-0be9f9ff4fce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:21:22.732328 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:22.732264 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/414b871d-0343-42a6-b560-0be9f9ff4fce-workload-certs" pod "414b871d-0343-42a6-b560-0be9f9ff4fce" (UID: "414b871d-0343-42a6-b560-0be9f9ff4fce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:21:22.732591 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:22.732341 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/414b871d-0343-42a6-b560-0be9f9ff4fce-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:21:22.732591 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:22.732363 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/414b871d-0343-42a6-b560-0be9f9ff4fce-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:21:22.732591 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:22.732377 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/414b871d-0343-42a6-b560-0be9f9ff4fce-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:21:22.732591 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:22.732490 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/414b871d-0343-42a6-b560-0be9f9ff4fce-istiod-ca-cert" pod "414b871d-0343-42a6-b560-0be9f9ff4fce" (UID: "414b871d-0343-42a6-b560-0be9f9ff4fce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:21:22.734127 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:22.734103 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/414b871d-0343-42a6-b560-0be9f9ff4fce-istio-podinfo" pod "414b871d-0343-42a6-b560-0be9f9ff4fce" (UID: "414b871d-0343-42a6-b560-0be9f9ff4fce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:21:22.734246 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:22.734194 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/414b871d-0343-42a6-b560-0be9f9ff4fce-istio-envoy" pod "414b871d-0343-42a6-b560-0be9f9ff4fce" (UID: "414b871d-0343-42a6-b560-0be9f9ff4fce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:21:22.734296 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:22.734261 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/414b871d-0343-42a6-b560-0be9f9ff4fce-kube-api-access-ghnnn" pod "414b871d-0343-42a6-b560-0be9f9ff4fce" (UID: "414b871d-0343-42a6-b560-0be9f9ff4fce"). InnerVolumeSpecName "kube-api-access-ghnnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:21:22.734296 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:22.734277 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/414b871d-0343-42a6-b560-0be9f9ff4fce-istio-token" pod "414b871d-0343-42a6-b560-0be9f9ff4fce" (UID: "414b871d-0343-42a6-b560-0be9f9ff4fce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:21:22.833884 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:22.833848 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/414b871d-0343-42a6-b560-0be9f9ff4fce-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:21:22.833884 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:22.833884 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghnnn\" (UniqueName: \"kubernetes.io/projected/414b871d-0343-42a6-b560-0be9f9ff4fce-kube-api-access-ghnnn\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:21:22.834160 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:22.833897 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/414b871d-0343-42a6-b560-0be9f9ff4fce-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:21:22.834160 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:22.833906 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/414b871d-0343-42a6-b560-0be9f9ff4fce-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:21:22.834160 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:22.833915 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/414b871d-0343-42a6-b560-0be9f9ff4fce-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:21:22.834160 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:22.833923 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/414b871d-0343-42a6-b560-0be9f9ff4fce-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:21:22.977155 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:22.977110 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:23.428229 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:23.428190 6572 generic.go:356] "Generic (PLEG): container finished" podID="414b871d-0343-42a6-b560-0be9f9ff4fce" containerID="8f2d924a182f67c7f12aab304b2d722eac721eced21b25f79feab7136924d149" exitCode=0 Oct 30 02:21:23.428393 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:23.428249 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" Oct 30 02:21:23.428393 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:23.428278 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" event={"ID":"414b871d-0343-42a6-b560-0be9f9ff4fce","Type":"ContainerDied","Data":"8f2d924a182f67c7f12aab304b2d722eac721eced21b25f79feab7136924d149"} Oct 30 02:21:23.428393 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:23.428334 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p" event={"ID":"414b871d-0343-42a6-b560-0be9f9ff4fce","Type":"ContainerDied","Data":"8d74e2be1beddcf7980a28d5b40d0c2b4c665a270a8ae6c77803a08021693ebd"} Oct 30 02:21:23.428393 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:23.428362 6572 scope.go:122] "RemoveContainer" containerID="8f2d924a182f67c7f12aab304b2d722eac721eced21b25f79feab7136924d149" Oct 30 02:21:23.440432 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:23.440408 6572 scope.go:122] "RemoveContainer" containerID="11c56fe9916e904a2d2119cedb366a949764a0794f4ea155e5fd1c4223fcf754" Oct 30 02:21:23.451078 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:23.451054 6572 scope.go:122] "RemoveContainer" containerID="8f2d924a182f67c7f12aab304b2d722eac721eced21b25f79feab7136924d149" Oct 30 02:21:23.451352 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:21:23.451332 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f2d924a182f67c7f12aab304b2d722eac721eced21b25f79feab7136924d149\": container with ID starting with 8f2d924a182f67c7f12aab304b2d722eac721eced21b25f79feab7136924d149 not found: ID does not exist" containerID="8f2d924a182f67c7f12aab304b2d722eac721eced21b25f79feab7136924d149" Oct 30 02:21:23.451426 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:23.451362 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f2d924a182f67c7f12aab304b2d722eac721eced21b25f79feab7136924d149"} err="failed to get container status \"8f2d924a182f67c7f12aab304b2d722eac721eced21b25f79feab7136924d149\": rpc error: code = NotFound desc = could not find container \"8f2d924a182f67c7f12aab304b2d722eac721eced21b25f79feab7136924d149\": container with ID starting with 8f2d924a182f67c7f12aab304b2d722eac721eced21b25f79feab7136924d149 not found: ID does not exist" Oct 30 02:21:23.451426 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:23.451378 6572 scope.go:122] "RemoveContainer" containerID="11c56fe9916e904a2d2119cedb366a949764a0794f4ea155e5fd1c4223fcf754" Oct 30 02:21:23.451678 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:21:23.451647 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11c56fe9916e904a2d2119cedb366a949764a0794f4ea155e5fd1c4223fcf754\": container with ID starting with 11c56fe9916e904a2d2119cedb366a949764a0794f4ea155e5fd1c4223fcf754 not found: ID does not exist" containerID="11c56fe9916e904a2d2119cedb366a949764a0794f4ea155e5fd1c4223fcf754" Oct 30 02:21:23.451718 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:23.451691 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11c56fe9916e904a2d2119cedb366a949764a0794f4ea155e5fd1c4223fcf754"} err="failed to get container status \"11c56fe9916e904a2d2119cedb366a949764a0794f4ea155e5fd1c4223fcf754\": rpc error: code = NotFound desc = could not find container \"11c56fe9916e904a2d2119cedb366a949764a0794f4ea155e5fd1c4223fcf754\": container with ID starting with 11c56fe9916e904a2d2119cedb366a949764a0794f4ea155e5fd1c4223fcf754 not found: ID does not exist" Oct 30 02:21:23.479909 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:23.479870 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p"] Oct 30 02:21:23.491146 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:23.491112 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-abb30f-openshift-default-59bd8b78df-hhq7p"] Oct 30 02:21:24.543398 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:24.543362 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="414b871d-0343-42a6-b560-0be9f9ff4fce" path="/var/lib/kubelet/pods/414b871d-0343-42a6-b560-0be9f9ff4fce/volumes" Oct 30 02:21:24.592444 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:24.592412 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:26.083359 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:26.083322 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:27.587976 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:27.587939 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:27.830315 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:27.830279 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:28.404393 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:28.404357 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-861cd2-6d4589fbd7-hr89s"] Oct 30 02:21:28.600433 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:28.600351 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:28.802460 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:28.802425 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:28.999285 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:28.999246 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:29.420766 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:29.420708 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:29.611599 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:29.611559 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:29.838487 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:29.838385 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:29.872777 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:29.872713 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-861cd2-g5bmz"] Oct 30 02:21:29.883523 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:29.883487 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-861cd2-g5bmz"] Oct 30 02:21:30.044146 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:30.044109 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:30.541526 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:30.541490 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cea286f-8506-49b5-b530-d2fc80ffd298" path="/var/lib/kubelet/pods/3cea286f-8506-49b5-b530-d2fc80ffd298/volumes" Oct 30 02:21:30.545696 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:30.545666 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:31.278700 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:31.278644 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:32.793061 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:32.793026 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:33.405469 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:33.405422 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-861cd2-6d4589fbd7-hr89s" podUID="963a53c2-66a3-415b-a7b0-2dcf1b48daf0" containerName="payload-pre-processing" containerID="cri-o://97c74579079ad79aa475823bfa55533b63d4d5696c92c0966e6ac3ef93f47fbf" gracePeriod=25 Oct 30 02:21:34.288034 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:34.287990 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:34.842285 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:34.842242 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-861cd2-6d4589fbd7-hr89s" podUID="963a53c2-66a3-415b-a7b0-2dcf1b48daf0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:21:35.784254 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:35.784212 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:36.871135 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:36.871094 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:37.066237 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:37.066203 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:37.267812 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:37.267743 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:37.462634 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:37.462600 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:37.746411 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:37.746370 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:37.910422 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:37.910382 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cf82f8-pl66j"] Oct 30 02:21:37.910902 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:37.910842 6572 cpu_manager.go:414] "containerMap: removing container" podUID="22e0c074-1c61-435e-b086-0c06657d2ab9" containerName="istio-proxy" Oct 30 02:21:37.910902 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:37.910855 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22e0c074-1c61-435e-b086-0c06657d2ab9" containerName="istio-proxy" Oct 30 02:21:37.910902 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:37.910868 6572 cpu_manager.go:414] "containerMap: removing container" podUID="3cea286f-8506-49b5-b530-d2fc80ffd298" containerName="revoke-keys" Oct 30 02:21:37.910902 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:37.910873 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cea286f-8506-49b5-b530-d2fc80ffd298" containerName="revoke-keys" Oct 30 02:21:37.910902 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:37.910886 6572 cpu_manager.go:414] "containerMap: removing container" podUID="414b871d-0343-42a6-b560-0be9f9ff4fce" containerName="istio-proxy" Oct 30 02:21:37.910902 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:37.910893 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="414b871d-0343-42a6-b560-0be9f9ff4fce" containerName="istio-proxy" Oct 30 02:21:37.910902 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:37.910901 6572 cpu_manager.go:414] "containerMap: removing container" podUID="414b871d-0343-42a6-b560-0be9f9ff4fce" containerName="istio-proxy" Oct 30 02:21:37.910902 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:37.910908 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="414b871d-0343-42a6-b560-0be9f9ff4fce" containerName="istio-proxy" Oct 30 02:21:37.911288 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:37.910971 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="414b871d-0343-42a6-b560-0be9f9ff4fce" containerName="istio-proxy" Oct 30 02:21:37.911288 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:37.910980 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="414b871d-0343-42a6-b560-0be9f9ff4fce" containerName="istio-proxy" Oct 30 02:21:37.911288 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:37.910993 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cea286f-8506-49b5-b530-d2fc80ffd298" containerName="revoke-keys" Oct 30 02:21:37.914513 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:37.914491 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cf82f8-pl66j" Oct 30 02:21:37.929499 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:37.929470 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:21:37.940979 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:37.940946 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:21:37.940979 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:37.940947 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:21:37.941203 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:37.940953 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:21:37.943823 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:37.943791 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cf82f8-pl66j"] Oct 30 02:21:38.093397 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:38.093293 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqkg8\" (UniqueName: \"kubernetes.io/projected/733ae104-96c5-40e8-a4cd-8afedb8bf8b8-kube-api-access-sqkg8\") pod \"maas-api-revoke-keys-e2e-worker-w1-cf82f8-pl66j\" (UID: \"733ae104-96c5-40e8-a4cd-8afedb8bf8b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cf82f8-pl66j" Oct 30 02:21:38.093562 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:38.093444 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/733ae104-96c5-40e8-a4cd-8afedb8bf8b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-cf82f8-pl66j\" (UID: \"733ae104-96c5-40e8-a4cd-8afedb8bf8b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cf82f8-pl66j" Oct 30 02:21:38.093562 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:38.093502 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/733ae104-96c5-40e8-a4cd-8afedb8bf8b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-cf82f8-pl66j\" (UID: \"733ae104-96c5-40e8-a4cd-8afedb8bf8b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cf82f8-pl66j" Oct 30 02:21:38.194240 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:38.194190 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqkg8\" (UniqueName: \"kubernetes.io/projected/733ae104-96c5-40e8-a4cd-8afedb8bf8b8-kube-api-access-sqkg8\") pod \"maas-api-revoke-keys-e2e-worker-w1-cf82f8-pl66j\" (UID: \"733ae104-96c5-40e8-a4cd-8afedb8bf8b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cf82f8-pl66j" Oct 30 02:21:38.194435 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:38.194271 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/733ae104-96c5-40e8-a4cd-8afedb8bf8b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-cf82f8-pl66j\" (UID: \"733ae104-96c5-40e8-a4cd-8afedb8bf8b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cf82f8-pl66j" Oct 30 02:21:38.194435 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:38.194305 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/733ae104-96c5-40e8-a4cd-8afedb8bf8b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-cf82f8-pl66j\" (UID: \"733ae104-96c5-40e8-a4cd-8afedb8bf8b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cf82f8-pl66j" Oct 30 02:21:38.194975 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:38.194955 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/733ae104-96c5-40e8-a4cd-8afedb8bf8b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-cf82f8-pl66j\" (UID: \"733ae104-96c5-40e8-a4cd-8afedb8bf8b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cf82f8-pl66j" Oct 30 02:21:38.196844 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:38.196815 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/733ae104-96c5-40e8-a4cd-8afedb8bf8b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-cf82f8-pl66j\" (UID: \"733ae104-96c5-40e8-a4cd-8afedb8bf8b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cf82f8-pl66j" Oct 30 02:21:38.212907 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:38.212866 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqkg8\" (UniqueName: \"kubernetes.io/projected/733ae104-96c5-40e8-a4cd-8afedb8bf8b8-kube-api-access-sqkg8\") pod \"maas-api-revoke-keys-e2e-worker-w1-cf82f8-pl66j\" (UID: \"733ae104-96c5-40e8-a4cd-8afedb8bf8b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cf82f8-pl66j" Oct 30 02:21:38.229526 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:38.229484 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cf82f8-pl66j" Oct 30 02:21:38.532296 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:38.532266 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cf82f8-pl66j"] Oct 30 02:21:38.533982 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:21:38.533948 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod733ae104_96c5_40e8_a4cd_8afedb8bf8b8.slice/crio-fd311a1e42d47711bf60149d26e03e25d1a58790836370fb5174e1e83e4bb563 WatchSource:0}: Error finding container fd311a1e42d47711bf60149d26e03e25d1a58790836370fb5174e1e83e4bb563: Status 404 returned error can't find the container with id fd311a1e42d47711bf60149d26e03e25d1a58790836370fb5174e1e83e4bb563 Oct 30 02:21:38.546335 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:38.546297 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cf82f8-pl66j" event={"ID":"733ae104-96c5-40e8-a4cd-8afedb8bf8b8","Type":"ContainerStarted","Data":"fd311a1e42d47711bf60149d26e03e25d1a58790836370fb5174e1e83e4bb563"} Oct 30 02:21:38.973451 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:38.973418 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:39.028297 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:39.028258 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-679e07df-98pc7"] Oct 30 02:21:39.030549 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:39.030520 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-679e07df-98pc7"] Oct 30 02:21:39.555800 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:39.555737 6572 generic.go:356] "Generic (PLEG): container finished" podID="733ae104-96c5-40e8-a4cd-8afedb8bf8b8" containerID="3ed0a469917628c95c94205f65fea615f4ecef2ecc85d3c62cee3b90e98ac23c" exitCode=0 Oct 30 02:21:39.556036 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:39.555855 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cf82f8-pl66j" event={"ID":"733ae104-96c5-40e8-a4cd-8afedb8bf8b8","Type":"ContainerDied","Data":"3ed0a469917628c95c94205f65fea615f4ecef2ecc85d3c62cee3b90e98ac23c"} Oct 30 02:21:40.479359 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:40.479319 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:40.541386 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:40.541347 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd" path="/var/lib/kubelet/pods/e19c3c2f-04ff-47c0-a75e-5db37c9c0cdd/volumes" Oct 30 02:21:40.697694 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:40.697663 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cf82f8-pl66j" Oct 30 02:21:40.820411 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:40.820318 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/733ae104-96c5-40e8-a4cd-8afedb8bf8b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/733ae104-96c5-40e8-a4cd-8afedb8bf8b8-maas-api-xks-ca\") pod \"733ae104-96c5-40e8-a4cd-8afedb8bf8b8\" (UID: \"733ae104-96c5-40e8-a4cd-8afedb8bf8b8\") " Oct 30 02:21:40.820411 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:40.820357 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/733ae104-96c5-40e8-a4cd-8afedb8bf8b8-kube-api-access-sqkg8\" (UniqueName: \"kubernetes.io/projected/733ae104-96c5-40e8-a4cd-8afedb8bf8b8-kube-api-access-sqkg8\") pod \"733ae104-96c5-40e8-a4cd-8afedb8bf8b8\" (UID: \"733ae104-96c5-40e8-a4cd-8afedb8bf8b8\") " Oct 30 02:21:40.820625 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:40.820441 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/733ae104-96c5-40e8-a4cd-8afedb8bf8b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/733ae104-96c5-40e8-a4cd-8afedb8bf8b8-maas-api-openshift-ca\") pod \"733ae104-96c5-40e8-a4cd-8afedb8bf8b8\" (UID: \"733ae104-96c5-40e8-a4cd-8afedb8bf8b8\") " Oct 30 02:21:40.820846 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:40.820823 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/733ae104-96c5-40e8-a4cd-8afedb8bf8b8-maas-api-openshift-ca" pod "733ae104-96c5-40e8-a4cd-8afedb8bf8b8" (UID: "733ae104-96c5-40e8-a4cd-8afedb8bf8b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:21:40.822457 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:40.822430 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/733ae104-96c5-40e8-a4cd-8afedb8bf8b8-maas-api-xks-ca" pod "733ae104-96c5-40e8-a4cd-8afedb8bf8b8" (UID: "733ae104-96c5-40e8-a4cd-8afedb8bf8b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:21:40.822589 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:40.822572 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/733ae104-96c5-40e8-a4cd-8afedb8bf8b8-kube-api-access-sqkg8" pod "733ae104-96c5-40e8-a4cd-8afedb8bf8b8" (UID: "733ae104-96c5-40e8-a4cd-8afedb8bf8b8"). InnerVolumeSpecName "kube-api-access-sqkg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:21:40.921738 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:40.921697 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/733ae104-96c5-40e8-a4cd-8afedb8bf8b8-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:21:40.921738 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:40.921741 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqkg8\" (UniqueName: \"kubernetes.io/projected/733ae104-96c5-40e8-a4cd-8afedb8bf8b8-kube-api-access-sqkg8\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:21:40.921975 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:40.921764 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/733ae104-96c5-40e8-a4cd-8afedb8bf8b8-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:21:41.572071 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:41.572039 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cf82f8-pl66j" Oct 30 02:21:41.572531 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:41.572084 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cf82f8-pl66j" event={"ID":"733ae104-96c5-40e8-a4cd-8afedb8bf8b8","Type":"ContainerDied","Data":"fd311a1e42d47711bf60149d26e03e25d1a58790836370fb5174e1e83e4bb563"} Oct 30 02:21:41.572531 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:41.572138 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd311a1e42d47711bf60149d26e03e25d1a58790836370fb5174e1e83e4bb563" Oct 30 02:21:42.097012 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:42.096970 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:42.879410 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:42.879377 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:43.674472 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:43.674437 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:44.687344 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:21:44.687306 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:21:44.842574 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:44.842531 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-861cd2-6d4589fbd7-hr89s" podUID="963a53c2-66a3-415b-a7b0-2dcf1b48daf0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:21:45.265834 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:45.265800 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:46.800929 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:46.800897 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:48.341035 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:48.340996 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-cf82f8-6fcc5c744b-gv5bq"] Oct 30 02:21:48.373813 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:48.373774 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-cf82f8-5bcf4d65c9-jv8q2"] Oct 30 02:21:48.430610 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:48.430551 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:48.644038 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:48.643948 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:49.033110 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:49.033067 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad9c077d-vmd8b"] Oct 30 02:21:49.036597 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:49.036557 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-deb53ecd-gfhc8"] Oct 30 02:21:49.045066 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:49.045020 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad9c077d-vmd8b"] Oct 30 02:21:49.047660 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:49.047620 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-deb53ecd-gfhc8"] Oct 30 02:21:49.995328 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:49.995295 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:50.540994 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:50.540957 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba3e43fb-87be-445f-875a-280d6787c4c7" path="/var/lib/kubelet/pods/ba3e43fb-87be-445f-875a-280d6787c4c7/volumes" Oct 30 02:21:50.541399 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:50.541385 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfa07f94-1c0a-4855-8849-5d3e753ab795" path="/var/lib/kubelet/pods/dfa07f94-1c0a-4855-8849-5d3e753ab795/volumes" Oct 30 02:21:51.025873 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:51.025836 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-561b8748-8lbtl"] Oct 30 02:21:51.036867 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:51.036839 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-561b8748-8lbtl"] Oct 30 02:21:51.155858 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:51.155824 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82eb752a-q7gw4"] Oct 30 02:21:51.156265 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:51.156253 6572 cpu_manager.go:414] "containerMap: removing container" podUID="733ae104-96c5-40e8-a4cd-8afedb8bf8b8" containerName="revoke-keys" Oct 30 02:21:51.156319 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:51.156266 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="733ae104-96c5-40e8-a4cd-8afedb8bf8b8" containerName="revoke-keys" Oct 30 02:21:51.156363 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:51.156332 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="733ae104-96c5-40e8-a4cd-8afedb8bf8b8" containerName="revoke-keys" Oct 30 02:21:51.160644 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:51.160626 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82eb752a-q7gw4" Oct 30 02:21:51.179820 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:51.179789 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:21:51.180110 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:51.180056 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:21:51.180239 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:51.180205 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82eb752a-q7gw4"] Oct 30 02:21:51.180435 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:51.180414 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:21:51.180527 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:51.180507 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:21:51.214766 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:51.214699 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e33def86-6f77-4c1f-8734-c575f37e6f6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82eb752a-q7gw4\" (UID: \"e33def86-6f77-4c1f-8734-c575f37e6f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82eb752a-q7gw4" Oct 30 02:21:51.214943 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:51.214875 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e33def86-6f77-4c1f-8734-c575f37e6f6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82eb752a-q7gw4\" (UID: \"e33def86-6f77-4c1f-8734-c575f37e6f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82eb752a-q7gw4" Oct 30 02:21:51.214998 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:51.214938 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8hkn\" (UniqueName: \"kubernetes.io/projected/e33def86-6f77-4c1f-8734-c575f37e6f6e-kube-api-access-x8hkn\") pod \"maas-api-revoke-sub-82eb752a-q7gw4\" (UID: \"e33def86-6f77-4c1f-8734-c575f37e6f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82eb752a-q7gw4" Oct 30 02:21:51.316074 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:51.315974 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e33def86-6f77-4c1f-8734-c575f37e6f6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82eb752a-q7gw4\" (UID: \"e33def86-6f77-4c1f-8734-c575f37e6f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82eb752a-q7gw4" Oct 30 02:21:51.316074 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:51.316065 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e33def86-6f77-4c1f-8734-c575f37e6f6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82eb752a-q7gw4\" (UID: \"e33def86-6f77-4c1f-8734-c575f37e6f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82eb752a-q7gw4" Oct 30 02:21:51.316317 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:51.316113 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8hkn\" (UniqueName: \"kubernetes.io/projected/e33def86-6f77-4c1f-8734-c575f37e6f6e-kube-api-access-x8hkn\") pod \"maas-api-revoke-sub-82eb752a-q7gw4\" (UID: \"e33def86-6f77-4c1f-8734-c575f37e6f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82eb752a-q7gw4" Oct 30 02:21:51.316676 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:51.316654 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e33def86-6f77-4c1f-8734-c575f37e6f6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82eb752a-q7gw4\" (UID: \"e33def86-6f77-4c1f-8734-c575f37e6f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82eb752a-q7gw4" Oct 30 02:21:51.318524 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:51.318504 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e33def86-6f77-4c1f-8734-c575f37e6f6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82eb752a-q7gw4\" (UID: \"e33def86-6f77-4c1f-8734-c575f37e6f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82eb752a-q7gw4" Oct 30 02:21:51.335652 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:51.335622 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8hkn\" (UniqueName: \"kubernetes.io/projected/e33def86-6f77-4c1f-8734-c575f37e6f6e-kube-api-access-x8hkn\") pod \"maas-api-revoke-sub-82eb752a-q7gw4\" (UID: \"e33def86-6f77-4c1f-8734-c575f37e6f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82eb752a-q7gw4" Oct 30 02:21:51.474419 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:51.474376 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82eb752a-q7gw4" Oct 30 02:21:51.776154 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:51.776127 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82eb752a-q7gw4"] Oct 30 02:21:51.778771 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:21:51.778724 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode33def86_6f77_4c1f_8734_c575f37e6f6e.slice/crio-7a5806719a712246f322b5c4dd97a97d9608684b14bc1373144d57a29204aa67 WatchSource:0}: Error finding container 7a5806719a712246f322b5c4dd97a97d9608684b14bc1373144d57a29204aa67: Status 404 returned error can't find the container with id 7a5806719a712246f322b5c4dd97a97d9608684b14bc1373144d57a29204aa67 Oct 30 02:21:52.541447 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:52.541416 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5211f68e-c27b-4d50-a49d-b47ea4adf58a" path="/var/lib/kubelet/pods/5211f68e-c27b-4d50-a49d-b47ea4adf58a/volumes" Oct 30 02:21:52.655478 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:52.655449 6572 generic.go:356] "Generic (PLEG): container finished" podID="e33def86-6f77-4c1f-8734-c575f37e6f6e" containerID="541e1e474012b71c782b8037cbae9c144966a0c7c18048157dfbe73748e055a1" exitCode=0 Oct 30 02:21:52.655607 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:52.655503 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82eb752a-q7gw4" event={"ID":"e33def86-6f77-4c1f-8734-c575f37e6f6e","Type":"ContainerDied","Data":"541e1e474012b71c782b8037cbae9c144966a0c7c18048157dfbe73748e055a1"} Oct 30 02:21:52.655607 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:52.655544 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82eb752a-q7gw4" event={"ID":"e33def86-6f77-4c1f-8734-c575f37e6f6e","Type":"ContainerStarted","Data":"7a5806719a712246f322b5c4dd97a97d9608684b14bc1373144d57a29204aa67"} Oct 30 02:21:53.342060 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:53.342000 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-cf82f8-6fcc5c744b-gv5bq" podUID="b1a65b5e-39ec-48ca-9bfc-34e207413e49" containerName="payload-processing" containerID="cri-o://782dcae789f98a60b662b84c9d6b7c664225e758873ec5c9815f53e11b8c44ea" gracePeriod=25 Oct 30 02:21:53.374565 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:53.374500 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cf82f8-5bcf4d65c9-jv8q2" podUID="4fcc6142-79ba-4dcc-b037-6b2fa054a9e3" containerName="payload-pre-processing" containerID="cri-o://b9a9d63d4cf89a45aa3e46991910ffb5a7c96d6d6d2a79bf257ec40262c57672" gracePeriod=25 Oct 30 02:21:53.666557 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:53.666437 6572 generic.go:356] "Generic (PLEG): container finished" podID="963a53c2-66a3-415b-a7b0-2dcf1b48daf0" containerID="97c74579079ad79aa475823bfa55533b63d4d5696c92c0966e6ac3ef93f47fbf" exitCode=0 Oct 30 02:21:53.667136 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:53.666844 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-861cd2-6d4589fbd7-hr89s" event={"ID":"963a53c2-66a3-415b-a7b0-2dcf1b48daf0","Type":"ContainerDied","Data":"97c74579079ad79aa475823bfa55533b63d4d5696c92c0966e6ac3ef93f47fbf"} Oct 30 02:21:53.667136 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:53.666930 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-861cd2-6d4589fbd7-hr89s" event={"ID":"963a53c2-66a3-415b-a7b0-2dcf1b48daf0","Type":"ContainerDied","Data":"f1079eb7650a7bcb198854ddfaa053c91ca89f4cb7c59f8e85b869d4a90adea9"} Oct 30 02:21:53.667136 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:53.666955 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1079eb7650a7bcb198854ddfaa053c91ca89f4cb7c59f8e85b869d4a90adea9" Oct 30 02:21:53.675905 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:53.671948 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-861cd2-6d4589fbd7-hr89s" Oct 30 02:21:53.741445 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:53.741416 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/963a53c2-66a3-415b-a7b0-2dcf1b48daf0-kube-api-access-6ptwj\" (UniqueName: \"kubernetes.io/projected/963a53c2-66a3-415b-a7b0-2dcf1b48daf0-kube-api-access-6ptwj\") pod \"963a53c2-66a3-415b-a7b0-2dcf1b48daf0\" (UID: \"963a53c2-66a3-415b-a7b0-2dcf1b48daf0\") " Oct 30 02:21:53.741659 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:53.741550 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/963a53c2-66a3-415b-a7b0-2dcf1b48daf0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/963a53c2-66a3-415b-a7b0-2dcf1b48daf0-plugins-config-volume\") pod \"963a53c2-66a3-415b-a7b0-2dcf1b48daf0\" (UID: \"963a53c2-66a3-415b-a7b0-2dcf1b48daf0\") " Oct 30 02:21:53.742215 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:53.742189 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/963a53c2-66a3-415b-a7b0-2dcf1b48daf0-plugins-config-volume" pod "963a53c2-66a3-415b-a7b0-2dcf1b48daf0" (UID: "963a53c2-66a3-415b-a7b0-2dcf1b48daf0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:21:53.744368 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:53.744337 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/963a53c2-66a3-415b-a7b0-2dcf1b48daf0-kube-api-access-6ptwj" pod "963a53c2-66a3-415b-a7b0-2dcf1b48daf0" (UID: "963a53c2-66a3-415b-a7b0-2dcf1b48daf0"). InnerVolumeSpecName "kube-api-access-6ptwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:21:53.839555 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:53.839522 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82eb752a-q7gw4" Oct 30 02:21:53.842570 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:53.842546 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/963a53c2-66a3-415b-a7b0-2dcf1b48daf0-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:21:53.842688 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:53.842571 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ptwj\" (UniqueName: \"kubernetes.io/projected/963a53c2-66a3-415b-a7b0-2dcf1b48daf0-kube-api-access-6ptwj\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:21:53.943807 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:53.943771 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e33def86-6f77-4c1f-8734-c575f37e6f6e-kube-api-access-x8hkn\" (UniqueName: \"kubernetes.io/projected/e33def86-6f77-4c1f-8734-c575f37e6f6e-kube-api-access-x8hkn\") pod \"e33def86-6f77-4c1f-8734-c575f37e6f6e\" (UID: \"e33def86-6f77-4c1f-8734-c575f37e6f6e\") " Oct 30 02:21:53.944019 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:53.943825 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e33def86-6f77-4c1f-8734-c575f37e6f6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e33def86-6f77-4c1f-8734-c575f37e6f6e-maas-api-openshift-ca\") pod \"e33def86-6f77-4c1f-8734-c575f37e6f6e\" (UID: \"e33def86-6f77-4c1f-8734-c575f37e6f6e\") " Oct 30 02:21:53.944019 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:53.943874 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e33def86-6f77-4c1f-8734-c575f37e6f6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e33def86-6f77-4c1f-8734-c575f37e6f6e-maas-api-xks-ca\") pod \"e33def86-6f77-4c1f-8734-c575f37e6f6e\" (UID: \"e33def86-6f77-4c1f-8734-c575f37e6f6e\") " Oct 30 02:21:53.944369 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:53.944345 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e33def86-6f77-4c1f-8734-c575f37e6f6e-maas-api-openshift-ca" pod "e33def86-6f77-4c1f-8734-c575f37e6f6e" (UID: "e33def86-6f77-4c1f-8734-c575f37e6f6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:21:53.946067 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:53.946044 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e33def86-6f77-4c1f-8734-c575f37e6f6e-maas-api-xks-ca" pod "e33def86-6f77-4c1f-8734-c575f37e6f6e" (UID: "e33def86-6f77-4c1f-8734-c575f37e6f6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:21:53.946304 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:53.946280 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e33def86-6f77-4c1f-8734-c575f37e6f6e-kube-api-access-x8hkn" pod "e33def86-6f77-4c1f-8734-c575f37e6f6e" (UID: "e33def86-6f77-4c1f-8734-c575f37e6f6e"). InnerVolumeSpecName "kube-api-access-x8hkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:21:54.044559 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:54.044522 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8hkn\" (UniqueName: \"kubernetes.io/projected/e33def86-6f77-4c1f-8734-c575f37e6f6e-kube-api-access-x8hkn\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:21:54.044559 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:54.044557 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e33def86-6f77-4c1f-8734-c575f37e6f6e-maas-api-openshift-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:21:54.044559 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:54.044567 6572 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e33def86-6f77-4c1f-8734-c575f37e6f6e-maas-api-xks-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:21:54.677385 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:54.677303 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82eb752a-q7gw4" Oct 30 02:21:54.677385 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:54.677322 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82eb752a-q7gw4" event={"ID":"e33def86-6f77-4c1f-8734-c575f37e6f6e","Type":"ContainerDied","Data":"7a5806719a712246f322b5c4dd97a97d9608684b14bc1373144d57a29204aa67"} Oct 30 02:21:54.677385 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:54.677383 6572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a5806719a712246f322b5c4dd97a97d9608684b14bc1373144d57a29204aa67" Oct 30 02:21:54.678047 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:54.677470 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-861cd2-6d4589fbd7-hr89s" Oct 30 02:21:54.751509 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:54.751476 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-861cd2-6d4589fbd7-hr89s"] Oct 30 02:21:54.761917 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:54.761879 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-861cd2-6d4589fbd7-hr89s"] Oct 30 02:21:56.541620 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:56.541587 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="963a53c2-66a3-415b-a7b0-2dcf1b48daf0" path="/var/lib/kubelet/pods/963a53c2-66a3-415b-a7b0-2dcf1b48daf0/volumes" Oct 30 02:21:59.508925 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:59.508878 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-cf82f8-6fcc5c744b-gv5bq" podUID="b1a65b5e-39ec-48ca-9bfc-34e207413e49" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:21:59.508925 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:21:59.508903 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cf82f8-5bcf4d65c9-jv8q2" podUID="4fcc6142-79ba-4dcc-b037-6b2fa054a9e3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:22:00.075920 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:00.075885 6572 scope.go:122] "RemoveContainer" containerID="f59bf48939839aa69b7b3e1c944925c8521f5751d9a8ab03054cb8926e99c8c8" Oct 30 02:22:00.086006 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:00.085984 6572 scope.go:122] "RemoveContainer" containerID="97c74579079ad79aa475823bfa55533b63d4d5696c92c0966e6ac3ef93f47fbf" Oct 30 02:22:00.095957 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:00.095930 6572 scope.go:122] "RemoveContainer" containerID="6c50384620024323388a9d0d5697c6fa7e741aa8a4847e7b589fc4a2a4b1769a" Oct 30 02:22:00.105081 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:00.105059 6572 scope.go:122] "RemoveContainer" containerID="9bf888c0d9d426697c33561d1471dc1dc92c01bb32f0ce0f429e466f88b6061a" Oct 30 02:22:00.113962 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:00.113943 6572 scope.go:122] "RemoveContainer" containerID="a073480a2f43747cda58293cfe3a706c208665652f4c1cf6ff871e4d40dba75e" Oct 30 02:22:00.122868 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:00.122847 6572 scope.go:122] "RemoveContainer" containerID="bf1b10e17cf04829cad9aa8c34891b8498af834494e4ee1a7fece3361dcd645d" Oct 30 02:22:06.410562 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:06.410529 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:06.858662 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:06.858565 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:06.866965 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:06.866936 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cf82f8-pl66j"] Oct 30 02:22:06.877797 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:06.877738 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-cf82f8-pl66j"] Oct 30 02:22:07.069167 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:07.069135 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:07.534215 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:07.534161 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:08.541056 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:08.541021 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="733ae104-96c5-40e8-a4cd-8afedb8bf8b8" path="/var/lib/kubelet/pods/733ae104-96c5-40e8-a4cd-8afedb8bf8b8/volumes" Oct 30 02:22:09.509428 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:09.509395 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-cf82f8-6fcc5c744b-gv5bq" podUID="b1a65b5e-39ec-48ca-9bfc-34e207413e49" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:22:09.509614 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:09.509387 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cf82f8-5bcf4d65c9-jv8q2" podUID="4fcc6142-79ba-4dcc-b037-6b2fa054a9e3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:22:09.687606 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:09.687573 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:13.640156 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:13.640129 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-cf82f8-6fcc5c744b-gv5bq" Oct 30 02:22:13.642469 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:13.642440 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:13.658993 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:13.658968 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cf82f8-5bcf4d65c9-jv8q2" Oct 30 02:22:13.731339 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:13.731308 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fcc6142-79ba-4dcc-b037-6b2fa054a9e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fcc6142-79ba-4dcc-b037-6b2fa054a9e3-plugins-config-volume\") pod \"4fcc6142-79ba-4dcc-b037-6b2fa054a9e3\" (UID: \"4fcc6142-79ba-4dcc-b037-6b2fa054a9e3\") " Oct 30 02:22:13.731539 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:13.731357 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1a65b5e-39ec-48ca-9bfc-34e207413e49-kube-api-access-27t48\" (UniqueName: \"kubernetes.io/projected/b1a65b5e-39ec-48ca-9bfc-34e207413e49-kube-api-access-27t48\") pod \"b1a65b5e-39ec-48ca-9bfc-34e207413e49\" (UID: \"b1a65b5e-39ec-48ca-9bfc-34e207413e49\") " Oct 30 02:22:13.731539 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:13.731440 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1a65b5e-39ec-48ca-9bfc-34e207413e49-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1a65b5e-39ec-48ca-9bfc-34e207413e49-plugins-config-volume\") pod \"b1a65b5e-39ec-48ca-9bfc-34e207413e49\" (UID: \"b1a65b5e-39ec-48ca-9bfc-34e207413e49\") " Oct 30 02:22:13.731539 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:13.731466 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fcc6142-79ba-4dcc-b037-6b2fa054a9e3-kube-api-access-qx5rp\" (UniqueName: \"kubernetes.io/projected/4fcc6142-79ba-4dcc-b037-6b2fa054a9e3-kube-api-access-qx5rp\") pod \"4fcc6142-79ba-4dcc-b037-6b2fa054a9e3\" (UID: \"4fcc6142-79ba-4dcc-b037-6b2fa054a9e3\") " Oct 30 02:22:13.731791 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:13.731767 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fcc6142-79ba-4dcc-b037-6b2fa054a9e3-plugins-config-volume" pod "4fcc6142-79ba-4dcc-b037-6b2fa054a9e3" (UID: "4fcc6142-79ba-4dcc-b037-6b2fa054a9e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:22:13.731899 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:13.731876 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1a65b5e-39ec-48ca-9bfc-34e207413e49-plugins-config-volume" pod "b1a65b5e-39ec-48ca-9bfc-34e207413e49" (UID: "b1a65b5e-39ec-48ca-9bfc-34e207413e49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:22:13.733606 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:13.733587 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1a65b5e-39ec-48ca-9bfc-34e207413e49-kube-api-access-27t48" pod "b1a65b5e-39ec-48ca-9bfc-34e207413e49" (UID: "b1a65b5e-39ec-48ca-9bfc-34e207413e49"). InnerVolumeSpecName "kube-api-access-27t48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:22:13.733685 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:13.733613 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fcc6142-79ba-4dcc-b037-6b2fa054a9e3-kube-api-access-qx5rp" pod "4fcc6142-79ba-4dcc-b037-6b2fa054a9e3" (UID: "4fcc6142-79ba-4dcc-b037-6b2fa054a9e3"). InnerVolumeSpecName "kube-api-access-qx5rp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:22:13.819267 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:13.819234 6572 generic.go:356] "Generic (PLEG): container finished" podID="4fcc6142-79ba-4dcc-b037-6b2fa054a9e3" containerID="b9a9d63d4cf89a45aa3e46991910ffb5a7c96d6d6d2a79bf257ec40262c57672" exitCode=0 Oct 30 02:22:13.819455 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:13.819277 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cf82f8-5bcf4d65c9-jv8q2" Oct 30 02:22:13.819455 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:13.819326 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cf82f8-5bcf4d65c9-jv8q2" event={"ID":"4fcc6142-79ba-4dcc-b037-6b2fa054a9e3","Type":"ContainerDied","Data":"b9a9d63d4cf89a45aa3e46991910ffb5a7c96d6d6d2a79bf257ec40262c57672"} Oct 30 02:22:13.819455 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:13.819388 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cf82f8-5bcf4d65c9-jv8q2" event={"ID":"4fcc6142-79ba-4dcc-b037-6b2fa054a9e3","Type":"ContainerDied","Data":"020562d7a532e4a0bb4d049ba4a45795d3047877f6f7c6eea0bb3559100efeeb"} Oct 30 02:22:13.819455 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:13.819419 6572 scope.go:122] "RemoveContainer" containerID="b9a9d63d4cf89a45aa3e46991910ffb5a7c96d6d6d2a79bf257ec40262c57672" Oct 30 02:22:13.821024 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:13.821000 6572 generic.go:356] "Generic (PLEG): container finished" podID="b1a65b5e-39ec-48ca-9bfc-34e207413e49" containerID="782dcae789f98a60b662b84c9d6b7c664225e758873ec5c9815f53e11b8c44ea" exitCode=0 Oct 30 02:22:13.821138 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:13.821028 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-cf82f8-6fcc5c744b-gv5bq" event={"ID":"b1a65b5e-39ec-48ca-9bfc-34e207413e49","Type":"ContainerDied","Data":"782dcae789f98a60b662b84c9d6b7c664225e758873ec5c9815f53e11b8c44ea"} Oct 30 02:22:13.821138 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:13.821090 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-cf82f8-6fcc5c744b-gv5bq" event={"ID":"b1a65b5e-39ec-48ca-9bfc-34e207413e49","Type":"ContainerDied","Data":"1dc891eebd4c7305e0208316a9fd5f2de2f05a0e112e1cbfb52c9a9af5194f93"} Oct 30 02:22:13.821138 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:13.821039 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-cf82f8-6fcc5c744b-gv5bq" Oct 30 02:22:13.830881 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:13.830863 6572 scope.go:122] "RemoveContainer" containerID="b9a9d63d4cf89a45aa3e46991910ffb5a7c96d6d6d2a79bf257ec40262c57672" Oct 30 02:22:13.831214 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:22:13.831194 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9a9d63d4cf89a45aa3e46991910ffb5a7c96d6d6d2a79bf257ec40262c57672\": container with ID starting with b9a9d63d4cf89a45aa3e46991910ffb5a7c96d6d6d2a79bf257ec40262c57672 not found: ID does not exist" containerID="b9a9d63d4cf89a45aa3e46991910ffb5a7c96d6d6d2a79bf257ec40262c57672" Oct 30 02:22:13.831307 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:13.831223 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9a9d63d4cf89a45aa3e46991910ffb5a7c96d6d6d2a79bf257ec40262c57672"} err="failed to get container status \"b9a9d63d4cf89a45aa3e46991910ffb5a7c96d6d6d2a79bf257ec40262c57672\": rpc error: code = NotFound desc = could not find container \"b9a9d63d4cf89a45aa3e46991910ffb5a7c96d6d6d2a79bf257ec40262c57672\": container with ID starting with b9a9d63d4cf89a45aa3e46991910ffb5a7c96d6d6d2a79bf257ec40262c57672 not found: ID does not exist" Oct 30 02:22:13.831307 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:13.831249 6572 scope.go:122] "RemoveContainer" containerID="782dcae789f98a60b662b84c9d6b7c664225e758873ec5c9815f53e11b8c44ea" Oct 30 02:22:13.832364 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:13.832346 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1a65b5e-39ec-48ca-9bfc-34e207413e49-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:22:13.832467 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:13.832364 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx5rp\" (UniqueName: \"kubernetes.io/projected/4fcc6142-79ba-4dcc-b037-6b2fa054a9e3-kube-api-access-qx5rp\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:22:13.832467 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:13.832378 6572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fcc6142-79ba-4dcc-b037-6b2fa054a9e3-plugins-config-volume\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:22:13.832467 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:13.832390 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27t48\" (UniqueName: \"kubernetes.io/projected/b1a65b5e-39ec-48ca-9bfc-34e207413e49-kube-api-access-27t48\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:22:13.841146 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:13.841126 6572 scope.go:122] "RemoveContainer" containerID="782dcae789f98a60b662b84c9d6b7c664225e758873ec5c9815f53e11b8c44ea" Oct 30 02:22:13.841404 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:22:13.841383 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"782dcae789f98a60b662b84c9d6b7c664225e758873ec5c9815f53e11b8c44ea\": container with ID starting with 782dcae789f98a60b662b84c9d6b7c664225e758873ec5c9815f53e11b8c44ea not found: ID does not exist" containerID="782dcae789f98a60b662b84c9d6b7c664225e758873ec5c9815f53e11b8c44ea" Oct 30 02:22:13.841467 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:13.841416 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"782dcae789f98a60b662b84c9d6b7c664225e758873ec5c9815f53e11b8c44ea"} err="failed to get container status \"782dcae789f98a60b662b84c9d6b7c664225e758873ec5c9815f53e11b8c44ea\": rpc error: code = NotFound desc = could not find container \"782dcae789f98a60b662b84c9d6b7c664225e758873ec5c9815f53e11b8c44ea\": container with ID starting with 782dcae789f98a60b662b84c9d6b7c664225e758873ec5c9815f53e11b8c44ea not found: ID does not exist" Oct 30 02:22:13.874842 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:13.874724 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-cf82f8-5bcf4d65c9-jv8q2"] Oct 30 02:22:13.878737 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:13.878710 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-cf82f8-5bcf4d65c9-jv8q2"] Oct 30 02:22:13.895726 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:13.895700 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v"] Oct 30 02:22:13.895962 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:13.895940 6572 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" podUID="b445121f-4bf6-4f3a-886c-a02f45574a57" containerName="istio-proxy" containerID="cri-o://e120bd8bc486c96df059b013155362b62a313e6abbdec528201637dd3a01de92" gracePeriod=30 Oct 30 02:22:13.916516 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:13.916482 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-cf82f8-6fcc5c744b-gv5bq"] Oct 30 02:22:13.924761 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:13.924723 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-cf82f8-6fcc5c744b-gv5bq"] Oct 30 02:22:14.541546 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:14.541503 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fcc6142-79ba-4dcc-b037-6b2fa054a9e3" path="/var/lib/kubelet/pods/4fcc6142-79ba-4dcc-b037-6b2fa054a9e3/volumes" Oct 30 02:22:14.541992 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:14.541968 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1a65b5e-39ec-48ca-9bfc-34e207413e49" path="/var/lib/kubelet/pods/b1a65b5e-39ec-48ca-9bfc-34e207413e49/volumes" Oct 30 02:22:14.687473 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:22:14.687433 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:22:15.096660 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:15.096583 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-f9c56f786-455r7_3d30861b-c70b-4c15-ac36-8ff66d801f09/authorino/0.log" Oct 30 02:22:15.297965 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:15.297914 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:16.361608 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:16.361572 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-vpxn7_76c97fcb-f62c-4de6-899b-3c23f809d1c6/manager/0.log" Oct 30 02:22:17.650347 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:17.650313 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-hf2jx_296d1c8b-da1d-4e25-8b3e-7a2c50337818/manager/0.log" Oct 30 02:22:18.817360 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:18.817320 6572 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:22:18.817802 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:18.817378 6572 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" podUID="b445121f-4bf6-4f3a-886c-a02f45574a57" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:22:19.154374 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.154347 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" Oct 30 02:22:19.268511 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.268480 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:19.278477 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.278450 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b445121f-4bf6-4f3a-886c-a02f45574a57-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b445121f-4bf6-4f3a-886c-a02f45574a57-workload-certs\") pod \"b445121f-4bf6-4f3a-886c-a02f45574a57\" (UID: \"b445121f-4bf6-4f3a-886c-a02f45574a57\") " Oct 30 02:22:19.278606 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.278485 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b445121f-4bf6-4f3a-886c-a02f45574a57-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b445121f-4bf6-4f3a-886c-a02f45574a57-istiod-ca-cert\") pod \"b445121f-4bf6-4f3a-886c-a02f45574a57\" (UID: \"b445121f-4bf6-4f3a-886c-a02f45574a57\") " Oct 30 02:22:19.278606 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.278520 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b445121f-4bf6-4f3a-886c-a02f45574a57-istio-token\" (UniqueName: \"kubernetes.io/projected/b445121f-4bf6-4f3a-886c-a02f45574a57-istio-token\") pod \"b445121f-4bf6-4f3a-886c-a02f45574a57\" (UID: \"b445121f-4bf6-4f3a-886c-a02f45574a57\") " Oct 30 02:22:19.278606 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.278592 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b445121f-4bf6-4f3a-886c-a02f45574a57-kube-api-access-8cb5s\" (UniqueName: \"kubernetes.io/projected/b445121f-4bf6-4f3a-886c-a02f45574a57-kube-api-access-8cb5s\") pod \"b445121f-4bf6-4f3a-886c-a02f45574a57\" (UID: \"b445121f-4bf6-4f3a-886c-a02f45574a57\") " Oct 30 02:22:19.278740 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.278644 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b445121f-4bf6-4f3a-886c-a02f45574a57-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b445121f-4bf6-4f3a-886c-a02f45574a57-istio-envoy\") pod \"b445121f-4bf6-4f3a-886c-a02f45574a57\" (UID: \"b445121f-4bf6-4f3a-886c-a02f45574a57\") " Oct 30 02:22:19.278740 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.278707 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b445121f-4bf6-4f3a-886c-a02f45574a57-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b445121f-4bf6-4f3a-886c-a02f45574a57-workload-socket\") pod \"b445121f-4bf6-4f3a-886c-a02f45574a57\" (UID: \"b445121f-4bf6-4f3a-886c-a02f45574a57\") " Oct 30 02:22:19.278865 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.278798 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b445121f-4bf6-4f3a-886c-a02f45574a57-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b445121f-4bf6-4f3a-886c-a02f45574a57-credential-socket\") pod \"b445121f-4bf6-4f3a-886c-a02f45574a57\" (UID: \"b445121f-4bf6-4f3a-886c-a02f45574a57\") " Oct 30 02:22:19.278865 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.278838 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b445121f-4bf6-4f3a-886c-a02f45574a57-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b445121f-4bf6-4f3a-886c-a02f45574a57-istio-podinfo\") pod \"b445121f-4bf6-4f3a-886c-a02f45574a57\" (UID: \"b445121f-4bf6-4f3a-886c-a02f45574a57\") " Oct 30 02:22:19.278984 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.278871 6572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b445121f-4bf6-4f3a-886c-a02f45574a57-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b445121f-4bf6-4f3a-886c-a02f45574a57-istio-data\") pod \"b445121f-4bf6-4f3a-886c-a02f45574a57\" (UID: \"b445121f-4bf6-4f3a-886c-a02f45574a57\") " Oct 30 02:22:19.279049 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.278798 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b445121f-4bf6-4f3a-886c-a02f45574a57-workload-certs" pod "b445121f-4bf6-4f3a-886c-a02f45574a57" (UID: "b445121f-4bf6-4f3a-886c-a02f45574a57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:22:19.279049 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.279025 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b445121f-4bf6-4f3a-886c-a02f45574a57-workload-socket" pod "b445121f-4bf6-4f3a-886c-a02f45574a57" (UID: "b445121f-4bf6-4f3a-886c-a02f45574a57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:22:19.279179 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.279050 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b445121f-4bf6-4f3a-886c-a02f45574a57-istiod-ca-cert" pod "b445121f-4bf6-4f3a-886c-a02f45574a57" (UID: "b445121f-4bf6-4f3a-886c-a02f45574a57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:22:19.279179 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.279081 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b445121f-4bf6-4f3a-886c-a02f45574a57-credential-socket" pod "b445121f-4bf6-4f3a-886c-a02f45574a57" (UID: "b445121f-4bf6-4f3a-886c-a02f45574a57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:22:19.279284 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.279250 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b445121f-4bf6-4f3a-886c-a02f45574a57-istio-data" pod "b445121f-4bf6-4f3a-886c-a02f45574a57" (UID: "b445121f-4bf6-4f3a-886c-a02f45574a57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:22:19.279341 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.279307 6572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b445121f-4bf6-4f3a-886c-a02f45574a57-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:22:19.279341 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.279327 6572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b445121f-4bf6-4f3a-886c-a02f45574a57-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:22:19.279439 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.279342 6572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b445121f-4bf6-4f3a-886c-a02f45574a57-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:22:19.279439 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.279356 6572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b445121f-4bf6-4f3a-886c-a02f45574a57-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:22:19.280763 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.280724 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b445121f-4bf6-4f3a-886c-a02f45574a57-kube-api-access-8cb5s" pod "b445121f-4bf6-4f3a-886c-a02f45574a57" (UID: "b445121f-4bf6-4f3a-886c-a02f45574a57"). InnerVolumeSpecName "kube-api-access-8cb5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:22:19.281057 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.281036 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b445121f-4bf6-4f3a-886c-a02f45574a57-istio-podinfo" pod "b445121f-4bf6-4f3a-886c-a02f45574a57" (UID: "b445121f-4bf6-4f3a-886c-a02f45574a57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:22:19.281188 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.281171 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b445121f-4bf6-4f3a-886c-a02f45574a57-istio-envoy" pod "b445121f-4bf6-4f3a-886c-a02f45574a57" (UID: "b445121f-4bf6-4f3a-886c-a02f45574a57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:22:19.281243 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.281190 6572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b445121f-4bf6-4f3a-886c-a02f45574a57-istio-token" pod "b445121f-4bf6-4f3a-886c-a02f45574a57" (UID: "b445121f-4bf6-4f3a-886c-a02f45574a57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:22:19.380411 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.380382 6572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b445121f-4bf6-4f3a-886c-a02f45574a57-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:22:19.380411 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.380407 6572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cb5s\" (UniqueName: \"kubernetes.io/projected/b445121f-4bf6-4f3a-886c-a02f45574a57-kube-api-access-8cb5s\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:22:19.380411 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.380417 6572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b445121f-4bf6-4f3a-886c-a02f45574a57-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:22:19.380616 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.380427 6572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b445121f-4bf6-4f3a-886c-a02f45574a57-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:22:19.380616 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.380436 6572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b445121f-4bf6-4f3a-886c-a02f45574a57-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\"" Oct 30 02:22:19.468128 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.468095 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:19.786339 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.786258 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-f9c56f786-455r7_3d30861b-c70b-4c15-ac36-8ff66d801f09/authorino/0.log" Oct 30 02:22:19.864477 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.864442 6572 generic.go:356] "Generic (PLEG): container finished" podID="b445121f-4bf6-4f3a-886c-a02f45574a57" containerID="e120bd8bc486c96df059b013155362b62a313e6abbdec528201637dd3a01de92" exitCode=0 Oct 30 02:22:19.865001 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.864476 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" event={"ID":"b445121f-4bf6-4f3a-886c-a02f45574a57","Type":"ContainerDied","Data":"e120bd8bc486c96df059b013155362b62a313e6abbdec528201637dd3a01de92"} Oct 30 02:22:19.865001 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.864496 6572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" Oct 30 02:22:19.865001 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.864539 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v" event={"ID":"b445121f-4bf6-4f3a-886c-a02f45574a57","Type":"ContainerDied","Data":"dc31336e7c0d77a2f9cbfd754f358bd0a97fee4a02b6240e47f707d8d7c3f501"} Oct 30 02:22:19.865001 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.864568 6572 scope.go:122] "RemoveContainer" containerID="e120bd8bc486c96df059b013155362b62a313e6abbdec528201637dd3a01de92" Oct 30 02:22:19.881885 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.881862 6572 scope.go:122] "RemoveContainer" containerID="e120bd8bc486c96df059b013155362b62a313e6abbdec528201637dd3a01de92" Oct 30 02:22:19.882162 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:22:19.882142 6572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e120bd8bc486c96df059b013155362b62a313e6abbdec528201637dd3a01de92\": container with ID starting with e120bd8bc486c96df059b013155362b62a313e6abbdec528201637dd3a01de92 not found: ID does not exist" containerID="e120bd8bc486c96df059b013155362b62a313e6abbdec528201637dd3a01de92" Oct 30 02:22:19.882211 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.882171 6572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e120bd8bc486c96df059b013155362b62a313e6abbdec528201637dd3a01de92"} err="failed to get container status \"e120bd8bc486c96df059b013155362b62a313e6abbdec528201637dd3a01de92\": rpc error: code = NotFound desc = could not find container \"e120bd8bc486c96df059b013155362b62a313e6abbdec528201637dd3a01de92\": container with ID starting with e120bd8bc486c96df059b013155362b62a313e6abbdec528201637dd3a01de92 not found: ID does not exist" Oct 30 02:22:19.926780 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.926730 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v"] Oct 30 02:22:19.938219 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:19.938197 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-zl59v"] Oct 30 02:22:20.140239 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:20.140151 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:20.319996 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:20.319964 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:20.540865 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:20.540835 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b445121f-4bf6-4f3a-886c-a02f45574a57" path="/var/lib/kubelet/pods/b445121f-4bf6-4f3a-886c-a02f45574a57/volumes" Oct 30 02:22:20.578947 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:20.578913 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:26.262690 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:26.262648 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:30.738995 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:30.738961 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-hf2jx_296d1c8b-da1d-4e25-8b3e-7a2c50337818/manager/0.log" Oct 30 02:22:33.239565 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:33.239533 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-4abfaf4e-nvxw8_ed6145a7-d13c-4efa-ad6d-eaa6c5d90974/revoke-keys/0.log" Oct 30 02:22:33.430928 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:33.430877 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-543807da-gxx84_e203d4b8-3f30-4d07-9c6f-431af65d9ec2/revoke-keys/0.log" Oct 30 02:22:33.625973 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:33.625867 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-5db98a24-7k4zk_a36c6723-df85-48fa-9b09-e466c2c28500/revoke-keys/0.log" Oct 30 02:22:33.798494 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:33.798458 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-82eb752a-q7gw4_e33def86-6f77-4c1f-8734-c575f37e6f6e/revoke-keys/0.log" Oct 30 02:22:33.958300 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:33.958262 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:33.973511 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:33.973486 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-8fdbaa16-bhwkq_1229a151-ffd5-4691-bc3a-7b4afdd0c5c3/revoke-keys/0.log" Oct 30 02:22:34.150121 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:34.150084 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-99fd05c4-wd9cz_af284c10-c406-439b-98ad-c0a5a35b9a8e/revoke-keys/0.log" Oct 30 02:22:34.339229 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:34.339144 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9a4f6408-jc7bj_8e87a3a0-b554-423f-b1cf-ff9516ec89b6/revoke-keys/0.log" Oct 30 02:22:34.528989 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:34.528959 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9ed7ea96-xhnnm_760b1b1c-26ad-4c65-a696-7f62a9917a93/revoke-keys/0.log" Oct 30 02:22:34.712319 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:34.712288 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a56df390-2dpvv_70e0bbb2-2d70-4f55-9aa1-91e0929b53c2/revoke-keys/0.log" Oct 30 02:22:36.440917 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:36.440887 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-f9c56f786-455r7_3d30861b-c70b-4c15-ac36-8ff66d801f09/authorino/0.log" Oct 30 02:22:37.592142 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:37.592112 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-vpxn7_76c97fcb-f62c-4de6-899b-3c23f809d1c6/manager/0.log" Oct 30 02:22:38.549160 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:38.549129 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-ngtfq_4773cabd-39d7-4386-852c-ae771c56e7f7/discovery/0.log" Oct 30 02:22:38.760529 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:38.760480 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66f46d7b58-bszt4_3d65df0f-5ee7-4774-b73d-2f8adb718fad/kube-auth-proxy/0.log" Oct 30 02:22:41.960415 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:41.960385 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-clwkt_c92a422d-2da3-4ce1-9155-a23b8eed9c90/main/0.log" Oct 30 02:22:42.529076 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:42.529047 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-dnvm5_24fa6193-db5f-447d-9c2d-3e62c06e59de/main/0.log" Oct 30 02:22:42.716976 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:42.716943 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-l8q6f_02ae87f3-5a71-4a60-af7b-98965dfd5862/main/0.log" Oct 30 02:22:44.029712 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:44.029675 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99fd05c4-wd9cz"] Oct 30 02:22:44.040140 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:44.040112 6572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5db98a24-7k4zk"] Oct 30 02:22:44.042208 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:44.042189 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99fd05c4-wd9cz"] Oct 30 02:22:44.043968 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:44.043949 6572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5db98a24-7k4zk"] Oct 30 02:22:44.549643 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:44.549609 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a36c6723-df85-48fa-9b09-e466c2c28500" path="/var/lib/kubelet/pods/a36c6723-df85-48fa-9b09-e466c2c28500/volumes" Oct 30 02:22:44.550068 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:44.550054 6572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af284c10-c406-439b-98ad-c0a5a35b9a8e" path="/var/lib/kubelet/pods/af284c10-c406-439b-98ad-c0a5a35b9a8e/volumes" Oct 30 02:22:44.686689 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:22:44.686649 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:22:53.934553 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:53.934517 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:54.144909 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:54.144875 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:59.727251 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:22:59.727213 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-j7l6k_88f5d337-fa16-4430-a82e-5f433354c164/global-pull-secret-syncer/0.log" Oct 30 02:23:00.185330 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:00.185294 6572 scope.go:122] "RemoveContainer" containerID="8277a0e52c0b0f27058b15b8f32c99426659d55b9c2389c64ba9e1b21b8ad422" Oct 30 02:23:00.195229 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:00.195205 6572 scope.go:122] "RemoveContainer" containerID="2e07265b9705ec06899bf35e7c895d3e43ce66b81f008d8ba64fcf9b860b1bd0" Oct 30 02:23:00.204084 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:00.204064 6572 scope.go:122] "RemoveContainer" containerID="dfc84eef8dfada8b8f69b0ac6e7233031feda3010b1f0a6875574f446d9089de" Oct 30 02:23:00.212590 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:00.212572 6572 scope.go:122] "RemoveContainer" containerID="e41838dbfa6605dd6be4f541609fa8feb160c7a8cac51522af133d6fc6dc3f66" Oct 30 02:23:00.221129 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:00.221105 6572 scope.go:122] "RemoveContainer" containerID="21c40c702c16de5c187a7111fdd4a51f38b49be37abc9b4269ee7904add9fd10" Oct 30 02:23:00.229268 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:00.229239 6572 scope.go:122] "RemoveContainer" containerID="eccde603d684e44a03331f5fc3ca7e67e6ff59ab48c7550dbaf31e9535197719" Oct 30 02:23:00.240052 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:00.239952 6572 scope.go:122] "RemoveContainer" containerID="a9cc05bdd2d284c300796157eea3e56887ec86bbc2645fb920a5888304af6570" Oct 30 02:23:00.249906 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:00.249888 6572 scope.go:122] "RemoveContainer" containerID="94e93b07e2ed8f31a0b8e46a268a51fbeef3ad56c350248cd7f3f83d850dfc82" Oct 30 02:23:00.258098 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:00.258078 6572 scope.go:122] "RemoveContainer" containerID="e0f16c01b5af3a9e9be408f1d4fe642888b5909bb7f24715cdf4e486dbfd27e1" Oct 30 02:23:00.479628 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:00.479556 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" Oct 30 02:23:01.223442 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:01.223407 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-223.ec2.internal_129d2d6479043c5ffd993e1a7481e29a/haproxy/0.log" Oct 30 02:23:07.796960 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:07.796922 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-4gtf9_a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab/kube-state-metrics/0.log" Oct 30 02:23:07.912114 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:07.912081 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-4gtf9_a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab/kube-rbac-proxy-main/0.log" Oct 30 02:23:08.024394 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:08.024364 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-4gtf9_a8793e92-1abf-4fcc-9a0b-ea16bdf7e4ab/kube-rbac-proxy-self/0.log" Oct 30 02:23:08.295540 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:08.295503 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-f7fkx_0ae3164f-fec1-4210-863e-70e49e6bd26a/monitoring-plugin/0.log" Oct 30 02:23:08.801226 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:08.801193 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tqcs7_f3928614-3b83-4bf6-9e3f-04a71bae52bd/node-exporter/0.log" Oct 30 02:23:08.918321 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:08.918285 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tqcs7_f3928614-3b83-4bf6-9e3f-04a71bae52bd/kube-rbac-proxy/0.log" Oct 30 02:23:09.031911 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:09.031884 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tqcs7_f3928614-3b83-4bf6-9e3f-04a71bae52bd/init-textfile/0.log" Oct 30 02:23:09.510614 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:09.510585 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-cnw75_cf531672-6ee1-4188-b38d-2899bf9d6a0f/kube-rbac-proxy-main/0.log" Oct 30 02:23:09.626543 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:09.626517 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-cnw75_cf531672-6ee1-4188-b38d-2899bf9d6a0f/kube-rbac-proxy-self/0.log" Oct 30 02:23:09.741587 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:09.741547 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-cnw75_cf531672-6ee1-4188-b38d-2899bf9d6a0f/openshift-state-metrics/0.log" Oct 30 02:23:09.811801 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:09.811687 6572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67v6w_262c01e0-8469-4fbe-aaa4-8a8c50551fce/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:09.883545 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:09.883513 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_326173d8-e9a0-4c44-8af9-d400b03828ac/prometheus/0.log" Oct 30 02:23:09.996002 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:09.995968 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_326173d8-e9a0-4c44-8af9-d400b03828ac/config-reloader/0.log" Oct 30 02:23:10.110050 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:10.109968 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_326173d8-e9a0-4c44-8af9-d400b03828ac/thanos-sidecar/0.log" Oct 30 02:23:10.223187 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:10.223161 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_326173d8-e9a0-4c44-8af9-d400b03828ac/kube-rbac-proxy-web/0.log" Oct 30 02:23:10.338909 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:10.338879 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_326173d8-e9a0-4c44-8af9-d400b03828ac/kube-rbac-proxy/0.log" Oct 30 02:23:10.452407 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:10.452373 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_326173d8-e9a0-4c44-8af9-d400b03828ac/kube-rbac-proxy-thanos/0.log" Oct 30 02:23:10.595058 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:10.595025 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_326173d8-e9a0-4c44-8af9-d400b03828ac/init-config-reloader/0.log" Oct 30 02:23:10.804225 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:10.804123 6572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9dvcx/perf-node-gather-daemonset-87dx2"] Oct 30 02:23:10.804561 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:10.804546 6572 cpu_manager.go:414] "containerMap: removing container" podUID="e33def86-6f77-4c1f-8734-c575f37e6f6e" containerName="revoke-keys" Oct 30 02:23:10.804626 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:10.804562 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e33def86-6f77-4c1f-8734-c575f37e6f6e" containerName="revoke-keys" Oct 30 02:23:10.804626 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:10.804574 6572 cpu_manager.go:414] "containerMap: removing container" podUID="963a53c2-66a3-415b-a7b0-2dcf1b48daf0" containerName="payload-pre-processing" Oct 30 02:23:10.804626 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:10.804580 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="963a53c2-66a3-415b-a7b0-2dcf1b48daf0" containerName="payload-pre-processing" Oct 30 02:23:10.804626 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:10.804603 6572 cpu_manager.go:414] "containerMap: removing container" podUID="b1a65b5e-39ec-48ca-9bfc-34e207413e49" containerName="payload-processing" Oct 30 02:23:10.804626 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:10.804608 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1a65b5e-39ec-48ca-9bfc-34e207413e49" containerName="payload-processing" Oct 30 02:23:10.804626 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:10.804617 6572 cpu_manager.go:414] "containerMap: removing container" podUID="4fcc6142-79ba-4dcc-b037-6b2fa054a9e3" containerName="payload-pre-processing" Oct 30 02:23:10.804626 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:10.804622 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fcc6142-79ba-4dcc-b037-6b2fa054a9e3" containerName="payload-pre-processing" Oct 30 02:23:10.804899 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:10.804646 6572 cpu_manager.go:414] "containerMap: removing container" podUID="b445121f-4bf6-4f3a-886c-a02f45574a57" containerName="istio-proxy" Oct 30 02:23:10.804899 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:10.804650 6572 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b445121f-4bf6-4f3a-886c-a02f45574a57" containerName="istio-proxy" Oct 30 02:23:10.804899 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:10.804713 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="b445121f-4bf6-4f3a-886c-a02f45574a57" containerName="istio-proxy" Oct 30 02:23:10.804899 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:10.804722 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="e33def86-6f77-4c1f-8734-c575f37e6f6e" containerName="revoke-keys" Oct 30 02:23:10.804899 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:10.804728 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="963a53c2-66a3-415b-a7b0-2dcf1b48daf0" containerName="payload-pre-processing" Oct 30 02:23:10.804899 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:10.804736 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1a65b5e-39ec-48ca-9bfc-34e207413e49" containerName="payload-processing" Oct 30 02:23:10.804899 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:10.804757 6572 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fcc6142-79ba-4dcc-b037-6b2fa054a9e3" containerName="payload-pre-processing" Oct 30 02:23:10.812148 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:10.812121 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-87dx2" Oct 30 02:23:10.824641 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:10.824612 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9dvcx/perf-node-gather-daemonset-87dx2"] Oct 30 02:23:10.825946 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:10.825919 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9dvcx\"/\"openshift-service-ca.crt\"" Oct 30 02:23:10.833588 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:10.833560 6572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9dvcx\"/\"kube-root-ca.crt\"" Oct 30 02:23:10.837494 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:10.837468 6572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9dvcx\"/\"default-dockercfg-v8hcx\"" Oct 30 02:23:10.931941 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:10.931900 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knt2c\" (UniqueName: \"kubernetes.io/projected/138c80a5-009d-4cbc-87db-1ae1de141ccc-kube-api-access-knt2c\") pod \"perf-node-gather-daemonset-87dx2\" (UID: \"138c80a5-009d-4cbc-87db-1ae1de141ccc\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-87dx2" Oct 30 02:23:10.931941 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:10.931940 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/138c80a5-009d-4cbc-87db-1ae1de141ccc-sys\") pod \"perf-node-gather-daemonset-87dx2\" (UID: \"138c80a5-009d-4cbc-87db-1ae1de141ccc\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-87dx2" Oct 30 02:23:10.932185 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:10.931969 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/138c80a5-009d-4cbc-87db-1ae1de141ccc-lib-modules\") pod \"perf-node-gather-daemonset-87dx2\" (UID: \"138c80a5-009d-4cbc-87db-1ae1de141ccc\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-87dx2" Oct 30 02:23:10.932185 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:10.932085 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/138c80a5-009d-4cbc-87db-1ae1de141ccc-proc\") pod \"perf-node-gather-daemonset-87dx2\" (UID: \"138c80a5-009d-4cbc-87db-1ae1de141ccc\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-87dx2" Oct 30 02:23:10.932185 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:10.932127 6572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/138c80a5-009d-4cbc-87db-1ae1de141ccc-podres\") pod \"perf-node-gather-daemonset-87dx2\" (UID: \"138c80a5-009d-4cbc-87db-1ae1de141ccc\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-87dx2" Oct 30 02:23:11.032575 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:11.032539 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/138c80a5-009d-4cbc-87db-1ae1de141ccc-proc\") pod \"perf-node-gather-daemonset-87dx2\" (UID: \"138c80a5-009d-4cbc-87db-1ae1de141ccc\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-87dx2" Oct 30 02:23:11.032789 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:11.032590 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/138c80a5-009d-4cbc-87db-1ae1de141ccc-podres\") pod \"perf-node-gather-daemonset-87dx2\" (UID: \"138c80a5-009d-4cbc-87db-1ae1de141ccc\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-87dx2" Oct 30 02:23:11.032789 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:11.032687 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/138c80a5-009d-4cbc-87db-1ae1de141ccc-proc\") pod \"perf-node-gather-daemonset-87dx2\" (UID: \"138c80a5-009d-4cbc-87db-1ae1de141ccc\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-87dx2" Oct 30 02:23:11.032789 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:11.032696 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knt2c\" (UniqueName: \"kubernetes.io/projected/138c80a5-009d-4cbc-87db-1ae1de141ccc-kube-api-access-knt2c\") pod \"perf-node-gather-daemonset-87dx2\" (UID: \"138c80a5-009d-4cbc-87db-1ae1de141ccc\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-87dx2" Oct 30 02:23:11.032789 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:11.032771 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/138c80a5-009d-4cbc-87db-1ae1de141ccc-sys\") pod \"perf-node-gather-daemonset-87dx2\" (UID: \"138c80a5-009d-4cbc-87db-1ae1de141ccc\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-87dx2" Oct 30 02:23:11.032789 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:11.032771 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/138c80a5-009d-4cbc-87db-1ae1de141ccc-podres\") pod \"perf-node-gather-daemonset-87dx2\" (UID: \"138c80a5-009d-4cbc-87db-1ae1de141ccc\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-87dx2" Oct 30 02:23:11.033006 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:11.032814 6572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/138c80a5-009d-4cbc-87db-1ae1de141ccc-lib-modules\") pod \"perf-node-gather-daemonset-87dx2\" (UID: \"138c80a5-009d-4cbc-87db-1ae1de141ccc\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-87dx2" Oct 30 02:23:11.033006 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:11.032837 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/138c80a5-009d-4cbc-87db-1ae1de141ccc-sys\") pod \"perf-node-gather-daemonset-87dx2\" (UID: \"138c80a5-009d-4cbc-87db-1ae1de141ccc\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-87dx2" Oct 30 02:23:11.033006 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:11.032946 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/138c80a5-009d-4cbc-87db-1ae1de141ccc-lib-modules\") pod \"perf-node-gather-daemonset-87dx2\" (UID: \"138c80a5-009d-4cbc-87db-1ae1de141ccc\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-87dx2" Oct 30 02:23:11.044806 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:11.044772 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76696fb494-9gc8p_9bc3b964-6e84-4860-a5bf-0a81f15b0d97/telemeter-client/0.log" Oct 30 02:23:11.051126 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:11.051105 6572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knt2c\" (UniqueName: \"kubernetes.io/projected/138c80a5-009d-4cbc-87db-1ae1de141ccc-kube-api-access-knt2c\") pod \"perf-node-gather-daemonset-87dx2\" (UID: \"138c80a5-009d-4cbc-87db-1ae1de141ccc\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-87dx2" Oct 30 02:23:11.124988 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:11.124893 6572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-87dx2" Oct 30 02:23:11.160734 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:11.160706 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76696fb494-9gc8p_9bc3b964-6e84-4860-a5bf-0a81f15b0d97/reload/0.log" Oct 30 02:23:11.276447 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:11.276396 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76696fb494-9gc8p_9bc3b964-6e84-4860-a5bf-0a81f15b0d97/kube-rbac-proxy/0.log" Oct 30 02:23:11.405055 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:11.404974 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-578df9b8f5-9z7nm_9e829899-8a60-4792-b4cc-fabcaff0469f/thanos-query/0.log" Oct 30 02:23:11.420305 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:11.420273 6572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9dvcx/perf-node-gather-daemonset-87dx2"] Oct 30 02:23:11.422442 ip-10-0-138-223 kubenswrapper[6572]: W1030 02:23:11.422366 6572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod138c80a5_009d_4cbc_87db_1ae1de141ccc.slice/crio-9b1a7fe46b3dfa921692977698e329e2971f62bc6ff2edc713418c5be92f5f11 WatchSource:0}: Error finding container 9b1a7fe46b3dfa921692977698e329e2971f62bc6ff2edc713418c5be92f5f11: Status 404 returned error can't find the container with id 9b1a7fe46b3dfa921692977698e329e2971f62bc6ff2edc713418c5be92f5f11 Oct 30 02:23:11.522355 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:11.522335 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-578df9b8f5-9z7nm_9e829899-8a60-4792-b4cc-fabcaff0469f/kube-rbac-proxy-web/0.log" Oct 30 02:23:11.636856 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:11.636825 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-578df9b8f5-9z7nm_9e829899-8a60-4792-b4cc-fabcaff0469f/kube-rbac-proxy/0.log" Oct 30 02:23:11.766056 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:11.766025 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-578df9b8f5-9z7nm_9e829899-8a60-4792-b4cc-fabcaff0469f/prom-label-proxy/0.log" Oct 30 02:23:11.899829 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:11.899798 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-578df9b8f5-9z7nm_9e829899-8a60-4792-b4cc-fabcaff0469f/kube-rbac-proxy-rules/0.log" Oct 30 02:23:12.014353 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:12.014324 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-578df9b8f5-9z7nm_9e829899-8a60-4792-b4cc-fabcaff0469f/kube-rbac-proxy-metrics/0.log" Oct 30 02:23:12.232377 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:12.232344 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-87dx2" event={"ID":"138c80a5-009d-4cbc-87db-1ae1de141ccc","Type":"ContainerStarted","Data":"b0d6c8c3a27915e786917b6576015afe5bdd14f1498e2dae7d9b7cef02761f28"} Oct 30 02:23:12.232596 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:12.232397 6572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-87dx2" event={"ID":"138c80a5-009d-4cbc-87db-1ae1de141ccc","Type":"ContainerStarted","Data":"9b1a7fe46b3dfa921692977698e329e2971f62bc6ff2edc713418c5be92f5f11"} Oct 30 02:23:12.232596 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:12.232421 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-87dx2" Oct 30 02:23:12.270373 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:12.270321 6572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-87dx2" podStartSLOduration=2.270304504 podStartE2EDuration="2.270304504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:23:12.26772573 +0000 UTC m=+3917.844151132" watchObservedRunningTime="2026-10-30 02:23:12.270304504 +0000 UTC m=+3917.846729905" Oct 30 02:23:13.572736 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:13.572706 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-hf2jx_296d1c8b-da1d-4e25-8b3e-7a2c50337818/manager/0.log" Oct 30 02:23:13.711548 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:13.711516 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-hf2jx_296d1c8b-da1d-4e25-8b3e-7a2c50337818/manager/0.log" Oct 30 02:23:14.686504 ip-10-0-138-223 kubenswrapper[6572]: E1030 02:23:14.686463 6572 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6vtnc" containerName="extractor" Oct 30 02:23:18.249452 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:18.249426 6572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-87dx2" Oct 30 02:23:18.452961 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:18.452930 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-hf2jx_296d1c8b-da1d-4e25-8b3e-7a2c50337818/manager/0.log" Oct 30 02:23:18.558153 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:18.558065 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-hf2jx_296d1c8b-da1d-4e25-8b3e-7a2c50337818/manager/0.log" Oct 30 02:23:20.456266 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:20.456234 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-98r9d_00c444f0-9313-4c0a-b9c8-79a953eaba17/dns/0.log" Oct 30 02:23:20.573576 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:20.573545 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-98r9d_00c444f0-9313-4c0a-b9c8-79a953eaba17/kube-rbac-proxy/0.log" Oct 30 02:23:21.188184 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:21.188151 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4dl5p_a6babafa-f85a-4a1b-9fe9-8bac54198754/dns-node-resolver/0.log" Oct 30 02:23:22.987738 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:22.987704 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-54hlq_e7922f23-5ba7-4931-8d19-598e2c4fe041/node-ca/0.log" Oct 30 02:23:23.491605 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:23.491566 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-hf2jx_296d1c8b-da1d-4e25-8b3e-7a2c50337818/manager/0.log" Oct 30 02:23:23.530193 ip-10-0-138-223 kubenswrapper[6572]: I1030 02:23:23.530144 6572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-hf2jx_296d1c8b-da1d-4e25-8b3e-7a2c50337818/manager/0.log"